Distributed computing system training method and device, program product and medium

Abstract

The invention discloses a training method and device of a distributed computing system, a program product and a medium, which are used for solving the problems of performance difference and low communication efficiency of heterogeneous computing nodes. According to the scheme, the calculation capability of each calculation node is fully considered, so that dynamic allocation and iterative optimization of tasks are realized; specifically, the problems of idle resources and low efficiency are avoided by acquiring the computing power of each computing node and allocating the training task matched with the computing power to the computing node; the time length of single iteration is determined, and the number of iterations in the communication interval is adjusted according to the time length, so that each computing node can perform data exchange at a proper time, network bandwidth occupation and delay caused by frequent communication are reduced, the overall efficiency of the training process is optimized, mutual waiting between the computing nodes is avoided, and the network performance is improved. When the computing node runs the training process, the power consumption and heat generation are reduced, and the communication overhead is also reduced, so that the long-term stability and reliability of the computing node are improved.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing systems, and in particular to a training method, device, program product and medium for a distributed computing system. Background Art

[0002] With the continuous exploration of training data scale and model parameters, deep learning models have been widely used in computer vision, natural language processing, and autonomous driving. However, the memory and computing power of single-machine training often cannot meet the needs, so the distributed training mode based on multiple machines and multiple cards has become the mainstream strategy. This mode accelerates model training through the collaboration of multiple computing workers.

[0003] Although many excellent distributed deep learning frameworks have been proposed, they usually assume that computing nodes are homogeneous, that is, they have the same performance and hardware characteristics. However, in actual training scenarios, each computing node exhibits significant heterogeneity, resulting in significant differences in computing power, memory capacity, and hardware characteristics. This heterogeneity not only affects computing power, but also involves differences in communication bandwidth between computing nodes, especially in different network devices and layers, where communication efficiency is often low.

[0004] The traditional overall synchronous training method requires all heterogeneous computing nodes to maintain the same iteration pace, resulting in faster computing nodes waiting for slower computing nodes. This "drag effect" makes the training process restricted by the slowest node, significantly reducing the training speed. It can be seen that the performance differences of heterogeneous computing nodes usually cause idle resources, and the frequent communication between heterogeneous computing nodes will also reduce the training efficiency. When processing this training process, computing nodes (that is, physical hardware devices, such as servers or CPUs (Central Processing Units) / GPUs (Graphics Processing Units) in servers) face the problems of uneven resource utilization and inefficiency. In addition, faster computing nodes cannot effectively reduce power consumption while waiting, which may cause additional power consumption and heat generation, thereby increasing the hardware burden and affecting long-term stability and reliability. At the same time, frequent data transmission and redundant exchanges increase network bandwidth occupancy and latency, further increasing the communication overhead of computing nodes.

[0005] Therefore, it is necessary to propose an optimization scheme for heterogeneous distributed computing systems. This scheme should fully consider the performance differences of each node, while considering the computing and communication capabilities of heterogeneous computing nodes, so as to improve the resource utilization and operation efficiency of computing nodes and reduce the communication overhead of computing nodes. Summary of the invention

[0006] The purpose of the present invention is to provide a training method, device, program product and medium for a distributed computing system, which not only optimizes the overall efficiency of the training process, but also avoids mutual waiting between computing nodes. When the computing nodes run this training process, they reduce power consumption and heat generation, and also reduce communication overhead, thereby improving the long-term stability and reliability of the computing nodes.

[0007] On the one hand, the present application provides a training method for a distributed computing system, which is applied to a distributed computing system, wherein the distributed computing system includes multiple heterogeneous computing nodes, and the training method for the distributed computing system includes: obtaining the computing power of each of the computing nodes; assigning a training task of a target model to each of the computing nodes according to the computing power of each of the computing nodes, and the task volume of the training task is positively correlated with the computing power; determining the single iteration duration of each of the computing nodes to execute the training task corresponding to itself; determining the number of iterations of each of the computing nodes within a communication interval according to the single iteration duration corresponding to each of the computing nodes; obtaining the local model parameters of each of the computing nodes after completing their respective iterations at every communication interval, and determining the global model parameters based on all the local model parameters until the iteration end condition is met.

[0008] Among them, obtaining the computing capacity of each of the computing nodes includes: obtaining performance indicator parameters of each of the computing nodes; determining a first capability parameter of each of the computing nodes based on the performance indicator parameters; controlling each of the computing nodes to perform a preset test task, and determining a second capability parameter based on the test result of each of the computing nodes performing the preset test task; determining the computing capacity of each of the computing nodes based on the first capability parameter and the second capability parameter corresponding to each of the computing nodes.

[0009] The performance indicator parameters include a combination of one or more of floating-point operation performance, memory bandwidth, memory latency, input / output throughput, error rate and unit energy efficiency; determining the first capability parameter of each computing node according to the performance indicator parameters includes: obtaining weight coefficients corresponding to the floating-point operation performance, the memory bandwidth, the memory latency, the input / output throughput and the error rate respectively; determining the first capability parameter according to the weight coefficients corresponding to the floating-point operation performance, the memory bandwidth, the memory latency, the input / output throughput and the error rate respectively, and the unit energy efficiency; the first capability parameter is positively correlated with the floating-point operation performance, the memory bandwidth, the input / output throughput and the unit energy efficiency, and the first capability parameter is negatively correlated with the memory latency and the error rate.

[0010] The first capability parameter is determined according to the weight coefficients corresponding to the floating-point operation performance, the memory bandwidth, the memory latency, the input / output throughput, and the error rate, and the unit energy efficiency, including: Calculate the first capability parameter; COM1 is the first capability parameter, C is the floating point operation performance, is the weight of the floating-point operation performance, M is the memory bandwidth, is the weight of the memory bandwidth, IO is the input / output throughput, is the weight of the input / output throughput, D is the memory delay, is the weight of the memory delay, ET is the error rate, is the weight of the error rate, and W is the unit energy efficiency.

[0011] Among them, controlling each of the computing nodes to execute a preset test task, and determining a second capability parameter based on the test results of each of the computing nodes executing the preset test task, includes: controlling each of the computing nodes to execute multiple preset test tasks; counting the running time and classification accuracy of the computing nodes executing multiple preset test tasks; determining the unit accuracy of each of the computing nodes based on the running time and the classification accuracy, and the unit accuracy is the second capability parameter.

[0012] Among them, the training task of the target model is allocated to each computing node according to the computing capacity of each computing node, including: determining the capacity ratio of each computing node according to the computing capacity of each computing node; determining the sample capacity allocated to each computing node according to the capacity ratio of each computing node and the initialization sample capacity of the target model.

[0013] The method further includes: obtaining current network status parameters, and adjusting the communication interval time in real time according to the current network status parameters.

[0014] Among them, obtaining the current network status parameters includes: monitoring the network delay and / or packet loss rate of the current network, determining the current network congestion level according to the network delay and / or the packet loss rate, and the network status parameters include the network congestion level.

[0015] Among them, it also includes: obtaining the priority of each of the training tasks; adjusting the communication interval time in real time according to the current network status parameters, including: adjusting the communication interval time in real time according to the current network status parameters and the priority of each of the training tasks.

[0016] Among them, the network status parameter is the degree of network congestion, and the communication interval time is adjusted in real time according to the current network status parameter and the priority of each training task, including: calculating the target communication interval time according to CIT=CIT0 / (1+NCD+DTP), and updating the communication interval time to the target communication interval time; CIT is the target communication interval time, CIT0 is the initial communication interval time before updating, NCD is the degree of network congestion, and DTP is the priority.

[0017] Among them, obtaining the priority of each of the training tasks includes: obtaining a first weight, a first score, a second weight and a second score for each of the training tasks, the first weight and the first score being used to characterize the importance of the training task, and the second weight and the second score being used to characterize the urgency of the training task; determining the priority of each of the training tasks according to the first weight, the first score, the second weight and the second score corresponding to each of the training tasks.

[0018] Among them, the priority of each training task is determined according to the first weight, the first score, the second weight and the second score corresponding to each training task, including: calculating the priority of each training task according to DTP=WI·IS+WU·US; DTP is the priority, WI is the first weight, IS is the first score, WU is the second weight, and US is the second score.

[0019] On the other hand, the present application provides a training device for a distributed computing system, comprising: a memory for storing a computer program; and a processor for implementing the steps of the above-mentioned training method for a distributed computing system when executing the computer program.

[0020] On the other hand, the present application provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned training method for the distributed computing system when executed by a processor.

[0021] On the other hand, the present application provides a non-volatile storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the training method for the distributed computing system described above are implemented.

[0022] The present invention provides a training method, device, program product and medium for a distributed computing system, which are used to solve the problems of performance differences and low communication efficiency of heterogeneous computing nodes. The solution realizes dynamic allocation and iterative optimization of tasks by fully considering the computing power of each computing node; specifically, by obtaining the computing power of each computing node and allocating training tasks that match it, the problems of idle resources and low efficiency are avoided; the duration of a single iteration is determined and the number of iterations within the communication interval is adjusted accordingly, so that each computing node can exchange data at an appropriate time, reducing the network bandwidth occupation and delay caused by frequent communication, which not only optimizes the overall efficiency of the training process, but also avoids mutual waiting between computing nodes. When the hardware device runs this training process, it reduces power consumption and heat generation, and also reduces communication overhead, thereby improving the long-term stability and reliability of the hardware device. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of synchronous communication between heterogeneous computing nodes provided by the present invention;

[0025] Figure 2 A schematic diagram of asynchronous communication between heterogeneous computing nodes provided by the present invention;

[0026] Figure 3 A flow chart of a training method for a distributed computing system provided by the present invention;

[0027] Figure 4 A schematic diagram of a training device for a distributed computing system provided by the present invention;

[0028] Figure 5 A schematic diagram of a non-volatile storage medium provided by the present invention. DETAILED DESCRIPTION

[0029] The core of the present invention is to provide a training method, device, program product and medium for a distributed computing system, which not only optimizes the overall efficiency of the training process, but also avoids mutual waiting between computing nodes. When the computing nodes run this training process, they reduce power consumption and heat generation, and also reduce communication overhead, thereby improving the long-term stability and reliability of the computing nodes.

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] First, before describing this embodiment, the communication topology and communication synchronization mode of the distributed computing system are introduced. The communication topology includes a parameter server topology and an All Reduce topology; the communication synchronization mode includes a synchronous communication update and an asynchronous communication update.

[0032] The parameter server topology includes two types: central server nodes and heterogeneous computing nodes. Among them, the heterogeneous computing nodes are responsible for the actual training tasks of the local local models, sending local local model parameters (such as gradient information) to the central server nodes, and receiving the global model parameters sent by the central server nodes. The central server node aggregates the local model parameters of multiple heterogeneous computing nodes and updates the global model, outputs the global model model parameters and returns the global model parameters to each computing node. The All reduce topology only contains computing nodes without central server nodes, and does not support asynchronous communication. The distributed computing system in this application is a parameter server topology.

[0033] Communication synchronization methods can be divided into synchronous update strategies and asynchronous update strategies. The synchronous update strategy means that after a computing node completes the current round of iterative training tasks, it must wait for other computing nodes to complete their respective current rounds of iterations. Only when all computing nodes have completed the current round of training tasks together, the system will enter the next round of iterations. The synchronous update strategy is limited by the node with the worst performance, causing other computing nodes to idle resources and wait, resulting in a certain waste of computing resources (such as Figure 1 As shown in Figure 1.1). The asynchronous update strategy means that after a computing node completes the current round of iteration, it will automatically enter the next iteration and update without waiting for other computing nodes. This overcomes the problem of slow model training convergence through some periodic synchronous updates. The asynchronous update strategy is suitable for distributed training environments composed of heterogeneous computing devices, which can fully improve the utilization of devices.

[0034] exist Figure 1In the figure, there are three heterogeneous computing nodes (the central server node is not shown) participating in the training task. Due to the differences in their respective computing performance, their computing time, communication time (including the time to send information to the central parameter server node and the time to receive information sent by the central server node), and waiting time are different. For example, in the iteration process of this task, computing node 1 has the worst performance, executing the sending, computing, and receiving stages respectively, and there is no waiting stage. Computing node 3 has the strongest performance. In addition to the sending, computing, and receiving nodes, it has the longest waiting time, which causes its device resource limitations. When all heterogeneous nodes complete the iteration of the current round, all nodes repeat the above iteration. It can be seen that the synchronization strategy adopted by multi-heterogeneous distributed systems is inefficient.

[0035] exist Figure 2 In the embodiment, the heterogeneous computing nodes adopt asynchronous communication measures, that is, after completing a single training, the node does not need to wait for other nodes and directly proceeds to the next iterative training. Specifically, during the iterative training process, the computing node 3 with the strongest performance completed 4 iterations within the synchronization interval, and the computing node 1 with the weakest performance completed 2 iterations. Compared with the synchronous communication method, the biggest advantage of asynchronous communication is that the computing nodes with strong performance eliminate the waiting time as much as possible, which improves the utilization rate of device resources. This embodiment uses an asynchronous communication strategy.

[0036] On the one hand, if Figure 3 As shown, the present application provides a training method for a distributed computing system, which is applied to a distributed computing system, wherein the distributed computing system includes a plurality of heterogeneous computing nodes, and the training method for the distributed computing system includes:

[0037] S11: Obtain the computing capacity of each computing node.

[0038] In this step, computing power usually includes indicators such as computing speed, available memory, and hardware characteristics of computing nodes. For example, for GPU nodes, their computing power may be determined by their floating-point computing performance (such as the number of floating-point operations that can be processed per second) and the size of video memory; while for CPU nodes, it may involve parameters such as the number of cores, main frequency, and cache size. The core goal of this step is to accurately evaluate the performance of each computing node so that in the subsequent task allocation, the complexity of the training task can be reasonably matched with the processing power of each node. In this way, the problems of idle resources and low training efficiency caused by performance differences can be effectively avoided. In addition, accurate computing power evaluation also provides data support for dynamically adjusting task allocation strategies to ensure that during the training process, fast nodes will not produce a "drag effect" due to waiting for slow nodes, thereby improving the overall training efficiency. Ultimately, this accurate acquisition and analysis of computing power helps to optimize the resource utilization of distributed computing systems and ensure that training tasks can be efficiently allocated and executed among various heterogeneous nodes.

[0039] S12: Allocate a training task of the target model to each computing node according to the computing power of each computing node, and the amount of the training task is positively correlated with the computing power.

[0040] This step aims to assign training tasks of the target model to each computing node according to its computing power, ensuring that the amount of tasks is positively correlated with computing power. The core of this process is to rationally allocate resources to improve training efficiency and shorten training time. First, after evaluating the computing power of each node, tasks with higher complexity can be assigned to nodes with stronger computing power, while relatively simple tasks can be assigned to nodes with weaker computing power, based on the performance differences of the nodes. This strategy effectively avoids the problem of idle or overloaded resources caused by mismatched task allocation.

[0041] Specifically, the complexity of the task can be measured by factors such as the number of training samples, the amount of model parameters, and the number of iterations. In this way, each node can be ensured to operate efficiently within its processing capacity, thereby maximizing its resource utilization. In addition, reasonable task allocation also reduces the waiting time between nodes and avoids the "drag effect" that occurs during the overall training process. In distributed training, the collaboration of computing nodes is crucial. Through effective task allocation, load balancing between nodes can be achieved, the operating efficiency of the entire training system can be improved, and the final model can be ensured to converge faster.

[0042] S13: Determine the duration of a single iteration of the training task that each computing node executes corresponding to itself.

[0043] In this step, the determination of a single iteration duration depends on multiple factors, including the hardware configuration of the computing node, the complexity of the task, and the current system load. First, the computing power of the computing node directly affects the speed at which it processes data. Nodes with stronger computing power can usually complete an iteration in a shorter time. Secondly, the complexity of the training task will also significantly affect the iteration duration. For example, processing larger data sets or more complex models often requires more computing time. By monitoring and analyzing the actual running time of each node when performing training tasks, more accurate single iteration duration information can be obtained. This step not only provides an important basis for subsequent task allocation and communication arrangements, but also helps identify possible performance bottlenecks, thereby performing dynamic optimization. In addition, accurate estimation of a single iteration duration helps coordinate the training progress between heterogeneous nodes, ensure that each node completes the corresponding training task at the right time, further reduce the waiting time between nodes, and improve overall training efficiency.

[0044] S14: Determine the number of iterations of each computing node within the communication interval according to the single iteration duration corresponding to each computing node.

[0045] In distributed training, the duration of a single iteration of each computing node varies due to performance differences, so a reasonable iteration frequency needs to be formulated based on the specific situation of each node. The communication interval provides a time frame for coordination between nodes. The computing nodes perform multiple iterations within this time period, which helps to improve resource utilization. At the same time, by calculating the number of iterations that each computing node can perform within the communication interval, it can ensure that each computing node participates in training as much as possible within its capabilities, rather than just passively waiting for other nodes to complete. This process not only reduces the waste of resources caused by inconsistent performance, but also increases the training speed, making the entire model training process smoother.

[0046] S15: At each communication interval, obtain the local model parameters of each computing node after completing its own number of iterations, and determine the global model parameters based on all the local model parameters until the iteration end condition is met.

[0047] In this step, by regularly collecting the local model parameters of each node, the knowledge obtained by different computing nodes during the training process can be summarized to form a more comprehensive and accurate global model. This process involves each computing node reporting its local training results to the master node (such as the central parameter processor) after completing a certain number of training iterations. Subsequently, these local model parameters are integrated to update the global model parameters to reflect the training results of all nodes. Such an update mechanism ensures that the model is continuously optimized within each iteration cycle, improving the learning effect and convergence speed of the model. It can be seen that this step not only ensures knowledge sharing among the computing nodes, but also effectively promotes the overall efficiency of model training, ensuring that the final target model can converge quickly and accurately.

[0048] For example, in an application scenario of image classification, it is assumed that there is a distributed computing system including multiple heterogeneous computing nodes, such as multiple GPU servers and / or CPU servers with different performance (the computing node can be any hardware device with computing power, not limited to the above examples). The training method of the distributed computing system may specifically include the following steps: by monitoring the performance indicators of the GPU and / or CPU (such as floating-point computing power, memory bandwidth, etc.), determine the computing power of the GPU and / or CPU (such as the computing power of the GPU is greater than the computing power of the CPU); assuming that the target model is a convolutional neural network (CNN) and there are 10,000 image samples for training (10,000 image samples are training tasks), then it can be decided to allocate 80% of the samples (8,000) to the GPU node and 20% of the samples (2,000) to the CPU node based on the computing power. Each iteration of the training task of the GPU node takes about 2 seconds, while the CPU node takes about 10 seconds due to its lower performance. Within the set communication interval of 20 seconds, the GPU node can complete 10 iterations (20 seconds / 2 seconds), while the CPU node can only complete 2 iterations (20 seconds / 10 seconds). Every 20 seconds, the GPU node sends its trained local model parameters to the main server, and the CPU node also sends its local model parameters. After receiving the local model parameters of all computing nodes, the main server merges the parameters and updates the global model parameters. Training continues until the accuracy of the global model on the validation set reaches the preset standard, or the preset number of iterations is reached.

[0049] It can be seen that in the image classification task, the reasonable allocation of training tasks according to the performance differences of computing nodes can significantly improve the resource utilization of computing nodes. For example, more samples are allocated to GPU nodes with stronger performance to give full play to their computing power, while fewer samples are allocated to CPU nodes to avoid idle resources. In addition, setting a suitable communication interval and reducing frequent data transmission can effectively reduce the communication overhead between computing nodes, thereby reducing network bandwidth occupancy and latency and improving operation efficiency.

[0050] In summary, the training method for a distributed computing system provided by the present application realizes dynamic allocation and iterative optimization of tasks by fully considering the computing power of each computing node; specifically, by obtaining the computing power of each computing node and allocating training tasks that match it, the problems of idle resources and inefficiency are avoided; the duration of a single iteration is determined and the number of iterations within the communication interval is adjusted accordingly, so that each computing node can exchange data at an appropriate time, reducing network bandwidth occupancy and delays caused by frequent communications, which not only optimizes the overall efficiency of the training process, but also avoids mutual waiting between computing nodes. When the computing nodes run this training process, they reduce power consumption and heat generation, and also reduce communication overhead, thereby improving the long-term stability and reliability of the computing nodes.

[0051] Based on the above embodiments:

[0052] As an optional embodiment, the computing capacity of each computing node is obtained, including: obtaining performance indicator parameters of each computing node; determining a first capability parameter of each computing node based on the performance indicator parameters; controlling each computing node to perform a preset test task, and determining a second capability parameter based on the test results of each computing node performing the preset test task; and determining the computing capacity of each computing node based on the first capability parameter and the second capability parameter corresponding to each computing node.

[0053] In this embodiment, the process of obtaining the computing power of each computing node involves multiple steps to ensure accurate evaluation of the performance of the node. First, by obtaining the performance index parameters of each computing node, the first capability parameters of the computing node are obtained. These indicators provide basic data for subsequent capability evaluation, so that the system can understand the potential performance of each node at the hardware level. Next, each computing node is controlled to perform preset test tasks in order to verify its computing power from the perspective of actual operation. These test tasks can be benchmarks for specific computing tasks, which are designed to simulate real workloads and measure the performance of nodes under such loads. According to the test results of each computing node when performing these test tasks, the second capability parameters can be obtained, which are more targeted and can reflect the operating efficiency and stability of the node in actual applications. Finally, the first capability parameters and the second capability parameters will be combined to determine the comprehensive computing power of each computing node. This comprehensive evaluation not only takes into account the theoretical performance of the computing node, but also reflects the operating conditions that it may encounter in actual work.

[0054] Through such multi-dimensional evaluation, we can more accurately understand the actual capabilities of each computing node, provide data support for subsequent task allocation and optimization strategies, and thus improve the performance and efficiency of the entire distributed training system.

[0055] As an optional embodiment, the performance indicator parameters include a combination of one or more of floating-point operation performance, memory bandwidth, memory latency, input / output throughput, error rate and unit energy efficiency; determining the first capability parameter of each computing node according to the performance indicator parameters includes: obtaining weight coefficients corresponding to the floating-point operation performance, memory bandwidth, memory latency, input / output throughput, and error rate respectively; determining the first capability parameter according to the weight coefficients and unit energy efficiency corresponding to the floating-point operation performance, memory bandwidth, memory latency, input / output throughput, and error rate respectively; the first capability parameter is positively correlated with the floating-point operation performance, memory bandwidth, input / output throughput, and unit energy efficiency, and the first capability parameter is negatively correlated with the memory latency and error rate.

[0056] In this embodiment, the selection of performance indicator parameters is crucial for accurately evaluating the capabilities of computing nodes. These parameters include floating-point operation performance, memory bandwidth, memory latency, input / output throughput, error rate, and unit energy efficiency, etc., which together reflect the overall performance of the node when performing computing tasks. First, in order to integrate these different performance indicators into a unified evaluation standard, a corresponding weight coefficient is assigned to each performance indicator, and these weight coefficients reflect the importance of each indicator in the overall computing capacity evaluation. Next, these weight coefficients are combined with the actual measured values ​​of each performance indicator to calculate the first capability parameter of each computing node. This parameter is a comprehensive indicator that can effectively reflect the overall performance level of the node. According to the design, the first capability parameter is positively correlated with floating-point operation performance, memory bandwidth, input / output throughput, and unit energy efficiency, which means that the higher these indicators are, the stronger the computing power of the node is; and negatively correlated with memory latency and error rate, which means that the lower these indicators are, the better the performance of the node is.

[0057] Floating-point operation performance: used to measure the speed at which processors (such as CPUs, GPUs, and FPGAs (Field-Programmable Gate Arrays)) perform floating-point operations, usually expressed in floating-point operations per second. For example, GFLOPS (Giga Floating Point Operations Per Second) means one billion floating-point operations per second, and TFLOPS (Tera Floating Point Operations Per Second) means one trillion floating-point operations per second. The greater the number of operations per second, the stronger the computing power of the heterogeneous node.

[0058] Memory bandwidth, which measures the speed at which data is transferred and read from memory, is generally measured in GB / s. The larger the memory bandwidth, the faster the computing node can read from memory. During the modeling process, the theoretical maximum memory bandwidth of each heterogeneous node is used for modeling.

[0059] Memory latency, which indicates the time interval between a processor requesting data and the data arriving at the processor, is usually measured in nanoseconds (ns). The smaller the memory latency, the stronger the computing node's processing power. During the modeling process, the theoretical minimum memory latency of each heterogeneous computing node is used for modeling.

[0060] Input / output throughput, which indicates the read and write speed of a heterogeneous node storage system, is generally measured in GB / s. The higher the I / O throughput, the stronger the node processing capability. During the modeling process, the theoretical maximum input / output throughput of each heterogeneous computing node is used for modeling.

[0061] Error rate, this indicator indicates the frequency of errors when heterogeneous nodes execute tasks. The lower the error rate, the more reliable the performance of the computing node. The present invention proposes a new method for calculating the error rate of computing nodes. The workflow of the method is as follows: Take two heterogeneous computing nodes, GPU and FPGA, executing the same computing task as an example.

[0062] Step 1: Given a labeled computing task (with a correct answer), run the task T times on the GPU and FPGA respectively.

[0063] Step 2: Use separately , calculate the GPU error rate And the error rate of FPGA , T error is the number of task errors, and T is the total number of task executions.

[0064] Step 3: Compare the above two error rates.

[0065] Step 4: If the error rates are equal, execute the task T times again and use the above error rate calculation method to obtain the actual error rate of the heterogeneous computing nodes.

[0066] Step 5: Repeat the above process until the two heterogeneous computing nodes obtain the exact error rate ranking.

[0067] Unit energy efficiency and energy efficiency ratio are important indicators for measuring the performance and energy efficiency of computing devices, especially in heterogeneous computing environments, where different computing units (such as CPU, GPU, FPGA, etc.) may have significant differences in energy efficiency. The unit energy efficiency of the present invention is defined as the computing performance that can be provided per watt of electrical energy, expressed in FLOPS / W. The higher the unit energy efficiency ratio, the stronger the computing node performance.

[0068] Through this comprehensive evaluation mechanism, the system can provide more accurate capability parameters, laying the foundation for subsequent task allocation and optimization decisions, thereby improving the efficiency and effectiveness of distributed training.

[0069] As an optional embodiment, determining the first capability parameter according to the weight coefficients and unit energy efficiency corresponding to the floating-point operation performance, memory bandwidth, memory latency, input / output throughput, and error rate, respectively, includes: Calculate the first capability parameter; COM1 is the first capability parameter, C is the floating point operation performance, is the weight of floating-point operation performance, M is the memory bandwidth, is the weight of memory bandwidth, IO is the input / output throughput, is the weight of input / output throughput, D is the memory latency, is the weight of memory delay, ET is the error rate, is the weight of the error rate, and W is the unit energy efficiency.

[0070] Among them, the parameters ,…, is the weight coefficient of each factor index and it takes a positive number between 0 and 1. They satisfy .

[0071] As an optional embodiment, each computing node is controlled to execute a preset test task, and a second capability parameter is determined based on the test results of each computing node executing the preset test task, including: controlling each computing node to execute multiple preset test tasks; counting the running time and classification accuracy of the computing node executing multiple preset test tasks; and determining the unit accuracy of each computing node based on the running time and classification accuracy, wherein the unit accuracy is the second capability parameter.

[0072] In this embodiment, each computing node is controlled to perform a preset test task, and then the second capability parameter is determined to evaluate the actual computing performance of the heterogeneous nodes. First, by controlling each computing node to perform the preset test task multiple times, more reliable performance data can be obtained, thereby eliminating the deviation caused by the uncertainty of a single test result. Then, the running time and classification accuracy used by each computing node in these test tasks are counted. These indicators reflect the efficiency and accuracy of the node when processing a specific task.

[0073] On this basis, unit accuracy is defined as the ratio of the final classification accuracy of each computing node to its corresponding running time. This unit accuracy calculation method provides a quantitative method for evaluating the relative performance of each heterogeneous node under the same task conditions. For example, when using a classic convolutional neural network for image classification tasks on a dataset, by training 100 epochs on a single GPU and FPGA respectively, the specific data of the running time and final classification accuracy of these two heterogeneous computing nodes can be obtained. In this way, not only can the performance of different hardware platforms when performing the same task be accurately measured, but also data support can be provided for subsequent task allocation and optimization strategies to ensure maximum resource utilization. In addition, the unit accuracy defined by the formula provides a clear and quantitative standard for evaluating the performance of heterogeneous nodes in practical applications, enabling more effective performance comparison and selection in complex computing environments.

[0074] As an optional embodiment, the computing capacity of each computing node=the first capacity parameter+the second capacity parameter.

[0075] As an optional embodiment, the training task of the target model is assigned to each computing node according to the computing capacity of each computing node, including: determining the capacity ratio of each computing node according to the computing capacity of each computing node; determining the sample capacity allocated to each computing node according to the capacity ratio of each computing node and the initialization sample capacity of the target model.

[0076] This embodiment aims to reasonably allocate the training tasks of the target model according to the computing power of each computing node, thereby improving the overall training efficiency. First, according to the specific performance indicators of each computing node, its capacity ratio is calculated, which reflects the relative contribution of each node in the overall computing power. Through such a capacity assessment, more accurate task allocation can be achieved, so that nodes with strong performance can obtain more computing resources, while nodes with weaker performance can obtain relatively less task load. This allocation strategy helps to improve the resource utilization of heterogeneous devices.

[0077] On this basis, combined with the initialization sample capacity of the target model, the sample capacity that each computing node should process is determined. Specifically, by appropriately increasing the sample size of nodes with strong computing power and reducing the sample size of nodes with weak computing power, not only can the data between nodes be synchronized, but also the execution time of the overall training task can be optimized to the maximum extent. In this way, the computing resources in the training process can be effectively utilized, avoiding performance bottlenecks caused by uneven sample distribution.

[0078] In order to implement this task allocation strategy, first, when the training task is initialized, the sample capacity of each heterogeneous computing node is set to the same to ensure fairness. Then, the comprehensive evaluation score of each node's ability (that is, the computing power mentioned above) is calculated to reflect its performance in the current training task. Then, according to the score proportion of each heterogeneous node, the initialization sample capacity is readjusted to achieve a reasonable distribution of the sample size. With this method, the system can flexibly respond to the performance differences of different nodes, improve the training efficiency in a heterogeneous computing environment, and ensure the efficiency and effectiveness of the overall model training process.

[0079] Taking three heterogeneous computing nodes (respectively denoted as computing node a, computing node b, and computing node c) as an example, the specific execution and calculation process of the allocation strategy is introduced.

[0080] Step 1: Initialize the sample capacity of heterogeneous computing nodes a, b, and c as ; Step 2: Use the above steps to calculate the comprehensive evaluation scores of the computing capabilities of the three computing nodes COM a ,COM b ,COM c ; Step 3: Calculate the score proportions respectively , , , Ra, Rb, and Rc represent the calculation score proportions of computing nodes a, b, and c respectively; Step 4: The sample capacity of computing node a is adjusted to , the sample capacity of computing node b is adjusted to , the sample capacity of computing node c is adjusted to .

[0081] As an optional embodiment, the method further includes: obtaining current network status parameters, and adjusting the communication interval time in real time according to the current network status parameters.

[0082] This embodiment aims to optimize communication efficiency and reduce network congestion by dynamically adjusting the communication interval between heterogeneous nodes by acquiring the current network status parameters in real time. Specifically, in a distributed computing environment, the training status and network status of heterogeneous nodes may change at any time, which requires the communication strategy to be able to flexibly respond to these changes. By monitoring the real-time status of the network, such as bandwidth utilization, latency, packet loss rate and other key parameters, potential signs of network congestion or increased latency can be identified.

[0083] On this basis, the communication calibration strategy will dynamically adjust the frequency or size of data transmission according to these network status parameters. For example, when the network is in good condition and the bandwidth is sufficient, the communication frequency can be appropriately increased to speed up the synchronization and update of model parameters and ensure the timeliness of the data. When the network is congested or delayed, the communication frequency will be reduced or the size of the data packet will be adjusted accordingly to reduce the network burden, thereby effectively reducing communication delays. This adaptive communication strategy ensures the consistency and reliability of data transmission between nodes, while maximizing communication efficiency and reducing training time delays caused by network problems.

[0084] By introducing a dynamic and adaptive communication interval adjustment mechanism, not only the collaborative training capability between heterogeneous nodes is improved, but also the smoothness and efficiency of the model training process under different network conditions is ensured, further optimizing the overall distributed training effect.

[0085] As an optional embodiment, obtaining current network status parameters includes: monitoring network delay and / or packet loss rate of the current network, and determining the current network congestion level according to the network delay and / or packet loss rate; the network status parameters include the network congestion level.

[0086] Specifically, network latency and packet loss rate are important indicators for measuring network performance. Network latency refers to the time required for data to be transmitted in the network, while packet loss rate refers to the proportion of data packets lost during data transmission to the total number of data packets sent. When network latency increases or packet loss rate rises, it usually means that the network is congested, which may lead to a decrease in the efficiency of data transmission.

[0087] In this framework, a formula for network congestion degree (NCD) can be defined, where the value of NCD is composed of two parameters: Represents network delay, represents the packet loss rate. Through a simple linear combination ( ), which can comprehensively consider these two key indicators to effectively quantify the current network status. If the NCD value is high, it means that the network is seriously congested, and measures can be taken, such as extending the communication interval or reducing the frequency of data transmission, to relieve the network burden. On the contrary, if the NCD value is low, it means that the network is in good condition, and the frequency of data transmission can be increased to ensure timely synchronization of data between nodes.

[0088] This dynamic monitoring and evaluation mechanism based on network status parameters not only enhances the system's adaptability to changes in network conditions, but also enables efficient communication management between heterogeneous nodes, improving the performance and stability of overall distributed training.

[0089] As an optional embodiment, it also includes: obtaining the priority of each training task; adjusting the communication interval time in real time according to the current network status parameters, including: adjusting the communication interval time in real time according to the current network status parameters and the priority of each training task.

[0090] In this embodiment, the priority of each training task is also obtained, and the communication interval is adjusted in real time in combination with the current network status parameters to optimize the efficiency and response speed of distributed training. First, each training task is assigned a priority at the beginning according to its importance and urgency. Tasks with higher priorities are usually those that have a greater impact on the model training results or require quick feedback. Through this priority allocation, it is possible to clearly identify which tasks should be prioritized among multiple training tasks, thereby effectively managing computing resources and communication traffic.

[0091] On this basis, the current network status parameters, including network latency, packet loss rate, etc., will be continuously monitored to assess the degree of network congestion. When the network is in good condition and the bandwidth is sufficient, the communication interval can be shortened for higher-priority training tasks, and the relevant data or model parameters can be quickly transmitted to ensure that these critical tasks can be updated and responded to in a timely manner. On the contrary, if the network is congested or the latency increases, the system will appropriately extend the communication interval, especially for lower-priority tasks, to avoid overloading the network and ensure the smoothness of overall communication.

[0092] This method of dynamically adjusting communication strategies based on task priority and network status can not only effectively balance the urgency of task processing and network conditions, but also achieve more efficient resource allocation and data transmission between heterogeneous nodes, further improving the performance and stability of distributed training.

[0093] As an optional embodiment, the network status parameter is the degree of network congestion, and the communication interval time is adjusted in real time according to the current network status parameter and the priority of each training task, including: calculating the target communication interval time according to CIT=CIT0 / (1+NCD+DTP), and updating the communication interval time to the target communication interval time; CIT is the target communication interval time, CIT0 is the initial communication interval time before the update, NCD is the degree of network congestion, and DTP is the priority.

[0094] In this embodiment, the target communication interval time (CIT) is dynamically calculated to optimize the data transmission efficiency in distributed training. First, the calculation formula of CIT is CIT=CIT0 / (1+NCD+DTP). This formula reflects the impact of network status and task priority on communication efficiency by adjusting the initial communication interval time. Specifically, when the network congestion degree (NCD) increases, it means that the network load increases and data transmission may become slow or unstable. In this case, the "1+NCD" in the formula increases, resulting in a corresponding increase in the target communication interval time (CIT), thereby extending the time interval for data transmission to reduce network pressure and ensure stable operation of the system. Conversely, when the network condition is good, the NCD is small, and the target communication interval time will be reduced, which helps to transmit data more frequently and improve the response speed of training tasks. On the other hand, priority (DTP) also plays an important role in the calculation. For training tasks with higher priority, the DTP value is larger, which further reduces the target communication interval time, allowing these important tasks to perform data transmission and parameter updates faster. For tasks with lower priority, the DTP value is smaller, which will correspondingly lengthen the target communication interval, thereby reasonably allocating network resources and avoiding the impact on high-priority tasks.

[0095] Through this dynamic adjustment mechanism based on network congestion level and task priority, it is possible to respond flexibly to different network conditions, optimize communication efficiency, and improve the performance and stability of overall distributed training.

[0096] As an optional embodiment, obtaining the priority of each training task includes: obtaining a first weight, a first score, a second weight and a second score for each training task, the first weight and the first score being used to characterize the importance of the training task, and the second weight and the second score being used to characterize the urgency of the training task; determining the priority of each training task according to the first weight, the first score, the second weight and the second score corresponding to each training task.

[0097] In this embodiment, the priority of each training task is determined by obtaining the weight and score of each training task, so as to reasonably allocate resources and optimize the order of task execution in a distributed training environment. Specifically, this embodiment defines two dimensions of weights and scores: the first dimension is used to characterize the importance of the training task, and the second dimension is used to characterize the urgency of the training task. Specifically, the first weight reflects the contribution of the task to the overall goal, and the value range is 0 to 1, where 1 means that the task contributes the most to the goal; the first score is an integer from 1 to 10, representing the importance of the task, and the higher the score, the more significant the impact of the task on the goal. On the other hand, the second weight reflects the urgency of task completion, and the range is also 0 to 1, 1 means that the task needs to be processed immediately; and the second score is based on the deadline or response pressure of the task, and the value range is 1 to 10, and the higher the score means the higher the urgency of the task. By combining the first weight, the first score, the second weight and the second score, the priority of each training task can be comprehensively evaluated to ensure that key tasks can be processed first when resources are limited.

[0098] For example, if a task is both more important and more urgent, it will be assigned a higher priority, so that it will be given priority during execution to ensure that it is completed in a timely manner; while for tasks with lower importance or urgency, execution can be appropriately postponed. This priority determination method based on weights and scores makes the management of training tasks more flexible and efficient, helping to optimize the overall training process, improve resource utilization and training results.

[0099] As an optional embodiment, determining the priority of each training task according to the first weight, the first score, the second weight, and the second score corresponding to each training task includes:

[0100] Calculate the priority of each training task according to DTP=WI·IS+WU·US;

[0101] DTP is the priority, WI is the first weight, IS is the first score, WU is the second weight, and US is the second score.

[0102] In this formula, the first weight WI and the first score IS jointly measure the importance of the task to the overall goal, reflecting the degree of contribution of the task; while the second weight WU and the second score US focus on the urgency of the task and characterize the time required to complete the task. Through this weighted combination, the importance and urgency of the task can be comprehensively considered. For example, if a task is highly important and needs to be completed immediately, its corresponding first weight and score will make it occupy a larger proportion in the priority calculation, so that it will be scheduled for execution first.

[0103] This embodiment aims to reasonably sort different training tasks in a quantitative manner, so that when resources are limited or competition is fierce, those tasks that are both important and urgent can be prioritized to ensure the efficiency and effectiveness of model training. At the same time, through this flexible priority calculation mechanism, it can dynamically adapt to the changing needs of different tasks and achieve an efficient training process.

[0104] On the other hand, Figure 4 As shown, the present application provides a training device for a distributed computing system, including: a memory 41 for storing a computer program; a processor 42 for implementing the steps of the above-mentioned training method for a distributed computing system when executing the computer program. For other introductions to the training device for a distributed computing system, please refer to the above-mentioned embodiments, and the present application will not repeat them here.

[0105] On the other hand, the present application provides a computer program product, including a computer program / instruction, which implements the steps of the training method of the above-mentioned distributed computing system when the computer program / instruction is executed by a processor. For other introductions to the computer program product, please refer to the above-mentioned embodiments, and the present application will not repeat them here.

[0106] On the other hand, Figure 5 As shown, the present application provides a non-volatile storage medium 51, on which a computer program 52 is stored, and when the computer program 52 is executed by a processor, the steps of the training method of the above-mentioned distributed computing system are implemented. For other introductions to the non-volatile storage medium 51, please refer to the above-mentioned embodiments, and the present application will not repeat them here.

[0107] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0108] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a distributed computing system, characterized in that: Applied to a distributed computing system, the distributed computing system includes a plurality of heterogeneous computing nodes, and the training method of the distributed computing system includes: Obtaining the computing capacity of each of the computing nodes; Allocating a training task of the target model to each computing node according to the computing capacity of each computing node, wherein the amount of the training task is positively correlated with the computing capacity; Determine the duration of a single iteration of the training task that each computing node performs corresponding to itself; Determine the number of iterations of each computing node within the communication interval according to the single iteration duration corresponding to each computing node; At each communication interval, local model parameters of each computing node after completing respective iterations are obtained, and global model parameters are determined according to all the local model parameters until an iteration end condition is met.

2. The training method of a distributed computing system according to claim 1, characterized in that: Obtaining the computing capacity of each computing node includes: Obtaining performance indicator parameters of each of the computing nodes; Determine a first capability parameter of each of the computing nodes according to the performance indicator parameter; Controlling each of the computing nodes to execute a preset test task, and determining a second capability parameter according to a test result of each of the computing nodes executing the preset test task; The computing capability of each computing node is determined according to the first capability parameter and the second capability parameter corresponding to each computing node.

3. The training method of a distributed computing system according to claim 2, characterized in that: The performance indicator parameters include one or more combinations of floating point operation performance, memory bandwidth, memory latency, input / output throughput, error rate and unit energy efficiency; Determining a first capability parameter of each computing node according to the performance indicator parameter includes: Obtain weight coefficients corresponding to the floating-point operation performance, the memory bandwidth, the memory latency, the input / output throughput, and the error rate, respectively; Determine the first capability parameter according to the weight coefficients corresponding to the floating-point operation performance, the memory bandwidth, the memory latency, the input / output throughput, and the error rate, and the unit energy efficiency; The first capability parameter is positively correlated with the floating-point operation performance, the memory bandwidth, the input / output throughput, and the unit energy efficiency, and the first capability parameter is negatively correlated with the memory delay and the error rate.

4. The training method of a distributed computing system according to claim 3, characterized in that: Determining the first capability parameter according to the weight coefficients corresponding to the floating-point operation performance, the memory bandwidth, the memory latency, the input / output throughput, and the error rate, respectively, and the unit energy efficiency, includes: according to calculating the first capability parameter; COM1 is the first capability parameter, C is the floating point operation performance, is the weight of the floating-point operation performance, M is the memory bandwidth, is the weight of the memory bandwidth, IO is the input / output throughput, is the weight of the input / output throughput, D is the memory delay, is the weight of the memory delay, ET is the error rate, is the weight of the error rate, and W is the unit energy efficiency.

5. The training method of a distributed computing system according to claim 2, characterized in that: Controlling each of the computing nodes to execute a preset test task, and determining a second capability parameter according to a test result of each of the computing nodes executing the preset test task, including: Control each of the computing nodes to execute multiple preset test tasks; Counting the running time and classification accuracy of the computing node executing the preset test task multiple times; The unit accuracy of each computing node is determined according to the running time and the classification accuracy, and the unit accuracy is the second capability parameter.

6. The training method of a distributed computing system according to claim 1, characterized in that: Allocating a training task of a target model to each computing node according to the computing capacity of each computing node includes: Determine the capacity ratio of each computing node according to the computing capacity of each computing node; The sample capacity allocated to each of the computing nodes is determined according to the capacity ratio of each of the computing nodes and the initialization sample capacity of the target model.

7. The training method for a distributed computing system according to any one of claims 1 to 6, characterized in that: Also includes: Acquire current network status parameters, and adjust the communication interval time in real time according to the current network status parameters.

8. The training method of a distributed computing system according to claim 7, characterized in that: Get the current network status parameters, including: Monitor the network latency and / or packet loss rate of the current network; The current network congestion level is determined according to the network delay and / or the packet loss rate; and the network status parameter includes the network congestion level.

9. The training method of a distributed computing system according to claim 7, characterized in that: Also includes: Obtaining the priority of each of the training tasks; Adjusting the communication interval in real time according to the current network status parameters includes: The communication interval time is adjusted in real time according to the current network status parameters and the priority of each training task.

10. The training method of a distributed computing system according to claim 9, characterized in that: The network status parameter is the degree of network congestion, and adjusting the communication interval time in real time according to the current network status parameter and the priority of each training task includes: Calculate the target communication interval time according to CIT=CIT0 / (1+NCD+DTP), and update the communication interval time to the target communication interval time; CIT is the target communication interval time, CIT0 is the initial communication interval time before updating, NCD is the network congestion degree, and DTP is the priority.

11. The training method of a distributed computing system according to claim 9, characterized in that: Get the priority of each training task, including: Obtaining a first weight, a first score, a second weight, and a second score for each of the training tasks, wherein the first weight and the first score are used to characterize the importance of the training task, and the second weight and the second score are used to characterize the urgency of the training task; The priority of each of the training tasks is determined according to the first weight, the first score, the second weight, and the second score corresponding to each of the training tasks.

12. The training method of a distributed computing system according to claim 11, characterized in that: Determining the priority of each of the training tasks according to the first weight, the first score, the second weight, and the second score corresponding to each of the training tasks includes: Calculate the priority of each training task according to DTP=WI·IS+WU·US; DTP is the priority, WI is the first weight, IS is the first score, WU is the second weight, and US is the second score.

13. A training device for a distributed computing system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the training method for a distributed computing system as described in any one of claims 1 to 12 when executing a computer program.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the training method for a distributed computing system according to any one of claims 1 to 12 are implemented.

15. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the training method for the distributed computing system as described in any one of claims 1 to 12 are implemented.

Images