Communication scheduling method, working node, parameter server, switch and system

Abstract

The embodiment of the invention provides a communication scheduling method, a working node, a parameter server, a switch and a system, and relates to the field of artificial intelligence. The method comprises the following steps: a parameter server sends training parameters to a plurality of working nodes based on a communication scheduling strategy, and schedules parameter gradient uploading of the plurality of working nodes; wherein the parameter gradient is obtained by performing forward propagation calculation on the pulled training parameters by the plurality of working nodes and then performing backward propagation calculation on each structural layer of the machine learning model in sequence. According to the scheme, the communication scheduling strategy is formulated based on the bandwidth data of the working node and the parameter server and the computing performance of the working node, so that the working node with high bandwidth and low computing performance can preferentially pull the training parameter, the bandwidth resource utilization rate is improved, the overall training duration is shortened, and the training efficiency is improved. And meanwhile, the data storage time of the high-computing-performance working nodes can be shortened, and the storage burden of a memory is reduced, so that the model training efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a communication scheduling method, a worker node, a parameter server, a switch and a system. BACKGROUND

[0002] A distributed machine learning system accelerates training and inference of large-scale datasets and complex models by distributing computing tasks to multiple nodes, and is suitable for scenarios such as large-scale data processing, complex model training, real-time data analysis and cross-regional data integration.

[0003] A distributed machine learning system usually includes a parameter server responsible for global model synchronization, a worker node performing local computation, and a communication framework for communication. In related solutions, the parameter gradients of the worker nodes are uploaded using a WFBP (Wait-Free Back Propagation) parallel strategy to perform synchronous training of the model. However, in this solution, the worker nodes start model training only after the parameter server sends training parameters to all worker nodes, which prolongs the total time cost of synchronous training. Therefore, the current problem to be solved is how to improve the efficiency of model training. SUMMARY

[0004] The embodiments of the present application provide a communication scheduling method, a worker node, a parameter server, a switch and a system to improve the efficiency of model training.

[0005] In a first aspect, the embodiments of the present application provide a communication scheduling method applied to a parameter server in a distributed machine learning system. The method includes sending training parameters to a plurality of worker nodes based on a communication scheduling strategy, wherein the communication scheduling strategy is determined based on bandwidth data between the worker nodes and the parameter server and computing performance of the worker nodes; and scheduling parameter gradient uploading of the plurality of worker nodes according to the communication scheduling strategy, wherein the parameter gradient is obtained by sequentially performing back propagation calculation on each structural layer of a machine learning model after forward propagation calculation of the training parameters pulled by the plurality of worker nodes.

[0006] In a second aspect, the embodiments of the present application provide a communication scheduling method applied to a plurality of worker nodes in a distributed machine learning system. The method includes pulling training parameters from a parameter server based on a communication scheduling strategy of the parameter server, wherein the communication scheduling strategy represents a priority of each worker node, and the priority is determined based on bandwidth data between the worker nodes and the parameter server and computing performance of the worker nodes; and sequentially performing, after forward propagation calculation of the pulled training parameters, back propagation calculation on each structural layer of a machine learning model to obtain a parameter gradient, and uploading the parameter gradient of the structural layer to the parameter server based on the communication scheduling strategy of the parameter server.

[0007] In a third aspect, an embodiment of the present application provides a parameter server configured to: send training parameters to a plurality of worker nodes based on a communication scheduling strategy; wherein the communication scheduling strategy is determined based on bandwidth data between the worker nodes and the parameter server and computing performance of the worker nodes; and schedule parameter gradient uploads from the plurality of worker nodes according to the communication scheduling strategy; wherein the parameter gradient is obtained by sequentially performing back propagation computation on each structural layer of a machine learning model after forward propagation computation of the training parameters pulled by the plurality of worker nodes.

[0008] In a fourth aspect, an embodiment of the present application provides a switch configured to: send training parameters to a plurality of worker nodes based on a communication scheduling strategy; wherein the communication scheduling strategy is determined based on bandwidth data between the worker nodes and the parameter server and computing performance of the worker nodes; and schedule parameter gradient uploads from the plurality of worker nodes according to the communication scheduling strategy; wherein the parameter gradient is obtained by sequentially performing back propagation computation on each structural layer of a machine learning model after forward propagation computation of the training parameters pulled by the plurality of worker nodes.

[0009] In a fifth aspect, an embodiment of the present application provides a worker node configured to: pull training parameters from a parameter server based on a communication scheduling strategy of the parameter server; wherein the communication scheduling strategy indicates a priority of each worker node, and the priority is determined based on bandwidth data between the worker nodes and the parameter server and computing performance of the worker nodes; and sequentially perform, after forward propagation computation of the pulled training parameters, back propagation computation on each structural layer of a machine learning model to obtain a parameter gradient of the structural layer, and upload the parameter gradient of the structural layer to the parameter server based on the communication scheduling strategy of the parameter server.

[0010] In a sixth aspect, an embodiment of the present application provides a distributed machine learning system comprising a plurality of the above worker nodes and the above parameter server.

[0011] In a seventh aspect, an embodiment of the present application provides a distributed machine learning system comprising a plurality of the above worker nodes, the above parameter server, and the above switch.

[0012] In the communication scheduling method, the working node, the parameter server, the switch and the system provided by the embodiments of the present application, the parameter server sends training parameters to a plurality of working nodes based on a communication scheduling strategy, and schedules parameter gradient uploading of the plurality of working nodes according to the communication scheduling strategy; wherein the parameter gradient is obtained by sequentially performing back propagation calculation on each structure layer of a machine learning model after forward propagation calculation of the training parameters pulled by the plurality of working nodes. The scheme of the present application can make the working nodes with high bandwidth and low computing performance preferentially pull training parameters by formulating a communication scheduling strategy based on bandwidth data of the working nodes and the parameter server and computing performance of the working nodes, thereby improving bandwidth resource utilization and reducing overall training time, and meanwhile reducing data storage time of the working nodes with high computing performance and reducing storage memory storage burden, so as to improve the efficiency of model training. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0014] Figure 1 A flowchart of the communication scheduling method provided by the embodiments of the present application is shown in FIG. 1.

[0015] Figure 2 A flowchart of the communication scheduling method provided by the embodiments of the present application is shown in FIG. 1.

[0016] Figure 3 A flowchart of the communication scheduling method provided by the embodiments of the present application is shown in FIG. 1.

[0017] Figure 4 A flowchart of the communication scheduling method provided by the embodiments of the present application is shown in FIG. 1.

[0018] Figure 5 A flowchart of the communication scheduling method provided by the embodiments of the present application is shown in FIG. 1.

[0019] Figure 6 A structure diagram of the distributed machine learning system provided by the embodiments of the present application is shown in FIG. 1.

[0020] Figure 7 A structure diagram of the distributed machine learning system provided by the embodiments of the present application is shown in FIG. 1.

[0021] Figure 8 A structure diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 1.

[0022] The specific embodiments of the application have been shown and described in the above drawings and text. These drawings and text are not meant to limit the scope of the inventive concept in any way but are merely meant to illustrate the inventive concept to one of ordinary skill in the art by reference to a particular embodiment. DETAILED DESCRIPTION

[0023] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The same reference numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the application as detailed in the appended claims.

[0024] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, public order and good customs are not violated, and appropriate operation portals are provided for users to choose authorization or refusal.

[0025] And the present application involves big data analysis of user information (including but not limited to personal biological characteristics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and uses artificial intelligence technology for automatic decision-making, and makes technical solutions based on automatic decision-making results that have a significant impact on personal rights and interests, provides corresponding operation portals for users to choose to agree or refuse automatic decision-making results; if the user chooses to refuse, the expert decision-making process is entered.

[0026] It should be noted that the communication scheduling method, working node, parameter server, switch and system provided by the present application can be used in the field of artificial intelligence, and can also be used in any field other than the field of artificial intelligence. The application field of the communication scheduling method, working node, parameter server, switch and system in the present application is not limited.

[0027] A distributed machine learning system is a hardware and software architecture that utilizes the computing, storage, and network resources of multiple computers to collaboratively train models or process massive amounts of data. It is typically used for training ultra-large-scale deep neural networks, large-scale graph neural networks, or fine-tuning pre-trained large models in natural language processing. A distributed training system usually consists of the following components: a cluster of computing nodes, also known as a cluster of worker nodes, composed of multiple physical or virtual servers, each equipped with a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or dedicated accelerator, responsible for executing part of the model's computational tasks; a parameter server, one or more server nodes dedicated to storing, distributing, and updating the latest global parameters of the model; and a scheduling and coordination framework responsible for managing cluster resources (task scheduling, such as allocating models to nodes), and node fault detection and recovery.

[0028] During the model training phase, the parameter server sends global parameters, such as weights and biases, along with multiple subsets of training data to the worker nodes. Each worker node applies the received global parameters to its local model copy and executes the following steps: first, it performs forward propagation, calculating the output (prediction) using its local model copy and the assigned data subset; then, it calculates the loss based on the prediction and labels; finally, it performs backpropagation, calculating the gradients of the model weights. Further, the worker nodes send the calculated gradients to the parameter server, allowing the system to aggregate gradients from all nodes to obtain new global parameters. In practical applications, training can be divided into synchronous and asynchronous training based on different training processes. Synchronous training involves waiting for gradients from all nodes to arrive, performing a global average, and then updating the global parameters. All nodes then use the updated parameters for the next iteration. Asynchronous training involves nodes immediately sending their calculated gradients to the parameter server for updates, without waiting for other nodes. In asynchronous training, the parameter server updates continuously, and nodes may use slightly older parameters in subsequent training iterations.

[0029] In related schemes, WFBP communication scheduling is typically used during model training. This improves training efficiency by overlapping gradient communication during backpropagation with computational operations as much as possible. In this scheme, when a worker node completes gradient calculation for a specific network layer on its assigned data shard, the gradient data for that layer is immediately and asynchronously pushed to the parameter server for aggregation or updating, without waiting for gradient calculations to complete for other network layers on the same node. This reduces idle time within a single node to some extent. However, the system still needs to complete gradient aggregation for all nodes for that iteration at the parameter server and broadcast the updated model parameters to all worker nodes before all nodes can use these consistent global parameters to start the next iteration of forward propagation training. Ultimately, the overall training cluster speed of this scheme is limited by the time it takes for the slowest communicating node to receive parameters, thus lengthening the overall time consumed by synchronous training.

[0030] The technical content provided in this application aims to solve the above-mentioned technical problems of the prior art. The communication scheduling method, worker nodes, parameter server, switch, and system provided in this application include: the parameter server sending training parameters to multiple worker nodes based on a communication scheduling strategy; and scheduling the upload of parameter gradients from multiple worker nodes according to the communication scheduling strategy. The parameter gradients are obtained by multiple worker nodes performing forward propagation calculations on the retrieved training parameters, followed by backward propagation calculations on each structural layer of the machine learning model. The solution in this application, by formulating a communication scheduling strategy based on the bandwidth data between the worker nodes and the parameter server and the computing performance of the worker nodes, allows worker nodes with high bandwidth and low computing performance to prioritize the retrieval of training parameters, improving bandwidth resource utilization and reducing the overall training time. Simultaneously, it reduces the data storage time of high-computing-performance worker nodes and lowers the storage burden on memory, thereby improving the efficiency of model training.

[0031] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0032] Figure 1 This is a flowchart illustrating the communication scheduling method provided in an embodiment of this application. Figure 1 As shown, this method is applied to a parameter server in a distributed machine learning system, and the method includes:

[0033] S101. Based on the communication scheduling strategy, send training parameters to multiple working nodes; wherein, the communication scheduling strategy is determined based on the bandwidth data between the working nodes and the parameter server, as well as the computing performance of the working nodes.

[0034] S102. According to the communication scheduling strategy, schedule the upload of parameter gradients of multiple working nodes; wherein, the parameter gradients are obtained by multiple working nodes performing forward propagation calculations on the training parameters pulled, and then performing backward propagation calculations on each structural layer of the machine learning model in turn.

[0035] In some embodiments, the content of this example can be applied to synchronous training of a distributed machine learning system or asynchronous training of a distributed machine learning system; this example does not limit this.

[0036] For example, in distributed machine learning training, bandwidth refers to the speed or capacity of the data transmission channel between worker nodes and parameter servers, typically measured in bits per second. Specifically, bandwidth depends on several factors: physical distance, such as whether the worker nodes and parameter servers are in the same data center; the quality of network infrastructure, such as routers and switches; current network congestion; and the overhead of the communication protocol.

[0037] For example, bandwidth may be high in a high-speed local area network, but it may decrease in cross-regional deployments due to increased latency and packet loss rates. In some embodiments, the system dynamically assesses bandwidth using real-time network probing tools or historical data monitoring, which helps avoid bottlenecks when transmitting critical data.

[0038] For example, the computing performance of a worker node refers to its ability to process and compute data, primarily influenced by its hardware configuration and workload. This includes CPU and GPU processing speeds (e.g., floating-point operations per second), memory size, cache efficiency, and current system utilization (e.g., whether other processes are consuming resources). Determining computing performance typically relies on performance benchmarking tools to measure metrics such as inference latency and throughput.

[0039] In practical applications, during training, the system can periodically collect resource usage data of worker nodes, such as GPU utilization, and obtain real-time status by querying hardware sensors or operating system interfaces. If worker nodes are frequently under high load, their computing performance will be considered low.

[0040] In some embodiments, a communication scheduling strategy can be determined based on the closed-loop time of each worker node. For example, the time required for each worker node to complete the entire closed loop of "fetching parameters, calculating, and uploading gradients" can be estimated first. Specifically, the estimate is based on the currently measured effective bandwidth of the worker node to calculate the communication time and the average time required for it to calculate the most recent batch of data to calculate the total time.

[0041] Furthermore, the estimated loop closure time of all worker nodes can be determined in real time, and whenever a communication action such as parameter fetching or gradient uploading needs to be scheduled, the worker nodes with the shortest estimated loop closure time are prioritized for operation. This example can be applied to one-step training scenarios of the model, enabling faster-reacting worker nodes to complete their work as early as possible and enter the next round, without being dragged down by slower worker nodes, thereby improving the overall iteration efficiency.

[0042] In some alternative embodiments, worker nodes can be assigned different weights based on their computational performance, such as the average time taken to process a batch, with higher weights for stronger computing power. The parameter server treats communication resources, such as bandwidth and communication windows, as a total allocable amount. During communication scheduling, whether a worker node is pulling parameters or uploading gradients, communication resources are allocated according to the worker node's weight. Worker nodes with stronger computing power receive a higher proportion of communication resources, such as earlier time slots, larger time windows, or higher scheduling priority. Simultaneously, the real-time bandwidth data of a worker node is used to control the speed or upper limit of data transmission within its allocated communication window. This example scheme prioritizes the communication needs of fast-computing worker nodes while not completely blocking the communication of slow-computing worker nodes, thus improving cluster resource utilization.

[0043] In some embodiments, a communication scheduling strategy can also be used to balance the loan data and computational performance of worker nodes. On one hand, worker nodes with high bandwidth but low computational performance are given higher priority. Specifically, high-bandwidth worker nodes can complete data transmission quickly, but due to slow computation, they generate gradients at a slower rate. To reduce overall training time, gradient uploads from these worker nodes can be prioritized. For example, worker node A has high bandwidth but low computational performance, resulting in long forward and backward propagation times; once gradients are generated, A is allowed to upload immediately because its upload speed is fast and it does not occupy bandwidth for a long time. In this way, the waiting time for other worker nodes is reduced, and the system avoids A's slow computation blocking the gradient aggregation process.

[0044] On the other hand, worker nodes with high computational performance but low bandwidth are given low priority. High-performance worker nodes generate gradients quickly, but their slow upload speeds due to low bandwidth allow the upload of node B to be delayed, enabling the high-bandwidth worker node to complete its upload first. In this example, the slow upload of worker node B reduces system congestion, allowing other worker nodes to advance their tasks faster, thus reducing overall waiting time.

[0045] In this example, during the gradient upload phase, each worker node employs a pipeline mechanism that overlaps computation and communication. This means that instead of waiting for all layers' backpropagation calculations to complete before uploading gradients, the upload operation for each layer's gradient is initiated immediately after calculation, allowing network transmission and the backpropagation calculations of the remaining layers to run in parallel. For instance, during model training, worker node A submits the gradient of the first layer to the upload queue immediately after completing the backpropagation of the first layer, while a background thread continues calculating the gradient of the second layer. When the network module transmits the first layer's gradient, node A's GPU is executing the second layer's backpropagation in parallel. Once the second layer's gradient is generated, its upload is immediately initiated, and this cycle continues until all layers' gradients have been calculated and transmitted. This asynchronous approach fully utilizes the worker nodes' computational resources and network bandwidth, significantly reducing the idle time spent waiting for uploads after computation is complete, as is typical in traditional synchronous uploads.

[0046] In the communication scheduling method provided in this application, the parameter server sends training parameters to multiple worker nodes based on a communication scheduling strategy; and schedules the upload of parameter gradients from multiple worker nodes according to the communication scheduling strategy. The parameter gradients are obtained by multiple worker nodes performing forward propagation calculations on the retrieved training parameters, followed by backward propagation calculations on each structural layer of the machine learning model. This application's solution, by formulating a communication scheduling strategy based on the bandwidth data between worker nodes and the parameter server and the computational performance of the worker nodes, allows high-bandwidth and low-computation-performance worker nodes to prioritize retrieving training parameters, improving bandwidth resource utilization and reducing overall training time. Simultaneously, it reduces data storage time for high-computation-performance worker nodes and lowers memory storage burden, thereby improving model training efficiency.

[0047] Figure 2 This is a flowchart illustrating the communication scheduling method provided in an embodiment of this application. Figure 2 As shown, the method also includes:

[0048] S201. Obtain upload time data for multiple worker nodes during the first training phase; wherein, the upload time data includes: the start time and upload time of the parameter gradient of each worker node in each structural layer;

[0049] S202. Determine multiple priority orders, and for each priority order, perform the following: calculate the additional communication cost under that priority order based on the upload time data; and calculate the total training time corresponding to that priority order based on the additional communication cost.

[0050] S203. Determine the communication scheduling strategy based on the priority order of the shortest total training time.

[0051] In some embodiments, the first training phase of this example can be a pre-training phase of the model. Specifically, during the pre-training phase, the model structure is fixed and the load pattern is stable, enabling the collection of representative temporal features without affecting the efficiency of formal training. Optionally, the first training phase can also set a specific data collection window for formal training, as long as the model structure does not change during that window.

[0052] For example, when determining multiple priority orders, the worker nodes are usually subjected to a full permutation or a finite sequence is generated based on heuristic rules. For example, the priority orders that may be formed for a high-bandwidth, high-performance worker node A, a high-performance, low-bandwidth worker node B, and a worker node with balanced bandwidth and performance include A>B>C, B>C>A, C>A>B, etc., where each order represents the scheduling sequence of the gradient upload queue.

[0053] Furthermore, when calculating the additional communication cost based on upload time data, it is necessary to analyze the gradient upload behavior of each structural layer: apply the priority order to the timestamps and time logs collected during pre-training, simulate the actual gradient upload process, and compare the gap between the ideal parallel upload time and the actual execution time. This gap is composed of the accumulated blocking time caused by the high-priority nodes occupying bandwidth, which forces the low-priority nodes to wait. For example, in the order B>C>A, if the low-bandwidth upload of worker node B continues to occupy the link, the gradient upload operations of worker nodes C and A will be delayed. The sum of these delays is the additional communication cost of this order.

[0054] In some embodiments, the calculation of the total training time integrates the computation and communication phases: based on the computation time of the complete backpropagation, the additional communication costs under this priority order are added, and the time spent on parameter server gradient aggregation and model updates can also be considered. The sum of these three is the estimated training iteration time. Finally, the priority order that minimizes this total time can be selected as the formal communication scheduling strategy.

[0055] The solution presented in this example optimizes the communication strategy for random scheduling by quantitatively calculating the impact of different scheduling strategies on communication congestion. This transforms the time wasted on communication waiting into effective training time, thereby improving the efficiency of model training.

[0056] Figure 3This is a flowchart illustrating the communication scheduling method provided in an embodiment of this application. Figure 3 As shown, in S202, based on upload time data, the additional communication cost under the initial priority order is calculated, including:

[0057] S301. For any adjacent parameter gradient upload, if the end time of the upload of the previous parameter gradient is less than the start time of the upload of the next parameter gradient, then the time interval between the end time of the upload of the previous parameter gradient and the start time of the upload of the next parameter gradient shall be used as the first additional communication cost.

[0058] S302. If the end time of the upload of the gradient of the previous parameter is greater than the start time of the upload of the gradient of the next parameter, then the time interval between the end time of the upload of the gradient of the previous parameter and the start time of the upload of the gradient of the next parameter shall be used as the second additional communication cost.

[0059] S303. Calculate the difference between the second additional communication cost and the first additional communication cost for each adjacent parameter gradient upload, and use the sum of the differences as the additional communication cost.

[0060] In practical applications, when multiple worker nodes' parameter gradient upload streams are scheduled for execution, the start and end times of each gradient upload task can be monitored. Specifically, for any two consecutively uploaded gradient tasks, such as worker node A uploading its Lth layer gradient followed immediately by worker node B uploading its L+1th layer gradient, the end time of worker node A's Lth layer upload task is first compared with the start time of worker node B's L+1th layer upload task.

[0061] On the one hand, if the end time of the previous task is earlier than the start time of the subsequent task, it indicates that there is an idle time interval between the two uploads, which is defined as the first additional communication cost. The first additional communication cost reflects the gap loss due to underutilization of network bandwidth. For example, if the Lth layer upload task of worker node A completes transmission at t=5 seconds while the L+1th layer upload task of worker node B starts at t=5.3 seconds, then the 0.3-second interval is the first additional communication cost.

[0062] On the other hand, if the end time of the preceding task is later than the start time of the following task, it means that the following task is forcibly delayed in starting because it is waiting for the bandwidth released by the preceding task. The overlapping period between their timestamps is actually the blocking time, which is recorded as the second additional communication cost. For example, if the Lth layer upload task of worker node A ends at t=5.2 seconds, while the L+1th layer upload task of worker node B, which should have started at t=5 seconds, is delayed until t=5.2 seconds, then the 0.2-second waiting time is the second additional communication cost.

[0063] In this example, the first additional communication cost represents wasted bandwidth, and the second additional communication cost represents wasted waiting time due to communication conflicts. The difference between the second cost and the first cost in each pair of adjacent upload tasks is globally accumulated, and the final sum is the additional communication cost of the overall communication process.

[0064] For example, if the number of worker nodes is n, and the number of structural layers or gradient uploads is m, the start time of the upload of the m-th parameter gradient of the first worker node can be expressed as: The upload time can be expressed as The following set of start times can be created first. Collection of upload times :

[0065]

[0066]

[0067] Furthermore, according to a certain priority order, the set of starting times can be... Obtain by arranging And correspondingly, the upload time collection will be compiled. Obtain by arranging Additional communication costs The following loop statement can be used for calculation:

[0068] = 0

[0069] For i in (1, n×m):

[0070] {

[0071] If :

[0072]

[0073] Else:

[0074]

[0075] }

[0076] Where i is Traversing variables in the context of additional communication costs A settlement is performed after any adjacent parameter gradient is uploaded.

[0077] The scheme in this example quantifies the loss of network resource utilization efficiency and conflict severity under the current scheduling strategy by calculating the cumulative value of communication costs for different priority orders, and thus can determine the communication scheduling strategy that minimizes additional communication costs.

[0078] Figure 4 This is a flowchart illustrating the communication scheduling method provided in an embodiment of this application. Figure 4 As shown in S202, the total training time corresponding to the initial priority order is calculated based on the additional communication cost, including:

[0079] S401. For each worker node, sum and calculate the following training time: the parameter fetching time of the worker node and each worker node before it, the forward propagation calculation time and backward propagation calculation time of the worker node, and the upload time of the last parameter gradient upload of the worker node.

[0080] S402. The sum of the longest training time and the additional communication cost is taken as the total training time.

[0081] For example, a complete training timeline is constructed for each worker node under each priority order to calculate the total training duration. Specifically, the scheduling process under a specific priority order can be calculated: for worker node A, its training duration needs to accumulate the parameter fetching time from the start of global training (i.e., the transmission time of downloading model parameters from the parameter server), the subsequent independently executed forward propagation computation time, the backpropagation computation time executed layer by layer including the internal gradient generation time, and the upload time of the last parameter gradient upload, such as the time to complete the final layer gradient transmission.

[0082] In some embodiments, when the priority order is A > B > C, the training time of worker node B will include the pull time of node A (because A's priority pull may occupy bandwidth), B's own forward and backward computation time, and the time spent on B's last gradient upload. Similarly, the training time of node C will be the sum of the pull times of A and B, C's own computation time, and upload time. By comparing the training times of all worker nodes, the timeline of the node with the longest time is selected. For example, if node C needs 20 seconds to wait for the preceding bandwidth, while node B only needs 15 seconds, then 20 seconds is taken, and the additional communication costs previously calculated under this priority order, such as the 3-second gap loss caused by inter-node communication blockage, are added to obtain a total training time of 23 seconds.

[0083] The scheme in this example takes into account the time consumed by the worker nodes to retrieve parameters under different priority orders, as well as the computation time and communication cost of the worker nodes. It can quantitatively calculate the total training time under different priority orders, providing a quantitative comparison standard for determining the optimal communication scheduling scheme.

[0084] As yet another example, based on any example, the parameter fetching time for a worker node is determined based on the number of training parameters and the bandwidth data of that worker node.

[0085] In some embodiments, the parameter fetching time of a worker node can be dynamically calculated based on the total amount of training parameters and the effective bandwidth available to the worker node in real time. Specifically, the scale of parameters to be transmitted in the current iteration can be obtained first, for example, worker node A needs to fetch fully connected layer parameters containing 5 million weights. At the same time, bandwidth data collected by network monitoring can be combined, such as the available bandwidth from worker node A to the parameter server dropping from the theoretical peak of 1Gbps to the actual 600Mbps due to network congestion. Finally, the total number of parameter bytes is divided by the effective bandwidth value to obtain the theoretical transmission time, such as 500 MB ÷ 75 MB / s ≈ 6.67 seconds.

[0086] In some alternative embodiments, the calculation may also take into account protocol overhead and resource contention mechanisms during parallel transmission. That is, when multiple nodes pull concurrently, the system dynamically allocates bandwidth resources based on a priority strategy. For example, node A gets 60% of the bandwidth while node B only gets 40%.

[0087] The scheme in this example calculates the parameter retrieval time of the working node based on the number of training parameters and the bandwidth data of the working ground, which can improve the accuracy of parameter retrieval time calculation.

[0088] Figure 5 This is a flowchart illustrating the communication scheduling method provided in an embodiment of this application. Figure 5 As shown, Figure 5 The upper part of the diagram illustrates a situation where the worker node does not fetch parameters according to the communication scheduling strategy, i.e., there is no inter-node scheduling. The lower part of the diagram illustrates a situation where the worker node fetches parameters according to the communication scheduling strategy. Different colors represent different worker nodes, and T... pull The parameters are fetched for the worker nodes. f1-f4 represent the forward propagation calculations for the worker nodes, b1-b4 represent the backward propagation calculations, and 4-1 represents the parameter gradient upload. As shown in the figure, after communication scheduling for parameter fetching by the worker nodes, the training time for all worker nodes in one round is shorter than that without communication scheduling for parameter fetching, thus improving the efficiency of model training.

[0089] In the communication scheduling method provided in this application, the parameter server sends training parameters to multiple worker nodes based on a communication scheduling strategy; and schedules the upload of parameter gradients from multiple worker nodes according to the communication scheduling strategy. The parameter gradients are obtained by multiple worker nodes performing forward propagation calculations on the retrieved training parameters, followed by backward propagation calculations on each structural layer of the machine learning model. This application's solution, by formulating a communication scheduling strategy based on the bandwidth data between worker nodes and the parameter server and the computational performance of the worker nodes, allows high-bandwidth and low-computation-performance worker nodes to prioritize retrieving training parameters, improving bandwidth resource utilization and reducing overall training time. Simultaneously, it reduces data storage time for high-computation-performance worker nodes and lowers memory storage burden, thereby improving model training efficiency.

[0090] This application also provides a communication scheduling method applied to multiple worker nodes in a distributed machine learning system. The method includes:

[0091] The communication scheduling strategy based on the parameter server pulls training parameters from the parameter server; where the communication scheduling strategy represents the priority of each worker node, and the priority is determined based on the bandwidth data between the worker node and the parameter server, as well as the computing performance of the worker node.

[0092] After performing forward propagation calculations on the retrieved training parameters, the following steps are performed sequentially for each structural layer of the machine learning model: performing backward propagation calculations on the structural layer to obtain the parameter gradients, and uploading the parameter gradients of the structural layer to the parameter server based on the parameter server's communication scheduling strategy.

[0093] The communication scheduling method provided in this embodiment is based on multiple worker nodes in a distributed machine learning system. Its implementation principle and technical effect are similar to those of the communication scheduling method based on the parameter server in a distributed machine learning system, and will not be described in detail here.

[0094] This application embodiment also provides a parameter server, used for:

[0095] Based on a communication scheduling strategy, training parameters are sent to multiple worker nodes. The communication scheduling strategy is determined based on the bandwidth data between the worker nodes and the parameter server, as well as the computing performance of the worker nodes. According to the communication scheduling strategy, the parameter gradients of multiple worker nodes are scheduled to be uploaded. The parameter gradients are obtained by multiple worker nodes performing forward propagation calculations on the retrieved training parameters, and then performing backward propagation calculations on each structural layer of the machine learning model in sequence.

[0096] As yet another example, parameter servers are also used for:

[0097] Acquire upload time data for multiple worker nodes during the first training phase; the upload time data includes: the start time and upload time of the parameter gradient of each worker node in each structural layer; determine multiple priority orders, and for each priority order, perform: calculate the additional communication cost under that priority order based on the upload time data; and calculate the total training time corresponding to that priority order based on the additional communication cost; determine the communication scheduling strategy based on the priority order with the shortest total training time.

[0098] As yet another example, the parameter server is specifically used for:

[0099] For any adjacent parameter gradient upload, if the end time of the previous parameter gradient upload is less than the start time of the next parameter gradient upload, the time interval between the end time of the previous parameter gradient upload and the start time of the next parameter gradient upload is used as the first additional communication cost; if the end time of the previous parameter gradient upload is greater than the start time of the next parameter gradient upload, the time interval between the end time of the previous parameter gradient upload and the start time of the next parameter gradient upload is used as the second additional communication cost; calculate the difference between the second additional communication cost and the first additional communication cost for each adjacent parameter gradient upload, and use the sum of the differences as the additional communication cost.

[0100] As yet another example, the parameter server is specifically used for:

[0101] For each worker node, the following training durations are summed: the parameter fetching duration of the worker node and each worker node before it, the forward and backward propagation computation durations of the worker node, and the upload time of the last parameter gradient upload of the worker node; the sum of the longest training duration and the additional communication cost is taken as the total training duration.

[0102] As yet another example, the parameter fetching time for a worker node is determined based on the number of training parameters and the bandwidth data of that worker node.

[0103] The parameter server provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0104] This application embodiment also provides a switch for: sending training parameters to multiple worker nodes based on a communication scheduling strategy; wherein the communication scheduling strategy is determined based on the bandwidth data between the worker nodes and the parameter server, as well as the computing performance of the worker nodes; scheduling the upload of parameter gradients of multiple worker nodes according to the communication scheduling strategy; wherein the parameter gradients are obtained by multiple worker nodes performing forward propagation calculations on the training parameters they have pulled, and then performing backward propagation calculations on each structural layer of the machine learning model in sequence.

[0105] It should be noted that the switch in this example should be programmable, commonly referred to as a programmable switch or software-defined network switch. Programmability allows users to dynamically adjust the switch's behavior by writing specific programs or rules to adapt to different communication needs and strategies.

[0106] In some embodiments, switches require high-performance processors and sufficient memory to handle complex scheduling algorithms and large data flows. In practical applications, switches also need multiple high-speed network interfaces to support high-bandwidth data transmission.

[0107] For example, a switch needs to be able to identify and process IP packets in order to determine the forwarding path of the packets according to communication scheduling policies. Specifically, the switch can determine the direction of data flow by parsing packet header information such as source IP address, destination IP address, and port number. The switch monitors the bandwidth usage between each working node and the parameter server in real time. Based on this data, the switch can dynamically adjust the priority of data flows to optimize the use of network resources.

[0108] In some embodiments, the switch is responsible for scheduling the transmission of training parameters and the uploading of parameter gradients according to a pre-defined communication scheduling strategy. Specifically, the switch can achieve this by adjusting the priority of data packets, selecting different routing paths, or controlling the data flow rate.

[0109] In some alternative implementations, the switch can implement load balancing and flow control policies, such as dynamically adjusting the path of data flows and limiting the bandwidth usage of certain flows, to reduce network congestion and improve the fairness of resource allocation.

[0110] As yet another example, switches are also used for:

[0111] Acquire upload time data for multiple worker nodes during the first training phase; the upload time data includes: the start time and upload time of the parameter gradient of each worker node in each structural layer; determine multiple priority orders, and for each priority order, perform: calculate the additional communication cost under that priority order based on the upload time data; and calculate the total training time corresponding to that priority order based on the additional communication cost; determine the communication scheduling strategy based on the priority order with the shortest total training time.

[0112] As yet another example, a switch is specifically used for:

[0113] For any adjacent parameter gradient upload, if the end time of the previous parameter gradient upload is less than the start time of the next parameter gradient upload, the time interval between the end time of the previous parameter gradient upload and the start time of the next parameter gradient upload is used as the first additional communication cost; if the end time of the previous parameter gradient upload is greater than the start time of the next parameter gradient upload, the time interval between the end time of the previous parameter gradient upload and the start time of the next parameter gradient upload is used as the second additional communication cost; calculate the difference between the second additional communication cost and the first additional communication cost for each adjacent parameter gradient upload, and use the sum of the differences as the additional communication cost.

[0114] As yet another example, a switch is specifically used for:

[0115] For each worker node, the following training durations are summed: the parameter fetching duration of the worker node and each worker node before it, the forward and backward propagation computation durations of the worker node, and the upload time of the last parameter gradient upload of the worker node; the sum of the longest training duration and the additional communication cost is taken as the total training duration.

[0116] As yet another example, the parameter fetching time for a worker node is determined based on the number of training parameters and the bandwidth data of that worker node.

[0117] The switch provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0118] This application embodiment also provides a working node for: pulling training parameters from a parameter server based on a communication scheduling strategy of the parameter server; wherein, the communication scheduling strategy characterizes the priority of each working node, and the priority is determined based on the bandwidth data between the working node and the parameter server, as well as the computing performance of the working node; after performing forward propagation calculation on the pulled training parameters, performing backward propagation calculation on each structural layer of the machine learning model to obtain the parameter gradient, and uploading the parameter gradient of the structural layer to the parameter server based on the communication scheduling strategy of the parameter server.

[0119] The working node provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0120] Figure 6 This is a schematic diagram of the structure of the distributed machine learning system provided in the embodiments of this application, such as... Figure 6 As shown, it includes multiple working nodes as in any of the above embodiments and a parameter server as in any of the above embodiments.

[0121] Figure 7 This is a schematic diagram of the structure of the distributed machine learning system provided in the embodiments of this application, such as... Figure 7 As shown, it includes multiple worker nodes, parameter servers, and switches as described in any of the above embodiments.

[0122] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device provided in this embodiment includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can call logical instructions in the memory 292 to execute the method described above.

[0123] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0124] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, that is, it implements the methods in the above method examples.

[0125] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0126] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0127] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0128] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0129] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0130] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0131] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0132] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0133] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A communication scheduling method, characterized in that, A parameter server applied in a distributed machine learning system, the method comprising: Training parameters are sent to multiple worker nodes based on a communication scheduling strategy; wherein the communication scheduling strategy is determined based on the bandwidth data between the worker nodes and the parameter server, as well as the computing performance of the worker nodes. According to the communication scheduling strategy, the parameter gradients of multiple working nodes are scheduled to be uploaded; wherein, the parameter gradients are obtained by the multiple working nodes performing forward propagation calculations on the training parameters pulled, and then performing backward propagation calculations on each structural layer of the machine learning model in turn.

2. The method according to claim 1, characterized in that, The method further includes: The upload time data of multiple worker nodes in the first training phase is obtained; wherein, the upload time data includes: the start time and upload time of the parameter gradient of each worker node in each structural layer; Multiple priority orders are determined, and for each priority order, the following steps are performed: based on the upload time data, the additional communication cost under that priority order is calculated; and based on the additional communication cost, the total training time corresponding to that priority order is calculated. The communication scheduling strategy is determined based on the priority order of the shortest total training time.

3. The method according to claim 2, characterized in that, The calculation of additional communication costs based on the upload time data under the priority order includes: For any adjacent parameter gradient upload, if the end time of the upload of the previous parameter gradient is less than the start time of the upload of the next parameter gradient, then the time interval between the end time of the upload of the previous parameter gradient and the start time of the upload of the next parameter gradient is taken as the first additional communication cost. If the end time of the upload of the gradient of the previous parameter is greater than the start time of the upload of the gradient of the next parameter, then the time interval between the end time of the upload of the gradient of the previous parameter and the start time of the upload of the gradient of the next parameter is used as the second additional communication cost. Calculate the difference between the second additional communication cost and the first additional communication cost for each adjacent parameter gradient upload, and use the sum of the differences as the additional communication cost.

4. The method according to claim 2, characterized in that, The step of calculating the total training time corresponding to the initial priority order based on the additional communication cost includes: For each worker node, the following training durations are summed: the parameter fetching duration of the worker node and each worker node before it, the forward and backward propagation computation durations of the worker node, and the upload time of the last parameter gradient upload of the worker node. The total training time is the sum of the longest training time and the additional communication cost.

5. The method according to claim 4, characterized in that, The parameter retrieval time of the working node is determined based on the number of training parameters and the bandwidth data of the working node.

6. A communication scheduling method, characterized in that, The method, applied to multiple worker nodes in a distributed machine learning system, includes: The communication scheduling strategy based on the parameter server pulls training parameters from the parameter server; wherein, the communication scheduling strategy characterizes the priority of each working node, and the priority is determined based on the bandwidth data between the working node and the parameter server, as well as the computing performance of the working node. After performing forward propagation calculations on the retrieved training parameters, the following steps are performed sequentially for each structural layer of the machine learning model: performing backward propagation calculations on the structural layer to obtain the parameter gradients, and uploading the parameter gradients of the structural layer to the parameter server based on the communication scheduling strategy of the parameter server.

7. A parameter server, characterized in that, Used for: Training parameters are sent to multiple worker nodes based on a communication scheduling strategy; wherein the communication scheduling strategy is determined based on the bandwidth data between the worker nodes and the parameter server, as well as the computing performance of the worker nodes. According to the communication scheduling strategy, the parameter gradients of multiple working nodes are scheduled to be uploaded; wherein, the parameter gradients are obtained by the multiple working nodes performing forward propagation calculations on the training parameters pulled, and then performing backward propagation calculations on each structural layer of the machine learning model in turn.

8. A switch, characterized in that, Used for: The training parameter retrieval of multiple worker nodes is scheduled based on a communication scheduling strategy; wherein, the communication scheduling strategy is determined based on the bandwidth data between the worker nodes and the parameter server, as well as the computing performance of the worker nodes. According to the communication scheduling strategy, the parameter gradients of multiple working nodes are scheduled to be uploaded; wherein, the parameter gradients are obtained by the multiple working nodes performing forward propagation calculations on the training parameters pulled, and then performing backward propagation calculations on each structural layer of the machine learning model in turn.

9. A working node, characterized in that, Used for: The communication scheduling strategy based on the parameter server pulls training parameters from the parameter server; wherein, the communication scheduling strategy characterizes the priority of each working node, and the priority is determined based on the bandwidth data between the working node and the parameter server, as well as the computing performance of the working node. After performing forward propagation calculations on the retrieved training parameters, the following steps are performed sequentially for each structural layer of the machine learning model: performing backward propagation calculations on the structural layer to obtain the parameter gradients, and uploading the parameter gradients of the structural layer to the parameter server based on the communication scheduling strategy of the parameter server.

10. A distributed machine learning system, characterized in that, It includes multiple working nodes as described in claim 9 and a parameter server as described in claim 7.

11. A distributed machine learning system, characterized in that, It includes multiple working nodes as described in claim 9, a parameter server, and a switch as described in claim 8.

Images