Distributed training communication compression method and device and distributed training system

Abstract

The invention provides a distributed training communication compression method and device and a distributed training system. The method comprises the following steps: grouping global gradient information of each node in a distributed cluster; selecting the first K gradient values with the maximum gradient absolute value in each group, setting the rest unselected gradient values as 0, and generating a sparse gradient matrix; the sparse gradient matrix is shared among the nodes. According to the method, the communication traffic among the nodes in the edge device training cluster can be effectively reduced, so that more efficient distributed cooperative training is realized, and the precision of the model is kept.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing, and specifically relates to a distributed training communication compression method, device, and distributed training system suitable for edge computing scenarios. Background Art

[0002] Most of the current distributed deep neural network training is based on cloud server cluster environments, relying on abundant computing resources and high-speed networks for large-scale model training. However, with the rise of edge computing, there is a growing demand for on-site intelligent processing using massive distributed data directly at the edge. There are huge challenges in deploying distributed training to resource-constrained edge devices. The hardware conditions of edge devices, such as bandwidth and computing power, are generally weak, and distributed training is extremely dependent on high-frequency communication between nodes. Existing studies have shown that communication overhead can account for more than 80% of the total time of distributed training. Therefore, the distributed training scheme of the existing cloud server environment is not suitable for the edge, and communication optimization methods for edge devices are urgently needed.

[0003] Therefore, it is easy to think of compressing the communication volume in distributed training to reduce the communication volume between nodes and thus speed up the overall training speed. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a distributed training communication compression method, device, and distributed training system suitable for edge computing scenarios, which can reduce the communication volume of edge devices while obtaining a distributed acceleration effect, thereby achieving the purpose of efficient distributed training in an edge environment.

[0005] In order to achieve the above object, the present invention provides a distributed training communication compression method, which is applicable to edge computing scenarios. The method comprises:

[0006] Group the global gradient information of each node in the distributed cluster;

[0007] In each group, select the first K gradient values ​​with the largest absolute value of the gradient, set the remaining unselected gradient values ​​to 0, and generate a sparse gradient matrix;

[0008] The sparse gradient matrix is ​​shared among the nodes.

[0009] In some embodiments, the group size and the gradient compression rate are set;

[0010] Grouping the global gradient information according to the group size;

[0011] In each group, the value of K is determined according to the product of the gradient compression rate and the group size.

[0012] In some embodiments, sharing the sparse gradient matrix among nodes comprises the following steps:

[0013] a1) Each node divides its own sparse gradient matrix according to the total number of nodes N to obtain N equal sub-gradient data blocks;

[0014] a2) Each node selects one of the sub-gradient data blocks to send to its neighboring nodes;

[0015] a3) repeating step a2) until each node sends each sub-gradient data block of its sparse gradient matrix at least once;

[0016] a4) Each node accumulates the sub-gradient data blocks received from other nodes and a sub-gradient data block held by itself to obtain an aggregated gradient data block.

[0017] In some embodiments, sharing the sparse gradient matrix among the nodes further comprises the following steps:

[0018] b1) Each node sends the aggregated gradient data block it holds to its neighboring nodes;

[0019] b2) the neighboring node replaces the original gradient data block at the corresponding position with the received aggregated gradient data block;

[0020] b3) the neighboring node sends the received aggregated gradient data block to another neighboring node, and the other neighboring node replaces the original gradient data block at the corresponding position with the received aggregated gradient data block;

[0021] b4) Repeat steps b1)-b4) until each node contains N aggregated gradient data blocks, and the N aggregated gradient data blocks are aggregated into a complete gradient matrix.

[0022] In some embodiments, it further comprises:

[0023] Adjust the learning rate during distributed training to compensate for loss in model accuracy.

[0024] In some embodiments, a Linear Scaling Rule is used to compensate the learning rate, including:

[0025] According to the increase in the number of nodes in distributed training, the learning rate is increased linearly to compensate for the change in the model iteration step size caused by the increase in the number of nodes.

[0026] In some embodiments, it further comprises:

[0027] The distributed cluster is constructed using a given distributed cluster framework, and data is distributed to each node of the distributed cluster before training begins. The initial data for training each node is different.

[0028] Another aspect of the present invention further provides a distributed training communication compression device, comprising:

[0029] The grouping module is used to group the global gradient information of each node in the distributed cluster;

[0030] The gradient compression module is used to select the first K gradient values ​​with the largest absolute value of the gradient in each group, set the remaining unselected gradient values ​​to 0, and generate a sparse gradient matrix;

[0031] The sharing module is used to share the sparse gradient matrix among the nodes.

[0032] In some embodiments, within each group, the value of K is determined according to the product of a preset gradient compression rate and the group size.

[0033] In another aspect, the present invention further provides a distributed training system, comprising:

[0034] Multiple edge device nodes, each edge device node is configured to execute the above-mentioned distributed training communication compression method.

[0035] It can be seen from the above scheme that the advantages of the present invention are:

[0036] The distributed training communication compression method disclosed by the present invention is suitable for edge computing scenarios. It uses the Group-TopK algorithm to group the global gradient information of each node in the distributed cluster; selects the top K gradient values ​​with the largest absolute value of the gradient in each group, sets the remaining unselected gradient values ​​to 0, and generates a sparse gradient matrix; and shares the sparse gradient matrix between the nodes. This method can effectively reduce the amount of communication between nodes in the edge device training cluster, thereby achieving more efficient distributed collaborative training and maintaining the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the overall process of a distributed training communication compression method provided by an embodiment of the present invention is shown;

[0038] Figure 2 The comparison of the sparsification principles of Group-TopK and TopK is shown;

[0039] Figure 3 A schematic diagram of the Scatter-Reduce phase is shown;

[0040] Figure 4A schematic diagram of the Allgather phase is shown;

[0041] Figure 5 The experimental effect of using linear scaling rules is shown (Resnet18 neural network model);

[0042] Figure 6 The experimental effect of using linear scaling rules is shown (Resnet50 neural network model);

[0043] Figure 7 A schematic block diagram of a distributed training communication compression device is shown.

[0044] in:

[0045] 200: distributed training communication compression device;

[0046] 210: grouping module;

[0047] 220: gradient compression module;

[0048] 230: Shared module. DETAILED DESCRIPTION

[0049] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0050] See also Figure 1 As shown in Figure 1 The figure shows an overall flow chart of a distributed training communication compression method provided by an embodiment of the present invention.

[0051] The distributed training communication compression method suitable for edge computing scenarios includes the following steps:

[0052] Step S1: group the global gradient information of each node in the distributed cluster.

[0053] In this embodiment, the distributed cluster is built using a given distributed cluster framework, such as Pytorch's own distributed cluster framework Pytorch DDP, and each node of the distributed cluster is configured as an edge device node. In addition, data is distributed to each node of the distributed cluster before training begins, and the initial data for training each node is different.

[0054] In one iteration of distributed training, all nodes calculate their own global gradient information and then aggregate the gradient information. In order to reduce the amount of gradient information, existing methods usually use TopK (Top K selection algorithm) to sort the global gradient information and only retain the gradient information with the largest gradient values ​​of the top K. However, there are some problems with the TopK method, especially when the gradient distribution of the model is sparse, or most of the gradients are small, TopK sparsification may not be applicable. In this case, most of the gradients will be truncated, which may cause the model to fail to train normally. In other words, TopK sparsification has problems when the gradient distribution is uneven.

[0055] Therefore, in this embodiment, the existing TopK method is considered to be improved, and the Group-TopK method is designed to optimize the situation of uneven gradient distribution. Specifically, Group-TopK is used to divide the global gradient information into interval groups, and then each group uses the TopK method to select the gradient information with the largest first K gradient values. This method can effectively retain the local information of the gradient while keeping the overall gradient sparsity unchanged, thereby avoiding the gradient truncation problem caused by uneven distribution of gradient information.

[0056] The Group-TopK algorithm has two hyperparameters: group size g and gradient compression rate s. The group size g determines how many groups the gradient matrix is ​​divided into, while the gradient compression rate s determines the proportion of gradient values ​​retained in each group.

[0057] Specifically, the global gradient information of each node is regrouped according to a preset grouping size g. Usually, each node is grouped according to the number of its global gradient information elements by the preset grouping size g. For example, if the global gradient information of a node has n elements, then the global gradient information will be divided into n / g groups.

[0058] Figure 2 The comparison of the sparsification principles between Group-TopK and TopK is shown. Figure 2In (a), different colors are used to represent the size of the values ​​in the gradient matrix, which has a dimension of 25x16. In (b), the original gradient matrix is ​​reshaped into a 20x20 grid, and the group size is set to g=20. In (c), the Group-TopK algorithm is used to compress the gradient information by retaining the largest k=2 values ​​in each group and setting the remaining values ​​to zero, thereby achieving a gradient compression rate of 10%. (d) shows the TopK algorithm, which maintains the same compression rate as Group-TopK, that is, retaining the global largest 10% of the values ​​and setting the remaining values ​​to zero. By comparing (c) and (d), it is found that Group-TopK exhibits more uniform characteristics in selecting important gradient values, while TopK may tend to be locally concentrated. Therefore, using Group-TopK for gradient compression can avoid the uneven distribution of selected gradients to a certain extent, thereby enhancing the convergence performance of model training.

[0059] Step S2: Select the first K gradient values ​​with the largest absolute gradient values ​​in each group, set the remaining unselected gradient values ​​to 0, and generate a sparse gradient matrix.

[0060] In this embodiment, after the global gradient information of each node is grouped, the TopK method is further used in each group to select the top K gradient values ​​with the largest absolute gradient values ​​and retain them, and the remaining unselected gradient values ​​are set to 0 to generate a sparse gradient matrix, which reduces the communication volume and retains local information.

[0061] Specifically, in each group, the value of K is determined according to the product of the gradient compression rate and the group size, that is, K=s·g.

[0062] Through Group-TopK calculation, a sparse gradient matrix is ​​obtained. This sparse gradient matrix not only significantly reduces the amount of data that needs to be transmitted, but also retains the most important information in the gradient.

[0063] In addition, in some embodiments, the sparse gradient matrix is ​​further converted to a sparse matrix format for storage, which is more efficient because it only records non-zero elements and their positions.

[0064] In addition, in this embodiment, the gradient compression rate and the group size can be adjusted according to actual conditions to obtain the optimal gradient compression effect.

[0065] Step S3: Share the sparse gradient matrix among the nodes.

[0066] In this embodiment, after each node generates its own sparse gradient matrix, it further shares the sparse gradient matrix with other nodes.

[0067] In order to further reduce the communication overhead, a decentralized ring communication method is adopted, which can effectively avoid overloading the bandwidth of the central node, thereby limiting the overall training speed.

[0068] Specifically, in some embodiments, a Ring-AllReduce decentralized communication mode is adopted, and each node communicates with its neighbor nodes to obtain data based on the Ring-AllReduce decentralized communication architecture until the data of all nodes are consistent. At this time, the sparse gradient matrix of each node is shared and enters the next round of iteration.

[0069] Specifically, the sharing process consists of two stages. The first stage, the Scatter-Reduce stage, is described in Figure 3 As shown in , first, in step a1), each node divides its own sparse gradient matrix according to the total number of nodes N to obtain N equal sub-gradient data blocks; then in step a2), each node selects one of the sub-gradient data blocks to send to its neighboring nodes, such as Figure 3 As shown in Figure (a); Step a3) repeats step a2) until each node sends each sub-gradient data block of its sparse gradient matrix at least once, as ... Figure 3 As shown in Figure (b); the last step a4) each node accumulates the sub-gradient data blocks received from other nodes and a sub-gradient data block held by itself to obtain an aggregated gradient data block, such as Figure 3 As shown in Figure (c).

[0070] The second stage, the Allgather stage, see Figure 4 As shown in , in this process: first, step b1) each node sends the aggregated gradient data block it holds to its neighboring nodes, such as Figure 4 As shown in Figure (a); step b2) the neighbor node replaces the original gradient data block at the corresponding position with the received aggregated gradient data block; at the same time, step b3) the neighbor node sends the received aggregated gradient data block to another neighbor node, and the other neighbor node replaces the original gradient data block at the corresponding position with the received aggregated gradient data block, such as Figure 4 As shown in Figure (b); finally, b4) repeats steps b1)-b4) until each node contains N aggregated gradient data blocks, and the N aggregated gradient data blocks are aggregated into a complete gradient matrix, as shown in Figure 4 As shown in Figure (c), at this time, each node contains the aggregated gradient data blocks "a1+b1+c1", "a2+b2+c2", and "a3+b3+c3". These three aggregated gradient data blocks "a1+b1+c1", "a2+b2+c2", and "a3+b3+c3" are aggregated into a complete gradient matrix. At this time, each node finally contains complete gradient information.

[0071] Since the number of nodes doubles compared to a single node during distributed training, in other words, the batch size of the training will also double. At this time, the step size of the model iteration is also enlarged, and the learning rate needs to be compensated to make the model converge quickly. In one embodiment, the Linear Scaling Rule is used to adjust the learning rate during the distributed training process to compensate for the loss of model accuracy.

[0072] In addition, introducing Group-TopK into distributed training for gradient compression is itself lossy training, which will affect the convergence effect of the model to a certain extent. During the training process, the learning rate is adjusted in conjunction with the Linear Scaling Rule strategy to ensure a relatively smooth training process for the model and reduce the adverse effects of gradient compression.

[0073] The specific principle of Linear Scaling Rule is as follows:

[0074] The most classic stochastic gradient descent (SGD) in neural network training. The optimization goal in neural network training is to reduce the loss function L(w):

[0075]

[0076] Among them, w is the weight of the network, X is the labeled data set, and L(w) is used to calculate the loss calculation function of the small batch sample data x∈X and the corresponding label y. In order to find the optimal solution of this formula, the stochastic gradient descent formula is derived:

[0077]

[0078] Among them, t is the current iteration number, η is the learning rate, n is the batch size, and β is a small batch of data belonging to x.

[0079] According to Formula 2, assuming that the learning rate is η and the batch size is n, after k SGD iterations, we can get

[0080]

[0081] From another perspective, when performing data parallel distributed training, assuming there are k nodes, the batch data volume is equivalent to expanding by k times. We can get another formula:

[0082]

[0083] In fact, whether running on a single machine or on multiple nodes, the iterative calculation is actually the same batch of data, so these two methods should achieve the same effect. Assume that when j < k is approximately equal to In most cases, this actually doesn't have much impact. Then it is observed that only when the learning rate η of distributed training is η = k·η, will there be Based on the reasoning of the above formula, when performing distributed training, the learning rate linear scaling rule needs to be used: when the batch data volume expands to k times the original, the learning rate also needs to expand to k times the original. That is, according to the increase ratio of the number of nodes in distributed training, linearly increase this learning rate to compensate for the change in the model iteration step length caused by the increase in the number of nodes.

[0084] In addition, the following two points need to be noted: First, it will change very violently at the initial stage of model training, and at this time, it cannot be determined Second, the learning rate η cannot be infinitely large and still needs to be set reasonably.

[0085] The following shows the specific impact brought by applying the Linear Scaling Rule in the gradient compression strategy Grouo-TopK. Among them, the yellow curve represents not applying the Linear Scaling Rule during distributed training, and the blue curve represents applying the Linear Scaling Rule during distributed training. Experiments are carried out on the Resnet18 and Resnet50 convolutional network models respectively, and the experimental results are as Figure 5 、 Figure 6 shown, where Figure 5 shows the experimental effect of using the Resnet18 neural network, Figure 6 shows the experimental effect of using the Resnet50 neural network. It can be seen that with the加持 of the Linear Scaling Rule, the gradient compression strategy Grouo-TopK has restored the training accuracy as expected.

[0086] In addition, to verify the advantages of Group-TopK, we re-implemented the fixed sparsity rate method with a pre-set gradient threshold, and according to its implementation characteristics, we called it Sample-TopK. In addition, we also re-implemented the currently most widely used TopK sparsification method and called it Global-TopK according to its implementation characteristics.

[0087] This experiment was conducted on the CIFAR-10 dataset, using a common convolutional neural network model for distributed training and testing. The CIFAR-10 dataset contains 60,000 color images of size 33232, divided into 10 categories. Among them, 50,000 images are used as training sets and 10,000 images are used as test sets. In the experiment, cross entropy is used as the training loss function, and two common convolutional network models, ResNet18 and ResNet50, are used. The training uses the momentum stochastic gradient descent method to update the model parameter weights, and the initial learning rate is set to 0.01 (and the Linear Scaling Rule strategy is used). The total number of training rounds is 10, of which the first round is used as warmup, and gradient sparseness is not used in this stage. The batch size of each process during training is 64.

[0088] The main evaluation indicators of the experiment include the time increase of each round of training and the accuracy of model training. This distributed training was conducted in a Jetson-Nano cluster environment with limited simulated edge computing resources, using 4 computing nodes and a network transmission bandwidth of 1Gbps between nodes.

[0089] Table 1 shows the experimental results, where "base" represents the training results when Pytorch DDP is run directly; "Sample-TopK" represents the results of the run using the fixed sparsity rate preset threshold method; "Global-TopK" represents the results of the run using the TopK sparsification method; "Group-TopK" represents the results of the run using the Group-TopK algorithm. The sparsity rate in the table represents the ratio of the compressed sparse gradient matrix to the complete gradient matrix.

[0090] Table 1 Comparison of Group-TopK with other methods, the experimental platform is JetsonNano

[0091]

[0092] As can be seen from Table 1, due to factors such as random sampling and threshold selection, Sample-TopK is not as fast as Global-TopK and Group-TopK under the same sparsity conditions, and its accuracy is also poor due to the uncertainty of its threshold selection.

[0093] Under the same sparsity rate conditions, Group-TopK has a performance advantage over Global-TopK's global sorting because it is implemented by group sorting. Compared with the most widely used gradient sparsification algorithm, it can be seen that Group-TopK has a slightly better acceleration effect on training than Global-TopK.

[0094] In addition, given that the TopK sparsification algorithm may not be applicable when the gradient distribution of the model is sparse, that is, when most gradients are small. In this case, most of the gradient details will be lost, which may cause the model to not converge well. Group-TopK has made improvements to this problem. As can be seen from Table 1, under the same sparsity, Group-TopK is also the best at maintaining accuracy. Specifically, in the case of extremely high sparsity such as 0.01, the accuracy is reduced by 1.54% on Resnet18 and 1.67% on Resnet50. This degree of accuracy loss is also within a reasonable range.

[0095] In addition, to verify the performance of Group-TopK on more experimental platforms, we conducted a series of experiments on Jetson Orin NX, and other basic experimental conditions were consistent with the previous JetsonNano. We compared the performance of different gradient sparsification methods on Resnet18 and Resnet50 models. Table 2 shows the accuracy at different sparsity ratios, the training time per epoch, and the speedup (Speedup) relative to the baseline method (Base).

[0096] Table 2 Comparison of Group-TopK with other methods, the experimental platform is Jetson Orin NX

[0097]

[0098]

[0099] First, the baseline performance (Base) is used as a reference. When the Sparse Ratio is 0, Resnet18 and Resnet50 achieve accuracies of 78.45% and 81.74% respectively, and the training time for each epoch is 136.65 seconds and 311.56 seconds respectively.

[0100] Secondly, when the Sparse Ratio is 0.05, Sample-TopK slightly improves the Speedup while slightly decreasing the accuracy, and Global-TopK decreases the accuracy while maintaining a relatively good Speedup. However, under this condition, Group-TopK decreases less in accuracy and achieves the best Speedup.

[0101] Furthermore, under the condition of high sparsity of Sparse Ratio of 0.01, the accuracy of Sample-TopK drops significantly, while the accuracy of Group-TopK drops less under the same sparsity. Under high sparsity conditions, Group-TopK shows better speedup than other methods, and provides better performance while maintaining high accuracy.

[0102] Finally, by comparing with Sample-TopK and Global-TopK, Group-TopK continues to show excellent performance on the Jetson OrinNX platform, proving its versatility and reliability on different hardware platforms. Under high sparsity conditions, Group-TopK is able to maintain a relatively high accuracy rate, showing its applicability and robustness in resource-constrained environments. Overall, the experimental results on the Jetson OrinNX platform consolidate Group-TopK as an excellent choice for distributed training in edge computing environments.

[0103] In summary, the distributed training communication compression method disclosed by the present invention is suitable for edge computing scenarios. The Group-TopK algorithm is used to group the global gradient information of each node in the distributed cluster; the top K gradient values ​​with the largest absolute value of the gradient are selected in each group, and the remaining unselected gradient values ​​are set to 0 to generate a sparse gradient matrix; the sparse gradient matrix is ​​shared among the nodes. The use of the Group-TopK algorithm can effectively reduce the communication overhead between the edge node and the server, while minimizing the impact on the model training accuracy. This method is particularly effective in a distributed training environment because it can significantly improve communication efficiency while ensuring model performance. In addition, in order to further reduce communication overhead, a decentralized ring communication method is adopted to realize the sharing of the sparse gradient matrix between nodes, which can effectively avoid excessive bandwidth burden on the central node, thereby limiting the overall training speed. At the same time, a learning rate linear scaling strategy is also applied to offset the decrease in model prediction accuracy caused by model compression.

[0104] In addition, corresponding to the above-mentioned embodiment of the distributed training communication compression method, the present invention also provides an embodiment of the distributed training communication compression device.

[0105] See also Figure 7 , an embodiment of the present invention provides a distributed training communication compression device 200, the device comprising:

[0106] The grouping module 210 is used to group the global gradient information of each node in the distributed cluster.

[0107] The gradient compression module 220 is used to select the first K gradient values ​​with the largest absolute gradient values ​​in each group, and set the remaining unselected gradient values ​​to 0 to generate a sparse gradient matrix.

[0108] The sharing module 230 is used to share the sparse gradient matrix among the nodes.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each module of the described device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0110] According to an embodiment of the present invention, the present invention also provides a distributed training system, the system includes multiple edge device nodes, and each edge device node is configured to execute the above-mentioned distributed training communication compression method.

[0111] According to an embodiment of the present invention, the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned distributed training communication compression method is implemented.

[0112] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and the present invention is not limited here.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed training communication compression method, suitable for edge computing scenarios, characterized in that: Include: Group the global gradient information of each node in the distributed cluster; In each group, select the first K gradient values ​​with the largest absolute value of the gradient, set the remaining unselected gradient values ​​to 0, and generate a sparse gradient matrix; The sparse gradient matrix is ​​shared among the nodes.

2. The method according to claim 1, characterized in that Set the group size and gradient compression rate; Grouping the global gradient information according to the group size; In each group, the value of K is determined according to the product of the gradient compression rate and the group size.

3. The method according to claim 1, characterized in that: Sharing the sparse gradient matrix between nodes includes the following steps: a1) Each node divides its own sparse gradient matrix according to the total number of nodes N to obtain N equal sub-gradient data blocks; a2) Each node selects one of the sub-gradient data blocks to send to its neighboring nodes; a3) repeating step a2) until each node sends each sub-gradient data block of its sparse gradient matrix at least once; a4) Each node accumulates the sub-gradient data blocks received from other nodes and a sub-gradient data block held by itself to obtain an aggregated gradient data block.

4. The method according to claim 3, characterized in that Sharing the sparse gradient matrix between nodes also includes the following steps: b1) Each node sends the aggregated gradient data block it holds to its neighboring nodes; b2) the neighboring node replaces the original gradient data block at the corresponding position with the received aggregated gradient data block; b3) the neighboring node sends the received aggregated gradient data block to another neighboring node, and the other neighboring node replaces the original gradient data block at the corresponding position with the received aggregated gradient data block; b4) Repeat steps b1)-b4) until each node contains N aggregated gradient data blocks, and the N aggregated gradient data blocks are aggregated into a complete gradient matrix.

5. The method according to claim 1, characterized in that Also includes: Adjust the learning rate during distributed training to compensate for loss in model accuracy.

6. The method according to claim 5, characterized in that Use the Linear Scaling Rule to compensate for the learning rate, including: According to the increase in the number of nodes in distributed training, the learning rate is increased linearly to compensate for the change in the model iteration step size caused by the increase in the number of nodes.

7. The method according to claim 1, characterized in that Also includes: The distributed cluster is constructed using a given distributed cluster framework, and data is distributed to each node of the distributed cluster before training begins. The initial data for training each node is different.

8. A distributed training communication compression device, characterized in that: Include: The grouping module is used to group the global gradient information of each node in the distributed cluster; The gradient compression module is used to select the first K gradient values ​​with the largest absolute value of the gradient in each group, set the remaining unselected gradient values ​​to 0, and generate a sparse gradient matrix; The sharing module is used to share the sparse gradient matrix among the nodes.

9. The device according to claim 9, characterized in that In each group, the value of K is determined according to the product of the preset gradient compression rate and the group size.

10. A distributed training system, characterized in that: Include: A plurality of edge device nodes, each edge device node is configured to execute the method according to any one of claims 1 to 7.

Images