Optimized gradient compression method and device for enhancing distributed deep learning

By constructing a communication threshold table in distributed deep learning using the Cocktails system and offloading the compression task to the CPU, the communication overhead and resource contention issues of gradient quantization methods are resolved, improving the efficiency and accuracy of model training, especially showing significant performance advantages in high-bandwidth environments.

CN118917384AInactive Publication Date: 2024-11-08NAT UNIV OF DEFENSE TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410990914.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In distributed deep learning, gradient quantization methods suffer from problems such as high communication overhead, GPU computing resource contention, and model convergence accuracy being affected by hyperparameters, especially when small gradient tensor transmission is not effective.

Method used

The Cocktails system is introduced, which builds a communication threshold table during the pre-training stage, utilizes the CPU to offload compression operations, and employs model backup technology to avoid unnecessary compression and optimize the gradient compression process.

Benefits of technology

It reduces contention for GPU computing and memory resources, increases training throughput, improves model convergence accuracy and communication efficiency, and shows significant performance advantages, especially in high-bandwidth environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917384A_ABST
    Figure CN118917384A_ABST
Patent Text Reader

Abstract

The invention discloses an optimized gradient compression method and device for enhancing distributed deep learning, and the method comprises the steps: building a communication threshold table through a cocktail system in a pre-training stage; in a formal training stage, a working node performs gradient calculation by using a local backup model to obtain a training gradient value; processing the training gradient value according to the communication threshold table to obtain a gradient value needing to be compressed, and sending the gradient value to a server node; and the server node processes the gradient value needing to be compressed to realize gradient compression. According to the method, a Cocktails system is introduced, on the basis of minimizing unnecessary compression operation, a compression task is transferred to a CPU to be executed, and the restriction of a communication task between nodes on calculation operation is relieved. In a high bandwidth environment, tensor calculation operations become sensitive to delays caused by compression operations on the GPU.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of gradient compression technology, and in particular to an optimized gradient compression method and device for enhancing distributed deep learning. Background Art

[0002] In data parallel distributed deep learning (DDL), each node has a copy of the DNN (Deep Neural Networks) model. In a homogeneous computing power cluster environment, the training data set is evenly divided into multiple partitions at the beginning of each iteration, and each node gets one partition. At the beginning of the iteration, each node inputs a small batch of training data in its own partition into the DNN model, and then independently performs forward propagation and back propagation to generate local gradient tensors. In order to avoid the training falling into the local optimal solution, each node needs to synchronize and aggregate the local gradient tensors and then update the model parameters. This process has expensive communication overhead because the gradient data from each node will compete for limited network bandwidth resources in the same physical link at the same time. In addition, the synchronization protocol constrains the node that completes the gradient calculation task first to wait for the slow node to start each iteration together, and its cost should be included in the synchronization communication overhead.

[0003] 1. Synchronous communication in DDL

[0004] There is a dependency between communication and computation in DDL. Therefore, to truly understand the communication overhead problem in DDL, it is necessary to clarify the parameter transfer process involved in the training process. Currently, the most common model training method in DDL is S-SGD (Symmetrical Stochastic Gradient Descent) or its derivative variants. In order to understand the working details of S-SGD, the process of node processing gradients and weights is introduced using the classic parameter server (PS) architecture as an example. The PS architecture abstracts the process into two roles according to the tasks performed by the process in the node. The working node is responsible for using the current model weight parameter w i Training sample data generates local gradient tensor And send it to the server, and the server node is responsible for aggregating the gradients from all working nodes and then starts updating the global weights. After the update operation is completed, each worker extracts the updated model weight w from the server i+1 , and then start the next iteration. S-SGD refines the model through training cycle iterations. The parameter transfer process between the worker and the server in each iteration constitutes the communication in DDL.

[0005] 2. Gradient Compression

[0006] In order to solve the communication bottleneck problem, a potential approach is to use the gradient quantization algorithm to map the floating point format gradient to a low bit format before sending it to reduce the data transmission overhead. Compared with full-precision communication, the gradient quantization algorithm adds the process of encoding the gradient before the worker node sends the gradient, and decoding the compressed gradient before the server aggregates the gradient. In order to maintain the convergence accuracy of model training, the gradient quantization algorithm will use the residual accumulation mechanism to avoid data loss. Taking the 2-bit compression algorithm as an example, if the compression threshold is set to 0.5, the gradients with absolute values ​​greater than the compression threshold will be encoded as 01 and 10 and sent. After that, the server will decode the received 01 and 10 into 0.5 and -0.5 and then aggregate the gradients. In this process, the difference between the gradient and the threshold will be accumulated in the cache until the absolute value of the accumulated residual is greater than the threshold and then sent to the server. Although this method avoids information loss, it still relies on reasonable hyperparameter settings to ensure convergence accuracy, and further increases the memory pressure and computational overhead of the GPU. Due to the existence of the above additional compression overhead, the gradient quantization algorithm often cannot achieve theoretical benefits during model training.

[0007] 3. Challenges and motivations

[0008] Challenge 1: Performing gradient quantization on the GPU will occupy some computing resources and delay the timing of tensor calculation. If the gradient quantization is offloaded to the CPU, it will lead to greater compression overhead.

[0009] Challenge 2: Gradient quantization does not always bring positive communication optimization benefits, which is particularly evident when nodes transmit small amounts of gradients. In this case, the main communication overhead is the startup overhead. At this time, encoding the gradient into a low-bit format does not bring effective data transmission overhead benefits, but instead introduces additional compression overhead. The definition of the range of small-scale gradient tensors is affected by the training environment and there is no fixed standard.

[0010] Challenge 3: When using gradient quantization, the convergence accuracy of the model is greatly affected by the gradient encoding threshold. Once the setting of this threshold deviates from the gradient distribution, the convergence accuracy will be seriously lost or even the model will not converge. However, we explored the gradient distribution of different models in the experiment and found that they do not have certain predictable characteristics. This means that when training different models, developers need to have rich prior knowledge to choose appropriate hyperparameters. Of course, some current methods solve the accuracy problem by momentum correction or setting multiple thresholds, but these methods themselves introduce additional hyperparameters. Summary of the invention

[0011] The technical problem to be solved by the present invention is to provide an optimized gradient compression method and device for enhancing distributed deep learning. In order to avoid contention for GPU computing and memory resources, Cocktails offloads the compression operation to the CPU and adopts model backup technology to decouple the reading of model parameters that depend on cross-working nodes, thereby hiding the CPU compression cost and part of the communication overhead.

[0012] In order to solve the above technical problems, the first aspect of the embodiments of the present invention discloses an optimized gradient compression method for enhancing distributed deep learning, the method comprising:

[0013] S1, in the pre-training stage, the communication threshold table is constructed using the cocktail system;

[0014] S2, in the formal training phase, the working node uses the locally backed-up model to perform gradient calculations and obtain training gradient values;

[0015] S3, processing the training gradient value according to the communication threshold table to obtain the gradient value that needs to be compressed, and sending it to the server node;

[0016] S4: The server node processes the gradient value that needs to be compressed to achieve gradient compression.

[0017] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the use of the cocktail system to construct a communication threshold table includes:

[0018] S11, obtain N pre-training gradients, where N is a positive integer;

[0019] S12, processing the N pre-training gradients to obtain a communication threshold table.

[0020] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the N pre-trained gradients to obtain a communication threshold table includes:

[0021] S121, processing the N pre-training gradients to obtain the communication overhead of the N pre-training gradients in a compressed state and the communication overhead of the N pre-training gradients in a non-compressed state; the communication overhead includes a communication operation startup overhead and a data transmission overhead;

[0022] S122, processing the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients;

[0023] S123, sorting the communication overheads of the N pre-trained gradients in a compressed state and in a non-compressed state according to the tensor sizes of the N pre-trained gradients to obtain a communication threshold table.

[0024] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the N pre-trained gradients to obtain the tensor size of the N pre-trained gradients includes:

[0025] S1221, processing the N pre-trained gradients to obtain compressed communication cost information, compressed processing cost information, and uncompressed communication cost information of the N pre-trained gradients;

[0026] S1222, process the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients.

[0027] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients are processed to obtain the tensor size of the N pre-trained gradients, including:

[0028] S12221, using a gradient compression information calculation model, processing the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain gradient compression information;

[0029] The gradient compression information calculation model expression is:

[0030]

[0031] Among them, τ is the gradient compression information, To compress communication cost information, To compress the processing cost information, t c is the uncompressed communication cost information,

[0032] S12222, process the gradient compression information to obtain tensor sizes of N pre-trained gradients.

[0033] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the training gradient value according to the communication threshold table to obtain the gradient value to be compressed and sending it to the server node includes:

[0034] S31, the working node processes the communication threshold table to obtain a pre-training gradient in which the compression gain is greater than 1 for two consecutive times;

[0035] S32, taking the average of the tensor sizes of the pre-trained gradients with compression gains greater than 1 for two consecutive times as the communication threshold;

[0036] S33, processing the training gradient value according to the communication threshold, obtaining the gradient value that needs to be compressed, and sending it to the server node.

[0037] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the training gradient value according to the communication threshold to obtain the gradient value to be compressed and sending it to the server node includes:

[0038] S331, obtaining a training gradient value;

[0039] S332, when the tensor size of the training gradient value is smaller than the communication threshold, sending the training gradient value to a server node;

[0040] When the tensor size of the training gradient value is greater than the communication threshold, encoding the tensor size of the training gradient value to obtain an encoded tensor size of the training gradient value;

[0041] The tensor size of the training gradient value and the encoded tensor size of the training gradient value are sent to a server node.

[0042] A second aspect of an embodiment of the present invention discloses an optimized gradient compression device for enhancing distributed deep learning, the device comprising:

[0043] A communication threshold table construction module is used to construct a communication threshold table using the cocktail system during the pre-training phase;

[0044] The training gradient value calculation module is used to calculate the gradient of the working node using the locally backed-up model to obtain the training gradient value during the formal training phase;

[0045] A training gradient value processing module, used to process the training gradient value according to the communication threshold table, obtain the gradient value to be compressed, and send it to the server node;

[0046] The gradient compression module is used for the server node to process the gradient value that needs to be compressed to achieve gradient compression.

[0047] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the use of the cocktail system to construct a communication threshold table includes:

[0048] S11, obtain N pre-training gradients, where N is a positive integer;

[0049] S12, processing the N pre-training gradients to obtain a communication threshold table.

[0050] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the N pre-trained gradients to obtain a communication threshold table includes:

[0051] S121, processing the N pre-training gradients to obtain the communication overhead of the N pre-training gradients in a compressed state and the communication overhead of the N pre-training gradients in a non-compressed state; the communication overhead includes a communication operation startup overhead and a data transmission overhead;

[0052] S122, processing the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients;

[0053] S123, sorting the communication overheads of the N pre-trained gradients in a compressed state and in a non-compressed state according to the tensor sizes of the N pre-trained gradients to obtain a communication threshold table.

[0054] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the N pre-trained gradients to obtain the tensor size of the N pre-trained gradients includes:

[0055] S1221, processing the N pre-trained gradients to obtain compressed communication cost information, compressed processing cost information, and uncompressed communication cost information of the N pre-trained gradients;

[0056] S1222, process the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients.

[0057] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients are processed to obtain the tensor size of the N pre-trained gradients, including:

[0058] S12221, using a gradient compression information calculation model, processing the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain gradient compression information;

[0059] The gradient compression information calculation model expression is:

[0060]

[0061] Among them, τ is the gradient compression information, To compress communication cost information, To compress the processing cost information, t c is the uncompressed communication cost information,

[0062] S12222, process the gradient compression information to obtain tensor sizes of N pre-trained gradients.

[0063] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the training gradient value according to the communication threshold table to obtain the gradient value to be compressed and sending it to the server node includes:

[0064] S31, the working node processes the communication threshold table to obtain a pre-training gradient in which the compression gain is greater than 1 for two consecutive times;

[0065] S32, taking the average of the tensor sizes of the pre-trained gradients with compression gains greater than 1 for two consecutive times as the communication threshold;

[0066] S33, processing the training gradient value according to the communication threshold, obtaining the gradient value that needs to be compressed, and sending it to the server node.

[0067] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the training gradient value according to the communication threshold to obtain the gradient value to be compressed and sending it to the server node includes:

[0068] S331, obtaining a training gradient value;

[0069] S332, when the tensor size of the training gradient value is smaller than the communication threshold, sending the training gradient value to a server node;

[0070] When the tensor size of the training gradient value is greater than the communication threshold, encoding the tensor size of the training gradient value to obtain an encoded tensor size of the training gradient value;

[0071] The tensor size of the training gradient value and the encoded tensor size of the training gradient value are sent to a server node.

[0072] A third aspect of the present invention discloses another optimized gradient compression device for enhancing distributed deep learning, the device comprising:

[0073] A memory storing executable program code;

[0074] a processor coupled to the memory;

[0075] The processor calls the executable program code stored in the memory to execute part or all of the steps in the optimized gradient compression method for enhancing distributed deep learning disclosed in the first aspect of an embodiment of the present invention.

[0076] The fourth aspect of the present invention discloses a computer-storable medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the optimized gradient compression method for enhancing distributed deep learning disclosed in the first aspect of an embodiment of the present invention.

[0077] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0078] In order to solve the problems existing with the gradient quantization method, the present invention introduces the Cocktails system. The system aims to minimize unnecessary compression operations, transfer compression tasks to the CPU for execution, and alleviate the constraints of inter-node communication tasks on computing operations. In a high-bandwidth environment, tensor computing operations become sensitive to the delay caused by compression operations on the GPU. At this time, the CPU compression strategy has performance advantages in more scenarios, and the compression offloading of Cocktails is more likely to produce performance advantages. In large-scale training modes, Cocktails will be more likely to gain advantages. Larger batches will increase the tensor computation cost of each iteration, but will not change the communication cost of the gradient tensor, because the scale of the gradient tensor is only related to the model type. Cocktails can take advantage of model backup to increase the parallelism of computing and communication operations. Cocktails can narrow the performance gap between different gradient quantization compression algorithms. In a high-bandwidth environment, Cocktails exhibits higher communication thresholds and communication masking rates, resulting in more significant performance advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0080] Figure 1 It is a flow chart of an optimized gradient compression method for enhancing distributed deep learning disclosed in an embodiment of the present invention;

[0081] Figure 2 It is a schematic diagram of comparing the accuracy in image recognition and NLP disclosed in an embodiment of the present invention;

[0082] Figure 3 is the speedup ratio of different models disclosed in the embodiments of the present invention;

[0083] Figure 4 is a comparison diagram of accuracies under different compression thresholds, communication thresholds and learning rates disclosed in an embodiment of the present invention;

[0084] Figure 5 It is a structural schematic diagram of an optimized gradient compression device for enhancing distributed deep learning disclosed in an embodiment of the present invention;

[0085] Figure 6 It is a structural schematic diagram of another optimized gradient compression device for enhancing distributed deep learning disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0087] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0088] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0089] To avoid contention for GPU computing and memory resources, Cocktails offloads compression operations to the CPU and uses model backup technology to decouple read dependencies from model parameters across worker nodes, thereby hiding the CPU compression cost and part of the communication overhead. Experimental evaluation on a cluster with 16 V100 GPUs shows that for typical DDL training tasks, the cocktail system improves training throughput by 74.3% and 68.7% under 1-bit and 2-bit compression methods, respectively. In addition, it achieves a 1.1% to 2.3% improvement in convergence accuracy and reduces GPU memory consumption by 10.3%.

[0090] The present invention discloses an optimized gradient compression method and device for enhancing distributed deep learning, the method comprising: in the pre-training stage, using the cocktail system to construct a communication threshold table; in the formal training stage, the working node uses the locally backed-up model to perform gradient calculation to obtain a training gradient value; according to the communication threshold table, the training gradient value is processed to obtain a gradient value to be compressed, and sent to a server node; the server node processes the gradient value to be compressed to achieve gradient compression. The present invention introduces the cocktail system, which transfers the compression task to the CPU for execution on the basis of minimizing unnecessary compression operations, and alleviates the constraints of the inter-node communication task on the computing operation. In a high-bandwidth environment, tensor computing operations become sensitive to the delay caused by the compression operation on the GPU. At this time, the CPU compression strategy has performance advantages in more scenarios, and the compression offloading of cocktails is more likely to produce performance advantages. In a large-scale training mode, cocktails will more easily gain advantages. Larger batches will increase the tensor calculation cost of each iteration, but will not change the communication cost of the gradient tensor, because the scale of the gradient tensor is only related to the model type. Cocktails can take advantage of model backup to increase the parallelism of computation and communication operations. Cocktails can narrow the performance gap between different gradient quantization compression algorithms. In high-bandwidth environments, Cocktails exhibit higher communication thresholds and communication masking rates, resulting in more significant performance advantages. The following are detailed descriptions of each.

[0091] Embodiment 1

[0092] See also Figure 1 , Figure 1 is a flow chart of an optimized gradient compression method for enhancing distributed deep learning disclosed in an embodiment of the present invention. Figure 1 The described optimized gradient compression method for enhancing distributed deep learning is applied to the field of gradient compression, minimizing unnecessary compression operations, transferring compression tasks to the CPU for execution, and alleviating the constraints of inter-node communication tasks on computing operations. The present invention can be mainly applied to big data analysis, cloud computing, unmanned driving, etc. Distributed communication optimization is the basis for ensuring efficient and stable operation of distributed systems. It involves the collaborative work of multiple levels such as network, software, and hardware, and is an important component that cannot be ignored in the design of distributed systems. The embodiments of the present invention are not limited. As Figure 1 As shown, the optimized gradient compression method for enhancing distributed deep learning may include the following operations:

[0093] S1, in the pre-training stage, the communication threshold table is constructed using the cocktail system;

[0094] S2, in the formal training phase, the working node uses the locally backed-up model to perform gradient calculations and obtain training gradient values;

[0095] S3, processing the training gradient value according to the communication threshold table to obtain the gradient value that needs to be compressed, and sending it to the server node;

[0096] S4: The server node processes the gradient value that needs to be compressed to achieve gradient compression.

[0097] Optionally, the using the cocktail system to construct a communication threshold table includes:

[0098] S11, obtain N pre-training gradients, where N is a positive integer;

[0099] S12, processing the N pre-training gradients to obtain a communication threshold table.

[0100] Optionally, the processing of the N pre-trained gradients to obtain a communication threshold table includes:

[0101] S121, processing the N pre-training gradients to obtain the communication overhead of the N pre-training gradients in a compressed state and the communication overhead of the N pre-training gradients in a non-compressed state; the communication overhead includes a communication operation startup overhead and a data transmission overhead;

[0102] S122, processing the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients;

[0103] S123, sorting the communication overheads of the N pre-trained gradients in a compressed state and in a non-compressed state according to the tensor sizes of the N pre-trained gradients to obtain a communication threshold table.

[0104] Optionally, the processing the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients includes:

[0105] S1221, processing the N pre-trained gradients to obtain compressed communication cost information, compressed processing cost information, and uncompressed communication cost information of the N pre-trained gradients;

[0106] S1222, process the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients.

[0107] Optionally, the processing of the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain the tensor size of the N pre-trained gradients includes:

[0108] S12221, using a gradient compression information calculation model, processing the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain gradient compression information;

[0109] The gradient compression information calculation model expression is:

[0110]

[0111] Among them, τ is the gradient compression information, To compress communication cost information, To compress the processing cost information, t c is the uncompressed communication cost information,

[0112] S12222, process the gradient compression information to obtain tensor sizes of N pre-trained gradients.

[0113] Optionally, processing the training gradient value according to the communication threshold table to obtain the gradient value to be compressed, and sending it to the server node includes:

[0114] S31, the working node processes the communication threshold table to obtain a pre-training gradient in which the compression gain is greater than 1 for two consecutive times;

[0115] S32, taking the average of the tensor sizes of the pre-trained gradients with compression gains greater than 1 for two consecutive times as the communication threshold;

[0116] S33, processing the training gradient value according to the communication threshold, obtaining the gradient value that needs to be compressed, and sending it to the server node.

[0117] Optionally, processing the training gradient value according to the communication threshold to obtain the gradient value to be compressed, and sending it to the server node includes:

[0118] S331, obtaining a training gradient value;

[0119] S332, when the tensor size of the training gradient value is smaller than the communication threshold, sending the training gradient value to a server node;

[0120] When the tensor size of the training gradient value is greater than the communication threshold, encoding the tensor size of the training gradient value to obtain an encoded tensor size of the training gradient value;

[0121] The tensor size of the training gradient value and the encoded tensor size of the training gradient value are sent to a server node.

[0122] It can be seen that in order to solve the problems existing with the gradient quantization method, the present invention introduces the Cocktails system. The system aims to minimize unnecessary compression operations, transfer compression tasks to the CPU for execution, and alleviate the constraints of inter-node communication tasks on computing operations. In a high-bandwidth environment, tensor computing operations become sensitive to the delay caused by compression operations on the GPU. At this time, the CPU compression strategy has performance advantages in more scenarios, and the compression offloading of Cocktails is more likely to produce performance advantages. In large-scale training modes, Cocktails will be more likely to gain advantages. Larger batches will increase the tensor calculation cost of each iteration, but will not change the communication cost of the gradient tensor, because the scale of the gradient tensor is only related to the model type. Cocktails can take advantage of model backup to increase the parallelism of computing and communication operations. Cocktails can narrow the performance gap between different gradient quantization compression algorithms. In a high-bandwidth environment, Cocktails exhibits a higher communication threshold and communication masking rate, resulting in more significant performance advantages.

[0123] Embodiment 2

[0124] This embodiment discloses another optimized gradient compression method for enhancing distributed deep learning, including:

[0125] In order to solve the above challenges related to the gradient quantization method, the present invention introduces a Cocktails system, which aims to minimize unnecessary compression operations, transfer compression tasks to the CPU for execution, and alleviate the constraints of inter-node communication tasks on computing operations.

[0126] To determine which compressed gradients can bring substantial benefits, Cocktails monitors the communication overhead of gradients before and after compression over multiple training iterations and records the data in a communication threshold table. It will then derive the communication threshold from this table to identify the gradients that really need to be compressed. In each iteration of formal training, the worker node uses the locally backed-up model to calculate the gradients, and then determines whether each gradient should be compressed and transmitted to the server node based on the communication threshold. In traditional compression schemes, the communication overhead between nodes may cause the worker node to be unable to start the calculation of the next iteration in time. However, Cocktails solves this problem by having the worker node start each iteration using the backup model instead of the global model on the server. To ensure the freshness and synchronization of the model, the worker node obtains the globally updated weights from the server node during the gradient calculation task, updates them with the local gradients, and then starts the next iteration. This design brings two benefits: First, the calculation operations of the adjacent iterations of the worker node run in parallel with the communication operations of the current iteration, thereby alleviating the adverse effects of the compression operation on the calculation overhead. Secondly, using floating-point gradients instead of compressed gradients to update the backup model on the working node alleviates the problem that small gradients caused by compression threshold filtering cannot be timely involved in weight updates.

[0127] Compression optimization strategy: The basic principle of the compression optimization strategy of Cocktails is to avoid compressing small-scale gradient tensors to ensure accuracy and improve performance. The optimization strategy design of the compression method in this paper is based on the following key principles: First, the communication overhead consists of two parts: the communication operation startup overhead and the data transmission overhead. Second, for small data volumes, the main component of the communication overhead is the startup cost of the communication operation, and within a certain data interval (for example, 1 byte to 1 kilobyte), the communication overhead of the data tends to be consistent. Third, for large-scale tensors, there is a positive correlation between the data transmission overhead and the data volume, that is, the larger the data volume, the higher the transmission overhead. Fourth, the compression operation will introduce additional encoding and decoding overheads, which should be taken into account when calculating the communication overhead after compression. Finally, without considering the interference caused by system performance fluctuations, during the model training process, the operating cost of each node in each iteration is relatively stable.

[0128] Model training is essentially a process of using data samples to continuously use the feedback information generated by training iterations to correct model parameters. Therefore, the working node needs to obtain the latest model parameters to be corrected before starting a new iteration. Based on this point of view, Cocktails proposes to back up the model parameters updated by the server node in the previous iteration on the worker during each iteration. The model parameters will not be directly used to start the training iteration, but the local gradients generated by the working node in the current iteration will be updated to replace the fresh model parameters generated by the server to start the next iteration. Formula (1) and Formula (2) represent the weight updates on the server node and the working node, respectively.

[0129]

[0130] Among them, w i is the global gradient of the server at step i, N is the number of nodes in distributed training, is the gradient of the jth working node at the i-th step, is the local weight of the jth working node at the i-th step, and η is the learning rate.

[0131] Compression offloading: GPUs perform concurrent tensor calculation tasks involved in the forward and backward propagation of model training in parallel. However, there is no strict concurrent requirement for the encoding and decoding of gradient tensors. Performing compression operations on the GPU will occupy the computing power resources required for tensor calculation. Therefore, Cocktails attempts to offload compression-related calculation operations from the GPU to the CPU. However, this approach will bring new challenges: copying data from the GPU to the CPU will bring additional operation overhead and delay the CPU's timing for gradient encoding. In addition, the CPU performs compression operations more slowly than the GPU, which will bring greater compression overhead. In order to reduce the impact of data copying, Cocktails has designed a non-blocking data transmission method. It does not require the GPU to transfer data to the CPU in a specified order, but determines whether the data has been successfully transferred based on the key value of each tensor. In this way, the data copy operation of the working node is parallel to the gradient calculation and gradient compression, and its overhead can be largely concealed. Using CPU compression can conceal the computational overhead of compression to a certain extent, but compared with GPU compression, the slow CPU compression speed is likely to be unable to completely conceal it, and the data copy that cannot be concealed here will be accumulated into the compression overhead. The compression operation overhead on the CPU may be close to or even exceed the overhead of gradient tensor calculation, which greatly delays the timing of communication startup, resulting in the worker node being unable to obtain the updated weights as early as possible to start the next iteration. Therefore, Cocktails also needs a method that can solve the worker node's dependency on reading new weights to truly obtain the performance benefits brought by compression offloading.

[0132] Model backup: The rationality of model backup lies in that after each node uses local gradient information to update the backup model parameters, the local gradients generated in the new iteration will still be aggregated on the server node. Therefore, the difference between using the backup model parameters of the working node and using the global model parameters of the server node to start the iteration is actually the difference in the timing of feedback information aggregation. The former is to aggregate the feedback information after participating in the new iteration, while the latter is to aggregate the feedback information before starting the new iteration. Cocktails uses the effect of model backup. After the worker completes the weight update, it can directly start the next iteration, avoiding the long wait caused by communication overhead.

[0133] Embodiment 3

[0134] Cocktails can dynamically determine the optimal tensor value and pass it to all working nodes to implement the proposed compression strategy.

[0135] Establishment of communication threshold table: For any gradient, if it compresses the communication cost and compression processing costs The sum exceeds the uncompressed communication cost (t c ), the gradient is considered to be a small-scale gradient tensor. During the pre-training phase, Cocktails records the communication overhead of each gradient in compressed and uncompressed states, and sorts these records from small to large according to the size of the tensor, and integrates them into the communication threshold table. To reduce the impact of network fluctuations, the working node will average the data generated by multiple rounds of iterations.

[0136] Select the optimal gradient value: After pre-training, the worker node will traverse the communication threshold table from top to bottom, looking for the first group of rows with compression gains greater than 1. Then, it will use the corresponding tensor size as the communication threshold. This communication threshold will be broadcast to each worker node as a basis for determining whether to compress specific gradients in subsequent formal training (steps 4-6). Gradient tensors less than the communication threshold will be directly transmitted to the server node; conversely, gradient tensors greater than the communication threshold will be encoded into a low-bit format and decoded back to a floating-point format on the server node before participating in gradient aggregation. Due to the discrete distribution characteristics of tensor size, the communication threshold table and communication threshold generated by Cocktails during training will vary from model to model.

[0137] The task of recording communication overhead is performed by a thread independent of training, and does not introduce significant additional overhead. The performance improvement can be obtained from the compression overhead saved by not compressing a small number of gradients and the corresponding communication overhead. Thanks to the reduced encoding cost, communication of small-scale tensors can be started earlier, thus reducing the waiting time between consecutive iterative calculations.

[0138] This embodiment experimentally evaluates computer vision and natural language processing tasks. The accuracy test experiment includes training ResNet-50, Inception-Bn on CIFAR-10, training VGG-16 on ImageNet ILSVRC2012, and training BERT-base on SQuAD. The hyperparameter sensitivity test includes training ResNet152 and Inception-Bn on CIFAR-10. The performance test includes training ResNet-152 on CIFAR-10, training Inception-Bn and VGG-16 on ImageNet ILSVRC2012, and training BERT-base on SQuAD. The experiment of analyzing cost composition includes training ResNet-152 on CIFAR-10 and Inception-Bn on ImageNet. The memory analysis experiment includes training Inception-Bn, AlexNet, and VGG-16 on ImageNet.

[0139] Accuracy experiment: The accuracy comparison experiment includes 6 components: 1-bit and 2-bit compression algorithms, 1-bit and 2-bit compression algorithms using Cocktails optimization compression strategy, and 1-bit and 2-bit compression algorithms using all Cocktails optimization strategies. In order to fairly compare the convergence accuracy of each component, each algorithm uses the same parameter settings and experimental configuration, and all hyperparameters (batch size, compression threshold, learning rate) are set according to the official recommendations of MXNet. The accuracy change curves of each method are shown in the figure below. Figure 2 As shown in (a), the following phenomena can be observed:

[0140] First, without using compression optimization or using the same compression optimization technology, the convergence accuracy of the 2-bit compression algorithm is 0.2% to 1.1% higher than that of the 1-bit compression algorithm because it can express richer information.

[0141] Secondly, with the same number of training times, the optimized Cocktails compression strategy effectively improves the convergence accuracy of the compression algorithm. 1 bit (CG Opt.) and 2 bits (CG Opt.) can help the 1-bit and 2-bit compression algorithms achieve 0.7% to 1.7% and 0.7% to 1.6% accuracy improvement, respectively.

[0142] Third, after applying all optimization techniques, the accuracy of Cocktails' 1-bit and 2-bit compression algorithms improved by 1.1% to 2.3% and 1.1% to 1.6%, respectively.

[0143] Fourth, since small-scale tensors are transmitted in full-precision communication and the backup model on the worker node is updated with full-precision gradients, Cocktails can help the compression algorithm find suitable weights faster in the early stages of model training, thereby helping the node find the optimal solution in the convergence phase. This phenomenon is more obvious in the training of ResNet-152 and Inception-Bn models because these models contain more small-scale gradient tensors.

[0144] This example fine-tuned the BERT-base for two epochs on the question-answering task on SQuAD and repeated the experiment six times to compare the F1 scores of various methods. Figure 2 As can be seen in (b), in the NLP model training task, the accuracy difference between the 1-bit compression algorithm and the 2-bit compression algorithm is relatively small, within 0.3%. In addition, the optimized compression strategy is the key to improving the accuracy of the 1-bit and 2-bit compression algorithms, which can improve the accuracy by 0.4% to 0.5%. Based on 2-bit (CG Opt.), combined with the local full-precision update technology, 2-bit (Cocktails) achieved the standard accuracy of full-precision training (88.5%).

[0145] Performance experiment: We selected representative models such as ResNet-152, VGG16, Inception-Bn, and BERT-base to verify the performance optimization effect of Cocktails on compression algorithms. The main training cost of the ResNet-152 model comes from dense small-scale tensor convolution calculations, which classifies it as a computationally intensive model. The main cost of VGG-16 comes from the fully connected layer communication that produces large-scale gradient tensors, which defines it as a communication-intensive model. The gradient tensor calculation and communication costs of Inception-Bn are relatively balanced, while BERT-base combines complex calculations with frequent communications. Figure 3 (a) shows the performance of various algorithms when training multiple models in a 10 Gbps network environment.

[0146] In order to more intuitively reflect the performance differences between different training methods, the GPU compression strategy is used to normalize the training speed of other methods based on the training speed of the 1-bit compression algorithm. The following phenomena can be observed:

[0147] First, when training computationally intensive models like ResNet-152, the 1-bit compression algorithm is not faster than the 2-bit compression algorithm, because it is challenging to take advantage of the high communication compression rate in this case. In contrast, the 1-bit compression algorithm shows better performance in the VGG-16 and BERT-base model training scenarios. As the batch size increases, the performance advantage of the 1-bit compression algorithm becomes more obvious.

[0148] Second, in computationally intensive model training scenarios, there is no significant performance difference between using GPU compression and CPU compression strategies, and this phenomenon is not affected by changes in batch size. When training VGG-16, using CPU compression can actually produce better performance in large batch training. However, when training the BERT-base model, the relative speed of the CPU and GPU compression strategies is related to the batch size and the type of compression algorithm used, and there is no fixed rule to follow.

[0149] Third, in low-bandwidth training environments, Cocktails significantly improves the performance of compression algorithms in ResNet-152 training scenarios. Cocktails optimizes the communication of small-scale tensors, and its model backup technology uses the intensive computational overhead during ResNet-152 training to mask the communication overhead. For other models, Cocktails provides a 4% to 16% performance improvement for compression algorithms. In these cases, low network bandwidth leads to an extreme imbalance between computational and communication costs, making effective communication shielding difficult. In addition, the main communication overhead of models such as VGG-16 does not come from small-scale gradient tensor communication. Cocktails tend to use a smaller communication threshold in low-bandwidth situations, limiting the range of gradient communications that can be optimized.

[0150] In order to eliminate the interference caused by the low-bandwidth network, the 10gbps Ethernet was replaced with 100gbps Ethernet and tested under the same configuration. Figure 3 As shown in (b), when a higher bandwidth network is used for distributed training, the performance of each algorithm changes slightly:

[0151] First, during the training of VGG-16 and Inception-Bn, the 1-bit compression algorithm showed a more significant performance advantage than the 2-bit compression algorithm, especially when using large batches to train Inception-Bn, the 1-bit compression algorithm was 27% to 32% faster than the 2-bit compression algorithm.

[0152] Second, in high-bandwidth environments, tensor computation operations become sensitive to the latency caused by compression operations on the GPU. At this point, the CPU compression strategy has performance advantages in more scenarios, and Cocktails' compression offload is more likely to produce performance advantages.

[0153] Third, Cocktails will be more likely to gain advantages in large batch training mode. Larger batches increase the tensor computation cost of each iteration, but do not change the communication cost of the gradient tensor, because the size of the gradient tensor is only related to the model type. Cocktails can take advantage of model backup to increase the parallelism of computation and communication operations.

[0154] Fourth, Cocktails can narrow the performance gap between different gradient quantization compression algorithms. When training Inception-Bn, after Cocktails optimization, the performance of the 2-bit compression algorithm is comparable to that of the 1-bit compression algorithm. This improvement is related to the optimization of small gradient compression. In theory, when the communication threshold reaches infinity, there will be no performance difference between compression algorithms.

[0155] Finally, in high-bandwidth environments, Cocktails exhibit higher communication thresholds and communication masking ratios, leading to more significant performance advantages.

[0156] In the training of ResNet-152, VGG-16, Inception-Bn and BERT base models, the cocktail improved the performance of the 1-bit compression algorithm by 74.3%, 39.5%, 29.9% and 56.6%, and the performance of the 2-bit compression algorithm by 75.9%, 37.4%, 68.7% and 50.1%, respectively.

[0157] Parameter sensitivity experiments: To study the factors that affect the accuracy improvement of the Cocktails compression algorithm, we examined three hyperparameters: compression threshold, communication threshold, and learning rate.

[0158] First, in terms of learning rate. When the compression threshold is 0.5, both the 1-bit and 2-bit compression algorithms have the smallest loss. This threshold is more consistent with the gradient distribution of most models and is recommended by the official guide. Secondly, the 1-bit compression algorithm cannot achieve convergence of ResNet-152 when the learning rate is adjusted using the linear scaling principle (the learning rate increases proportionally with the batch size); however, it achieves convergence when the learning rate is adjusted using the square root scaling strategy. Without using other accuracy optimization methods, the convergence of this method depends on a reasonable learning rate setting. Third, when an inappropriate compression threshold is used, the convergence of each compression algorithm becomes unpredictable. In addition, there is no correlation between the size of the deviation from the ideal threshold (0.5) and the degradation of model convergence. In addition, the experience gained from setting the compression threshold for an established algorithm may not necessarily provide information for the optimization of other algorithms. For example, during the training of Inception-Bn, the compression threshold of the 1-bit compression algorithm was 0.7, which was better than 0.6, while the 2-bit compression algorithm exhibited the opposite behavior.

[0159] Secondly, in terms of compression thresholds. The use of compression optimization strategies may cause the optimal compression threshold of the compression algorithm to shift. Therefore, when training on the same model, different algorithms may not share the same optimal compression threshold, and even the same algorithm may show changes in its optimal compression threshold in different model training scenarios. This phenomenon occurs due to differences in tensor distribution between different models and differences in cost characteristics of different algorithms. Cocktails assigns different communication thresholds to accommodate these differences, resulting in changes in the proportion of compressed tensors. In this case, Cocktails avoids pursuing the optimal compression threshold setting because it recognizes that this will impose additional burdens on users. Setting the compression threshold to 0.5, Cocktails effectively limits the loss difference to within 0.6% compared to the optimal compression threshold setting.

[0160] Finally, the communication threshold aspect. The 1-bit and 2-bit compression algorithms without compression optimization strategies can be regarded as 1-bit (CG Opt) and 2-bit (CG Opt) algorithms with a communication threshold of zero. Therefore, it is crucial to verify the optimal communication threshold setting to improve the accuracy of the compression algorithm. Figure 2 (b) shows the minimum loss value of each algorithm under different communication thresholds, and the compression threshold is fixed at 0.5. It is observed that when the tensor with a data volume less than 1500B is not compressed and communicated, the accuracy of the compression algorithm is significantly improved. Moreover, as the communication threshold increases, the accuracy benefit of the compression algorithm is gradually improved. This result is exciting because Figure 2 The data in show that under traditional computing power and network configuration (V100 GPU and 100Gbps Ethernet), 1-bit and 2-bit compression algorithms can only achieve real performance advantages when the compressed tensor is 80KB or higher, and the communication threshold is far more than 1500B. In theory, when the communication threshold is set to infinity, 1-bit (CG Opt) and 2-bit (CG Opt) can be regarded as full-precision gradient synchronous communication algorithms. In actual model training scenarios, reaching a certain communication threshold can produce optimal benefits. For example, when a linear scaling learning rate strategy is adopted, setting the communication threshold to 15KB can enable the 2-bit compression algorithm to achieve the standard accuracy of full-precision training: ResNet-152 and Inception-Bn achieve 93.9% and 92.9% Top-1 accuracy respectively. Figure 4 Comparison chart of accuracy under different compression thresholds, communication thresholds and learning rates.

[0161] Embodiment 4

[0162] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an optimized gradient compression device for enhancing distributed deep learning disclosed in an embodiment of the present invention. Figure 5 The described optimized gradient compression device for enhancing distributed deep learning is applied to the field of gradient compression, minimizing unnecessary compression operations, transferring compression tasks to the CPU for execution, and alleviating the constraints of inter-node communication tasks on computing operations. The embodiments of the present invention are not limited. Figure 5 As shown, the optimized gradient compression device for enhancing distributed deep learning may include the following operations:

[0163] S301, a communication threshold table construction module, used to construct a communication threshold table using the cocktail system in the pre-training stage;

[0164] S302, a training gradient value calculation module, which is used for the working node to perform gradient calculation using the locally backed-up model to obtain the training gradient value during the formal training phase;

[0165] S303, a training gradient value processing module, used to process the training gradient value according to the communication threshold table, obtain the gradient value to be compressed, and send it to the server node;

[0166] S304, a gradient compression module is used for the server node to process the gradient value that needs to be compressed to achieve gradient compression.

[0167] Embodiment 5

[0168] See also Figure 6 , Figure 6 is a schematic diagram of another optimized gradient compression device for enhancing distributed deep learning disclosed in an embodiment of the present invention. Figure 6 The described optimized gradient compression device for enhancing distributed deep learning is applied to the field of gradient compression, minimizing unnecessary compression operations, transferring compression tasks to the CPU for execution, and alleviating the constraints of inter-node communication tasks on computing operations. The embodiments of the present invention are not limited. Figure 6 As shown, the optimized gradient compression device for enhancing distributed deep learning may include the following operations:

[0169] A memory 401 storing executable program codes;

[0170] a processor 402 coupled to the memory 401;

[0171] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the optimized gradient compression method for enhancing distributed deep learning described in Embodiment 1, Embodiment 2, and Embodiment 3.

[0172] Embodiment 6

[0173] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the optimized gradient compression method for enhancing distributed deep learning described in Embodiment 1, Embodiment 2, and Embodiment 3.

[0174] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0175] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0176] Finally, it should be noted that the optimized gradient compression method and device for enhancing distributed deep learning disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimized gradient compression method for enhancing distributed deep learning, characterized in that: The method comprises: S1, in the pre-training stage, the communication threshold table is constructed using the cocktail system; S2, in the formal training phase, the working node uses the locally backed-up model to perform gradient calculations and obtain training gradient values; S3, processing the training gradient value according to the communication threshold table to obtain the gradient value that needs to be compressed, and sending it to the server node; S4: The server node processes the gradient value that needs to be compressed to achieve gradient compression.

2. The optimized gradient compression method for enhancing distributed deep learning according to claim 1, characterized in that: The method of using the cocktail system to construct a communication threshold table includes: S11, obtain N pre-training gradients, where N is a positive integer; S12, processing the N pre-training gradients to obtain a communication threshold table.

3. The optimized gradient compression method for enhancing distributed deep learning according to claim 2, characterized in that: The processing of the N pre-trained gradients to obtain a communication threshold table includes: S121, processing the N pre-training gradients to obtain the communication overhead of the N pre-training gradients in a compressed state and the communication overhead of the N pre-training gradients in a non-compressed state; the communication overhead includes a communication operation startup overhead and a data transmission overhead; S122, processing the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients; S123, sorting the communication overheads of the N pre-trained gradients in a compressed state and in a non-compressed state according to the tensor sizes of the N pre-trained gradients to obtain a communication threshold table.

4. The optimized gradient compression method for enhancing distributed deep learning according to claim 3, characterized in that: The processing of the N pre-trained gradients to obtain the tensor sizes of the N pre-trained gradients includes: S1221, processing the N pre-trained gradients to obtain compressed communication cost information, compressed processing cost information, and uncompressed communication cost information of the N pre-trained gradients; S1222, process the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain tensor sizes of the N pre-trained gradients.

5. The optimized gradient compression method for enhancing distributed deep learning according to claim 4, characterized in that: The processing of the compressed communication cost information, the compressed processing cost information and the uncompressed communication cost information of the N pre-trained gradients to obtain the tensor size of the N pre-trained gradients includes: S12221, using a gradient compression information calculation model, processing the compressed communication cost information, the compressed processing cost information, and the uncompressed communication cost information of the N pre-trained gradients to obtain gradient compression information; The gradient compression information calculation model expression is: Among them, τ is the gradient compression information, To compress communication cost information, To compress the processing cost information, t c is the uncompressed communication cost information, S12222, process the gradient compression information to obtain tensor sizes of N pre-trained gradients.

6. The optimized gradient compression method for enhancing distributed deep learning according to claim 1, characterized in that: The step of processing the training gradient value according to the communication threshold table to obtain the gradient value to be compressed and sending the gradient value to the server node includes: S31, the working node processes the communication threshold table to obtain a pre-training gradient in which the compression gain is greater than 1 for two consecutive times; S32, taking the average of the tensor sizes of the pre-trained gradients with compression gains greater than 1 for two consecutive times as the communication threshold; S33, processing the training gradient value according to the communication threshold, obtaining the gradient value that needs to be compressed, and sending it to the server node.

7. The optimized gradient compression method for enhancing distributed deep learning according to claim 6, characterized in that: The step of processing the training gradient value according to the communication threshold to obtain the gradient value to be compressed and sending the gradient value to the server node includes: S331, obtaining a training gradient value; S332, when the tensor size of the training gradient value is smaller than the communication threshold, sending the training gradient value to a server node; When the tensor size of the training gradient value is greater than the communication threshold, encoding the tensor size of the training gradient value to obtain an encoded tensor size of the training gradient value; The tensor size of the training gradient value and the encoded tensor size of the training gradient value are sent to a server node.

8. An optimized gradient compression device for enhancing distributed deep learning, characterized in that: The device comprises: A communication threshold table construction module is used to construct a communication threshold table using the cocktail system during the pre-training phase; The training gradient value calculation module is used to calculate the gradient of the working node using the locally backed-up model to obtain the training gradient value during the formal training phase; A training gradient value processing module, used to process the training gradient value according to the communication threshold table, obtain the gradient value to be compressed, and send it to the server node; The gradient compression module is used for the server node to process the gradient value that needs to be compressed to achieve gradient compression.

9. An optimized gradient compression device for enhancing distributed deep learning, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the optimized gradient compression method for enhancing distributed deep learning as described in any one of claims 1-7.

10. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, which, when called, are used to execute the optimized gradient compression method for enhancing distributed deep learning as described in any one of claims 1-7.