Multistage fine-grained gradient compression method for neural network distributed learning

Abstract

The invention discloses a multi-level fine-grained gradient compression method for neural network distributed learning. The method comprises the following steps: initializing a global model at a server side; in each round of communication between the server side and the client side, a corresponding current global model is distributed to the selected client side through the server; in each client, performing local training on the received current global model by using the local data set to obtain a local model and calculate a cumulative gradient; performing granularity compression on the cumulative gradient of the local model based on an error feedback mechanism and a multi-stage feedback bit width distribution algorithm to obtain compressed gradient data and uploading the compressed gradient data to a server side; and at the server side, the compression gradient data uploaded by each client side is decompressed, and the global model is updated through weighted average. According to the method, sparsification is regarded as a special case of quantization, a multi-stage feedback bit width distribution algorithm is introduced, and the optimal quantization bit width is dynamically and accurately distributed for each gradient parameter, so that granularity compression is realized.

Description

Technical Field

[0001] This invention belongs to the field of distributed machine learning technology, specifically relating to a multi-level fine-grained gradient compression method for distributed learning of neural networks. Background Technology

[0002] As the complexity of deep neural networks (DNNs) continues to increase, their parameter size grows exponentially, leading to extremely time-consuming training. Distributed training strategies have been introduced into the neural network training process to accelerate training speed and reduce training time, becoming a fundamental technology in research fields such as large model pre-training and federated learning. However, throughout the training process, neural network training requires multiple iterations, and frequent gradient transfers are necessary between computing nodes. The amount of data transmitted in each communication is almost equivalent to the size of the model parameters. This high communication overhead reduces the acceleration efficiency of distributed training, hindering the widespread application of this technology.

[0003] To reduce communication overhead, the academic community has proposed various methods, including sparsification, quantization, matrix factorization, and knowledge distillation. Sparsification methods, such as Deep Gradient Compression (DGC), significantly reduce communication overhead by selectively transmitting gradients with larger absolute values ​​and introduce error feedback mechanisms to mitigate information loss. Quantization methods, such as SignSGD, compress gradients into 1-bit sign values ​​(+1 / -1), reducing the communication cost per parameter from 32 bits to 1 bit.

[0004] However, existing technologies have significant limitations. Sparsification can be viewed as a special form of quantization, using only two bit widths (0 bits and 32 bits) to handle unimportant and important parameters respectively. This binarization's compression granularity is too coarse; using a 32-bit width to transmit gradients may introduce information redundancy, while compressing gradients to 0 bits may ultimately lead to suboptimal model convergence. Quantization methods, on the other hand, uniformly compress all gradients to a specified fixed bit width, resulting in significant information loss and a decline in model convergence performance, while limiting the maximum achievable compression ratio to 32:1. Clearly, existing technologies lack finer-grained quantization levels. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a multi-level fine-grained gradient compression method for distributed learning of neural networks. This method achieves finer-grained compression than sparsification and fixed-bit quantization, thereby minimizing the loss of gradient information during compression. This ensures that distributed learning neural networks can still converge quickly and stably and achieve high accuracy even at high compression ratios. Furthermore, it solves the problem that existing methods, under limited communication budgets, suffer from insufficient compression granularity and are unable to adaptively and finely allocate bit widths based on the importance of gradients, thus making it difficult to achieve an optimal balance between compression ratio and model performance.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: a multi-level fine-grained gradient compression method for distributed learning of neural networks, comprising the following steps: Initialize the global model on the server side; the global model is a neural network model based on distributed learning. In each round of communication between the server and the client, execute: The server distributes the corresponding current global model to the selected clients. In each client, the received current global model is trained locally using the local dataset to obtain a local model, and the cumulative gradient of the local model is calculated. The cumulative gradient of the local model is compressed in a granular manner based on the error feedback mechanism and the multi-level feedback bit width allocation algorithm to obtain compressed gradient data and upload it to the server. On the server side, the compressed gradient data uploaded by each client is decompressed, and the global model is updated by weighted averaging.

[0007] Furthermore, in each round of communication, the accumulated gradient of the local model is compressed in a granular manner based on the error feedback mechanism and the multi-level feedback bit width allocation algorithm, including: Based on the error feedback mechanism, the residual of the local model parameters after local training is calculated in the current communication round; The absolute value of the residuals of each parameter is used as the importance measure of the corresponding parameter, and then a parameter importance matrix is ​​constructed. Based on the parameter importance matrix and the client's communication budget, a bit width allocation matrix is ​​constructed through a multi-level feedback bit width allocation algorithm; in the bit width allocation matrix, each parameter corresponds to a quantized bit width. Based on the bit width allocation matrix, the residuals of each parameter are symmetrically quantized, and the scaling factor required for storage quantization is calculated. Based on the residuals of each parameter and their symmetric quantization results, the compression error in the current communication round is calculated; the compression error is accumulated into the gradient calculation in the next communication round. The integer values ​​and scaling factors of the symmetric quantization results are used as compressed gradient data and uploaded to the server.

[0008] Furthermore, a bit width allocation matrix is ​​constructed using a multi-level feedback bit width allocation algorithm, including: S100, Predefined discrete bit width set B; S200. Initialize the bit width level of all parameters of the local model to zero; the bit width level is used to map the local model parameters to the corresponding bit width in the discrete bit width set. S300. Determine the dynamic range of importance scores for local model parameters in the current communication round, as well as the importance decay step size; S400. From all parameters of the local model, select the top few parameters with the highest current importance scores; S500, For each selected parameter, calculate the bit width increment; S600, sequentially determine whether the remaining communication budget is at least greater than or equal to the bit width increment of one parameter; If so, increase the bit width level of the corresponding parameter, update the remaining communication budget and decay its importance score, and return to step S400; If not, proceed to step S700; S700. Based on the current bit width level of each parameter, map it to the discrete bit width set to construct the bit width allocation matrix.

[0009] Furthermore, in step S100, the size of the discrete bit width set is determined based on the data types supported by the client hardware configuration and manual configuration.

[0010] Furthermore, in step S300, the importance decay step size for: In the formula, and These represent the current communication rounds. The maximum and minimum values ​​of the importance of local model parameters. This represents the step size factor used to control the fineness of the allocation.

[0011] Furthermore, in step S500, the bit width increment... for: In the formula, Represents the set of discrete bit widths. Indicates the current communication round Selected parameters bit width level, Indicates communication rounds Selected parameters Bit width level.

[0012] Furthermore, in step S600, the current communication round Selected parameters Upgraded bit width level for: In the formula, Indicates the parameter at the current iteration number. Bit width level before upgrade.

[0013] Furthermore, the current communication rounds Selected parameters Importance score after decay for: In the formula, Indicates the current communication round Selected parameters Importance score before decay This indicates the importance decay step size.

[0014] Furthermore, based on the bit width allocation matrix, the method for symmetric quantization of the residuals of each parameter is as follows: Based on the bit width allocated to each parameter, map its floating-point value to... At discrete quantization levels; where, Indicates the current communication round The following parameters The bit width.

[0015] The beneficial effects of this invention are as follows: (1) It achieves an extremely high compression ratio while maintaining high model accuracy, and solves the problem of insufficient granularity in sparsification and quantization methods. By comparing the accuracy of the method of this invention with other methods in various model and dataset training scenarios, this method can still maintain extremely high accuracy under extremely high compression ratios and is suitable for complex tasks; at the same compression ratio, it maintains high accuracy and model precision, balancing compression ratio and model performance.

[0016] (2) It significantly reduces information loss during gradient compression and improves compression efficiency. By introducing compression efficiency to quantify the information retention of the method, and by comparing with existing methods, this method retains more key gradient information during compression, providing a direct technical explanation for maintaining model performance.

[0017] (3) It enhances the stability of the training process and avoids the risk of model collapse: The MFBA algorithm used in this method performs dynamic and smooth bit allocation based on gradient residuals. Compared with the existing STC method, which suffers from model collapse in the later stages of training, this method exhibits stable convergence characteristics in all experimental scenarios, avoiding the radical or unstable compression decisions that may occur with STC, and does not experience a sharp decline in performance.

[0018] (4) It maintains low computational overhead and has good practicality and scalability: Based on the complexity and runtime performance analysis of this method and existing methods, this method, under high compression ratio, has a time complexity that is linearly close to the model size, and a space complexity that is proportional to the number of model parameters. By adjusting the parallelism, the compression latency of this method can be significantly reduced; the running time is much lower than the local training time of the model and will not become a system bottleneck; furthermore, increasing the parallelism P improves the running efficiency without negatively affecting the model accuracy, providing flexibility for deployment and acceleration on actual hardware. Attached Figure Description

[0019] Figure 1 The flowchart of the multi-level fine-grained gradient compression method for distributed learning of neural networks provided by the present invention is shown.

[0020] Figure 2 The block diagram for implementing multi-level fine-grained gradient compression provided by the present invention is shown.

[0021] Figure 3 A comparison of compression efficiency curves for different methods provided by this invention. Detailed Implementation

[0022] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0023] This invention provides a multi-level fine-grained gradient compression method for distributed learning of neural networks. It treats sparsity as a special case of quantization and introduces a multilevel feedback bitwidth allocation (MFBA) algorithm to dynamically and accurately allocate the optimal quantization bit width for each gradient parameter, thereby achieving granular compression.

[0024] refer to Figure 1 Multi-level fine-grained gradient compression includes the following steps: Initialize the global model on the server side; the global model is a neural network model based on distributed learning. In each round of communication between the server and the client, execute: The server distributes the corresponding current global model to the selected clients. In each client, the received current global model is trained locally using the local dataset to obtain a local model, and the cumulative gradient of the local model is calculated. The cumulative gradient of the local model is compressed in a granular manner based on the error feedback mechanism and the multi-level feedback bit width allocation algorithm to obtain compressed gradient data and upload it to the server. On the server side, the compressed gradient data uploaded by each client is decompressed, and the global model is updated by weighted averaging.

[0025] In this embodiment, based on Figure 2 The gradient compression process framework shown initializes the global model on the server side. In each round of communication, the server selects a subset of clients and sets the current global model. Distributed to each client, on each client Using a local dataset, the received global model conduct indivual Local training yields an updated local model. And calculate the update amount of the local model, i.e., the accumulated gradient. .

[0026] In this embodiment, in each round of communication, the accumulated gradient of the local model is granularly compressed (MAPQ) based on the error feedback mechanism and the multi-level feedback bit width allocation algorithm, including: Based on the error feedback mechanism, the residuals of the local model parameters after local training are calculated for the current communication round. ;in, This represents the error of gradient compression in the previous communication round; The absolute value of the residuals of each parameter is used as a measure of the importance of the corresponding parameter, and then a parameter importance matrix is ​​constructed. Its dimensions are equal to the total number of global model parameters. Same; specifically, for parameters Its importance is expressed as This importance metric can effectively reflect the current contribution of the gradient to the model update; Based on the parameter importance matrix and the client's communication budget, a bit width allocation matrix is ​​constructed through a multi-level feedback bit width allocation algorithm; in the bit width allocation matrix, each parameter corresponds to a quantized bit width. Based on the bit width allocation matrix, the residuals of each parameter are symmetrically quantized, and the scaling factor required for storage quantization is calculated. ; Specifically, the symmetric quantization method is as follows: based on the bit width allocated to each parameter, the floating-point value is mapped to... At discrete quantization levels; where, Indicates the current communication round The following parameters bit width; Based on the residuals of each parameter and their symmetric quantization results, the compression error in the current communication round is calculated. , This indicates a quantization operation; where compression errors are accumulated into gradient calculations in the next communication round to ensure that information is not permanently lost due to compression. Use the integer values ​​and scaling factor of the symmetric quantization result as compressed gradient data. Upload it to the server.

[0027] Furthermore, the server decompresses the compressed gradients received from each client, then performs a weighted average based on the gradient averaging method to update the global model.

[0028] In this embodiment, the Multi-Level Feedback Bit Width Allocation (MFBA) algorithm is the core of the invention. Its design is inspired by multi-level feedback queue scheduling in operating systems, and its goal is to optimize communication budget. Under the constraints, the bit width is dynamically and reasonably allocated according to the importance score of the parameters using a greedy strategy.

[0029] In this embodiment, a bit width allocation matrix is ​​constructed using a multi-level feedback bit width allocation algorithm, including: S100, Predefined discrete bit width set B; S200. Initialize the bit width level of all parameters of the local model to zero; the bit width level is used to map the local model parameters to the corresponding bit width in the discrete bit width set. S300. Determine the dynamic range of importance scores for local model parameters in the current communication round, as well as the importance decay step size; S400. From all parameters of the local model, select the top few parameters with the highest current importance scores; S500, For each selected parameter, calculate the bit width increment; S600, sequentially determine whether the remaining communication budget is at least greater than or equal to the bit width increment of one parameter; If so, increase the bit width level of the corresponding parameter, update the remaining communication budget and decay its importance score, and return to step S400; If not, proceed to step S700; S700. Based on the current bit width level of each parameter, map it to the discrete bit width set to construct the bit width allocation matrix.

[0030] In step S100, the discrete bit width set The size is determined based on the data types supported by the client's hardware configuration and manual configuration; for example, ,Right now .

[0031] In step S200, the bit width level is used to map the local model parameters to the corresponding bit width in the discrete bit width set, i.e. During initialization, This means that all parameters are not initially transmitted.

[0032] In step S300, the importance decay step size for: In the formula, and These represent the current communication rounds. The maximum and minimum values ​​of the importance of local model parameters. This represents the step size factor used to control the fineness of the allocation.

[0033] In step S400, from all parameters, select the one with the highest current importance score. Parameters , This is called the degree of parallelism, which allows multiple parameters to be processed simultaneously to improve algorithm efficiency (usually through hardware parallelization, which can significantly improve the speed of the process in practical use). The selected parameters are represented as follows: In step S500, for each selected parameter Calculate the number of additional bits required to increase its bit width by one level, i.e., the bit width increment. for: In the formula, Represents the set of discrete bit widths. Indicates the current communication round Selected parameters bit width level, Indicates communication rounds Selected parameters Bit width level.

[0034] In step S600, when the remaining communication budget At that time, the current communication round Selected parameters Upgraded bit width level for: In the formula, Indicates the parameter at the current iteration number. Bit width level before upgrade.

[0035] The updated remaining communication budget is represented as follows: The importance score of the attenuation parameter is used to simulate that its resource requirements have been met, and the current communication round is obtained. Selected parameters Importance score after decay for: In the formula, Indicates the current communication round Selected parameters Importance score before decay This indicates the importance decay step size.

[0036] If the remaining communication budget is insufficient to increase the bit width of any candidate parameter, the loop terminates. After the loop ends, each parameter... All of them obtained a final bit width level. This allows us to obtain the quantized bit width, and then construct the bit width allocation matrix. .

[0037] The Multi-Level Feedback Bit Width Allocation (MFBA) algorithm provided by this invention acts as an "intelligent resource scheduler," treating limited communication bits (budget) as a resource and prioritizing their allocation to the gradients most important for updating the current model. By iteratively improving the accuracy (bit width) of important gradients while diminishing their importance, optimal allocation under budget constraints is achieved, preserving the key information in the original gradients to the maximum extent.

[0038] In this embodiment, as shown in Table 1, the final accuracy of MAPQ and other methods in federated learning is compared under various model and dataset training scenarios. Table 1: Accuracy Comparison of MAPQ with Other Methods Table 1 highlights the best and second-best performance using bold and underline, respectively. DGC, STC, and MAPQ all employ a compression ratio of 10,000:1, while signSGD and z-sign maintain their original 1 / 32 compression ratio (highlighted in the table). Table 1 shows that extensive experiments on seven datasets and seven models demonstrate that MAPQ maintains extremely high accuracy even at a very high compression ratio of 10,000:1 (0.01% compression). On some complex tasks (such as ResNet and ViT models on the SVHN and ImageNette datasets), its performance even surpasses that of uncompressed FedAvg.

[0039] Comparative analysis shows that, at the same compression ratio (0.01%), MAPQ's accuracy is significantly higher than that of the sparsity method DGC and the hybrid method STC in most scenarios. Furthermore, although the compression ratios of the quantization methods SignSGD and z-signFedAvg are only 32:1 (far lower than MAPQ's 10000:1), MAPQ's model accuracy still significantly surpasses theirs. This fully demonstrates the significant advantage of the granular compression proposed in this invention in balancing compression ratio and performance.

[0040] This invention quantifies the degree of information retention by introducing a compression efficiency metric; where compression efficiency is defined as the cosine similarity between the decompressed gradient and the original gradient. A higher cosine similarity indicates less information loss, expressed as: like Figure 3 As shown, throughout the training process, the compression efficiency curve of MAPQ is consistently significantly higher than that of the comparative methods such as DGC, STC, and SignSGD, and is very close to that of uncompressed FedAvg (cosine similarity is 1). This provides a direct technical explanation for MAPQ's ability to maintain model performance: it retains more key gradient information during the compression process.

[0041] Furthermore, a complexity analysis and runtime performance comparison are performed between the method of this invention and existing methods, specifically: The time complexity of the MAPQ algorithm is ,in For the number of model parameters, under high compression ratios, The magnitude is much smaller than Therefore, the overall complexity is linearly close to the model size. The space complexity is O(n). Only the importance matrix and bit-level matrix need to be stored, which are proportional to the number of model parameters.

[0042] As shown in Table 2, by adjusting the parallelism MAPQ's compression latency can be significantly reduced. Although its latency is slightly higher than some minimalist methods (such as SignSGD), it is far lower than the local model training time and will not become a system bottleneck. More importantly, Table 3 shows that the parallelism... Increasing the efficiency of MAPQ does not negatively impact model accuracy, providing flexibility for deployment and acceleration on actual hardware. In Table 2, the numbers after MAPQ indicate its parallelism. Since FedAvg does not involve gradient compression, its compression latency is always 0; in Table 3, the model accuracy in each experiment did not show significant changes under different P-values.

[0043] Table 2: Compression delay of different methods Table 3: Model accuracy under different values ​​of parallelism P The Multi-Level Feedback Bit Width Allocation (MFBA) algorithm in this invention aims to allocate bit widths based on parameters, and it employs a greedy strategy. Furthermore, optimization methods or scheduling strategies can also be used to achieve the goal of bit width allocation, such as the following scheme: The allocation scheme based on optimization theory formalizes the bit allocation problem as a constrained optimization problem. The objective function is to maximize the sum of the importance scores of all parameters and the information carried by the allocated bit width. The constraints are that the total number of bits does not exceed the communication budget, and the bit width is taken from a discrete bit width set. Then, integer programming or a relaxed continuous optimization plus rounding method is used to solve the problem. This scheme allocates from the perspective of global optimum, but the computational cost is high, which may not be suitable for online scenarios where rapid allocation is required in each round of communication.

[0044] Schemes based on grouping or hierarchical allocation: Instead of directly allocating each parameter, the parameters are first roughly grouped according to their importance (e.g., high, medium, and low groups). Then, the total budget is allocated to different groups in certain proportions (e.g., 70%, 20%, 10%), and parameters within each group are then allocated using uniform quantization or simple MFBA. Alternatively, allocation can be done by model hierarchy, assigning different total bit budgets to different layers, and then using MFBA within each layer. This scheme is a two-stage strategy of "coarse-grained allocation + fine-grained allocation," which can be seen as a simplification or variant of fully fine-grained MFBA allocation. It can reduce the computational complexity of the allocation algorithm, but the allocation accuracy may not be as good as MFBA, which operates directly on each parameter.

[0045] The learning-based allocation scheme uses a lightweight neural network (such as a small MLP) or a reinforcement learning agent. Taking the gradient information and historical states of the parameters as input, it directly outputs the bit width to be allocated for each parameter. This network or agent can be trained and optimized along with the main model during distributed deep learning. This is a "meta-learning" or "learning how to compress" approach, which may adaptively learn a better allocation strategy than a manually designed MFBA, but introduces additional model complexity and training instability. The MFBA in this invention is a deterministic, training-free rule, simpler, more reliable, and easier to implement, while not introducing excessive computational latency.

[0046] Based on the above scheme, it can be seen that although there are other methods that can also achieve gradient compression in the distributed learning process of neural network models, their principle is the same as that of MFBA in this invention. They are all used to implement the width allocation of model parameters, and their allocation accuracy and model training effect are not as good as the method of this invention. Therefore, it should be noted that this invention requests protection for a method of achieving granular gradient compression through bit width allocation. Other methods that use similar ideas to achieve model gradient compression are all within the scope of protection of this application.

[0047] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0048] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A multi-level fine-grained gradient compression method for distributed learning of neural networks, characterized in that, Includes the following steps: Initialize the global model on the server side; the global model is a neural network model based on distributed learning. In each round of communication between the server and the client, execute: The server distributes the corresponding current global model to the selected clients. In each client, the received current global model is trained locally using the local dataset to obtain a local model, and the cumulative gradient of the local model is calculated. The cumulative gradient of the local model is compressed in a granular manner based on the error feedback mechanism and the multi-level feedback bit width allocation algorithm to obtain compressed gradient data and upload it to the server. On the server side, the compressed gradient data uploaded by each client is decompressed, and the global model is updated by weighted averaging.

2. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 1, characterized in that, In each round of communication, the accumulated gradient of the local model is compressed in a granular manner based on the error feedback mechanism and the multi-level feedback bit width allocation algorithm, including: Based on the error feedback mechanism, the residual of the local model parameters after local training is calculated in the current communication round; The absolute value of the residuals of each parameter is used as the importance measure of the corresponding parameter, and then a parameter importance matrix is ​​constructed. Based on the parameter importance matrix and the client's communication budget, a bit width allocation matrix is ​​constructed through a multi-level feedback bit width allocation algorithm; in the bit width allocation matrix, each parameter corresponds to a quantized bit width. Based on the bit width allocation matrix, the residuals of each parameter are symmetrically quantized, and the scaling factor required for storage quantization is calculated. Based on the residuals of each parameter and their symmetric quantization results, the compression error in the current communication round is calculated; the compression error is accumulated into the gradient calculation in the next communication round. The integer values ​​and scaling factors of the symmetric quantization results are used as compressed gradient data and uploaded to the server.

3. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 2, characterized in that, A bit width allocation matrix is ​​constructed using a multi-level feedback bit width allocation algorithm, including: S100, Predefined discrete bit width set B; S200. Initialize the bit width level of all parameters of the local model to zero; the bit width level is used to map the local model parameters to the corresponding bit width in the discrete bit width set. S300. Determine the dynamic range of importance scores for local model parameters in the current communication round, as well as the importance decay step size; S400. From all parameters of the local model, select the top few parameters with the highest current importance scores; S500, For each selected parameter, calculate the bit width increment; S600, sequentially determine whether the remaining communication budget is at least greater than or equal to the bit width increment of one parameter; If so, increase the bit width level of the corresponding parameter, update the remaining communication budget and decay its importance score, and return to step S400; If not, proceed to step S700; S700. Based on the current bit width level of each parameter, map it to the discrete bit width set to construct the bit width allocation matrix.

4. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 3, characterized in that, In step S100, the size of the discrete bit width set is determined based on the data types supported by the client hardware configuration and manual configuration.

5. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 3, characterized in that, In step S300, the importance decay step size for: In the formula, and These represent the current communication rounds. The maximum and minimum values ​​of the importance of local model parameters. This represents the step size factor used to control the fineness of the allocation.

6. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 3, characterized in that, In step S500, the bit width increment for: In the formula, Represents the set of discrete bit widths. Indicates the current communication round Selected parameters bit width level, Indicates communication rounds Selected parameters Bit width level.

7. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 3, characterized in that, In step S600, the current communication round Selected parameters Upgraded bit width level for: In the formula, Indicates the parameter at the current iteration number. Bit width level before upgrade.

8. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 3, characterized in that, Current communication round Selected parameters Importance score after decay for: In the formula, Indicates the current communication round Selected parameters Importance score before decay This indicates the importance decay step size.

9. The multi-level fine-grained gradient compression method for distributed learning of neural networks according to claim 2, characterized in that, The method for symmetric quantization of the residuals of each parameter based on the bit width allocation matrix is ​​as follows: Based on the bit width allocated to each parameter, map its floating-point value to... At discrete quantization levels; where, Indicates the current communication round The following parameters The bit width.

Images