Federal learning communication compression method and system

Abstract

The invention relates to the field of neural networks and federated learning, and provides a communication compression method and system combining quantization and sparse algorithms, and the method comprises the steps: dividing a model gradient into a plurality of vectors with different dimensions according to the hierarchy of a neural network; performing data compression on all vectors through a quantization algorithm; transmitting the quantized and compressed data to a parameter self-checking module, and determining whether the gradient of the trained model is uploaded or not according to a self-adaptive threshold value and the quantized gradient contribution degree, so as to achieve the aim of sparse communication; and quantization errors and unuploaded gradient models are locally accumulated at working nodes, so that the influence of communication compression on model convergence precision is reduced. The combination of gradient quantization and a communication sparse method is used for solving the problems that the existing communication compression strength is not enough, the model training precision is reduced and the like.

Description

Technical Field

[0001] The present invention relates to the field of neural network and federated learning technology, and in particular to a federated learning communication compression method and system. Background Art

[0002] With the rapid development of emerging technologies such as the Internet of Things and edge computing, the amount of data generated at the edge of the network is growing exponentially, with a large amount of valuable data dispersed across various end devices. Traditional deep learning requires the centralized storage of large amounts of training data for training. However, in real-world environments, centralized data integration is often difficult to achieve, resulting in data silos and the formation of "data silos." Federated learning has, to some extent, addressed this data silo problem. McMahan et al. described a federated learning framework for deep learning tasks and proposed the classic FedAVG algorithm. Federated learning eliminates the need to share private data between nodes, ensuring that nodes have absolute control over their locally stored data. In a typical worker-central server architecture, worker nodes upload local model training information (gradients / parameters) to the central server, which then uses the information uploaded by the worker nodes to update the global model based on an aggregation algorithm. However, federated learning requires a significant amount of communication between nodes to achieve good model accuracy. Furthermore, as the scale of deep learning continues to increase, the number of model parameters is exploding, dramatically increasing the cost per communication in federated learning. Limited by network conditions and bandwidth, these communication costs prevent many edge nodes from participating in federated learning. High communication costs have become a major bottleneck in federated learning. Therefore, research on efficient communication in federated learning has become a necessary task. To reduce the high communication costs of federated learning, researchers have proposed a series of communication compression methods to reduce the communication costs in federated learning and distributed machine learning.

[0003] Neural network pruning is an early approach to model compression and acceleration. Srinivas et al. used pruning to remove redundant neurons, while Han et al. reduced the number of parameters and operations in the entire network, removed redundant connections, and quantized parameters. Vanhoucke et al. demonstrated that 8-bit quantization of parameters can significantly accelerate models with minimal loss of accuracy. While the primary goal of model compression research is to alleviate node storage and computational pressure, model compression also reduces communication costs by reducing network complexity and parameter size. Quantization methods were later directly applied to the study of communication-efficient distributed machine learning. In contrast to model compression, communication compression focuses solely on communication efficiency. By quantizing the content of communication, previous researchers proposed a series of gradient quantization methods to accelerate data-parallel distributed learning. Inspired by model latency, Storm et al. sparsified gradients, sending only those gradients above a certain threshold. Subsequently, Aji, Dryden et al. proposed different gradient selection strategies. Unlike the aforementioned gradient sparsification methods, Chen et al. used the entire node's gradient as the minimum unit for gradient selection. That is, a node either sends all local sub-gradients or none at all. This sparsification method reduces the number of node communications and is therefore also known as a communication sparsification method.

[0004] Quantization and sparsification can significantly reduce the communication cost of distributed machine learning. How to reduce the impact of gradient loss on model convergence under high compression is the focus of research on communication-efficient federated learning. Summary of the Invention

[0005] The present invention provides a communication compression method and system for federated learning, which is used to solve the problem that the compression strength in existing compression methods is not strong and the convergence of model accuracy is reduced after compression. The solution of the present application can meet the requirements of a federated learning communication compression method with high compression strength, low federated communication bandwidth and good convergence accuracy with high compression strength and good model convergence accuracy.

[0006] The present invention provides a federated learning communication compression method, comprising: Initialize the local deep learning model of the training node, train the model using the local dataset, and divide the model gradient into vectors of different dimensions according to the neural network layer; All vectors are compressed by the quantization algorithm RQSGD, and the original k-bit precision data value is compressed to b bits (b <k); The quantized and compressed data is transmitted to the parameter self-check module, and the adaptive threshold and the quantized gradient contribution are used to determine whether the model gradient of this training should be uploaded, thereby achieving the goal of sparse communication. The quantization error and the unuploaded gradient model are accumulated locally on the working node, thereby reducing the impact of communication compression on the model convergence accuracy.

[0007] According to the federated learning communication compression method provided by the present invention, after initializing the local deep learning model of the training node, the method further includes: Initialize the federated learning participating nodes. The central server initializes the CNN model for this training and sends the latest global model parameters to all working nodes. When the total number of training rounds is greater than 1, the global threshold is calculated based on the model gradient uploaded by the working node and sent to the working node together with the global model. Based on the deep learning model sent by the central node, the working node divides the model gradient into vectors of different dimensions according to the neural network layer; The working node memorizes the original deep learning model structure and restores the model vector according to subsequent needs.

[0008] According to the federated learning communication compression method provided by the present invention, all multidimensional vectors converted from model gradients are compressed by the RQSGD quantization algorithm to obtain a compressed vector representation, including: For a given number of n-dimensional vectors v ,right v Middle i Elements v i Perform b-bit quantization; use None is used as the quantization scaling factor, and a full-precision value is used to record the minimum absolute value in the current quantization vector, that is, the minimum factor min factor; The quantization error generated by RQSGD is accumulated locally at the node.

[0009] According to the federated learning communication compression method provided by the present invention, the adaptive parameter self-checking module includes: The quantized vector is used as the input of the self-check module; Calculate the distance between all vectors and the original global model sent by the central server in this round, using the vector Euclidean norm; The calculation result is compared with the threshold of this round sent by the central server. If it is greater than the threshold of this round, the working node sends the training model of this round to the central node; if it is not greater than the threshold, the training result of this round is retained locally and the weighted cumulative gradient error is calculated. After the central server receives the model gradients sent by all working nodes in this round, it uses the vector Euclidean norm to calculate the contribution of all local models and take the average value to obtain the average contribution made by the working nodes in the kth round. It then uses the average contribution of the working nodes in the dth round to estimate the threshold for the next round of training.

[0010] According to the federated learning communication compression method provided by the present invention, quantization errors and unuploaded gradient models are accumulated locally on the working node, including: After the working node updates the model using the local dataset, it checks whether the quantization error of the previous round is stored locally; If it exists, the weighted quantization error is accumulated into the updated model; if it does not exist, it means that this node did not communicate with the central node in the previous round, and the previously trained cumulative model increment is accumulated into the updated model; After the accumulation is complete, the working node uses the accumulated model to perform quantitative calculations and calculates the contribution value of this node using the quantized model. The node chooses to store the accumulated information locally according to the situation.

[0011] The present invention also provides a federated learning communication compression system, comprising: The deep learning model vectorization processing module defines the target model structure of this federated learning, initializes the model parameters, vectorizes the parameters, and obtains the input target compressed data of the quantization algorithm; Quantization algorithm module, used to quantize and compress the vectorized model parameters layer by layer, compressing the original k-bit data to b-bits, usually k=32, b=8 or 4; The communication sparse parameter self-check module sends the quantization results to the parameter self-check module, calculates the node contribution and compares it with the threshold of this round to screen out the nodes for uploading gradient updates, thereby achieving the purpose of communication sparseness.

[0012] The gradient error accumulation module is used to accumulate the quantization error and the gradient model retained by the working node locally on the working node to reduce the impact of communication compression on the convergence of model training.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-described federated learning communication compression methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned federated learning communication compression methods.

[0015] The present invention also provides a computer program product, comprising a computer program, which implements any of the above-mentioned federated learning communication compression methods when executed by a processor.

[0016] The solution provided in this application can meet the requirements of a federated learning communication compression method with high compression strength, low federated communication bandwidth and good convergence accuracy with high compression strength and good model convergence accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the federated learning communication compression method provided by an embodiment of the present invention; Figure 2 This is the second flow chart of the federated learning communication compression method provided by an embodiment of the present invention; Figure 3 Schematic diagram of interval mapping of the quantization algorithm provided in an embodiment of the present invention; Figure 4 1 is a schematic diagram of the structure of a federated learning communication compression system provided by an embodiment of the present invention; Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] Figure 1 This is one of the flow charts of the federated learning communication compression method provided by an embodiment of the present invention.

[0021] Figure 2 This is the second flow chart of the federated learning communication compression method provided by the embodiment of the present invention. like Figure 1 and Figure 2 As shown, this embodiment provides a federated learning communication compression method, including: Step 101: Initialize the local deep learning model of the training node, train the model using the local dataset, and divide the model gradient into vectors of different dimensions according to the neural network hierarchy; Step 102: Compress all vectors using the quantization algorithm RQSGD proposed in this paper, and compress the original k-bit precision data value to b bits (b <k); Step 103: The quantized and compressed data is transmitted to the parameter self-checking module, and the model gradient of this training is determined to be uploaded according to the adaptive threshold and the quantized gradient contribution, so as to achieve the purpose of sparse communication. Step 104: Accumulate the quantization error and the unuploaded gradient model locally on the working node, thereby reducing the impact of communication compression on the model convergence accuracy.

[0022] Gradient quantization algorithms reduce the accuracy of gradients, thereby reducing communication traffic. Most modern computers use 32 or 64 bits to represent floating-point numbers. Quantization reduces the number of bits used to represent each value, compressing communication traffic several times. This compression effect is particularly evident in deep learning, where parameter sizes can easily reach hundreds of thousands. Representing raw values ​​with low bits inevitably introduces quantization error, and quantization methods focus on reducing quantization error and its impact on model convergence.

[0023] Gradient sparsification refers to selectively sending some gradient values, reducing communication costs by discarding some gradient values ​​with smaller contributions. Research has found that the gradients calculated by nodes in distributed SGD are often sparse, with most gradient values ​​close to 0. Such gradient exchanges are not only redundant but also increase communication costs. The redundancy of gradient exchange provides theoretical feasibility for sparsification methods, and how to select gradients is the key to sparsification methods. In this embodiment, the model gradient is processed by applying quantization and sparsification algorithms, which can compress the communication volume in federated learning by several times.

[0024] In an exemplary embodiment, after initializing the local deep learning model of the training node, the method further includes: The central server initializes the deep learning model for this training and sends the latest global model parameters to all working nodes. When the total number of training rounds is greater than 1, the global threshold is calculated based on the model gradient uploaded by the working node and sent to the working node together with the global model. Based on the deep learning model sent by the central node, the working node divides the model gradient into vectors of different dimensions according to the neural network layer; The working node memorizes the original deep learning model structure and restores the model vector according to subsequent needs.

[0025] In an exemplary embodiment, all multidimensional vectors converted from model gradients are compressed using the RQSGD quantization algorithm to obtain a compressed vector representation, including: For a given number of n-dimensional vectors v ,right v Middle i Elements v iPerform b-bit quantization; use As a quantization scaling factor, a full-precision value is used to record the minimum absolute value in the current quantization vector, that is, the minimum factor min factor; The quantization error generated by RQSGD is accumulated locally at the node.

[0026] In an exemplary embodiment, the adaptive parameter self-checking module includes: The quantized vector is used as the input of the self-check module; Calculate the distance between all vectors and the original global model sent by the central server in this round, using the vector Euclidean norm; The calculation result is compared with the threshold of this round sent by the central server. If it is greater than the threshold of this round, the working node sends the training model of this round to the central node; if it is not greater than the threshold, the training result of this round is retained locally and the weighted cumulative gradient error is calculated. After the central server receives the model gradients sent by all working nodes in this round, it uses the vector Euclidean norm to calculate the contribution of all local models and take the average value to obtain the average contribution made by the working nodes in the kth round. It then uses the average contribution of the working nodes in the dth round to estimate the threshold for the next round of training.

[0027] In an exemplary embodiment, quantization errors and unuploaded gradient models are accumulated locally on the working node, including: After the working node updates the model using the local dataset, it checks whether the quantization error of the previous round is stored locally; If it exists, the weighted quantization error is accumulated into the updated model; if it does not exist, it means that this node did not communicate with the central node in the previous round, and the previously trained cumulative model increment is accumulated into the updated model; After the accumulation is completed, the working node uses the accumulated model for quantitative calculations and uses the quantized model for calculations. The node chooses to store the accumulated information locally according to the situation.

[0028] Figure 3 This is a mapping interval diagram of the quantization algorithm RQSGD provided by an embodiment of the present invention.

[0029] like Figure 3 As shown, the quantization level b=3, the RQSGD quantization process is divided into two steps. For a given n-dimensional vector v ,right v Middle i Elements v i Perform b-bit quantization, the process is as follows:

[0030] in is the scaling factor, and defines , Will Mapping to quantized space middle:

[0031] in is the scaling factor, Falling In the interval, l yes Integer between . express The quantization error, . The scaling factor used rather than . The values ​​can be distributed more evenly in the [0,1] interval, that is, more evenly distributed in the quantization space. The value is compressed to near 0, and the quantization space on the right side is not fully utilized. The above formula can easily quantize the smaller values ​​in the vector to 0, while most model gradients or incremental values ​​in deep learning are close to 0. Assume that Represents a model increment vector, , if b=2, the scaling factor ,like 99% of the value satisfy ,but 99% of the values ​​in the vector will be quantized to 0. These 0 values ​​are useless for model learning, a phenomenon we call invalid quantization. This not only slows down the model's convergence, but the transmission of a large number of 0 values ​​also results in significant communication waste. RQSGD corrects these quantized values ​​as shown below. This comes at the cost of using a full-precision value to record the minimum absolute value in the current quantized vector, known as the min factor:

[0032] For an n-dimensional vector v, if the full-precision bit size is 32 bits, RQSGD quantizes the size of 32n bits to 64+nbbits, which includes a full-precision scaling factor and a minimum factor, b<32. The following is an example of a specific implementation process of the federated learning communication compression method provided by this application.

[0033] like Figure 2 As shown, the federated learning communication compression method includes: 1. Deep Learning Model Gradient Vectorization: Initialize the federated learning participating nodes. The central server initializes the CNN model for this training and sends the latest global model parameters to all working nodes. When the overall number of training rounds is greater than 1, the global threshold is calculated based on the model gradient uploaded by the working node and sent to the working node together with the global model. Based on the deep learning model sent by the central node, the working node divides the model gradient into vectors of different dimensions according to the neural network hierarchy. The working node memorizes the original deep learning model structure and restores the vector model according to subsequent needs. 2. Model gradient RQSGD quantization compression: for multiple vectors converted layer by layer by the deep model v ,right v Middle i Elements v i Perform b-bit quantization; use As a quantization scaling factor, a full-precision value is used to record the minimum absolute value in the current quantization vector, that is, the minimum factor min factor; 3. The node performs parameter self-check: the quantized vector is used as the input of the self-check module; Calculate the distance between all vectors and the original global model sent by the central server in this round, using the vector Euclidean norm for calculation; compare the calculation result with the threshold of this round sent by the central server. If it is greater than the threshold of this round, the working node sends the training model of this round to the central node; if it is not greater than the threshold, the training result of this round is retained locally and the weighted cumulative gradient error is calculated; after the central server receives the model gradients sent by all working nodes in this round, it uses the vector Euclidean norm to calculate the contribution of all local models and take the average value to obtain the average contribution made by the working nodes in the dth round. The average contribution of the working nodes in the dth round is used to estimate the threshold for the next round of training; 4. Nodes accumulate gradient errors: After the working node updates the model using the local dataset, it checks whether the quantization error of the previous round is stored locally; If it exists, the weighted quantization error is accumulated into the updated model; if it does not exist, it means that this node did not communicate with the central node in the previous round, and the previously trained cumulative model increment is accumulated into the updated model; After the accumulation is completed, the working node uses the accumulated model for quantitative calculations and uses the quantized model for calculations. The node chooses to store the accumulated information locally according to the situation; 5. Train the defined deep model based on the established federated learning architecture.

[0034] The federated learning communication compression system provided by the present invention is described below. The federated learning communication compression system described below and the federated learning communication compression method described above can be referenced to each other.

[0035] Figure 4 2 is a schematic diagram of the structure of the federated learning communication compression system provided by an embodiment of the present invention.

[0036] like Figure 4 As shown, the federated learning communication compression system provided in this embodiment includes: The deep learning model vectorization processing module 401 is used to define the target model structure of the federated learning, initialize the model parameters, vectorize the parameters, and obtain the input target compressed data of the quantization algorithm; Quantization algorithm module 402, used to quantize and compress the vectorized model parameters layer by layer, compressing the original k-bit data to b-bits, usually k=32, b=8 or 4; The communication sparse parameter self-check module 403 is used to send the quantization results to the parameter self-check module, calculate the node contribution and compare it with the current round threshold to screen out the nodes for uploading gradient updates, thereby achieving the purpose of communication sparseness; The gradient error accumulation module 404 is used to accumulate the quantization error and the gradient model retained by the working node locally in the working node to reduce the impact of communication compression on the convergence of model training.

[0037] The specific implementation method of the federated learning communication compression system provided in this embodiment can be implemented with reference to the above embodiment and will not be repeated here.

[0038] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the federated learning communication compression method, which includes: Initialize the local deep learning model of the training node, train the model using the local dataset, and divide the model gradient into vectors of different dimensions according to the neural network layer; All vectors are compressed by the quantization algorithm RQSGD, and the original k-bit precision data value is compressed to b bits (b <k); The quantized and compressed data is transmitted to the parameter self-check module, and the adaptive threshold and the quantized gradient contribution are used to determine whether the model gradient of this training should be uploaded, thereby achieving the goal of sparse communication. The quantization error and the unuploaded gradient model are accumulated locally on the working node, thereby reducing the impact of communication compression on the model convergence accuracy.

[0039] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0040] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the federated learning communication compression method provided by the above methods, which includes: Initialize the local deep learning model of the training node, train the model using the local dataset, and divide the model gradient into vectors of different dimensions according to the neural network layer; All vectors are compressed by the quantization algorithm RQSGD, and the original k-bit precision data value is compressed to b bits (b <k); The quantized and compressed data is transmitted to the parameter self-check module, and the adaptive threshold and the quantized gradient contribution are used to determine whether the model gradient of this training should be uploaded, thereby achieving the goal of sparse communication. The quantization error and the unuploaded gradient model are accumulated locally on the working node, thereby reducing the impact of communication compression on the model convergence accuracy.

[0041] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the federated learning communication compression method provided by the above methods, the method comprising: Initialize the local deep learning model of the training node, train the model using the local dataset, and divide the model gradient into vectors of different dimensions according to the neural network layer; All vectors are compressed by the quantization algorithm RQSGD, and the original k-bit precision data value is compressed to b bits (b <k); The quantized and compressed data is transmitted to the parameter self-check module, and the adaptive threshold and the quantized gradient contribution are used to determine whether the model gradient of this training should be uploaded, thereby achieving the goal of sparse communication. The quantization error and the unuploaded gradient model are accumulated locally on the working node, thereby reducing the impact of communication compression on the model convergence accuracy.

[0042] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0043] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods of each embodiment or certain portions of the embodiments. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for compressed communication of federated learning, characterized in that: include: Initialize the local deep learning model of the training node, train the model using the local dataset, and divide the model gradient into vectors of different dimensions according to the neural network layer; All vectors are compressed by the quantization algorithm RQSGD, and the original k-bit precision data value is compressed to b bits (b <k); The quantized and compressed data is transmitted to the parameter self-check module, and the adaptive threshold and the quantized gradient contribution are used to determine whether the model gradient of this training should be uploaded, thereby achieving the goal of sparse communication. The quantization error and the unuploaded gradient model are accumulated locally on the working node, thereby reducing the impact of communication compression on the model convergence accuracy.

2. The method for compression of federated learning communication according to claim 1, characterized in that: After initializing the local deep learning model of the training node, the method further includes: Initialize the federated learning participating nodes. The central server initializes the CNN model for this training and sends the latest global model parameters to all working nodes. When the total number of training rounds is greater than 1, the global threshold is calculated based on the model gradient uploaded by the working node and sent to the working node together with the global model. Based on the deep learning model sent by the central node, the working node divides the model gradient into vectors of different dimensions according to the neural network layer; The working node memorizes the original deep learning model structure and restores the model vector according to subsequent needs.

3. The distributed deep learning communication compression method according to claim 1, characterized in that The RQSGD quantization algorithm compresses all multidimensional vectors converted from model gradients to obtain compressed vector representations, including: For a given number of n-dimensional vectors v ,right v Middle i Elements v i Perform b-bit quantization; use As a quantization scaling factor, a full-precision value is used to record the minimum absolute value in the current quantization vector, that is, the minimum factor min factor; The quantization error generated by RQSGD is accumulated locally at the node.

4. The method for compression of federated learning communication according to claim 1, wherein: The adaptive parameter self-checking module includes: The quantized vector is used as the input of the self-check module; Calculate the distance between all vectors and the original global model sent by the central server in this round, using the vector Euclidean norm; The calculation result is compared with the threshold of this round sent by the central server. If it is greater than the threshold of this round, the working node sends the training model of this round to the central node; if it is not greater than the threshold, the training result of this round is retained locally and the weighted cumulative gradient error is calculated. After the central server receives the model gradients sent by all working nodes in this round, it uses the vector Euclidean norm to calculate the contribution of all local models and take the average value to obtain the average contribution made by the working nodes in the kth round. It then uses the average contribution of the working nodes in the dth round to estimate the threshold for the next round of training.

5. The method for compression of federated learning communication according to claim 1, wherein: The quantization error and the unuploaded gradient model are accumulated locally on the working node, including: After the working node updates the model using the local dataset, it checks whether the quantization error of the previous round is stored locally; If it exists, the weighted quantization error is accumulated into the updated model; if it does not exist, it means that this node did not communicate with the central node in the previous round, and the previously trained cumulative model increment is accumulated into the updated model; After the accumulation is complete, the working node uses the accumulated model to perform quantitative calculations and calculates the contribution value of this node using the quantized model. The node chooses to store the accumulated information locally according to the situation.

6. Federated learning communication compression system, characterized by: include: The deep learning model vectorization processing module defines the target model structure of this federated learning, initializes the model parameters, vectorizes the parameters, and obtains the input target compressed data of the quantization algorithm; Quantization algorithm module, used to quantize and compress the vectorized model parameters layer by layer, compressing the original k-bit data to b-bits, usually k=32, b=8 or 4; The communication sparse parameter self-check module sends the quantization results to the parameter self-check module, calculates the node contribution and compares it with the threshold of this round to screen out the nodes for uploading gradient updates, thereby achieving the purpose of communication sparseness.

7. Gradient error accumulation module, used to accumulate the quantization error and the gradient model retained by the working node locally on the working node to reduce the impact of communication compression on the convergence of model training.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for predicting quay crane operation efficiency according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for predicting quay crane operation efficiency according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the federated learning communication compression method according to any one of claims 1 to 5 is implemented.

Images