A Model Parameter Update Method for Distributed Federated Learning
By using pooling operators to compress the gradient value matrix in distributed federated learning and combining it with a distributed gradient descent algorithm, the problems of accuracy loss and reduced convergence rate during model parameter updates are solved, achieving efficient model training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2023-01-13
- Publication Date
- 2026-05-26
AI Technical Summary
In existing distributed federated learning models, the communication compression scheme discards the local correlation information implied by gradient values during the model parameter update process, leading to a loss of accuracy and a decrease in convergence rate.
Pooling operators are used to compress the gradient value matrix, preserving gradient information while reducing communication. The distributed gradient descent algorithm is combined to update model parameters. Through the collaborative work of edge nodes and the central server, the gradient matrix is sparsified and unpooled.
While reducing communication volume, the model accuracy is maintained and the number of communication rounds is reduced, thus improving training efficiency and alleviating the communication bottleneck in the training process of large neural networks.
Smart Images
Figure CN116306912B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of machine learning, specifically relating to a method for updating model parameters in distributed federated learning. Background Technology
[0002] In recent years, deep neural networks have become an important method for solving many machine learning problems, especially convolutional neural networks, which have seen rapid development and have been applied in fields such as computer vision and natural language processing. With the increasing amount of training data and the growing complexity of deep neural networks, distributed machine learning has been widely used to train these large-scale deep neural networks, significantly reducing training time.
[0003] Federated learning is a popular and efficient distributed machine learning paradigm and framework. When using federated learning to model and train a model, a server-edge node architecture is consistently employed. The server is responsible for model updates and task scheduling for all edge nodes throughout the federated learning task. Edge nodes compute the model parameters assigned by the server using their local data and transmit the results back to the server. The server aggregates the received parameters to update the global parameters and then distributes the updated global parameters to each edge node. This process repeats until the model converges.
[0004] In some cases, neural networks require a large number of model parameters when processing large-scale training data. This makes communication overhead a major bottleneck in the training process, with communication time for parameters far exceeding computation time. To alleviate the communication pressure in these scenarios, existing technologies employ quantization-based and sparsity-based communication compression methods to reduce communication bandwidth during model parameter updates. However, because these two methods discard the local correlation information implicit in gradient values, they cause accuracy loss and reduced convergence rate. Summary of the Invention
[0005] The technical problem to be solved by this invention is to propose a distributed federated learning model parameter update method to solve the problems of accuracy loss and reduced convergence rate caused by discarding the local correlation information implicit in the gradient values in the communication compression scheme used in the model parameter update process in the prior art.
[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0007] A distributed federated learning model parameter update method is applied to a distributed architecture including a central server and multiple edge nodes. The method includes: the central server designating the edge nodes participating in training, initializing model parameters, loss functions, and training hyperparameter information, and distributing them to the designated edge nodes; the distributed architecture then executes the following steps according to iteration rounds:
[0008] S1. Each designated edge node uses the locally stored training samples to calculate the gradient matrix of the model parameters in the current round through the backpropagation algorithm.
[0009] S2. Each designated edge node uses a pooling operator to compress the gradient value matrix calculated in step S1, obtains the sparse gradient matrix of the current round, and uploads it to the central server.
[0010] S3. The central server updates the model parameters for the current round using the distributed gradient descent algorithm based on the sparse gradient matrix of each designated edge node in the current round, and obtains the model parameters for the next round.
[0011] S4. The central server determines whether the model parameters for the next round have reached the preset accuracy. If so, it outputs the model parameters for the next round as the final model parameters. Otherwise, it distributes the model parameters for the next round to each designated edge node, returns to step S1, and enters the next round of iteration.
[0012] Furthermore, the training hyperparameter information includes: pooling compression step size, local gradient descent learning rate of edge nodes, gradient residual value of each edge node, and global gradient descent learning rate.
[0013] Furthermore, in step S1, each designated edge node uses the locally stored training samples to calculate the gradient matrix of the model parameters for the current round using the backpropagation algorithm, specifically including:
[0014] The designated edge node randomly selects a training sample for the current round from the locally stored training samples. Using the selected training sample for the current round, it calculates the current loss value based on the loss function and the model parameters for the current round. Then, it uses the backpropagation algorithm to calculate the gradient matrix of the model parameters for the current round based on the current loss value.
[0015] Furthermore, in step S2, each designated edge node uses a pooling operator to compress the gradient value matrix calculated in step S1 to obtain the sparse gradient matrix for the current round, specifically including:
[0016] First, the designated edge nodes are compressed using pooling operators to compress the gradient matrix of the model parameters in the current round and the gradient residual values in the current round, resulting in the sparsed gradient matrix for the current round:
[0017]
[0018] in, Pool represents the sparsed gradient matrix of the i-th specified edge node in the k-th round; RIndicates the pooling operator; α represents the gradient residual value of the i-th designated edge node in the k-th round; α represents the local gradient descent learning rate of the edge node; This represents the model parameters computed in the k-th round for the i-th specified edge node. The gradient value matrix;
[0019] Next, update the gradient residual values for the next round:
[0020]
[0021] in, This represents the gradient residual value of the i-th specified edge node in the (k+1)-th round.
[0022] Furthermore, in step S2, the pooling operator is either a max pooling operator or an average pooling operator.
[0023] Furthermore, in step S3, the central server updates the model parameters using a distributed gradient descent algorithm based on the received sparse gradient matrices of each designated edge node for the current round, to obtain the model parameters for the next round, specifically including:
[0024] First, the central server processes the sparsed gradient matrix of each designated edge node for the current round. Perform unpooling to obtain the recovered gradient matrix.
[0025] Then, the model parameters from the previous round are updated using the distributed gradient descent algorithm to obtain the model parameters for the current round:
[0026]
[0027] Where, x k x represents the model parameters in the k-th round; k+1 γ represents the model parameters in the (k+1)th round; γ represents the global gradient descent learning rate. Indicates the number of specified edge nodes.
[0028] The beneficial effects of this invention are:
[0029] While compressing the gradient value matrix of the model parameters by pooling algorithm, the local correlation of gradient elements in the gradient matrix is preserved by using gradient residuals to achieve maximum preservation of gradient information. This reduces the amount of communication of uploaded gradient matrix information, alleviates the training time extension caused by high-load communication when training large neural networks, and ensures model accuracy as much as possible.
[0030] Therefore, compared with the existing communication compression methods based on quantization and sparsity in the process of updating model parameters, the present invention can achieve better model accuracy and fewer communication rounds with the same communication compression rate. Attached Figure Description
[0031] Figure 1 This is a flowchart of the model parameter update method for distributed federated learning in an embodiment of the present invention;
[0032] Figure 2 This is a schematic diagram illustrating the principle of data compression using the max pooling operator.
[0033] Figure 3 This is a schematic diagram illustrating the principle of data compression using the average pooling operator. Detailed Implementation
[0034] This invention aims to propose a distributed federated learning model parameter update method, addressing the accuracy loss and reduced convergence rate issues inherent in existing communication compression schemes used during model parameter updates, which discard local correlation information implicit in gradient values. In this invention, edge nodes utilize locally stored training samples to calculate the gradient matrix of the model parameters for the current iteration using the backpropagation algorithm. Then, a pooling operator is used to compress the gradient matrix, preserving gradient information while maintaining communication data volume, resulting in a sparse gradient matrix for the current iteration, which is then uploaded to the central server. The central server, based on the received sparse gradient matrices from each designated edge node, updates the model parameters for the current iteration using a distributed gradient descent algorithm to obtain the model parameters for the next iteration. If the obtained next-element model parameters achieve the required preset accuracy, they are output as the final model parameters. Otherwise, the next-element model parameters are redistributed to the edge nodes for the next iteration.
[0035] Example:
[0036] like Figure 1 As shown, the distributed federated learning model parameter update method in this embodiment includes the following implementation steps:
[0037] 1. The central server designates the edge nodes and initializes the relevant data:
[0038] In this step, the central server can select multiple edge nodes to participate in the model training task based on the actual load of all edge nodes. Each edge node locally stores the sample data (Dataset) that will be used to train the model parameters. i , where i represents the specified i-th edge node;
[0039] Next, the central server initializes the relevant data, including initializing the model parameters x, the training loss function f(x), and the training hyperparameters. The training hyperparameter information includes the pooling compression step size R, the local gradient descent learning rate α at the edge nodes, and the gradient residual parameter m at each edge node. i The global gradient descent learning rate γ; after initialization, the central server distributes the above data to the designated edge nodes for the edge nodes to update the gradient values of the parameters.
[0040] II. Calculate the gradient matrix of the model parameters for the current round at the edge nodes:
[0041] In this step, assuming the current iteration round is the kth round, the designated edge node retrieves the training sample dataset from its local storage. i Randomly select training samples from the current round of the dataset i dataset i ∈Dataset i Using the selected training sample dataset for the current round i Based on the loss function f(x) and the model parameters of the current round Calculate the current loss value, and then, based on the current loss, use the backpropagation algorithm to calculate the gradient matrix of the model parameters for the current round.
[0042] It should be noted that the amount of training samples selected by the edge nodes in this step will also affect the efficiency of the algorithm. The more training samples selected, the higher the model accuracy that the algorithm can achieve and the fewer communication rounds it takes to reach the target accuracy. However, the time it takes for the backpropagation algorithm to calculate the result will also be higher. Therefore, these factors can be considered when configuring the number of samples selected each time.
[0043] 3. The gradient value matrix of the edge nodes is compressed using the pooling operator to obtain a sparse gradient matrix:
[0044] In this step, firstly, the designated edge node uses a pooling operator to compress the gradient value matrix of the model parameters in the current round and the gradient residual value in the current round to obtain the sparse gradient matrix of the current round. Then, based on the gradient value matrix of the model parameters in the current round, the gradient residual value in the current round, and the calculated sparse gradient matrix of the current round, the gradient residual value of the next round is updated. This preserves as much of the untransmitted gradient value information as possible, improves model accuracy, and saves the computational load of the edge nodes to the greatest extent, speeding up the training process while reducing the number of communication rounds.
[0045] In practice, the pooling operator can be the max pooling operator MaxPool. R Or the average pooling operator AvgPool R The principle of the max pooling operator is illustrated in the following example. Figure 2 As shown, its mathematical representation is:
[0046] MaxPool, the maximum pooling compression operator R : Defined as, for input have:
[0047]
[0048] An example of the principle of the average pooling operator is as follows: Figure 3 As shown, its mathematical representation is:
[0049] For input There is: the average pooling compression operator AvgPool R Defined as, for input There is: AvgPool R (X) := Avg(x) iR:iR+R,jR∶jR+R ), where Avg(x iR:iR+R,jR∶jR+R ) represents the mean value of the area covered by each filter.
[0050] For ease of description, we will use Pool for both max pooling and average pooling operators. R Operator representation.
[0051] Assuming the current iteration is the kth iteration, the gradient matrix of the model parameters in the current iteration is compressed using the pooling operator at the specified edge nodes. and the gradient residual value of the current round Obtain the sparsed gradient matrix for the current round.
[0052]
[0053] in, Pool represents the sparsed gradient matrix of the i-th specified edge node in the k-th round; R Indicates the pooling operator; α represents the gradient residual value of the i-th designated edge node in the k-th round; α represents the local gradient descent learning rate of the edge node; This represents the model parameters computed in the k-th round for the i-th specified edge node. The gradient value matrix;
[0054] In order to update the gradient residual values for the next round The difference between the uncompressed gradient matrix and the compressed gradient matrix will be calculated and then summed into the gradient residual value of the current round. As the gradient residual value for the next round In terms of form, it is expressed as:
[0055]
[0056] The sparsed gradient matrix obtained above for the current round The values are uploaded to the central server for model parameter updates, and the gradient residuals for the next round are used for further updates. It is stored locally on the edge node for use in the next round of gradient value matrix compression.
[0057] It should be noted that when performing pooling operator compression, the larger the compression step size R, the lower the compression ratio, and vice versa. Those skilled in the art will know that the higher the compression ratio, the less network bandwidth is used. However, since less gradient value information is transmitted to the central server, the time to train the final model parameters is longer. Therefore, these factors can be considered when configuring the pooling compression step size R.
[0058] IV. The central server updates the model parameters based on the sparse gradient matrix of the current round:
[0059] In this step, firstly, the central server processes the sparse gradient matrix of each designated edge node for the current round. Perform unpooling to obtain the recovered gradient matrix. Then, based on the recovered gradient value matrix corresponding to each edge node The model parameters for the current round are obtained by updating the model parameters from the previous round using a distributed gradient descent algorithm.
[0060] In practice, once all the specified edge nodes have calculated their sparse gradient matrices... The data is uploaded to the central server, which then performs an unpooling step to expand the size of the sparsed gradient matrix, restoring it to its original size before sparsification. In form, it is... This is represented as follows: The gradient value matrices restored to their original size are then summed and multiplied by a global gradient descent learning rate γ. A gradient descent step is then run to update the model parameters x. k It is formally represented by the following formula:
[0061]
[0062] Where, x k x represents the model parameters in the k-th round; k+1γ represents the model parameters in the (k+1)th round; γ represents the global gradient descent learning rate. Indicates the number of specified edge nodes.
[0063] V. The central server determines whether the model parameters have reached the preset accuracy:
[0064] In this step, after the central server calculates the model parameters for the next round in step four, it first determines whether the accuracy of the model parameters has reached the preset accuracy of the model parameters. If so, the model parameters are output as the final trained model parameters. If the preset accuracy of the model parameters is not reached, the central server will redistribute the model parameters to all designated edge nodes, and these edge nodes will execute step two again. This process is repeated until the model parameters reach the preset accuracy of the model parameters.
[0065] Although the present invention has been described herein with reference to embodiments thereof, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.
Claims
1. A model parameter update method for distributed federated learning, applied to a distributed architecture including a central server and multiple edge nodes, characterized in that, The method includes: the central server designates the edge nodes participating in the training, initializes the model parameters, loss function and training hyperparameter information, and distributes them to the designated edge nodes; The distributed architecture executes the following steps according to the iteration rounds: S1. Each designated edge node uses the locally stored training samples to calculate the gradient matrix of the model parameters in the current round through the backpropagation algorithm. S2. Each designated edge node uses a pooling operator to compress the gradient value matrix calculated in step S1, obtains the sparse gradient matrix of the current round, and uploads it to the central server. S3. The central server updates the model parameters for the current round using the distributed gradient descent algorithm based on the sparse gradient matrix of each designated edge node in the current round, and obtains the model parameters for the next round. S4. The central server determines whether the model parameters for the next round have reached the preset accuracy. If so, it outputs the model parameters for the next round as the final model parameters. Otherwise, it distributes the model parameters for the next round to each designated edge node, returns to step S1, and enters the next round of iteration. In step S2, each designated edge node uses a pooling operator to compress the gradient value matrix calculated in step S1 to obtain the sparse gradient matrix for the current round, specifically including: First, the designated edge nodes are compressed using pooling operators to compress the gradient matrix of the model parameters in the current round and the gradient residual values in the current round, resulting in the sparsed gradient matrix for the current round: in, This represents the sparsed gradient matrix of the i-th designated edge node in the k-th round; Indicates the pooling operator; This represents the gradient residual value of the i-th specified edge node in the k-th round; This represents the learning rate for local gradient descent at edge nodes; This represents the model parameters computed in the k-th round for the i-th specified edge node. The gradient value matrix; Next, update the gradient residual values for the next round: in, This represents the gradient residual value of the i-th specified edge node in the (k+1)-th round; In step S3, the central server updates the model parameters using a distributed gradient descent algorithm based on the received sparse gradient matrices of each designated edge node for the current round, thereby obtaining the model parameters for the next round. Specifically, this includes: First, the central server processes the sparsed gradient matrix of each designated edge node for the current round. Perform unpooling to obtain the recovered gradient matrix. ; Then, the model parameters from the previous round are updated using the distributed gradient descent algorithm to obtain the model parameters for the current round: in, This represents the model parameters in the k-th round; This represents the model parameters in the (k+1)th round; This represents the global gradient descent learning rate; Indicates the number of specified edge nodes.
2. The model parameter update method for distributed federated learning as described in claim 1, characterized in that, The training hyperparameter information includes: pooling compression step size, local gradient descent learning rate of edge nodes, gradient residual value of each edge node, and global gradient descent learning rate.
3. The model parameter update method for distributed federated learning as described in claim 1, characterized in that, In step S1, each designated edge node uses locally stored training samples to calculate the gradient matrix of the model parameters for the current round using the backpropagation algorithm, specifically including: The designated edge node randomly selects a training sample for the current round from the locally stored training samples. Using the selected training sample for the current round, it calculates the current loss value based on the loss function and the model parameters for the current round. Then, it uses the backpropagation algorithm to calculate the gradient matrix of the model parameters for the current round based on the current loss value.
4. A method for updating model parameters in distributed federated learning as described in any one of claims 1-3, characterized in that, In step S2, the pooling operator is either a max pooling operator or an average pooling operator.