A momentum acceleration and error feedback based federated learning ternary quantization method

Abstract

The present application belongs to the field of federated learning communication overhead optimization, and particularly relates to a federated learning ternary quantization method based on momentum acceleration and error feedback, which comprises the following steps: a central server issues model parameters, a part of clients are randomly sampled to receive the model, the clients locally update the model, the difference Δ between the updated model and the model before updating is calculated, the difference Δ and the error e are calculated by a Ternary quantization method to obtain the final compression result, the new error of this round is updated, and finally the compression result is uploaded to the central server; the server aggregates the parameters and updates the global model by a momentum acceleration algorithm, and the above steps are executed for multiple rounds to finally obtain a trained target model. The present application uses the Ternary quantization method on the federated learning baseline model, greatly reduces the overhead of transmitting the model, and combines the Adam momentum acceleration and error feedback mechanism to ensure that the model can still converge quickly under the condition of reducing the communication overhead.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering technology, specifically to a federated learning ternary quantization method based on momentum acceleration and error feedback. Background Technology

[0002] With the advent of the era of massive data, abundant data provides excellent training opportunities for models using traditional centralized machine learning methods. However, massive data originates from independent clients. As society places increasing emphasis on privacy and data security, Google proposed the federated learning framework in 2016. In this context, due to the communication and collaboration between multiple devices, the performance of federated learning is limited by communication efficiency and bandwidth. Therefore, researchers need to explore efficient communication strategies to reduce communication overhead and improve training efficiency. Existing methods for reducing communication overhead mainly fall into two categories. The first is to reduce communication overhead by decreasing the number of communications between the central service and clients, such as the federated averaging algorithm. The idea is to increase the computational load of local clients, thereby reducing the number of parameter updates submitted, ultimately reducing communication overhead. The one-shot averaging algorithm communicates only once during the entire training process. It has proven effective in some strongly convex problems, but performs poorly in some non-convex problems and is highly dependent on the client's computing power.

[0003] Another way to reduce communication overhead is to reduce the amount of data transmitted in each round of communication. Since uncompressed models generally have high dimensionality, this leads to significant overhead during transmission. The main idea behind solving this problem is to compress the model by using a small amount of encoded labeling information. Currently, there are two main methods to reduce uplink communication overhead: one is to reduce the number of parameters in the model using low-rank and random masking methods; the other is to compress the model parameters using methods such as quantization, random rotation, and subsampling. For example, in model sparsity, not all information in the gradient is important. For some model parameters, even mapping 99% of the parameter information to zero and then uploading the sparse matrix to the server can achieve good results. However, only a few model parameters meet the conditions, so this method is not objective. In terms of quantization, the main approach is to discretize continuous values ​​and replace the original 32-bit, 64-bit, or other high-precision parameters with low-precision parameters such as 8 bits, 2 bits, or even 1 bit. This is a proven method to quantize the parameters in each gradient into binary symbols, which can reduce the amount of data transmitted by 32 times. However, the compressed model will collapse during downlink, causing a large error. Ultimately, the accuracy of the model depends on the choice of quantization precision.

[0004] While the two methods mentioned above are effective in reducing the communication overhead of federated learning frameworks, neither considers the loss of model accuracy or the impact of residual information from previous gradients. The final model performance is a crucial criterion that cannot be ignored. Therefore, historical information needs to be considered. Existing research uses error feedback mechanisms to compensate for losses caused by compression. The main idea is to accumulate compressed gradient information from previous rounds into the current round, compress it, and then upload it to the central server. This involves storing the accumulated second-order momentum of the gradients on the server to improve the learning ability of the neural network. However, abnormal changes in the second-order momentum can cause fluctuations in the training process. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a federated learning ternary quantization method based on momentum acceleration and error feedback. The method includes the following steps: a central server initializes the parameters of the model to be trained; during each training round, a fixed number of clients are randomly sampled to receive the global model sent by the central server. After performing multiple steps of stochastic gradient descent locally, these clients calculate the difference Δ between the trained model and the model sent by the central server. This difference Δ, along with the error e, is used to calculate the final compression result using the ternary quantization method, while simultaneously updating the new error generated in the current training round for each client. For clients not participating in this training round, the error remains unchanged from the previous round. Finally, the clients participating in the training upload the compressed result to the central server. The server then aggregates these parameters and updates the global model using the momentum acceleration algorithm, before distributing the new global model to the clients participating in the next training round. This process is repeated multiple times to finally obtain the trained target model.

[0006] The training process includes the following steps:

[0007] S1. Obtain the original dataset and divide it into training and test sets. Preprocess the training set data: take the first 80% of the data as the training dataset, and in each epoch, the training dataset will be shuffled and distributed in fixed batches; take the last 20% of the data as the test dataset, and in each epoch, the data will be distributed sequentially in fixed batches.

[0008] S2. Initialize the neural network model to be trained, global iteration epoch, global learning rate, total number of clients participating in training num_users, proportion of clients participating in training per epoch frac, local iteration epoch local_epoch, local learning rate, and other parameters.

[0009] S3, Central Unit issues global model;

[0010] S4. Randomly sample clients of a certain proportion from all clients, and assign training and test data loaders to the selected clients by index;

[0011] S5. Each client updates its model locally using the allocated data and a stochastic gradient descent optimizer. It calculates the difference between the locally updated model and the model distributed by the server, aggregates the model difference and quantization error (initially 0), and obtains a compressed model by aggregating the aggregated results using ternary quantization. At the same time, it updates the quantization error of the current round to the difference between before and after quantization. The clients participating in the training upload the quantized model.

[0012] S6. The central server aggregates model updates from clients and accelerates global model updates through the Adam optimization algorithm, then returns to step S3.

[0013] S7. Verify the effectiveness of the model parameters trained in each epoch using the test dataset;

[0014] S8. Repeat the above steps until the global iteration count is satisfied, then end the training and obtain the final trained model.

[0015] Furthermore, the central server aggregates the differences between the model after each training round and the model before training from each participating client, then averages them to update the global model, using the following formula:

[0016]

[0017] In the standard paradigm of federated learning, the central server updates the global model by aggregating the average of the models from each participating client after each round of training, using the following formula:

[0018]

[0019] It can be proven that

[0020] Furthermore, the adaptive gradient optimization formula for Adam is:

[0021] The first-order momentum update rule is:

[0022] The second-order momentum update rule is:

[0023] Furthermore, the client-side local gradient descent formula is:

[0024]

[0025] Furthermore, the loss function formula is as follows:

[0026]

[0027] Furthermore, the formula for the ternary quantization operator is:

[0028]

[0029] Furthermore, b t Indicates that it is signed(g) t The binary mask matrix of the processed gradient has values ​​that satisfy the following distribution, as shown in the formula:

[0030]

[0031] Furthermore, the formula for the error feedback mechanism is:

[0032]

[0033] Furthermore, the formula for calculating the global model update rule is as follows:

[0034]

[0035] The beneficial effects of this invention are:

[0036] (1) This invention uses the ternary quantization method to compress model parameters on the FedAvg model, randomly quantizing the floating-point gradient to three values ​​{-1, 0, +1}. While ensuring model accuracy, this method uses only 3 sign bits, which theoretically can improve the accuracy. It achieves twice the communication efficiency and greatly reduces the amount of gradient communication.

[0037] (2) The present invention uses an error feedback mechanism on the FedAvg model to compensate for the loss of model accuracy caused by quantization.

[0038] (3) This invention uses the Adam optimization algorithm on the FedAvg model to accelerate model convergence. While maintaining a high learning rate adaptability, it significantly reduces communication overhead. The client uploads the quantized model differences to the server instead of the model itself. The server aggregates these differences and then updates the model for the next round through Adam. Attached Figure Description

[0039] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0040] Figure 1 This is a diagram of the overall architecture of the present invention;

[0041] Figure 2 This is a flowchart of the local model update process on the client side in this invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] A federated learning ternary quantization method based on momentum acceleration and error feedback, such as... Figure 1 The diagram shows the overall architecture of the method of this invention. First, the central server initializes the parameters of the model to be trained. During each training round, a fixed proportion of randomly sampled clients receive the global model sent by the central server. These clients use a stochastic gradient descent optimizer locally. After iteratively updating the local_epoch step, they calculate the difference Δ between the trained model and the model sent by the central server. This difference Δ, along with the error e (also known as historical information, initially set to 0), is used to calculate the final compression result through ternary quantization. Simultaneously, the client updates the new error generated in the current training round. For clients not participating in this training, the error remains unchanged from the previous round. The clients participating in the training upload the compression result to the central server. The server then aggregates these parameters and updates the global model using the Adam algorithm. The new global model is then sent to each client participating in the next training round. When the number of training iterations meets the epoch requirement, the trained target model is finally obtained.

[0044] Figure 2 The following is a flowchart of the client-side local model update process in this invention:

[0045] S1. Obtain the original dataset and divide it into training and test sets. Preprocess the training set data: take the first 80% of the data as the training dataset. In each epoch, the training dataset will be shuffled and distributed in fixed batches. Then take 10% of the data as the test dataset, and finally use 10% of the data as the validation set. In each epoch, the data will be distributed sequentially in fixed batches.

[0046] The input to the dataset is:

[0047] The dataset used is CIFAR-10, a widely used image classification dataset in machine learning and computer vision. CIFAR-10 contains 60,000 32*32 pixel color images, divided into 10 different categories, with 6,000 images in each category. The dataset is split into 50,000 training images and 10,000 test images. For the test set, images are randomly cropped and randomly horizontally flipped to increase data diversity and normalize the image data, making the input data distribution closer to a standard normal distribution, which is beneficial for model training.

[0048] S2. Initialize the neural network model to be trained, global iteration epoch, global learning rate, total number of clients participating in training num_users, proportion of clients participating in training per epoch frac, local iteration epoch local_epoch, local learning rate, and other parameters.

[0049] The input to the model is:

[0050] The training model is ResNet, a deep learning model architecture used to address the performance degradation problem in deep neural network training.

[0051] S3, The central server distributes the global model;

[0052] Specifically, during the first round of training, the central server distributes the determined model to the clients participating in the training. Before the global iteration epoch is met, the central server needs to aggregate the model difference updates uploaded by all clients participating in the training and calculate the global model required for the next round of training.

[0053] S4. Randomly sample clients of a certain proportion from all clients, and assign training and test data loaders to the selected clients by index;

[0054] S5. Each client updates the model locally using the allocated data and a stochastic gradient descent optimizer, as shown in the formula:

[0055]

[0056] in, Represents the gradient. Let α represent the loss function, and α represent the local learning rate for each client. Let represent the model of the i-th client in round k during round t of training. Then, calculate the difference between the locally updated model and the model delivered by the server, using the formula:

[0057] Δt =W t+1 -W t #(3)

[0058] The compressed model is obtained by aggregating the model difference and quantization error (initially 0) using ternary quantization aggregation. Simultaneously, the quantization error of the current round is updated to the difference between before and after quantization. Clients participating in training upload the quantized model. The formula is:

[0059]

[0060] C represents the pseudo-gradient, which quantizes model differences and the previous quantization error through ternary quantization. The quantization steps include:

[0061] Step 1: Calculate the sign matrix of the pseudo-gradient:

[0062] Typically, gradient information is stored using floating-point numbers, and the sign of each term in the pseudo-gradient is calculated using the sign function. Specifically:

[0063] If the current value is negative, it is represented by -1; if the value is 0, it remains unchanged; if the value is 0, it is represented by 1. Ternary is an orthogonal sparse method that quantizes pseudo-gradients into ternary orders {-1, 0, 1}.

[0064] Step 2: Calculate the scaling factor:

[0065] To minimize the number of levels, a globally shared scalar, called a scaling factor, is needed to control the change in the value of each term in the final pseudo-gradient, thus restoring the influence of the true gradient as much as possible. Therefore, a very small amplitude is varied by calculating the infinite norm of the true gradient parameters, i.e., the term with the maximum absolute value in the gradient information. The specific calculation formula is as follows:

[0066]

[0067] Among them, s t G represents the scaling factor. t This represents the true gradient.

[0068] Step 3: Calculate the binary mask matrix of the pseudo-gradient:

[0069] The parameters of the binary mask matrix are distributed as {0, 1}. For each value in the pseudo-gradient, calculate its absolute value and scaling factor s. t The ratio of to represents the probability that the value is 1, and vice versa, the probability that the value is 0.

[0070] Specifically, the formula for calculating the binary mask matrix is:

[0071]

[0072] Among them, b tk Indicates that it is signed(g) t The value of g in the t-th row and k-th column of the gradient matrix after processing. tk This represents the value in the t-th row and k-th column of the true gradient parameter matrix.

[0073] Step 4: Calculate the pseudo gradient to be uploaded:

[0074] The pseudo-gradient matrix obtained by multiplying the sign matrix and the binary mask matrix using Hadamard is represented as follows: For each value, if the sign of the true gradient is 0, the corresponding term remains 0; if the sign of the true gradient is 1, the value of the corresponding term is compared with the binary mask matrix. The larger the ratio of this ratio to the scaling factor, the greater the influence of the term, and the higher the probability of retaining it (marked as 1); otherwise, the smaller the influence of the term, the lower the probability of retaining it (marked as 0). If the sign of the true gradient is -1, the calculation method is the same as for a sign of 1, but with the opposite effect. To reproduce the influence of the true gradient as accurately as possible, the amplitude of each term needs to be adjusted using a scaling factor. The pseudo-gradient calculation formula is:

[0075]

[0076] in, Identify pseudo gradients. The symbol is the Hadama.

[0077] S6. The central server aggregates model updates from clients and accelerates global model updates using the Adam optimization algorithm, returning to step S3; the aggregation formula for global model updates is:

[0078]

[0079] Where n represents the number of clients participating in the training. This represents the model update after quantization for the i-th client in the t-th round of training. The formula for updating the global model using the Adam optimization algorithm, representing an approximate pseudo-gradient, is as follows:

[0080]

[0081] in, This represents the corrected first-order moment estimate. Let represent the corrected second-order moment estimate, ∈ , and η represent the parameters that need to be adjusted during training. The adaptive gradient optimization Adam formula is:

[0082]

[0083] S7. Validate the effectiveness of the model parameters trained in each epoch using the test dataset; the validation metrics are calculated loss and model accuracy. The formula for calculating loss is:

[0084]

[0085] Among them, LOSS avg Let n represent the average loss, and n represent the total number of batches in the test dataset. i This represents the loss of the i-th batch. i Let represent the number of samples in the i-th batch, and labels represent the total number of samples; the accuracy is expressed as follows:

[0086]

[0087] Acc represents the accuracy, C represents the number of correctly predicted samples, and labels represents the total number of samples.

[0088] S8. Repeat the above steps until the global iteration count is satisfied, then end the training and obtain the final trained model.

[0089] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A momentum acceleration and error feedback based federated learning ternary quantization method, characterized in that, The method comprises the following steps: The central server initializes the model parameters to be trained, in each training process, a fixed number of clients are randomly selected to receive the global model sent by the central server, after the clients perform multi-step random gradient descent locally, the difference Δ between the model after training and the model sent by the central server is calculated, the difference Δ and the error e are calculated by the ternary quantization method to obtain the final compression result, and the new error generated by the current training round of the client is updated, and for those clients that do not participate in this training, the error remains unchanged from the last round, finally, the clients participating in the training upload the compression result to the central server, then the server aggregates the parameters and updates the global model by the momentum acceleration algorithm, and then the new global model is sent to the clients participating in the next round of training, so as to perform multiple rounds, and finally obtain the trained target model; The training process The method comprises the following process: S1, obtaining an original data set, dividing it into a training set and a test set, and preprocessing the data in the training set: taking the first 80% of the data in the data set as the training data set, and in each epoch, the training data set is shuffled and allocated in a fixed batch; taking the last 20% of the data as the test data set, and in each epoch, the data is allocated in a fixed batch according to the order; S2, initializing the neural network model to be trained, the global iteration round epoch, the global learning rate learning rate, the number of all clients participating in training num_users, the proportion of clients participating in training frac in each round, the local iteration round local_epoch, and the local learning rate local rate parameter; S3, the central server sends the global model; S4, randomly sampling frac proportion of clients from all clients, and distributing training data and test data loader to the selected clients through index; S5, each client updates the model locally using the random gradient descent optimizer, calculates the difference between the local updated model and the model sent by the server, aggregates the model difference and the initial quantization error, obtains the compressed model after ternary quantization aggregation, updates the quantization error of the current round to the difference before and after quantization, and uploads the quantized model to the client participating in the training; S6, the central server aggregates the model updates from the clients, accelerates the update of the global model by the Adam optimization algorithm, and returns to step S3; S7, verifying the effect of the model parameters trained in each epoch by the test data set; S8, repeat the above steps, when the global iteration number is met, end the training, and obtain the final training model.

2. The momentum acceleration and error feedback based federated learning ternary quantization method according to claim 1, characterized in that, The central server aggregates the difference between the model after each training of each client participating in the training and the model before the training, and then averages it to update the global model, the formula is: where W t+1 denotes the global model of the t+1th round, denotes the difference between the i-th client training model and the global model before training, denotes the model of the i-th client after the tth round of training, n denotes the number of clients, and in the standard paradigm of federated learning, the central server updates the global model by aggregating the average of the models of each participating client after each round of training, the formula is: It can be shown that 3. The momentum acceleration and error feedback based federated learning ternary quantization method according to claim 1, characterized in that, The adaptive gradient optimization Adam formula is: The first order momentum update rule is: The second order momentum update rule is: where β1 and β2 are hyperparameters that control the rate of decay, denotes the gradient, the first order momentum m t denotes the average of the gradient, used to estimate the overall trend of the gradient; the second order momentum v t denotes the variance of the gradient, used to estimate the degree of fluctuation of the gradient.

4. The momentum acceleration and error feedback based federated learning ternary quantization method according to claim 1, characterized in that, The local gradient descent formula of the client is: where a denotes the local learning rate, 0 i+1 denotes the model after the i+1th local iteration, denotes the gradient at the ith iteration.

5. The momentum acceleration and error feedback based federated learning ternary quantization method according to claim 1, characterized in that, The loss function formula is: wherein, represents the true label of the i-th sample, represents the predicted probability distribution of the model for the i-th sample, C represents the number of sample classifications of the dataset.

6. The momentum acceleration and error feedback based federated learning ternary quantization method according to claim 3, characterized in that, The ternary quantization operator formula is: wherein, denotes the quantized gradient, s t denotes a scaling factor for adjusting the size of the quantized gradient, g t denotes the original gradient, sign(g t ) denotes the sign of g t denotes the Hadamard product, b t denotes a binary mask matrix of the gradient after being processed by sign(g t ) and taking values satisfying the following distribution, the formula being:​ where g tk represents the value of the kth item in the gradient matrix, b tk represents the value of the kth item in the binary mask matrix, ranging from {0, 1}.

7. The momentum acceleration and error feedback based federated learning ternary quantization method according to claim 4, characterized in that, The error feedback mechanism formula is: Δ t = W t+1 - W t where e t represents the difference between the last round of model quantization and before quantization, Δ t represents the difference between the client local updated model and the server delivered model, represents the model difference after ternary quantization, for the client who does not participate in this round of training, the quantization error update formula is: e t+1 = e t .

8. The momentum acceleration and error feedback based federated learning ternary quantization method according to claim 1, characterized in that, The global model update rule calculation formula is: wherein, denotes the corrected first moment estimate, denotes the corrected second moment estimate, ∈, η denote parameters to be adjusted during training.

Images