Federated learning based stationary parameter freezing and sparsification method, system, and medium

Abstract

The present application relates to the technical field of data processing, and particularly relates to a stationary parameter freezing and sparsification method and system based on federated learning and a medium, the method uses a model to perform local training on a local training set at a client, and ensures that parameters that are zeroed and frozen remain unchanged during training to obtain a trained model; the trained model is sparsified to obtain a sparsified model; and the sparsified model is sent to a server for aggregation to obtain a global model with the highest final accuracy. In this way, during the freezing process, the unfrozen parameters are not bound by the frozen parameters, and thus can find a more optimal value along the correct convergence route. For the frozen parameters, after they are unfrozen, the unfrozen parameters find a more optimal point, and the unfrozen parameters are affected by the unfrozen parameters and find a more optimal point again, so that the entire model converges in a better direction, and thus the model accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a stable parameter freezing and sparsification method and system based on federated learning and a medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, people have made great breakthroughs in image recognition, speech recognition, natural language processing and other fields. The success of artificial intelligence technology depends on a large amount of data. As people become more sensitive to private data, data collection has become an important factor restricting the development of artificial intelligence. In a federated learning system, a parameter server is responsible for maintaining a global model, aggregating model parameters, and a client uses local private data to train a model and then synchronizes model updates with the parameter server.

[0003] However, in federated learning, clients are usually resource-constrained, including device computing resources, storage resources, and communication resources. In distributed machine learning, in order to reduce model synchronization communication overhead, a large number of research works on communication compression have been proposed. Some scholars believe that during model convergence, large gradients are more important than small gradients for model changes, so a threshold is set and only gradients with absolute values greater than the threshold are transmitted during model synchronization. However, the selection of the threshold is very difficult. Researchers have proposed a Top-k method that does not require a specific threshold to be specified, but only requires a transmission rate to be set. Some scholars believe that a transmission rate s% is set in advance, and then the top s% of positive and negative values are transmitted for model synchronization. Some scholars also set a transmission rate s% in advance, and then synchronize the model according to the top s% of the absolute values of the gradients. The above sparsification methods are based on distributed machine learning environments, and when directly applied to federated learning environments, they can significantly reduce model accuracy. SUMMARY

[0004] The present application provides a stable parameter freezing and sparsification method and system based on federated learning to solve the problem of low model accuracy.

[0005] To achieve the above purpose, the present application realizes the following technical solutions:

[0006] In a first aspect, the present application provides a stable parameter freezing and sparsification method based on federated learning, comprising:

[0007] The client uses the model to train locally on the local training set, and ensures that the frozen parameters that are set to zero remain unchanged during training to obtain a trained model;

[0008] The trained model is sparsified to obtain a sparsified model;

[0009] The sparse model is sent to a server for aggregation to obtain a final global model with the highest accuracy;

[0010] The data of the to-be-identified object is input into the global model, and an identification result output by the global model is obtained.

[0011] Optionally, before the trained model is sparse, the method further comprises:

[0012] During the training process, if the current training is a checking round, the stability of the model parameters is checked, the parameters that have been zeroed and frozen are unfrozen, and the stable parameters are zeroed and frozen for a checking period, and the parameters will not change during the backward propagation of the model training during the freezing period.

[0013] Optionally, the zeroing and freezing of the stable parameters for a checking period comprises:

[0014] First, the stable parameters are set to 0, and then the operation of freezing the parameters for a checking period is performed.

[0015] Optionally, the checking of the stability of the model parameters is realized in the following manner:

[0016]

[0017] In the formula, P t represents an effective disturbance, E t represents an exponential moving average of the model gradient, represents an exponential moving average of the absolute value of the model gradient, Δ t represents the model gradient at the tth update, and a represents a smoothing factor. When the parameter fluctuations can offset each other, the effective disturbance P t is close to 0, otherwise it is close to 1. If it is found that P t is less than a threshold value τ, it is considered that the parameter is stable.

[0018] Optionally, the sparse of the trained model comprises:

[0019] In the gradient set of the model, the gradients with the top s% absolute values are selected for transmission, and for the remaining gradients that are not transmitted, the residuals are accumulated in a local residual matrix for subsequent gradient selection to sparse the trained model.

[0020] Optionally, the aggregation manner adopted by the server when aggregating is as follows:

[0021]

[0022] In the formula, w t+1,i represents the weight of the i th parameter of the aggregated model, K represents the number of clients, n represents the total number of dataset samples of all clients, n k The number of dataset samples of the client is represented.

[0023] Optionally, when the client downloads the next round of model for parameter training, the corresponding sparse mask should also be downloaded, and the client restores the complete matrix locally using the sparse mask, and replaces the parameters not contained in the sparse matrix with the parameters received in the last round.

[0024] In a second aspect, the present application provides a stationary parameter frozen and sparse system based on federated learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of the first aspect.

[0025] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the method of the first aspect.

[0026] Advantages:

[0027] The stationary parameter frozen and sparse method based on federated learning provided by the present application uses the model to perform local training on the local training set at the client, ensures that the frozen parameters that are set to zero remain unchanged during training to obtain a trained model, sparsifies the trained model to obtain a sparse model, and sends the sparse model to the server for aggregation to obtain a global model with the highest final accuracy. In this way, during the freezing process, the unfrozen parameters are not restricted by the frozen parameters, and thus can find a more optimal value along the correct convergence route. For the frozen parameters, after being unfrozen, the unfrozen parameters find a more optimal point, and the unfrozen parameters are affected by them to find a more optimal point, for example, to find a more optimal local optimal point or global optimal point, so that the entire model converges in a better direction, thereby improving the model accuracy. Moreover, by sparsifying the model, the model synchronization overhead can be reduced and the training speed can be accelerated. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 One of the flowcharts of a stationary parameter frozen and sparse method based on federated learning according to a preferred embodiment of the present application;

[0029] Figure 2 One of the flowcharts of a stationary parameter frozen and sparse method based on federated learning according to a preferred embodiment of the present application;

[0030] Figure 3A schematic diagram of a comparison of the SPF of a preferred embodiment of the present application with a baseline algorithm in a LeNet-5 model;

[0031] Figure 4 A schematic diagram of a comparison of the SPF of a preferred embodiment of the present application with a baseline algorithm in an AlexNet model;

[0032] Figure 5 A schematic diagram of a comparison of the SPF of a preferred embodiment of the present application with a baseline algorithm in an LSTM model;

[0033] Figure 6 A schematic diagram of a comparison of the SPFS of a preferred embodiment of the present application with a baseline algorithm in a LeNet-5 model;

[0034] Figure 7 A schematic diagram of a comparison of the SPFS of a preferred embodiment of the present application with a baseline algorithm in an AlexNet model;

[0035] Figure 8 A schematic diagram of a comparison of the SPF of a preferred embodiment of the present application with a baseline algorithm in an LSTM model. DETAILED DESCRIPTION

[0036] The technical solutions of the present application will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0037] Unless otherwise defined, the technical terms or scientific terms used in the present application shall have the meanings commonly understood by those skilled in the art to which the present application belongs. The terms "first", "second" and the like used in the present application do not represent any order, number or importance, but are only used to distinguish different components. Similarly, the terms "one" or "a" and the like do not represent a quantity limitation, but represent the existence of at least one. The terms "connected" or "connected" and the like do not mean a physical or mechanical connection, but can include an electrical connection, whether direct or indirect. The terms "up", "down", "left", "right" and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships are also changed accordingly.

[0038] Please refer to Figures 1-2 The present application provides a stationary parameter freezing and sparse method based on federated learning, comprising:

[0039] locally training the client using model on a local training set, ensuring that the parameters that are zeroed and frozen remain unchanged during training to obtain a trained model;

[0040] sparsifying the trained model to obtain a sparsified model;

[0041] sending the sparsified model to the server for aggregation to obtain a final global model with the highest accuracy;

[0042] inputting the obtained data of the to-be-identified object into the global model to obtain an identification result output by the global model.

[0043] In the embodiment, first, a historical data set of a to-be-identified object is divided into a test set and a training set; and the training set is evenly divided into each client. Specifically, the to-be-identified object can be image information or natural language information. In other words, the federated learning-based stable parameter freezing and sparsification method of the application can be applied to the field of image recognition or the field of natural language processing.

[0044] The federated learning-based stable parameter freezing and sparsification method described above, in the freezing process, the unfrozen parameters are not bound by the frozen parameters, and thus can find a more optimal value along the correct convergence route. For the frozen parameters, after being unfrozen, because the unfrozen parameters find a more optimal point, the unfrozen parameters will be affected by them, and find a more optimal point again, for example, find a more optimal local optimal point or global optimal point, so that the entire model converges in a better direction, thereby improving the model accuracy. Moreover, by sparsifying the model, the model synchronization overhead can be reduced, and the training speed can be accelerated.

[0045] Optionally, before the trained model is sparsified to obtain a sparsified model, the method further includes:

[0046] During the training process, if the current training round is a check round, the stability of the model parameters is checked, the previously zeroed and frozen parameters are unfrozen, the stable parameters are zeroed and frozen for a check period, and the parameters do not change during the backward propagation of the model training during the freezing period.

[0047] The stable parameters are zeroed and frozen for a check period, including:

[0048] First, the stable parameters are set to 0, and then the parameters are frozen for a check period.

[0049] It needs to be explained that the check wheel in the present application refers to that assuming the check period is set to 5, every 5 model synchronization wheels is a check wheel, that is, the stability check is needed when the model synchronization is performed for the 5th, 10th, 15th, and so on times, that is, the check wheel. In the embodiment, after the check wheel identifies the stability parameter, the stability parameter is frozen. If the stability parameter is identified after the 1st check wheel, that is, the 5th model synchronization, the freezing period of the stability parameter is between the 1st and 2nd check wheels, that is, the 6th-9th model synchronization wheels.

[0050] Optionally, the stability of the model parameter is checked in the following manner:

[0051]

[0052] In the formula, P t represents the effective disturbance, E t represents the exponential moving average of the model gradient, represents the exponential moving average of the absolute value of the model gradient, Δ t represents the model gradient at the tth update, and a represents the smoothing factor. When the parameter fluctuation can offset each other, the effective disturbance P t is close to 0, otherwise, it is close to 1. If it is found that P t is less than a threshold value τ, it is considered that the parameter is stable.

[0053] In the optional embodiment, the stability of the parameter is checked periodically. If it is found that P t is less than a threshold value τ, it is considered that the parameter is stable. First, the stable parameter is set to 0, and then the freezing operation is performed on the parameter. During the freezing period, the parameter will not change when the back propagation of the model training is performed. When the next check period is reached, the parameter that was frozen last time is unfrozen, and then the stability of the parameter is rechecked. Since the P t of the parameter that was frozen last time will not change, if it is checked, it is less than τ, therefore, the parameter that was frozen last time will not be checked in the present period.

[0054] Optionally, the trained model is sparsified, including:

[0055] In the gradient set of the model, the gradients with the top s% absolute values are selected for transmission. For the remaining gradients that are not transmitted, the residuals are accumulated in a local residual matrix, which is used for gradient selection to sparsify the trained model.

[0056] In this way, in the optional embodiment, the trained model is sparsified by setting the model transmission rate, which can reduce the model synchronization overhead, speed up the training, and reduce the transmission amount of the gradient.

[0057] Optionally, the aggregation manner adopted by the server when aggregating is as follows:

[0058]

[0059] In the formula, w t+1,i represents the weight of the i-th parameter of the aggregated model, represents the weight of the i-th parameter in the sparse model before aggregation, K represents the number of clients, n represents the total number of dataset samples of all clients, and n k represents the number of dataset samples of the client.

[0060] It is worth pointing out that the aggregation manner in the prior art is for a model without sparsification, that is, most of the values in the model are not zero, such as:

[0061] m1 = [1, 3, 4, 2, 3, 4, 2, 3, 8, 8], m2 = [5, 2, 6, 4, 3, 4, 2, 2, 7, 3];

[0062] The manner in the prior art only needs to take a weighted average value, and is not applicable to the model of the present application.

[0063] The method in the present application adopts a sparsified model, in which:

[0064] m1 = [0, 0, 0, 0, 0, 0, 0, 0, 8, 8], m2 = [0, 0, 6, 0, 0, 0, 0, 0, 7, 0].

[0065] Therefore, only the model containing values can participate in the weighted average, and m aggregation = [0, 0, 6, 0, 0, 0, 0, 0, 7.5, 8]. Therefore, in the present application, when the weighted average of a single parameter is performed, it is necessary to consider whether the current client contains the parameter, and the denominator of the weight needs to be recalculated. After the calculation of the above formula, the parameter server still obtains a sparse matrix because the parameter may not be transmitted, and when the client downloads the next round of model for parameter training, the corresponding sparse mask should also be downloaded, and the client restores the complete matrix using the sparse mask locally, and replaces the parameters not contained in the sparse matrix with the parameters received in the last round.

[0066] In summary, the principle of the present application is that after the client downloads the global model, if it reaches the check round, it first checks the stability of the model parameters, then sets the stable parameters to zero and freezes for a check period. Then local training is carried out, after the local training is completed, the model gradient is first sparsified, and then uploaded to the parameter server to participate in model aggregation. In the training process of the model, periodic checks are carried out. In the checking process, stable parameters are identified, and the stable parameters are set to zero, and then frozen for a check period. During the freezing process, the unfrozen parameters are not bound by the frozen parameters, and thus can find a better value along the correct convergence route. For the frozen parameters, after they are unfrozen, because the unfrozen parameters find a better point, the unfrozen parameters will be affected by them, and find a better point (a better local optimal point or a global optimal point), so that the entire model converges in a better direction, thereby improving the model precision.

[0067] In a complete example, the experimental setup is as follows:

[0068] Hardware settings. The experiment was carried out on a server with an Ubuntu 18.04 operating system, a memory capacity of 128 GB, and 4 GeForce RTX 2080Ti graphics cards, simulating 16 client and 1 parameter server. At the same time, the download and upload bandwidth of each client was configured as 9 Mbps and 3 Mbps respectively, and the bandwidth of the central server was set to 10 Gbps.

[0069] Dataset settings. In the experiment, CIFAR-10 and CIFAR-100 image recognition datasets were used, both containing 60,000 images, of which 50,000 were used for training and 10,000 for testing. The difference is that CIFAR-10 contains 10 categories, while CIFAR-100 contains 100 categories. In addition, the IMDB review dataset was also used for natural language processing experiments, which contains 50,000 reviews that need to be classified as positive or negative.

[0070] Model settings. LeNet-5 was selected for CIFAR-10 classification, AlexNet for CIFAR-100 classification, and LSTM network with two recursive layers for IMDB classification. In order to maintain consistency in different experiments, the batch size was set to 10, and SGD was used as the optimizer. The learning rate of LSTM was set to 0.5, while the learning rate of other convolutional neural networks was set to 0.1. The model synchronization period was set to 10, and the stability check frequency of SPF and APF was set to 5. The smoothing factor a was set to 0.9, and the effective interference threshold τ was set to 0.05.

[0071] First, SPF (without sparsity) was compared with three baseline algorithms: FedAVG, FedSGD, and APF. FedAVG and FedSGD are classic federated learning algorithms, while APF is another frozen-based algorithm by Chen et al. Their test accuracy was compared across three model settings. Since the parameter synchronization time for the four algorithms is almost identical, the horizontal axis was set to the test wheel, and the vertical axis to the test accuracy. Figure 3 As shown in ,4,5, according to Figure 3 As can be seen, comparing SPF with baseline algorithms in the LeNet-5 model, the algorithm in this application outperforms all baseline algorithms in terms of model accuracy; according to Figure 4 As can be seen, comparing SPF with baseline algorithms in the AlexNet model, the algorithm in this application outperforms all baseline algorithms in terms of model accuracy; according to Figure 5 As can be seen, when comparing SPF with baseline algorithms in LSTM models, the algorithm in this application outperforms all baseline algorithms in terms of model accuracy. The SPF algorithm of this invention improves accuracy by approximately 2% in LeNet-5, approximately 4% in AlexNet, and even reaches 5% in LSTM. This is because the freeze-to-zero method in SPF allows the model to be trained to a better convergence point, improving the model's generalization ability. However, in terms of convergence speed, the number of epochs required to reach convergence in AlexNet and LSTM is slower than other baseline methods. Therefore, sparsity techniques are incorporated to compare SPFS with other baseline methods.

[0072] Then, in the SPFS experiment, the Top-k transmission rate was set to 0.1, 0.05, and 0.01 respectively, and the same communication rounds were trained. The training results were compared with APF, FedAVG, and SPF. Since FedSGD performs synchronization once per batch, its convergence time is definitely much longer than other algorithms, so it was not chosen for comparison. In the SPFS experiment, because the transmission time of the sparsified model is significantly reduced, the horizontal axis is set to the training time. Figure 6 As shown in Figures 7 and 8, according to Figure 6 As can be seen, in the LeNet-5 model, the algorithm (SPFS) proposed in this application, compared with the baseline algorithm, shows better convergence performance than other baseline methods when the transmission rate is 0.01; according to Figure 7 As can be seen, in the comparison of the proposed algorithm (SPFS) with baseline algorithms in the AlexNet model, SPFS has a faster convergence speed than other baseline algorithms, while its model accuracy falls between SPF and other baseline algorithms; according to Figure 8As can be seen, in comparison with the baseline algorithm in the LSTM model, the algorithm of the application is superior to each baseline algorithm in model accuracy; LeNet-5 has little effect on transmission speed before and after sparsification due to its small model, and SPFS cannot play its advantage of accelerating the synchronization process. In each figure, Accuracy represents the accuracy of the current model on the test set; Test Round represents the test round Time represents the time. In the training of other models, the convergence speed of the SPFS method is much faster than that of other methods, because SPFS reduces the upload load of synchronization and accelerates the model synchronization speed. The transmission amount of parameters is directly related to the training cost. Reducing the communication flow required for the model to reach convergence can reduce the training cost of the federated learning model. Since the Top-k sparsification method needs to extract the gradients ranked in the top s% of the absolute value, it needs to consume a certain amount of computing resources, but this computing resource is negligible compared to the reduction of communication cost. For the comparison of accuracy, although sparsification is added and the amount of parameters communicated each time is greatly reduced, the accuracy of the model does not decrease significantly, and the accuracy is still higher than that of the baseline method.

[0073] In addition, the application also provides a federated learning-based stationary parameter frozen sparsification system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0074] The federated learning-based stationary parameter frozen sparsification system can implement each embodiment of the above federated learning-based stationary parameter frozen sparsification method and achieve the same beneficial effects, which will not be described here.

[0075] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the method steps as described above.

[0076] The computer readable storage medium can implement each embodiment of the above federated learning-based stationary parameter frozen sparsification method and achieve the same beneficial effects, which will not be described here.

[0077] The above detailed the preferred embodiments of the application. It should be understood that those skilled in the art can make many modifications and changes without creative labor based on the concept of the application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment based on the prior art according to the concept of the application shall be within the protection scope determined by the claims.

Claims

1. A method for parameter freezing and sparsification based on federated learning, the method comprising: Comprise: The client uses the model to perform local training on a local training set, divides a historical data set of an object to be recognized into a test set and a training set; The training set is evenly divided into each client, and the object to be recognized is image information or natural language information; during training, the parameters that are zeroed and frozen remain unchanged to obtain a trained model; during the training process, if the current training belongs to a check round, the stability of the model parameters is checked, the parameters that were previously zeroed and frozen are unfrozen, the stable parameters are zeroed and frozen for a check period, and the parameters do not change during the freezing period when the model is trained by back propagation; The trained model is sparsified to obtain a sparsified model; The sparsified model is sent to the server for aggregation to obtain a final global model with the highest accuracy; the aggregation method used by the server when aggregating is as follows: where w t+1,i denotes the weight of the i-th parameter of the aggregated model, denotes the weight of the i-th parameter in the sparse model before aggregation, K denotes the number of clients, n denotes the sum of the number of dataset samples of all clients, n k denotes the number of dataset samples of a client; The data of the object to be recognized is input into the global model, and the recognition result output by the global model is obtained; The stability of the model parameters is achieved by the following method: where P t denotes the effective perturbation, E t denotes the exponential moving average of the model gradient, denotes the exponential moving average of the absolute value of the model gradient, Δ t denotes the model gradient at the t-th update, and α denotes the smoothing factor, when the parameter fluctuations can cancel each other out, the effective perturbation P t is close to 0, otherwise it is close to 1, where if P t is found to be smaller than a threshold τ, it is considered that the parameter is stable.

2. The federated learning based stationary parameter freezing and sparsification method according to claim 1, wherein, The stable parameters are set to 0, and then frozen for a check period. The trained model is sparsified, comprising:

3. The federated learning based stationary parameter freezing and sparsification method according to claim 1, wherein, In the gradient set of the model, select the top s% of the absolute value of the gradient for transmission, and for the remaining gradients that are not transmitted, accumulate in the local residual matrix for subsequent gradient selection to sparsify the trained model. When the client downloads the next round of model for parameter training, the corresponding sparse mask should also be downloaded, and the client restores the complete matrix using the sparse mask locally, and replaces the parameters not contained in the sparse matrix with the parameters received in the last round.

4. The federated learning based stationary parameter freezing and sparsification method according to claim 1, wherein, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 4.

5. A federated learning based stationary parameter freezing sparsification system comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-4.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images