Federal learning method, device and system adopting global variance reduction strategy

Abstract

The invention provides a federal learning method, device and system adopting a global variance reduction strategy, and the method is applied to a client, and comprises the steps: receiving a global model sent by a server; initializing the local model according to the global model; inputting local training data into the initialized local model, performing model training, and determining the local model subjected to model training and a loss function of the local model; according to the loss function of the local model, the local model subjected to model training is updated based on back propagation; when the number of times of updating processing reaches a preset threshold value, determining a local model change amount of the communication round; and determining the difference quantity of the local model change quantity according to the local model change quantity of the communication round. Through the method and the device, the difference quantity of the change quantity of the local model is calculated at the client, the data privacy of the client is protected, the model variance caused by the client is reduced, and the convergence rate and the generalization performance of the global model are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a federated learning method, device and system using a global variance reduction strategy. Background Art

[0002] With the development of the Internet of Things, the number of smart devices around the world has grown rapidly in the past few years, many of which are equipped with various sensors and increasingly powerful hardware, enabling them to collect and process data on an unprecedented scale. At the same time, deep learning has made breakthrough progress in fields such as computer vision, natural language processing, and speech recognition, a large part of which is due to the increasing abundance of data resources. Therefore, there is great potential in using the rich data provided by IoT devices to train deep learning models.

[0003] However, at the same time, data privacy has become an increasing concern for many users, and centralized data processing poses a high threat to user privacy. Based on a literature search of existing technologies, McMahan et al. proposed a method in the paper "Communication-efficient learning of deep networks from decentralized data" published in 2016, which uses multiple users to jointly train a deep learning model without any user uploading their data to the server, that is, federated learning solves the privacy protection problem. Specifically, first, due to unstable communication, only some clients download the latest global model each time; second, the client uses optimization algorithms such as stochastic gradient descent to train the downloaded model based on its local private data; finally, all clients participating in the training upload the model changes generated by their training to the server, and the server aggregates the model changes and linearly adds the aggregated results to the original global model to generate a new global model. These steps are repeated until a certain convergence criterion is met. Note that during the training process, the training data never leaves the local client because only the communication model or model changes are uploaded to the server. However, due to data heterogeneity, the model variance between clients is large, resulting in unstable convergence; in addition, due to unstable communication, clients can only partially participate, further increasing the model variance between clients participating in training and those not participating in training.

[0004] In machine learning, there are many methods to reduce random variance, such as the SAG algorithm and the SAGA algorithm, which are used to reduce the variance caused by sampling in the stochastic gradient descent algorithm. Specifically, the stochastic gradient descent algorithm only uses a part of the samples each time to calculate the gradient. Although the amount of calculation is reduced, the sampling variance caused by it leads to a decrease in the convergence rate. Roux et al. published "A stochastic gradient method with an exponential convergence rate for finite training sets" at the 2012 "Neural Information Processing Systems" conference, which proposed a method that can reduce variance without increasing the computational overhead, namely SAG. The SAG algorithm maintains an old gradient for each sample in memory. In the specific update process, a part of the samples are still randomly selected to calculate the gradient. The updated gradient replaces the old gradient of the corresponding sample. Finally, the gradient of all samples is used iteratively, including the old gradient and the latest gradient. Theoretical analysis shows that the SAG algorithm can accelerate convergence. Defazio et al. proposed a more efficient variance reduction technique, the SAGA algorithm, in their paper "SAGA: AFast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives" published at the 2014 Neural Information Processing Systems conference. The SAGA algorithm uses the same principle, but modifies the update rule so that the gradient update becomes unbiased in a statistical sense, further accelerating the convergence rate. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a federated learning method, device and system using a global variance reduction strategy.

[0006] To achieve the above objective, according to one aspect of the present disclosure, a federated learning method using a global variance reduction strategy is provided, which is applied to a client and includes:

[0007] Receiving a global model sent by a server, wherein the global model includes an initialized global model or an updated global model;

[0008] Initialize the local model according to the global model, and determine the initialized local model;

[0009] Inputting local training data into the initialized local model, performing model training on the initialized local model, and determining a loss function of the model-trained local model and the local model, wherein the loss function of the local model includes a cross entropy loss between a training label and a true label of the local training data;

[0010] According to the loss function of the local model, updating the local model after model training based on back propagation to determine an updated local model;

[0011] When the number of update processes reaches a preset threshold, determining the amount of change of the local model in the current communication round according to the last updated local model and the initialized local model;

[0012] The difference in the local model change amount is determined based on the local model change amount in the current communication round and the local model change amount participating in the model training in the previous communication round.

[0013] Optionally, the method further comprises:

[0014] The difference amount of the local model change amount is sent to the server.

[0015] Optionally, when the number of update processes reaches a preset threshold, determining the amount of change of the local model in the current communication round according to the last updated local model and the initialized local model includes:

[0016] When the number of update processing times reaches a preset threshold, the last updated local model is subtracted from the initialized local model to determine the change amount of the local model in the current communication round.

[0017] Optionally, determining the difference in the amount of change of the local model according to the amount of change of the local model in the current communication round and the amount of change of the local model participating in the model training in the previous communication round includes:

[0018] The difference in the local model change amount in the current communication round is determined by subtracting the local model change amount in the previous communication round from the local model change amount that participated in the model training.

[0019] According to a second aspect of the present disclosure, a federated learning method using a global variance reduction strategy is provided, which is applied to a server side and includes:

[0020] Initialize the global model and determine the initialized global model;

[0021] Sending the initialized global model to the client;

[0022] Receiving the difference of the local model change amount sent by the client;

[0023] Aggregate the difference in the amount of change of the local model of each client to determine an aggregate amount;

[0024] Linearly adding the aggregate amount to a global control variable to determine an updated global control variable, wherein the global control variable represents the mean value of the local model change amounts of all clients;

[0025] The global model is updated according to the updated global control variables to determine an updated global model.

[0026] Optionally, updating the global model according to the updated global control variable to determine the updated global model includes:

[0027] linearly adding the global model to the updated global control variable to determine the updated global model; or,

[0028] The global model, the aggregate amount, and the updated global control variable are linearly added to determine the updated global model.

[0029] Optionally, the method further comprises:

[0030] The updated global model is sent to the client, and the client executes the step of receiving the global model sent by the server.

[0031] According to a third aspect of the present disclosure, a federated learning device adopting a global variance reduction strategy is provided, which is applied to a client and includes:

[0032] A client receiving module, used for receiving a global model sent by a server, wherein the global model includes an initialized global model or an updated global model;

[0033] A client initialization module, used to initialize the local model according to the global model and determine the initialized local model;

[0034] A client model training module, used for inputting local training data into the initialized local model, performing model training on the initialized local model, and determining the model-trained local model and a loss function of the local model, wherein the loss function of the local model includes a cross entropy loss between a training label and a true label of the local training data;

[0035] A client local model updating module, used to update the local model after model training based on back propagation according to the loss function of the local model, and determine an updated local model;

[0036] The client local model change amount calculation module is used to determine the local model change amount of this communication round according to the last updated local model and the initialized local model when the number of update processing reaches a preset threshold;

[0037] The client local model change amount difference module is used to determine the difference in the local model change amount based on the local model change amount of the current communication round and the local model change amount participating in the model training in the previous communication round.

[0038] According to a fourth aspect of the present disclosure, a federated learning device using a global variance reduction strategy is provided, which is applied to a server side and includes:

[0039] The server-side initialization module is used to initialize the global model and determine the initialized global model.

[0040] A server-side sending module, used for sending the initialized global model to the client;

[0041] A server-side receiving module, used for receiving the difference amount of the local model change amount sent by the client;

[0042] A server-side global aggregation module, used for aggregating the difference in the amount of change of the local model of each client to determine an aggregate amount;

[0043] A server-side global control variable updating module, used for linearly adding the aggregate amount to the global control variable to determine an updated global control variable, wherein the global control variable includes a mean value of the local model change amounts of all clients;

[0044] The server-side global model updating module is used to update the global model according to the updated global control variables to determine the updated global model.

[0045] According to a fifth aspect of the present disclosure, a federated learning system adopting a global variance reduction strategy is provided, comprising:

[0046] The local model update module is used to update the local model based on back propagation on the client side;

[0047] A model change amount calculation module, configured to, when the number of update processing of the local model of the client reaches a preset threshold, determine the local model change amount of the current communication round by subtracting the last updated local model from the initialized local model;

[0048] A model change difference module is used to, at the client, determine the difference in the local model change amount by subtracting the local model change amount of the current communication round from the local model change amount of the previous communication round participating in the model training;

[0049] A global aggregation module, used for aggregating the difference of the local model change amount uploaded by the client on the server side, updating the global control variable, and determining the updated global control variable;

[0050] A global model updating module, configured to update the global model based on the updated global control variables at the server end and determine an updated global model;

[0051] A communication module is used to transmit the difference of the local model change amount of the client to the server, and is also used to transmit the initialized global model of the server to the client, or to transmit the updated global model of the server to the client.

[0052] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0053] Through the technical solution, when the number of updates of the local model of the client reaches a preset threshold, the local model change of this communication round is calculated, and the difference in the local model change is calculated based on the local model change and the local model change of the previous communication round. The model variance between clients caused by random participation in training and non-independent and identically distributed data is reduced, thereby improving the generalization performance of the global model and protecting the client's data privacy without introducing additional communication and computing overhead.

[0054] The embodiments of the present disclosure introduce global variance reduction technology on the server side, adopt the SAG variance reduction method or the SAGA variance reduction method to update the global model, reduce the model variance between clients, improve the convergence rate and generalization performance of the global model, and can adapt to various distributed application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0056] Figure 1 The present invention is a flowchart of a federated learning method using a global variance reduction strategy and applied to a client according to an exemplary embodiment.

[0057] Figure 2 The present invention is a flowchart of a federated learning method using a global variance reduction strategy and applied to a server according to an exemplary embodiment.

[0058] Figure 3 The present invention is a schematic diagram of signaling interaction between a client and a server according to an exemplary embodiment.

[0059] Figure 4 It is a block diagram of a federated learning device using a global variance reduction strategy and applied to a client according to an exemplary embodiment.

[0060] Figure 5 It is a block diagram of a federated learning device using a global variance reduction strategy and applied to a server side according to an exemplary embodiment.

[0061] Figure 6 The diagram is a schematic diagram of the architecture of a federated learning system using a global variance reduction strategy according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0063] Figure 1 The present invention is a flowchart of a federated learning method using a global variance reduction strategy and applied to a client according to an exemplary embodiment.

[0064] like Figure 1 As shown, the present disclosure provides a federated learning method using a global variance reduction strategy, which is applied to a client, and the client includes a local model, including S11 to S16.

[0065] S11, receiving the global model sent by the server.

[0066] The global model includes an initialized global model or an updated global model.

[0067] S12, initializing the local model according to the global model, and determining the initialized local model.

[0068] S13, inputting the local training data into the initialized local model, performing model training on the initialized local model, and determining the trained local model and the loss function of the local model.

[0069] Among them, the loss function of the local model includes the cross entropy loss between the training labels and the true labels of the local training data.

[0070] S14, according to the loss function of the local model, the local model that has been trained is updated based on back propagation to determine an updated local model.

[0071] S15, when the number of update processes reaches a preset threshold, determining the amount of change of the local model in this communication round according to the last updated local model and the initialized local model.

[0072] S16, determining the difference in the local model change amount according to the local model change amount in the current communication round and the local model change amount participating in the model training in the previous communication round.

[0073] Through the technical solution, when the number of updates of the local model of the client reaches a preset threshold, the local model change of this communication round is calculated, and the difference in the local model change is calculated based on the local model change and the local model change of the previous communication round. The model variance between clients caused by random participation in training and non-independent and identically distributed data is reduced, thereby improving the generalization performance of the global model and protecting the client's data privacy without introducing additional communication and computing overhead.

[0074] In a possible embodiment, S15, when the number of update processes reaches a preset threshold, determining the amount of change of the local model in the current communication round according to the last updated local model and the initialized local model, may include:

[0075] When the number of update processing reaches a preset threshold, the last updated local model is subtracted from the initialized local model to determine the change amount of the local model in this communication round.

[0076] The following formula is used to determine the amount of change in the local model for this communication round:

[0077]

[0078] in, represents the local model updated by node i for the last time (i.e., K times) in t communication rounds, represents the local model of node i initialized in t communication rounds, Represents the change in the local model of node i in t communication rounds.

[0079] In a possible embodiment, S16, determining the difference in the amount of change of the local model according to the amount of change of the local model in the current communication round and the amount of change of the local model participating in the model training in the previous communication round, may include:

[0080] The difference in the local model change amount in this communication round is determined by subtracting the local model change amount in the previous communication round from the local model change amount that participated in the model training.

[0081] The difference in the amount of change in the local model is determined using the following formula:

[0082]

[0083] in, represents the difference in the amount of change in the local model of node i in t communication rounds, represents the change in the local model of node i in t communication rounds, represents the change in the local model of node i in the t-τ communication round.

[0084] Since node i cannot participate in training in every communication round, the t-τ communication round represents the previous communication round closest to the t communication round.

[0085] In a possible embodiment, a federated learning method using a global variance reduction strategy is applied to a client and may also include S17.

[0086] S17, sending the difference of the local model change amount to the server.

[0087] Among them, only the difference in the amount of local model change is sent to the server, and there is no need to upload the original data of each client to the server. This can protect the data privacy of each client and reduce the model variance between clients caused by random participation in training and non-independent and identically distributed data.

[0088] Figure 2 The present invention is a flowchart of a federated learning method using a global variance reduction strategy and applied to a server according to an exemplary embodiment.

[0089] like Figure 2 As shown, the present disclosure also provides a federated learning method using a global variance reduction strategy, which is applied to the server side and includes S21 to S26.

[0090] S21, initializing the global model and determining the initialized global model.

[0091] S22, sending the initialized global model to the client.

[0092] Before training the local model on the client, the global model on the server needs to be initialized first, and the initialized global model is sent to the clients participating in the training in this communication round.

[0093] S23, receiving the difference of the local model change amount sent by the client.

[0094] The server receives the difference in the amount of change of the local model sent by each client participating in the training in this communication round.

[0095] S24, aggregating the difference in the amount of change of the local model of each client to determine an aggregate amount.

[0096] The aggregation processing method may adopt a weighted summation method.

[0097] S25, linearly adding the aggregate amount to the global control variable to determine an updated global control variable.

[0098] The global control variable represents the mean value of the local model change of all clients, including the clients that participated in the training in this communication round and the clients that did not participate in the training.

[0099] The global control variable when updating the global variable is the old global control variable in the current communication round.

[0100] S26, updating the global model according to the updated global control variables to determine an updated global model.

[0101] The updating rules followed by updating the global model according to the updated global control variables may include the SAG variance reduction method and the SAGA variance reduction method.

[0102] The SAG variance reduction method and the SAGA variance reduction method are both global variance reduction techniques.

[0103] Through the above technical solution, the global model is updated on the server side according to the updated global control variables, the global variance reduction technology is introduced, and the SAG variance reduction method or the SAGA variance reduction method is used to update the global model, thereby reducing the model variance between clients, improving the convergence rate and generalization performance of the global model, and being able to adapt to various distributed application scenarios.

[0104] In a possible embodiment, S24, updating the global model according to the updated global control variables to determine the updated global model, may include S241 or S242.

[0105] S241, linearly add the global model and the updated global control variables to determine the updated global model.

[0106] Among them, the SAG variance reduction method adopts the following formula:

[0107]

[0108] Among them, x t-1 represents the global model, xt represents the updated global model, h t represents the updated global control variable, m represents the number of all clients, and η represents the global learning rate.

[0109] or,

[0110] S242, linearly adding the global model, the aggregation amount, and the updated global control variable to determine the updated global model.

[0111] Among them, the SAGA variance reduction method adopts the following formula:

[0112]

[0113] Among them, x t-1 represents the global model, x t represents the updated global model, η represents the global learning rate, p represents the number of clients participating in the training, m represents the number of all clients, α represents the introduced hyperparameter,

[0114]

[0115] Indicates the amount of polymerization.

[0116] Step S241 indicates that the global model is updated using an update rule including the SAG variance reduction method, and step S242 indicates that the global model is updated using an update rule including the SAGA variance reduction method.

[0117] In a possible embodiment, a federated learning method using a global variance reduction strategy is applied to a server side and may also include S27.

[0118] S27, sending the updated global model to the client.

[0119] After the server executes step S27, the client executes the step of receiving the global model sent by the server, that is, the client executes step S11.

[0120] Figure 3 The present invention is a schematic diagram of signaling interaction between a client and a server according to an exemplary embodiment.

[0121] like Figure 3 As shown, the present disclosure also provides a signaling interaction method between a client and a server, which includes S101 to S114.

[0122] S101, the server initializes the global model and determines the initialized global model.

[0123] S102, the server sends the initialized global model to the client.

[0124] S103, the client receives the global model sent by the server.

[0125] S104: The client initializes the local model according to the global model and determines the initialized local model.

[0126] S105, the client inputs the local training data into the initialized local model, performs model training on the initialized local model, and determines the trained local model and the loss function of the local model, where the loss function of the local model includes the cross entropy loss between the training labels and the true labels of the local training data.

[0127] S106, the client updates the trained local model based on back propagation according to the loss function of the local model to determine an updated local model.

[0128] S107, when the number of update processings of the client reaches a preset threshold, the client determines the amount of change of the local model in this communication round according to the last updated local model and the initialized local model.

[0129] S108, the client determines the difference in the local model change amount according to the local model change amount in the current communication round and the local model change amount participating in the model training in the previous communication round.

[0130] S109, the client sends the difference of the local model change amount to the server.

[0131] S110: The server receives the difference of the local model change amount sent by the client.

[0132] S111, the server aggregates the difference of the local model change amount of each client to determine the aggregate amount.

[0133] S112, the server side linearly adds the aggregate amount to the global control variable to determine an updated global control variable, where the global control variable represents the average of the local model changes of all clients.

[0134] S113, the server updates the global model according to the updated global control variables to determine an updated global model.

[0135] S114, the server sends the updated global model to the client.

[0136] After step S101 to step S104 are executed, step S105 to step S106 are executed in a loop until the number of update processes of the client reaches a preset threshold, and then step S107 is executed.

[0137] After step S114 is executed, the process returns to step S103, and the client executes step S103 to start the next communication round of training for the local client until the global model converges and stops model training.

[0138] The technical solution provided by the above embodiment of the present disclosure is further described in detail below in conjunction with a specific application example.

[0139] The present disclosure provides a federated learning method using a global variance reduction strategy, which is applied in autonomous driving scenarios.

[0140] The in-vehicle system builds a local data set based on the driver's operation data, such as navigation destination, driving instructions, and vehicle sensor data, such as throttle, brake, and steering angle, and trains a model for predicting the next operation or instruction without leaking user privacy.

[0141] Based on a trusted third-party organization, which acts as the server and each vehicle-mounted system as the client, a communication network, i.e., a communication module, is built between the server and each client. There is no need for direct communication between user devices, and the original user input data set is not transmitted between the third-party organization and the vehicle-mounted system.

[0142] Based on a federated learning method using a global variance reduction strategy provided by the present disclosure, a deep learning model with good generalization performance is quickly trained as a model for predicting the next operation or instruction. The specific steps are as follows:

[0143] Step 1: The server initializes the global model, randomly selects a client vehicle to participate in the training, and sends the initialized global model to the randomly selected client vehicle to participate in the training.

[0144] Step 2: The client vehicle participating in the training initializes the local deep learning model according to the received initialized global model, and determines the initialized local deep learning model.

[0145] Step 3: Train the initialized local deep learning model based on the vehicle local data set, input historical driving behavior and operation data information into the vehicle system, train the model for predicting the driver's next operation or instruction prediction, and determine the loss function, which is the cross entropy loss between the input data and the target behavior: Based on the loss function of the deep learning model, the client updates the local deep learning model through the back propagation algorithm.

[0146] Step 4: Repeat step 3 K times (K ≥ 1), calculate and save the local model change of the client vehicle in this communication round, that is, use the last updated deep learning model and the initialized local deep learning model to make a difference, and determine the local model change of the client vehicle in this communication round. The specific formula is as follows:

[0147]

[0148] in, represents the local deep learning model updated by node i for the last time (i.e., K times) in t communication rounds, represents the initial local deep learning model of node i in t communication round, Represents the change in the local model of node i in t communication rounds.

[0149] Each node i represents each client vehicle.

[0150] Step 5: Calculate the difference in the local model change of the client vehicle, that is, subtract the local model change of the client vehicle in this communication round from the local model change generated by the client vehicle's most recent participation in training, determine the difference in the local model change of the client vehicle, and send it to the server. The specific formula is as follows:

[0151]

[0152] in, represents the difference in the amount of change in the local model of node i in t communication rounds, represents the change in the local model of node i in t communication rounds, represents the change in the local model of node i in the t-τ communication round.

[0153] Since node i cannot participate in training in every communication round, the t-τ communication round represents the previous communication round closest to the t communication round.

[0154] Step 6: The server receives the difference in the local model change of the client vehicle, and aggregates the difference in the local model change of different client vehicles by weighted summation, determines the aggregate amount, and linearly adds the aggregate amount and the global control variable to update the global control variable.

[0155] Step 7: Since the vehicles involved in the training are not fixed, the server updates the global model according to the global variance reduction technology SAG variance reduction method or SAGA variance reduction method. Specifically:

[0156] SAG variance reduction method: Linearly add the updated global control variables to the old global model to obtain an updated global model:

[0157]

[0158] Among them, x t-1 represents the old global model, x t represents the updated global model, h t represents the updated global control variable, m represents the number of all clients, and η represents the global learning rate.

[0159] The SAGA variance reduction method linearly adds the old global model, the aggregate, and the updated global control variables to obtain the updated global model:

[0160]

[0161] Among them, x t-1 represents the old global model, x t represents the updated global model, η represents the global learning rate, p represents the number of clients participating in the training, m represents the number of all clients, α represents the introduced hyperparameter,

[0162]

[0163] Indicates the amount of polymerization.

[0164] Step 8: After the server updates the global model, it sends the updated global model to the client.

[0165] The client vehicle repeats steps 2 to 8 above based on the new global model, and continues model training for the next communication round based on the updated global model and the vehicle local dataset until the global model converges.

[0166] In order to solve the problems of data heterogeneity and insufficient training data between different vehicles, a federated learning method is adopted. The server side cannot directly obtain the original data of the vehicle. The federated learning method using the global variance reduction strategy provided by the present invention is adopted to realize the effective update and optimization of the global model.

[0167] By adopting a federated learning method with a global variance reduction strategy, the vehicle's autonomous driving system can achieve cross-vehicle system learning while protecting the driver's privacy, improve the accuracy of the prediction of the next operation or command, and at the same time, effectively reduce the variance of deep learning model updates between different vehicles, achieving faster convergence of the global model and stronger generalization ability.

[0168] In a possible embodiment, the image datasets EMNIST, CIFRA-10, CIFAR-100 and the natural language data Shakespeare dataset are used to verify the global model trained by the federated learning method using the global variance reduction strategy provided by the present disclosure, and the methods compared with the method FedVR provided by the present disclosure include the method FedAvg in “Communication-efficient learning of deep networks from decentralized data” by McMahan et al., the method FedDyn in “Federated Learning Based on Dynamic Regularization” by Acer et al., the method SCAFFOLD in “scaffold: stochastic controlled averaging for federated learning” by Karimierddy et al., and the method FedAvgM in “Measuring the effects of non-identical data distribution for federated visual classification” by Hsu et al.

[0169] Table 1 is a schematic diagram showing the statistical results of the classification accuracy of the federated learning method using the global variance reduction strategy proposed in the present disclosure on CIFAR-10, CIFAR-100, EMNIST and data sets for different global learning rates and different global model update methods.

[0170]

[0171] Table 1

[0172] Among them, method 1 represents a method of updating the global model using the SAG variance reduction method, and method 2 represents a method of updating the global model using the SAGA variance reduction method.

[0173] It can be seen from Table 1 that, under the same global learning rate, the classification accuracy achieved by method 2 is higher than that achieved by method 1, indicating that the generalization of updating the global model using the SAGA variance reduction method is higher; when the global model is updated in the same way and different global learning rates are used, the classification accuracy produced by increasing the global learning rate is lower, which shows that increasing the global learning rate cannot improve the performance gain.

[0174] Table 2 shows the statistical results of the classification accuracy of the global model of the methods FedVR proposed in the present invention, FedDyn, SCAFFOLD, FedAvgM and FedAvg methods on the datasets ClFAR-10, ClFAR-100, EMNIST and Shakespeare at different client participation rates.

[0175]

[0176]

[0177] Table 2

[0178] As can be seen from Tables 1 and 2 above, it can be seen from the classification accuracy that the federated learning method FedVR proposed in the present disclosure using a global variance reduction strategy can achieve optimal performance in generalization in most data sets.

[0179] Figure 4 It is a block diagram of a federated learning device using a global variance reduction strategy and applied to a client according to an exemplary embodiment.

[0180] Based on the same concept, Figure 4 As shown, the present disclosure also provides a federated learning device 100 that adopts a global variance reduction strategy, which is applied to a client, and includes: a client receiving module 110, a client initialization module 120, a client model training module 130, a client local model updating module 140, a client local model change calculation module 150, and a client local model change difference module 160.

[0181] The client receiving module 110 is used to receive the global model sent by the server, where the global model includes an initialized global model or an updated global model;

[0182] The client initialization module 120 is used to initialize the local model according to the global model and determine the initialized local model;

[0183] The client model training module 130 is used to input local training data into the initialized local model, perform model training on the initialized local model, and determine the model-trained local model and the loss function of the local model, wherein the loss function of the local model includes a cross entropy loss between a training label and a true label of the local training data;

[0184] The client local model updating module 140 is used to update the local model after model training based on back propagation according to the loss function of the local model to determine an updated local model;

[0185] The client local model change amount calculation module 150 is used to determine the local model change amount of the current communication round according to the last updated local model and the initialized local model when the number of update processes reaches a preset threshold;

[0186] The client local model change amount difference module 160 is used to determine the difference in the local model change amount according to the local model change amount of the current communication round and the local model change amount participating in the model training in the previous communication round.

[0187] Through the technical solution, when the number of updates of the local model of the client reaches a preset threshold, the local model change of this communication round is calculated, and the difference in the local model change is calculated based on the local model change and the local model change of the previous communication round. The model variance between clients caused by random participation in training and non-independent and identically distributed data is reduced, thereby improving the generalization performance of the global model and protecting the client's data privacy without introducing additional communication and computing overhead.

[0188] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0189] Figure 5 It is a block diagram of a federated learning device using a global variance reduction strategy and applied to a server side according to an exemplary embodiment.

[0190] Based on the same concept, Figure 5 As shown, the present disclosure also provides a federated learning device 200 adopting a global variance reduction strategy, which is applied to the server side and includes: a server side receiving module 210, a server global aggregation module 220, a server global control variable updating module 230, and a server side global model updating module 240.

[0191] The server-side initialization module 210 is used to initialize the global model and determine the initialized global model;

[0192] The server-side sending module 220 is used to send the initialized global model to the client.

[0193] The server-side receiving module 230 is used to receive the difference of the local model change amount sent by the client;

[0194] The server-side global aggregation module 240 is used to aggregate the difference of the local model change amount of each client to determine the aggregation amount;

[0195] A server-side global control variable updating module 250, configured to linearly add the aggregate amount to the global control variable to determine an updated global control variable, wherein the global control variable includes a mean value of the local model change amounts of all clients;

[0196] The server-side global model updating module 260 is used to update the global model according to the updated global control variables to determine an updated global model.

[0197] Through the above technical solution, the global variance reduction technology is introduced on the server side, and the SAG variance reduction method or SAGA variance reduction method is used to update the global model, reduce the model variance between clients, improve the convergence rate and generalization performance of the global model, and adapt to various distributed application scenarios.

[0198] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0199] Figure 6 The diagram is a schematic diagram of the architecture of a federated learning system using a global variance reduction strategy according to an exemplary embodiment.

[0200] Based on the same concept, Figure 6 As shown, the present disclosure also provides a federated learning system 300 that adopts a global variance reduction strategy, including: a local model update module 310, a model change calculation module 320, a model change difference module 330, a global aggregation module 340, a global model update module 350, and a communication module 360.

[0201] A local model updating module 310 is used to update the local model based on back propagation at the client;

[0202] The model change amount calculation module 320 is used to determine the local model change amount of the current communication round by subtracting the last updated local model from the initialized local model when the number of update processing of the local model of the client reaches a preset threshold;

[0203] The model change amount difference module 330 is used to, at the client, determine the difference amount of the local model change amount by subtracting the local model change amount of the current communication round from the local model change amount of the previous communication round participating in the model training;

[0204] A global aggregation module 340 is used to aggregate the difference of the local model change amount uploaded by the client on the server side, update the global control variable, and determine the updated global control variable;

[0205] A global model updating module 350, configured to update the global model based on the updated global control variables at the server end and determine an updated global model;

[0206] The communication module 360 ​​is used to transmit the difference of the local model change amount of the client to the server, and is also used to transmit the initialized global model of the server to the client, or to transmit the updated global model of the server to the client.

[0207] like Figure 6 As shown, as an example, nodes 1 to m represent clients, and the local model updating module 310 performs model training and updating processing on the local model based on the private data of each node; the local model change amount calculation module 320 calculates the local model change amount of each node; the model change amount difference module 330 calculates the difference amount of the local model change amount of each node; nodes 1 to m transmit the difference amount of the local model change amount of each node participating in the training to the server side through the communication module 360; in the global aggregation module 340, the difference amount of the local model change amount of each node participating in the training received by the aggregation server is aggregated, and the global control variables are updated; in the global model updating module 350, a global variance reduction technology, such as the SAG variance reduction method or the SAGA variance reduction method, is used to update the global model based on the updated global control variables, and the updated global model is transmitted to the client through the communication module 360.

[0208] The communication module 360 ​​is also used to transmit the initialized model on the server side to the client side.

[0209] The above describes the specific embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which does not affect the essence of the present disclosure. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A federated learning method using a global variance reduction strategy, characterized in that: Applied to the client, including: Receiving a global model sent by a server, wherein the global model includes an initialized global model or an updated global model; Initialize the local model according to the global model, and determine the initialized local model; Inputting local training data into the initialized local model, performing model training on the initialized local model, and determining a loss function of the model-trained local model and the local model, wherein the loss function of the local model includes a cross entropy loss between a training label and a true label of the local training data; According to the loss function of the local model, updating the local model after model training based on back propagation to determine an updated local model; When the number of update processes reaches a preset threshold, determining the amount of change of the local model in the current communication round according to the last updated local model and the initialized local model; The difference in the local model change amount is determined based on the local model change amount in the current communication round and the local model change amount participating in the model training in the previous communication round.

2. The method according to claim 1, characterized in that: The method further comprises: The difference amount of the local model change amount is sent to the server.

3. The method according to claim 1, characterized in that When the number of update processing reaches a preset threshold, determining the amount of change of the local model in the current communication round according to the last updated local model and the initialized local model, including: When the number of update processing times reaches a preset threshold, the last updated local model is subtracted from the initialized local model to determine the change amount of the local model in the current communication round.

4. The method according to claim 1, characterized in that The determining the difference in the amount of change of the local model according to the amount of change of the local model in the current communication round and the amount of change of the local model participating in the model training in the previous communication round includes: The difference in the local model change amount in the current communication round is determined by subtracting the local model change amount in the previous communication round from the local model change amount that participated in the model training.

5. A federated learning method using a global variance reduction strategy, characterized in that: Applied to the server side, including: Initialize the global model and determine the initialized global model; Sending the initialized global model to the client; Receiving the difference of the local model change amount sent by the client; Aggregate the difference in the amount of change of the local model of each client to determine an aggregate amount; Linearly adding the aggregate amount to a global control variable to determine an updated global control variable, wherein the global control variable represents the mean value of the local model change amounts of all clients; The global model is updated according to the updated global control variables to determine an updated global model.

6. The method according to claim 5, characterized in that The updating process of the global model according to the updated global control variable to determine the updated global model includes: linearly adding the global model to the updated global control variable to determine the updated global model; or, The global model, the aggregate amount, and the updated global control variable are linearly added to determine the updated global model.

7. The method according to claim 5, characterized in that The method further comprises: The updated global model is sent to the client, and the client executes the step of receiving the global model sent by the server.

8. A federated learning device using a global variance reduction strategy, characterized in that: Applied to the client, including: A client receiving module, used for receiving a global model sent by a server, wherein the global model includes an initialized global model or an updated global model; A client initialization module, used to initialize the local model according to the global model and determine the initialized local model; A client model training module, used for inputting local training data into the initialized local model, performing model training on the initialized local model, and determining the model-trained local model and a loss function of the local model, wherein the loss function of the local model includes a cross entropy loss between a training label and a true label of the local training data; A client local model updating module, used to update the local model after model training based on back propagation according to the loss function of the local model, and determine an updated local model; The client local model change amount calculation module is used to determine the local model change amount of this communication round according to the last updated local model and the initialized local model when the number of update processing reaches a preset threshold; The client local model change amount difference module is used to determine the difference in the local model change amount based on the local model change amount of the current communication round and the local model change amount participating in the model training in the previous communication round.

9. A federated learning device using a global variance reduction strategy, characterized in that: Applied to the server side, including: The server-side initialization module is used to initialize the global model and determine the initialized global model. A server-side sending module, used for sending the initialized global model to the client; A server-side receiving module, used for receiving the difference amount of the local model change amount sent by the client; A server-side global aggregation module, used for aggregating the difference in the amount of change of the local model of each client to determine an aggregate amount; A server-side global control variable updating module, used for linearly adding the aggregate amount to the global control variable to determine an updated global control variable, wherein the global control variable includes a mean value of the local model change amounts of all clients; The server-side global model updating module is used to update the global model according to the updated global control variables to determine the updated global model.

10. A federated learning system using a global variance reduction strategy, characterized in that: include: The local model update module is used to update the local model based on back propagation on the client side; A model change amount calculation module, configured to, when the number of update processing of the local model of the client reaches a preset threshold, determine the local model change amount of the current communication round by subtracting the last updated local model from the initialized local model; A model change difference module is used to, at the client, determine the difference in the local model change amount by subtracting the local model change amount of the current communication round from the local model change amount of the previous communication round participating in the model training; A global aggregation module, used for aggregating the difference of the local model change amount uploaded by the client on the server side, updating the global control variable, and determining the updated global control variable; A global model updating module, configured to update the global model based on the updated global control variables at the server end and determine an updated global model; A communication module is used to transmit the difference of the local model change amount of the client to the server, and is also used to transmit the initialized global model of the server to the client, or to transmit the updated global model of the server to the client.

Images