Federal learning method and system

Abstract

The invention provides a federal learning method and system, and the method is applied to a client, and comprises the steps: receiving a global control variable and a global model sent by a server; initializing a local model according to the global model, and determining the initialized local model; inputting local training data into the initialized local model for model training, and determining a loss function of the local model; according to the loss function of the local model, the local model is updated based on back propagation and a model difference reduction algorithm, and an updated local model is determined; and after the local model is updated for a preset number of times, determining a local model change amount according to the last updated local model and the initialized local model. According to the method and the device, a model difference reduction algorithm is introduced into the client, so that model differences are reduced, local model overfitting is avoided, data privacy is protected, and the convergence rate of a global model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a federated learning method and system. Background Art

[0002] With the development of big data, artificial intelligence driven by massive data has made significant progress in various fields. These data often come from various types of clients, including computers, mobile phones, smart homes and other devices. However, the emphasis on data privacy and security has become a global trend, and the collection of client data is often restricted by relevant laws and regulations. For example, the General Data Protection Regulation (GDPR), a plan officially implemented by the European Union in 2018, shows that the research and business communities often collect and transfer data on the client side, and such data collection and transfer will be severely punished by the law. Similarly, the Cybersecurity Law of the People's Republic of China, which was implemented in 2017, also restricts network operators from collecting user data. Therefore, without leaking user data privacy, it is impossible to upload user data to the server to train machine learning models like traditional central learning, and new methods are urgently needed.

[0003] After searching the literature on the prior art, it was found that McMahan et al. proposed a method in the article "Communication-efficient learning of deep networks from decentralized data" published in 2016, namely federated learning. Specifically, the initialization model is first downloaded to the clients participating in the current round of training. Please note that due to unstable communication, not all clients will participate in each round of training. The clients participating in the training use local private data to train and update local model parameters. These updated models (or model changes) are sent to the server and aggregated to obtain an updated global model. The global model will be re-downloaded to the clients participating in the next round of training as the initialization model for the next iteration, and this process is repeated until the model converges. However, since the data of each client is not independent and identically distributed, the trained models often fall into their own local optimality, and the variance between them is large. The simple average of model parameters will significantly reduce the overall effect of the aggregated model.

[0004] Stochastic variance reduction technology is a method commonly used to reduce the variance of stochastic gradients in stochastic gradient descent optimization. Based on the literature search of the existing technology, Defazio et al. proposed a more efficient variance reduction technology, namely SAGA, in the paper "SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives" published at the 2014 "Neural Information Processing Systems" conference. SAGA is used to reduce the variance between gradients caused by sampling in the stochastic gradient descent algorithm. SAGA improves the convergence rate by reducing the gradient variance caused by sampling and making the gradient estimate unbiased. Its convergence speed is better than other variance reduction techniques such as SAG and SVRG under the condition of strong convexity, and is also applicable to non-convex situations. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a federated learning method and system.

[0006] To achieve the above objective, according to one aspect of the present disclosure, a federated learning method is provided, which is applied to a client, wherein the client includes a local model, including:

[0007] Receiving global control variables and global models sent by the server, wherein the global control variables include initialized global control variables and updated global control variables, and the global models include initialized global models and updated global models;

[0008] Initializing the local model according to the global model, and determining the initialized local model;

[0009] Inputting local training data into the initialized local model for model training, determining the model-trained local model and a loss function of the local model, wherein the loss function includes a cross entropy loss between a training label and a true label of the local training data;

[0010] According to the loss function of the local model, the model-trained local model is updated based on a back-propagation and model difference reduction algorithm to determine an updated local model;

[0011] When the number of update processes of the local model reaches a preset threshold, the local model change amount is determined according to the last updated local model and the initialized local model.

[0012] Optionally, updating the model-trained local model based on a back propagation and model difference reduction algorithm according to the loss function of the local model to determine an updated local model includes:

[0013] Subtract the gradient obtained based on back propagation from the mean value of the gradient involved in the previous round of model training to determine the result of the subtraction;

[0014] Sum the difference result and the global control variable to determine the sum result;

[0015] Multiplying the sum result with the updated learning rate of the client to determine the product result;

[0016] The product result is used to update the local model that has undergone model training to determine an updated local model.

[0017] Optionally, when the number of update processes of the local model reaches a preset threshold, determining the amount of change of the local model according to the last updated local model and the initialized local model includes:

[0018] When the number of update processing of the local model reaches a preset threshold, the last updated local model is subtracted from the initialized local model to determine the change amount of the local model.

[0019] Optionally, the method further comprises:

[0020] The local model change amount is uploaded to the server.

[0021] According to a second aspect of the present disclosure, a federated learning method is provided, which is applied to a server side, wherein the server side includes a global model, including:

[0022] Initializing global control variables and a global model, and determining the initialized global control variables and the initialized global model;

[0023] Sending the initialized global control variables and the initialized global model to a client;

[0024] Receive the local model changes uploaded by the client;

[0025] Aggregating the local model changes uploaded by the client to determine an aggregated amount of the local model changes;

[0026] According to the aggregated amount of the local model changes, the global model and the global control variables are updated to determine an updated global model and an updated global control variable.

[0027] The global control variables are updated to determine an updated global model and updated global control variables.

[0028] Optionally, updating the global model and the global control variables according to the aggregated amount of the local model changes to determine the updated global model and the updated global control variables includes:

[0029] Linearly adding the aggregated amount of the local model change amount to the global model to determine the updated global model;

[0030] The aggregate amount of the local model change amount is linearly added to the global control variable to determine the updated global control variable.

[0031] Optionally, the method further comprises:

[0032] The updated global model and the updated global control variables are sent to the client.

[0033] According to a third aspect of the present disclosure, a federated learning system is provided, which is applied to a client, wherein the client includes a local model, including:

[0034] The client receiving module is used to receive the global control variables and the global model sent by the server, wherein the global control variables include the initialized global control variables and the updated global control variables, and the global model includes the initialized global model and the updated global model;

[0035] A client initialization module, used to initialize the local model according to the global model and determine the initialized local model;

[0036] A client model training module, used for inputting local training data into the initialized local model for model training, determining the model-trained local model and a loss function of the local model, wherein the loss function includes a cross entropy loss between a training label and a true label of the local training data;

[0037] A client update module, configured to update the model-trained local model based on a back propagation and model difference reduction algorithm according to the loss function of the local model, and determine an updated local model;

[0038] The client local model change amount determination module is used to determine the local model change amount according to the last updated local model and the initialized local model when the number of update processing of the local model reaches a preset threshold.

[0039] According to a fourth aspect of the present disclosure, there is provided a federated learning system, which is applied to a server side, wherein the server side includes a global model, including:

[0040] A server-side initialization module, used for initializing global control variables and global models, and determining initialized global control variables and initialized global models;

[0041] A server-side sending module, used for sending the initialized global control variables and the initialized global model to the client;

[0042] The server-side receiving module is used to receive the local model changes uploaded by the client;

[0043] A server-side aggregation module, used to aggregate the local model changes uploaded by the client to determine an aggregated amount of the local model changes;

[0044] The server-side update module is used to update the global model and the global control variables according to the aggregate amount of the local model change amount, and determine the updated global model and the updated global control variables.

[0045] According to a fifth aspect of the present disclosure, a federated learning system is provided, including:

[0046] The local model update module is used to update the local model based on the back propagation and model difference reduction algorithms on the client side;

[0047] A model change amount calculation module, used to determine the local model change amount by subtracting the last updated local model from the initialized local model when the number of update processing of the local model of the client reaches a preset threshold;

[0048] A global aggregation module, used for aggregating the local model changes uploaded by the client on the server side and updating the global model;

[0049] A global control variable updating module, used for updating the global control variable based on the aggregate amount of the local model change amount at the server end;

[0050] A communication module is used to transmit the local model change of the client to the server, and is also used to transmit the initialized global control variables and the initialized global model of the server to the client, or to transmit the updated global model and the updated global control variables of the server to the client.

[0051] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0052] Through the above technical solution, global control variables are introduced on the client to assist local training, so that the update direction of the local model is biased towards the direction of the global control variables, and a model difference reduction algorithm is used to reduce the model differences of the local models of each client node, prevent the local model from overfitting and falling into the local optimum, and improve the convergence speed.

[0053] In the embodiment of the present disclosure, the client receives the global control variables and the global model sent by the server, and uploads the local model changes to the server without transmitting the local data of the client, thereby protecting the data privacy of the client.

[0054] In the embodiments of the present disclosure, the server side introduces variance reduction technology, updates the global model and global control variables based on aggregated local model changes, reduces the client model variance caused by non-independent distribution of data, and improves the generalization performance of the global model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0056] Figure 1 The figure is a flowchart of a federated learning method applied to a client according to an exemplary embodiment.

[0057] Figure 2 It is a flowchart of a federated learning method applied to a server according to an exemplary embodiment.

[0058] Figure 3 The figure is a schematic diagram of signaling interaction between a client and a server according to an exemplary embodiment.

[0059] Figure 4 It is a schematic diagram of test curves of various federated learning methods on CIFAR10 and CIFAR100 data sets according to an exemplary embodiment.

[0060] Figure 5 It is a test curve diagram of various federated learning methods on EMNIST and Shakespeare data sets according to an exemplary embodiment.

[0061] Figure 6 The invention is a block diagram of a federated learning system applied to a client according to an exemplary embodiment.

[0062] Figure 7 It is a block diagram of a federated learning system applied to a server according to an exemplary embodiment.

[0063] Figure 8The diagram is a schematic diagram of the architecture of a federated learning system according to an exemplary embodiment. DETAILED DESCRIPTION

[0064] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0065] Figure 1 The figure is a flowchart of a federated learning method applied to a client according to an exemplary embodiment.

[0066] like Figure 1 As shown, the present disclosure provides a federated learning, which is applied to a client, and the client includes a local model, including S11 to S15.

[0067] S11, receiving the global control variables and global model sent by the server.

[0068] The global control variables include initialized global control variables and updated global control variables; the global model includes initialized global model and updated global model.

[0069] S12, initializing the local model according to the global model, and determining the initialized local model.

[0070] S13, inputting the local training data into the initialized local model for model training, and determining the trained local model and the loss function of the local model.

[0071] The loss function includes the cross entropy loss between the training labels and the true labels of the local training data.

[0072] S14, according to the loss function of the local model, the local model that has been trained is updated based on the back propagation and model difference reduction algorithm to determine the updated local model.

[0073] S15, when the number of update processing of the local model reaches a preset threshold, determining the amount of change of the local model according to the last updated local model and the initialized local model.

[0074] Through the above technical solution, global control variables are introduced on the client to assist local training, so that the update direction of the local model is biased towards the direction of the global control variables, and a model difference reduction algorithm is used to reduce the model differences of the local models of each client node, prevent the local model from overfitting and falling into the local optimum, and improve the convergence speed.

[0075] In a possible embodiment, S14 may include S141 to S144.

[0076] S141, subtract the gradient obtained by back propagation from the mean of the gradients participating in the previous round of model training to determine the subtraction result.

[0077] S142, sum the difference result and the global control variable to determine the sum result.

[0078] S143, multiplying the sum result by the updated learning rate of the client to determine the product result.

[0079] S144, using the multiplication result to update the local model and determine an updated local model.

[0080] As an example, based on step S141 to step S144, the local model is updated using the following formula to determine an updated local model:

[0081]

[0082]

[0083] in, and Indicates the intermediate variable cached during the update process of the local model by client i (no actual meaning). represents the change in the local model of client i in the t-1th round of model training, h t-1 represents the global control variable of the t-1th round of model training, K represents the total number of update processing times of the local model, represents the learning rate of the client in the t-1th round of model training, represents the local model trained by client i at communication round t and number of local updates k, represents the local model trained by client i at communication round t with local update times k-1, represents the updated learning rate of the client in round t, It represents the gradient generated by the client i in the k-1th local update iteration in the tth round, and k represents the number of update processing times of the local model.

[0084] in That is, initialized to 0.

[0085] In a possible embodiment, S15, when the number of update processing of the local model reaches a preset threshold, determining the amount of change of the local model according to the last updated local model and the initialized local model, includes:

[0086] When the number of update processing of the local model reaches a preset threshold, the local model last updated is subtracted from the initialized model to determine the change amount of the local model.

[0087] In a possible embodiment, a federated learning method is applied to a client and further includes S16.

[0088] S16, uploading the local model changes to the server.

[0089] Among them, each client participating in this round of model training uploads its local model changes to the server.

[0090] The client of the present invention receives the global control variables and the global model sent by the server, and uploads the local model changes to the server without transmitting the local data of the client, thereby protecting the data privacy of the client.

[0091] Figure 2 It is a flowchart of a federated learning method applied to a server according to an exemplary embodiment.

[0092] like Figure 2 As shown, the present disclosure also provides a federated learning method, which is applied to a server side, and the server side includes a global model, including S21 to S25.

[0093] S21, initializing global control variables and a global model, and determining the initialized global control variables and the initialized global model.

[0094] S22, sending the initialized global control variables and the initialized global model to the client.

[0095] S23, receiving the local model change uploaded by the client.

[0096] S24, performing aggregation processing on the local model changes uploaded by the client to determine an aggregated amount of the local model changes.

[0097] The aggregation process adopts a weighted summation method, that is, a weighted sum is performed on each local model change amount uploaded by the client to determine the aggregate amount of the local model change amount.

[0098] S25: According to the aggregated amount of the local model changes, the global model is updated to determine an updated global model and an updated global control variable.

[0099] Through the above technical solution, variance reduction technology is introduced on the server side to update the global model and global control variables based on the aggregated local model changes, thereby reducing the client model variance caused by non-independent distribution of data and improving the generalization performance of the global model.

[0100] In a possible embodiment, S25, updating the global model according to the aggregated amount of the local model change amount, and determining the updated global model and the updated global control variable, includes:

[0101] The aggregate amount of local model changes is linearly added to the global model to determine the updated global model.

[0102] The aggregate amount of local model changes is linearly added to the global control variable to determine the updated global control variable.

[0103] In a possible embodiment, a federated learning method, applied on a server side, may also include S26.

[0104] S26, sending the updated global model and the updated global control variables to the client.

[0105] After the server executes step S26, the client executes step S11 to receive the global control variables and the global model sent by the server.

[0106] Figure 3 The figure is a schematic diagram of signaling interaction between a client and a server according to an exemplary embodiment.

[0107] like Figure 3 As shown, the present disclosure also provides a signaling interaction process of federated learning between a client and a server, which includes S101 to 112.

[0108] S101, the server initializes global control variables and a global model, and determines the initialized global control variables and the initialized global model.

[0109] S102, the server sends the initialized global control variables and the initialized global model to the client.

[0110] S103, the client receives the global control variables and the global model sent by the server.

[0111] The global control variables include initialized global control variables and updated global control variables, and the global model includes initialized global model and new global model.

[0112] S104: The client initializes the local model according to the global model and determines the initialized local model.

[0113] S105, the client inputs the local training data into the initialized local model for model training, and determines the trained local model and the loss function of the local model.

[0114] S106, the client updates the model-trained local model according to the loss function of the local model based on back propagation and model difference reduction algorithm to determine an updated local model.

[0115] S107: When the number of update processing of the local model reaches a preset threshold, the client determines the amount of change of the local model according to the last updated local model and the initialized local model.

[0116] S108, the client uploads the local model changes to the server.

[0117] S109: The server receives the local model change uploaded by the client.

[0118] S110, the server aggregates the local model changes uploaded by the client to determine an aggregated amount of the local model changes.

[0119] S111, the server updates the global model and the global control variables according to the aggregated amount of the local model changes, and determines the updated global model and the updated global control variables.

[0120] S112, the server sends the updated global model and the updated global control variables to the client.

[0121] After steps S101 to S104 are executed, steps S105 to S106 are executed in a loop until the number of updates of the local model reaches a preset threshold.

[0122] After step S112 is executed, the client returns to execute step S103 and starts the next round of training of the local client until the global model converges and stops model training.

[0123] The technical solution provided by the above embodiment of the present disclosure is further described in detail below in conjunction with a specific application example.

[0124] A federated learning method provided by the present disclosure is used for the next word prediction task of a mobile phone terminal. The next word prediction task is to use a deep learning model to predict the next word input by the user based on the historical input data of the user's mobile phone terminal to assist the user in inputting. In order to protect the privacy of the user, the server cannot directly obtain the input data of the mobile phone terminal of the mobile phone user, and the deep learning model requires a large amount of data for effective training. The federated learning method provided by the present disclosure is used to train a global model with excellent generalization performance without leaking the local data of each mobile phone terminal.

[0125] Among them, a communication network between the server and the mobile phone terminals of each user can be built based on a trusted third-party organization as the server and the mobile phone terminals of each user as the client. That is, a communication module. There is no need for direct communication between the mobile phone terminals of each user, and the original user input data set is not transmitted between the third-party organization and the mobile phone terminals of the users.

[0126] Based on the technical solution provided by the above embodiments of the present disclosure, a deep learning model with excellent generalization performance can be quickly trained. The specific implementation steps are as follows:

[0127] Step 1: The server initializes the global model and global control variables. Due to the instability of the communication environment, the server randomly selects some mobile terminals as clients participating in this round of training and sends them a unified initialized global model and initialized global control variables.

[0128] Step 2: The client receives the initialized global model and the initialized global control variables, and initializes the deep learning model of the local client based on the initialized global model, and determines the initialized deep learning model of the local client.

[0129] Step 3: Use the local user input data set to train the initialized local deep learning model, use the historical data input by the user as the input of the deep learning model, and train the deep learning model by predicting the next word to determine the loss function of the deep learning model. The loss function includes the cross entropy loss function of the predicted data and the target data. According to the loss function of the deep learning model, the client uses the initialized global control variables to update the local deep learning model based on the back propagation and model difference reduction algorithm. The specific formula is as follows:

[0130]

[0131] in and Indicates the intermediate variable cached during the update process of the local model by client i (no actual meaning). represents the change in the local model of client i in the t-1th round of model training, h t-1 represents the global control variable of the t-1th round of model training, K represents the total number of update processing times of the local model, represents the learning rate of the client in the t-1th round of model training, represents the local model trained by client i at communication round t and number of local updates k, represents the local model trained by client i at communication round t with local update times k-1, represents the updated learning rate of the client in round t, It represents the gradient generated by the client i in the k-1th local update iteration in the tth round, and k represents the number of update processing times of the local model.

[0132] Step 4: Repeat step 3 K times, K ≥ 1, save the mean gradient of the client participating in this round of training and calculate the change of the deep learning model of each mobile terminal participating in the training in this round, and upload it to the server.

[0133] Among them, the change amount of the deep learning model of the mobile phone terminal is determined by subtracting the deep learning model of the mobile phone terminal that was last updated from the deep learning model of the mobile phone terminal that has been initialized.

[0134] Step 5: The server performs global aggregation on the received changes in the deep learning model to determine a new global model. The global aggregation can be performed using an arithmetic average.

[0135] Step 6: The server updates the global control variable, which represents the gradient mean of the deep learning model of all mobile terminals (including those participating in the training in this round and those not participating in the training).

[0136]

[0137] Among them, h t represents the updated global control variable, h t-1 Represents the original global control variable, It represents the updated learning rate of the client in the tth round of model training, p represents the number of mobile terminals participating in the training in this round, and m represents the total number of mobile terminals.

[0138] in,

[0139]

[0140] The mean of the aggregated amount of changes in the local model.

[0141] Step 7: The server sends the updated global control variables and the updated global model to the mobile terminal that is the client participating in the next round of training.

[0142] Step 7: Repeat steps 2 to 7 until the global model converges, that is, the global next word prediction model converges, and stop model training.

[0143] Through the above technical solution, mobile terminals can collaboratively train a global next-word prediction model with superior generalization performance without sharing original data, accelerate the convergence speed of the global model, effectively protect the user's privacy data, and have strong practical application value.

[0144] In a possible embodiment, a federated learning method provided by the present disclosure is used to perform health monitoring and sports behavior prediction.

[0145] With the popularity of wearable devices (such as smart bracelets, smart watches, etc.), more and more users have begun to record their personal exercise, heart rate, sleep and other health data in real time. However, health data involves the user's privacy information, so uploading these data to the cloud or centralized server for processing has a greater risk of privacy leakage. In order to use these data for effective health prediction and exercise guidance while protecting user privacy, a federated learning method provided by the present disclosure can be used for health monitoring and exercise behavior prediction.

[0146] In smart bracelets or smart wearable devices, users' future health status or exercise behavior can be predicted by analyzing their exercise behavior data. In this scenario, the goal is to use the device data of multiple users to train a global deep learning model, namely a global health monitoring model, to help users better manage their health and exercise plans without leaking their personal data.

[0147] A communication network between the server and each user's device, i.e., a communication module, can be built based on a trusted third-party organization as the server and each user's device, such as a smart bracelet or other smart wearable device, as the client. There is no need for direct communication between the devices of each user, and the user's device data is not transmitted between the third-party organization and the user's device.

[0148] Step 1: The server initializes a global health monitoring model, which may include the user's exercise behavior classification module, heart rate prediction module, and sleep analysis module; the server also initializes global control variables to coordinate the local model updates of each client in the subsequent training process. Due to the instability of the communication environment, the server randomly selects some users' devices as clients participating in this round of training, and sends them a unified initialized global health monitoring model and initialized global control variables.

[0149] Step 2: Each user's device receives the global health monitoring model and global control variables sent by the server. The user's device uses local health monitoring data (such as exercise steps, heart rate, activity duration, etc.) to initialize the local model and determine the initialized local model.

[0150] Step 3: Each user's device trains a local model based on its own exercise data and health records, predicts future exercise behavior or health status (such as whether to rest, whether to over-exercise, etc.) based on the user's exercise data (such as daily steps, exercise type, heart rate, etc.), and updates the local model through back propagation and model difference reduction algorithms. The update process takes into account global control variables to ensure the consistency of updates on each client. The specific local update step uses the following formula:

[0151]

[0152] in and Indicates the intermediate variable cached during the update process of the local model by client i (no actual meaning). represents the change in the local model of client i in the t-1th round of model training, h t-1 represents the global control variable of the t-1th round of model training, K represents the total number of update processing times of the local model, represents the learning rate of the client in the t-1th round of model training, represents the local model trained by client i at communication round t and number of local updates k, represents the local model trained by client i at communication round t with local update times k-1, represents the updated learning rate of the client in round t, It represents the gradient generated by the client i in the k-1th local update iteration in the tth round, and k represents the number of update processing times of the local model.

[0153] Step 4: Repeat step 3 K times, K ≥ 1, save the gradient mean of the client participating in this round of training and calculate the local model change of each client participating in the training in this round, and upload it to the server.

[0154] The change amount of the client's local model is determined by subtracting the last updated local model of the client from the initialized local model.

[0155] Step 5: The server performs global aggregation on the received local model changes to determine a new global model. The global aggregation may be performed using an arithmetic average processing method.

[0156] Step 6: The server updates the global control variable, which represents the gradient mean of the local model of all clients (including those participating in the training in this round and those not participating in the training).

[0157]

[0158] Among them, h trepresents the updated global control variable, h t-1 Represents the original global control variable, represents the updated learning rate of the client in the tth round of model training, p represents the number of clients participating in the training in this round, and m represents the total number of clients.

[0159] in,

[0160]

[0161] The mean of the aggregated amount of changes in the local model.

[0162] Step 7: The server sends the updated global control variables and the updated global health monitoring model to the devices of the users who are the clients participating in the next round of training.

[0163] Step 8: Repeat steps 2 to 7 until the global health monitoring model reaches the expected accuracy or convergence standard. For example, when the accuracy of the model in predicting exercise behavior, health status or exercise load reaches a preset standard, the training process can be terminated.

[0164] Through this embodiment, multiple users' devices (such as smart bracelets, smart watches, etc.) can collaboratively train a global health monitoring model with strong generalization ability without exchanging original health data, and can predict the health status of users based on their exercise behavior, health data, etc., and provide personalized suggestions based on the exercise needs of different users, such as whether they need to rest, whether they have achieved their health goals, etc. At the same time, due to the privacy protection mechanism of federated learning, the user's personal data (such as steps, heart rate, etc.) is always retained on the local device and will not be uploaded to the server, effectively protecting the user's privacy information.

[0165] In a possible embodiment, the image datasets EMNIST, CIFRA-10, CIFAR-100 and the natural language data Shakespeare dataset are used to verify the model convergence speed and generalization performance of the federated learning method provided by the present disclosure and the method FedAvg in “Communication-efficient learning of deep networks from decentralized data” by McMahan et al., the method FedDyn in “Federated Learning Based on Dynamic Regularization” by Acer et al., the method SCAFFOLD in “scaffold: stochastic controlled averaging for federated learning” by Karimierddy et al., and the method FedAvgM in “Measuring the effects of non-identical data distribution for federated visual classification” by Hsu et al.

[0166] Figure 4 It is a schematic diagram of test curves of various federated learning methods on CIFAR10 and CIFAR100 data sets according to an exemplary embodiment. Figure 5 It is a test curve diagram of various federated learning methods on EMNIST and Shakespeare data sets according to an exemplary embodiment.

[0167]

[0168] Table 1

[0169] like Figure 4 , Figure 5 As shown in Table 1, FedMDVR represents the federated learning method proposed in the present disclosure.

[0170] The federated learning method FedMDVR proposed in this disclosure has the fastest convergence speed among all the above data sets.

[0171] Depend on Figure 4 , Figure 5 as well as Figure 1 It can be seen that the federated learning method FedMDVR proposed in the present disclosure has the highest model classification accuracy in the above dataset, and it achieves the best performance in the above dataset.

[0172] Figure 6The invention is a block diagram of a federated learning system applied to a client according to an exemplary embodiment.

[0173] Based on the same concept, the present disclosure also provides a federated learning system 100, which is applied to a client, and the client includes a local model, including: a client receiving module 110, a client initialization module 120, a client model training module 130, a client update module 140, and a client local model change amount determination module 150.

[0174] The client receiving module 110 is used to receive the global control variables and the global model sent by the server, wherein the global control variables include the initialized global control variables and the updated global control variables, and the global model includes the initialized global model and the updated global model;

[0175] A client initialization module 120, configured to initialize a local model according to the global model and determine an initialized local model;

[0176] The client model training module 130 is used to input local training data into the initialized local model for model training, and determine the model-trained local model and the loss function of the local model, wherein the loss function includes a cross entropy loss between a training label and a true label of the local training data;

[0177] The client updating module 140 is used to update the local model after model training based on the loss function of the local model and the back propagation and model difference reduction algorithm to determine the updated local model;

[0178] The client local model change amount determination module 150 is used to determine the local model change amount according to the last updated local model and the initialized local model when the number of update processes of the local model reaches a preset threshold.

[0179] Through the above technical solution, global control variables are introduced on the client to assist local training, so that the update direction of the local model is biased towards the direction of the global control variables, and a model difference reduction algorithm is used to reduce the model differences of the local models of each client node, prevent the local model from overfitting and falling into the local optimum, and improve the convergence speed.

[0180] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0181] Figure 7 It is a block diagram of a federated learning system applied to a server according to an exemplary embodiment.

[0182] Based on the same concept, the present disclosure also provides a federated learning system 200, which is applied to the server side. The server side includes a global model, including: a server-side initialization module 210, a server-side sending module 220, a server-side receiving module 230, a server-side aggregation module 240, and a server-side update module 250.

[0183] The server-side initialization module 210 is used to initialize the global control variables and the global model, and determine the initialized global control variables and the initialized global model;

[0184] A server-side sending module 220, configured to send the initialized global control variables and the initialized global model to the client;

[0185] The server-side receiving module 230 is used to receive the local model change uploaded by the client;

[0186] The server-side aggregation module 240 is used to aggregate the local model changes uploaded by the client to determine the aggregated amount of the local model changes;

[0187] The server-side updating module 250 is used to update the global model according to the aggregated amount of the local model changes, and determine the updated global model and the updated global control variables.

[0188] Through the above technical solution, variance reduction technology is introduced on the server side to update the global model and global control variables based on the aggregated local model changes, thereby reducing the client model variance caused by non-independent distribution of data and improving the generalization performance of the global model.

[0189] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0190] Figure 8 The diagram is a schematic diagram of the architecture of a federated learning system according to an exemplary embodiment.

[0191] In a possible embodiment, the present disclosure further provides a federated learning system 300 , including: a local model update module 310 , a model change calculation module 320 , a global aggregation module 330 , a global control variable update module 340 , and a communication module 350 .

[0192] Among them, the local model update module 310 and the model change calculation module 320 are set on the local client, the global aggregation module 330 and the global control variable update module 340 are set on the server side, and the communication module 350 is used to transmit the client's local model change and the client's global model and global control variables.

[0193] A local model updating module 310 is used to update the local model based on the back propagation and model difference reduction algorithm at the client;

[0194] The model change amount calculation module 320 is used to determine the local model change amount by subtracting the last updated local model from the initialized local model when the number of update processing of the local model of the client reaches a preset threshold;

[0195] A global aggregation module 330 is used to aggregate the local model changes uploaded by the client on the server side and update the global model;

[0196] A global control variable updating module 340, configured to update the global control variable based on the aggregate amount of the local model change amount at the server end;

[0197] The communication module 350 is used to transmit the local model change of the client to the server, and is also used to transmit the initialized global control variables and the initialized global model of the server to the client, or to transmit the updated global model and the updated global control variables of the server to the client.

[0198] like Figure 8 As shown, as an example, clients 1 to m use local privacy data to perform model training and update processing on the local model in the local model update module 310; the local model change amount is calculated in the model change amount calculation module 320; clients 1 to m transmit the local model change amount to the server side through the communication module 350, aggregate the local model change amount of each client in the global aggregation module 330, and update the global model; the global control variable is updated in the global control variable update module 340 based on the aggregated amount of local model change amount.

[0199] The communication module 350 is also used to transmit the global model and the updated global control variables updated by the server to the client.

[0200] The above describes the specific embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which does not affect the essence of the present disclosure. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A federated learning method, characterized in that: Applied to a client, the client includes a local model, including: Receiving global control variables and global models sent by the server, wherein the global control variables include initialized global control variables and updated global control variables, and the global models include initialized global models and updated global models; Initializing the local model according to the global model, and determining the initialized local model; Inputting local training data into the initialized local model for model training, determining the model-trained local model and a loss function of the local model, wherein the loss function includes a cross entropy loss between a training label and a true label of the local training data; According to the loss function of the local model, the model-trained local model is updated based on a back-propagation and model difference reduction algorithm to determine an updated local model; When the number of update processes of the local model reaches a preset threshold, the local model change amount is determined according to the last updated local model and the initialized local model.

2. The method according to claim 1, characterized in that The updating process of the model-trained local model based on the loss function of the local model and the back propagation and model difference reduction algorithm to determine the updated local model includes: Subtract the gradient obtained based on back propagation from the mean value of the gradient involved in the previous round of model training to determine the result of the subtraction; Sum the difference result and the global control variable to determine the sum result; Multiplying the sum result with the updated learning rate of the client to determine the product result; The product result is used to update the local model that has undergone model training to determine an updated local model.

3. The method according to claim 1, characterized in that When the number of update processing of the local model reaches a preset threshold, determining the amount of change of the local model according to the last updated local model and the initialized local model, comprises: When the number of update processing of the local model reaches a preset threshold, the last updated local model is subtracted from the initialized local model to determine the change amount of the local model.

4. The method according to claim 1, characterized in that The method further comprises: The local model change amount is uploaded to the server.

5. A federated learning method, characterized in that: Applied to the server side, the server side includes a global model, including: Initializing global control variables and a global model, and determining the initialized global control variables and the initialized global model; Sending the initialized global control variables and the initialized global model to a client; Receive the local model changes uploaded by the client; Aggregating the local model changes uploaded by the client to determine an aggregated amount of the local model changes; According to the aggregated amount of the local model changes, the global model and the global control variables are updated to determine an updated global model and an updated global control variable.

6. The method according to claim 5, characterized in that The updating process of the global model and the global control variable according to the aggregate amount of the local model change amount to determine the updated global model and the updated global control variable includes: Linearly adding the aggregated amount of the local model change amount to the global model to determine the updated global model; The aggregate amount of the local model change amount is linearly added to the global control variable to determine the updated global control variable.

7. The method according to claim 5, characterized in that The method further comprises: The updated global model and the updated global control variables are sent to the client.

8. A federated learning system, characterized in that: Applied to a client, the client includes a local model, including: The client receiving module is used to receive the global control variables and the global model sent by the server, wherein the global control variables include the initialized global control variables and the updated global control variables, and the global model includes the initialized global model and the updated global model; A client initialization module, used to initialize the local model according to the global model and determine the initialized local model; A client model training module, used for inputting local training data into the initialized local model for model training, determining the model-trained local model and a loss function of the local model, wherein the loss function includes a cross entropy loss between a training label and a true label of the local training data; A client update module, configured to update the model-trained local model based on a back propagation and model difference reduction algorithm according to the loss function of the local model, and determine an updated local model; The client local model change amount determination module is used to determine the local model change amount according to the last updated local model and the initialized local model when the number of update processing of the local model reaches a preset threshold.

9. A federated learning system, characterized in that: Applied to the server side, the server side includes a global model, including: A server-side initialization module, used for initializing global control variables and global models, and determining initialized global control variables and initialized global models; A server-side sending module, used for sending the initialized global control variables and the initialized global model to the client; The server-side receiving module is used to receive the local model changes uploaded by the client; A server-side aggregation module, used to aggregate the local model changes uploaded by the client to determine an aggregated amount of the local model changes; The server-side update module is used to update the global model and the global control variables according to the aggregate amount of the local model change amount, and determine the updated global model and the updated global control variables.

10. A federated learning system, characterized in that: include: The local model update module is used to update the local model based on the back propagation and model difference reduction algorithms on the client side; A model change amount calculation module, used to determine the local model change amount by subtracting the last updated local model from the initialized local model when the number of update processing of the local model of the client reaches a preset threshold; A global aggregation module, used for aggregating the local model changes uploaded by the client on the server side and updating the global model; A global control variable updating module, used for updating the global control variable based on the aggregate amount of the local model change amount at the server end; A communication module is used to transmit the local model change of the client to the server, and is also used to transmit the initialized global control variables and the initialized global model of the server to the client, or to transmit the updated global model and the updated global control variables of the server to the client.

Images