A federated learning method and a domain generalization federated system

Abstract

A federated learning method and a domain generalization federated system. The present application provides a federated learning method, comprising: a server sending an initialized server model to a plurality of clients as a client model; the server and the clients cooperating to complete multiple rounds of training to obtain a final server model and a client model, each round comprising: S1, each client training the latest obtained client model according to a local training set to obtain a client model after this round of training; S2, the server determines the consistency index of the parameter of each position in the updated direction of all the client models after this round of training, and aggregates the parameters that meet the preset condition of the consistency index to update the parameters of the current server model to obtain an updated server model; S3, each client determines the general parameters of the trained client model based on the client model before and after the training of the current round and the server model obtained in the current round, and updates the general parameters according to the server model of the current round to obtain the updated client model of the current round.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ubiquitous computing, transfer learning and federated learning, in particular to a federated learning method and a domain generalization federated system. BACKGROUND

[0002] In recent years, with the development of machine learning technology, various machine learning algorithms, especially deep learning, are widely used in various fields of real life. Machine learning usually uses a certain amount of data to mine and learn the patterns or trends of observed objects. For example, in the field of medical health, machine learning plays a huge role. Researchers collect health-related data (such as inertial sensors, images, sounds, etc.) to build training models to estimate individual health status. However, machine learning often requires a large amount of data for model construction. As people and organizations become more aware of the importance of self-data protection, it is becoming increasingly difficult and costly to obtain large amounts of data. Therefore, traditional model training by aggregating large amounts of data has become almost impossible in some fields. Most people or organizations expect to contribute to machine learning models and enjoy the convenience brought by machine learning, while also wanting their private data to be private and secure. In addition, various countries, organizations and institutions are actively improving laws and regulations related to data protection. Based on the above circumstances, in order to protect data privacy and security, data dispersion leads to data silos and cannot build high-precision models.

[0003] To solve the above problems, federated learning emerges as the times require. Google proposed the first federated learning method, FedAVG. FedAVG no longer requires aggregating scattered data, nor does it require local data to be exposed to other parties. FedAVG aggregates scattered information by exchanging network parameters, no longer requiring the transmission of local data, thereby achieving the purpose of data protection. In addition, many scholars and researchers have participated in federated learning research. For example, the scheme with the public number CN113869528A and the invention name of consensus representation extraction and diversity propagation disentangled personalized federated learning method extracts the local consensus representation extraction model and the unique representation extraction model corresponding to different nodes, uses mutual information to obtain the data distribution similarity between nodes, and aggregates the weights, so that the global consensus representation model has good generalization ability. The scheme with the public number CN114399055A and the invention name of a domain generalization method based on federated learning uses an adversarial learning network to align the feature distribution of the source domain data of the client and the reference feature distribution by category, and combines a federated learning architecture to aggregate the model parameters of multiple clients, thereby learning a model with good generalization ability in the target domain without target scene data under the condition of protecting the privacy of the source domain data. The scheme with the public number CN113420888A and the invention name of an unsupervised federated learning method based on generalization domain adaptation uses a distributed model parameter weighted average aggregation method based on class centroids, and combines a data interaction mode based on the number of communication rounds to improve the communication efficiency between clients and protect user privacy through an asynchronous learning method of feature extraction and classification learning.

[0004] Although various machine learning methods have been successfully applied in the federated field, the existing federated training method rarely considers the individualization of the client model and the generalization of the server model at the same time, and the server model lacks universality and the client model lacks pertinence. Therefore, there is an urgent need for a federated learning scheme that can protect data privacy and security and construct a high-precision and high-universality server model and a high-precision and individualized client model using scattered data. SUMMARY

[0005] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide a federated learning method and a domain generalization federated system.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] According to a first aspect of the present application, a federated learning method is provided, the method comprising: distributing, by a server, an initialized server model to a plurality of clients as an initial client model; and performing, by the server and the clients, a plurality of rounds of federated training, wherein each round of federated training comprises: S1, training, by each client, its client model using its local training set to obtain a plurality of client models after this round of training and uploading to the server, wherein the input data of the samples in the local training sets of different clients have the same dimension and the same set of label types, the current client model of the first round of training is the initial client model, and the current client model of other rounds of training is the client model updated after the previous round of training; S2, determining, by the server based on the plurality of client models uploaded in step S1, a consistency indicator of the parameter update direction at each position in the plurality of client models, and aggregating the parameters at the corresponding positions of the plurality of client models based on the consistency indicator to update the parameters of the current server model and issue to the plurality of clients; S3, determining, by each client, a general parameter of the client model after the current round of training based on the client model before the current round of training, the client model after the current round of training, and the server model obtained in the current round, and updating the general parameter in each client model according to the server model of the current round to obtain the client model updated in the current round.

[0008] In some embodiments of the present application, when the number of clients is less than or equal to a predetermined number threshold, the S2 comprises: calculating the parameter change value of each position of each client model after the current round of training and the corresponding position of the current server model; for all parameters at each position in all client models after the current round of training, calculating the consistency indicator of the parameter update direction at the position according to the number of positive parameter change values and the number of negative parameter change values at the corresponding position; and aggregating the parameters at the positions with a consistency indicator greater than or equal to a predetermined score in all client models after the current round of training to obtain an aggregation result.

[0009] In some embodiments of the present application, the consistency indicator of the parameter update direction is calculated as follows:

[0010]

[0011] wherein score j represents the consistency indicator of the jth position parameter, N represents the total number of client models after the current round of training, represents the parameter change value of the jth position of the ith client model after the current round of training relative to the jth position of the current server model.

[0012] In some embodiments of the present application, the parameters of the current server model are updated in step S2 as follows:

[0013]

[0014] wherein, denotes the parameter of the jth position obtained in the tth round of updating, denotes the parameter of the jth position of the server model obtained in the (t-1)th round of updating, score j denotes the consistency index of the parameter of the jth position, mask(·)∈{0,1} denotes the mask indicating whether the parameter of the jth position of the current server model needs to be updated, the parameter with the consistency index greater than or equal to a predetermined score is set to 1, and the values of the masks corresponding to the remaining parameters are set to 0, and N denotes the total number of client models after the current round of training, denotes the change value of the parameter of the jth position of the ith client model after the current round of training relative to the parameter at the jth position of the current server model, w i denotes the update weight parameter of the ith client model after the current round of training.

[0015] In some embodiments of the present application, when the number of clients exceeds a predetermined number threshold, the S2 comprises: grouping the plurality of client models after the current round of training and performing intra-group aggregation on each obtained group after grouping to obtain a representative model of each group; performing inter-group aggregation based on the representative model of each group to obtain an aggregation result, wherein the inter-group aggregation comprises the following steps: calculating the change value of the parameter of each position of each representative model relative to the parameter at the corresponding position of the current server model; obtaining all parameters for each position in all representative models, and calculating the consistency index of the parameter update direction of the position according to the number of positive parameter change values and the number of negative parameter change values at the corresponding position; and aggregating the parameters at the positions with the consistency index greater than or equal to a predetermined score in all representative models to obtain the aggregation result.

[0016] In some embodiments of the present application, the grouping and intra-group aggregation of each obtained group after grouping are performed in the following manner: obtaining the feature mean and feature variance of all samples calculated by each client using the features of all samples in the local training set extracted by the server model received in the last round; grouping the plurality of client models after the current round of training based on the feature mean and feature variance of all samples determined by each client to obtain a plurality of groups; and performing parameter aggregation on the parameters of the plurality of client models after the current round of training in each group to obtain a representative model of the group.

[0017] In some embodiments of the present application, the grouping manner comprises: inputting the feature mean value and feature variance determined by each client into a clustering algorithm for clustering to obtain a plurality of groups, wherein the clustering algorithm is a K-means clustering algorithm or a Gaussian mixture clustering algorithm.

[0018] In some embodiments of the present application, the parameter aggregation manner of the parameters of the plurality of client models in each group after the current round of training comprises:

[0019]

[0020] wherein, represents the model parameters of the representative model of the kth group in the tth round, m k represents the total number of client models after the current round of training in the kth group, represents the update weight parameter of the ith client model after the current round of training corresponding to the group, represents the parameters of the ith client model after the current round of training corresponding to the group.

[0021] In some embodiments of the present application, step S3 comprises: S31, calculating the training change value of each position parameter of the client model after the current round of training and the corresponding position parameter of the client model before the current round of training; S32, calculating the aggregation change value of each position parameter of the updated server model and the corresponding position parameter of the client model before the current round of training; S33, determining the general parameter of each client model according to the training change value and the aggregation change value corresponding to each parameter; S34, updating the general parameter in the client model after the current round of training according to the updated server model to obtain an updated client model.

[0022] In some embodiments of the present application, in step S33, the parameters with the same sign of the training change value and the aggregation change value are general parameters, and the parameters with different signs are personalized parameters, wherein the sign refers to the sign indicating a positive number or a negative number.

[0023] In some embodiments of the present application, the image is classified and predicted by using the client model or the server model to obtain a classification result of the image.

[0024] In some embodiments of the present application, the classification prediction of the image comprises: in the field of lung CT image recognition, the labels of image classification include virus infection, bacteria infection and normal; or in the field of road image recognition, the labels of image classification include cat, dog, human and car.

[0025] According to a second aspect of the present application, there is provided a domain generalization federated system, comprising: a server configured to obtain a final server model based on the method of the first aspect of the present application, and distribute the final server model to a client; and the client configured to deploy the final server model on the client for classification prediction of input data to be predicted by the client.

[0026] According to a third aspect of the present application, there is provided a domain generalization federated system, comprising: a server configured to obtain a final server model based on the method of the first aspect of the present application, and distribute the final server model to a client; and the client configured to deploy the final server model on the client, obtain a final client model based on the method of the first aspect of the present application, and use the final client model for classification prediction of input data to be predicted by the client.

[0027] According to a fourth aspect of the present application, there is provided an electronic device, comprising: one or more processors; and a memory configured to store executable instructions; the one or more processors are configured to implement the steps of the method of the first aspect of the present application by executing the executable instructions.

[0028] Compared with the prior art, the present application has the following advantages:

[0029] In each round of federated training, each client first trains a current client model using its local training set, and the server determines a consistency index of the trainable parameters in the same position in all trained client models based on the trained client model obtained by each client in the current round, aggregates the parameters that meet the preset condition in the consistency index to update the parameters of the current server model, so as to improve the universality of the server model, and finally, each client determines whether each parameter of the trained client model is a universal parameter or a personalized parameter based on the client model before the current round of training, the trained client model in the current round of training and the latest server model, so as to update the universal parameters and not update the personalized parameters, thereby ensuring the high accuracy of the client model, and at the same time, the generalization and personalization requirements of the model are also taken into account.

[0030] In the present application, when the number of clients is too large, the parameters of the server model are updated with reference to the representative model, which reduces the large conflict in the update direction when the number of clients is too large, and improves the learning efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0031] The embodiments of the present application will be further described below with reference to the accompanying drawings, in which:

[0032] Figure 1 A flowchart of a federated learning method according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0034] As mentioned in the background section, the existing federated learning methods rarely consider the personalization of the client model and the generalization of the server model at the same time, resulting in the lack of universality of the server model and the insufficient personalization ability of the client model.

[0035] In order to obtain a server model with higher universality and higher precision and a client model with higher personalization and higher precision, the present application provides a federated learning method, wherein the initialized server model is distributed by the server to multiple clients as the initial client model, and the server and the clients cooperate to complete multiple rounds of federated training to obtain the final server model and the final client model. In each round of federated training, first, each client trains the current client model with its local training set, second, the server determines the consistency index of the update direction of the parameters (trainable parameters) at each position in the multiple client models based on the trained client model obtained by each client in the current round, and the parameters meeting the preset condition are aggregated according to the consistency index to update the parameters of the current server model, thereby improving the universality of the server model, and finally, each client needs to determine whether each parameter of the trained client model is a universal parameter or a personalized parameter, so as to update the universal parameters and not update the personalized parameters, thereby ensuring the precision of the client model, and at the same time, the generalization and personalization requirements of the model are also taken into account.

[0036] Before the embodiments of the present application are specifically introduced, some terms used therein are explained as follows:

[0037] Trainable parameters: parameters in a neural network model (server model or client model) that can be updated according to the training set. For example, the weights, biases and other parameters of the convolutional neural network layer and / or the fully connected layer in some neural network models.

[0038] The present application will be described in detail below with reference to the drawings and embodiments.

[0039] According to one embodiment of the present application, a federated learning method is provided, which comprises distributing an initialized server model by a server to a plurality of clients as an initial client model, and participating in completing a plurality of rounds of federated training by the server and the clients. According to one embodiment of the present application, the server model can be an existing neural network classification model, for example, a ResNet model (such as a ResNet18 model, a ResNet34 model, or a ResNet50 model), an AlexNet model, a VGG model, etc., or also can be some neural network classification model defined by an implementer, and the present application does not make any limitation in this regard. According to one embodiment of the present application, referring to Figure 1 Each round of federated training comprises steps S1, S2 and S3. In order to better understand the present application, each step will be described in detail below in combination with specific embodiments.

[0040] In step S1, each client trains its client model using its local training set, obtains a plurality of client models after this round of training and uploads to the server, wherein the input data dimensions of the samples in the local training sets of different clients are the same and the label types of the samples are the same, the current client model of the first round of training is the initial client model, and the current client model of other rounds of training is the client model updated after the previous round of training. Wherein, each client trains to obtain one client model after this round of training using the local training set, and each round includes a plurality of clients participating in training, and the plurality of clients can correspond to obtain a plurality of client models after this round of training.

[0041] According to one embodiment of the present application, there are N different clients, denoted as {F1, F 12 ,…,F N}, the local data set of each client is denoted as D i , i = 1, 2, 3, … N, and the local data set D i of each client is divided into a local training set a validation data set and a test data set , wherein, represents the number of samples of the local training set of the i-th client, represents the a-th sample of the local training set of the i-th client, is the input data of the sample, is the label of the sample, represents the number of samples of the validation data set of the i-th client, represents the b-th sample of the validation data set of the i-th client, is the input data of the sample, is the label of the sample, a number of samples of a test data set of the i-th client, a c-th sample of a test data set of the i-th client, input data of the sample, a label of the sample, and

[0042] According to one embodiment of the present application, the input data of the sample has the same dimension and the same set of label types, as in Example One: in the field of lung CT image recognition, the input data of each sample is a human lung CT image of the same size, and the label types of the image classification include virus infection, bacteria infection and normal; in Example Two: in the field of road image recognition, the dimension of the input data of each sample is 3*224*224, 224 is the height and width of the picture, 3 is the RGB three-channel image of the picture, and the set of label types includes dog, cat, person and car. Illustratively, the client model can use Resnet18. It should be understood that the present application is not limited to the above-mentioned fields, but can also be applied to other fields. For example, in the field of Parkinson tremor recognition, the input data of each sample includes the nine-axis sensor data collected by a mobile phone held by an experimenter within 15 seconds, resulting in 9*t-dimensional data, 9 is the number of nine-axis sensor channels, and t is the total number of frames. The client model outputs the tremor level classification of the experimenter according to the input data, i.e., the set of label types includes zero level, first level, second level, third level, fourth level and fifth level. Zero level means no abnormal individual; first level means that the affected side of the limb appears, showing tremor or increased muscle tension; second level means that both sides of the limb are affected, but the balance function is normal; third level means that the balance function is abnormal, the patient's pull test is positive, but the daily life can be basically completed; fourth level means that the patient's balance function is seriously affected, and can barely stand and walk; fifth level means that the patient has difficulty standing, and generally needs to be in bed or sit in a wheelchair; illustratively, the client model can use Transformer.

[0043] In step S2, the server determines the consistency index of the parameter update direction at each position in the plurality of client models based on the plurality of client models uploaded in step S1, and aggregates the parameters at the corresponding positions of the plurality of client models based on the consistency index to update the parameters of the current server model, and delivers to the plurality of clients.

[0044] According to one embodiment of the present application, when the number of clients is less than or equal to a predetermined number threshold, the manner of aggregating the parameters in S2 comprises steps a1, a2 and a3. Wherein the predetermined number threshold of the number of clients can be set to 6 (it should be understood that this is only illustrative, and can also be 8, 12, etc., which can be set according to the specific needs, and the present application does not make any limitation thereon). The steps a1, a2 and a3 are described in detail below.

[0045] In step a1, the parameter change value of each position of each client model trained in this round relative to the corresponding position of the current server model is calculated.

[0046] According to one embodiment of the present application, each client model trained in this round is denoted as and indicates the i-th client model trained in the t-th round, and the current server model is updated in the t-1-th round, denoted as Wherein the j-th position of the client model is denoted as The j-th position of the server model is denoted as The trainable parameters of all positions of are denoted as The trainable parameters of all positions of The parameter change value of each client model trained in this round relative to the corresponding position of the server model is calculated as follows formula (1):

[0047]

[0048]

[0049] Wherein, indicates the parameter change value of all positions of the client model relative to the server model , indicates the parameter change value of the 0-th position of relative to , indicates the parameter change value of the 1-th position of relative to , indicates the parameter change value of the j-th position of relative to , indicates the parameter change value of all positions of relative to The parameter change value at the Mth position of the server model. It can be seen that since the server model and the plurality of client models are of the same structure, each trainable parameter corresponds to a position; therefore, the consistency of the parameter update direction of the plurality of client models at each position can be determined based on the server model, thereby obtaining a consistency index to guide the aggregation process, so as to improve the generalization and / or prediction accuracy of the server model.

[0050] In step a2, for all parameters of each position in all client models after this round of training, the consistency index of the parameter update direction of the position is calculated according to the number of positive parameter change values and the number of negative parameter change values at the corresponding position.

[0051] According to an embodiment of the present application, for all parameters of each position in all client models after this round of training, the absolute value of the difference between the number of positive parameter change values and the number of negative parameter change values at the corresponding position can be taken as the consistency index of the parameter update direction of the position. The consistency index of the parameter update direction is calculated as follows formula (2):

[0052]

[0053] Wherein, score j represents the consistency index of the parameter of the jth position in all client models, N represents the total number of client models after this round of training, represents the parameter change value of the jth position of the ith client model after this round of training relative to the parameter at the jth position of the current server model.

[0054] In step a3, the parameters at positions with consistency index greater than or equal to a predetermined score in all client models after this round of training are aggregated to obtain the aggregation result. In the present application, by counting the consistency index, when the consistency index is greater than or equal to a predetermined score, it means that the parameter update direction of the corresponding position of all client models is mostly the same, at this time, the result of aggregating the parameters at the position of all client models is used to update the server model, which better improves the generality of the server model.

[0055] According to an embodiment of the present application, the parameters of the current server model are updated in step S2 according to the following formula (3):

[0056]

[0057] Wherein, represents the parameter of the jth position of the server model obtained after the tth update, parameters of the jth position of the server model obtained in the t-1th round of updating, score j consistency indicator of the jth position of the parameters, mask(·)∈{0,1} indicates whether the current server model needs to be updated mask of the jth position of the parameters, the consistency indicator is greater than or equal to a predetermined score, the value of the mask corresponding to the parameters is set to 1, and the values of the masks corresponding to the remaining parameters are set to 0, and N represents the total number of client models after this round of training, w represents the change value of the jth position of the parameters of the ith client model after this round of training relative to the jth position of the parameters of the current server model, w i w represents the update weight parameter of the ith client model after this round of training. i is usually set to 1 / N.

[0058] In the above embodiment, the current server model is directly updated by the trained client model, and when the number of clients is large, the number of client models involved is large, and the possibility of conflict during updating is large, which reduces the training efficiency. Therefore, according to another embodiment of the present application, when the number of clients exceeds a predetermined number threshold, steps b1 and b2 are used to aggregate parameters in S2. Wherein:

[0059] In step b1, the plurality of client models after this round of training are grouped, and each group obtained after grouping is aggregated within the group to obtain a representative model of each group.

[0060] According to an embodiment of the present application, the grouping and the aggregation within each group obtained after grouping include steps b11, b12 and b13:

[0061] In step b11, the feature mean and feature variance of all samples calculated by each client using the features of all samples in the local training set extracted from the server model received in the last round are obtained. Wherein, the way the client determines the feature mean and feature variance of all samples includes: based on the features of all samples and the total number of samples in the feature sequence of the client, the feature mean of all samples is calculated; based on the features of all samples, the feature mean and the total number of samples in the feature sequence of the client, the feature variance of all samples is calculated.

[0062] According to one embodiment of the present application, the feature sequence of the client is extracted by using the last received server model to extract the feature of each sample of the local training set of the client, and the set of features of all samples is taken as the feature sequence of the corresponding client. The last received server model is the last updated server model, and the feature of each sample is the output of the previous neural network layer of the final output layer after the input data of the sample is input into the last updated server model. The feature sequence of the local training set of each client is extracted locally by each client using the last updated server model and uploaded to the server. Because the last updated server model has been distributed to each client during the last round of training, the sample data will not be leaked to the server when the feature sequence of the local training set of each client is extracted, ensuring data security and reducing transmission cost by only transmitting the feature mean and feature variance. The features of all samples of the client are extracted by using the same server model, and the features of the samples obtained are meaningful, the grouping is more accurate, and the grouping is determined once before each aggregation of the server model. The more accurate the server model obtained after multiple rounds of training is, the more accurate the grouping obtained is, and the higher the accuracy of the final obtained server model is.

[0063] According to one embodiment of the present application, the generation method of the feature sequence of the i-th client is as follows formula (4):

[0064]

[0065] Wherein, represents the feature sequence of the i-th client, represents the feature extractor of the last updated server model, represents the feature of the first sample used to train the current i-th client model, represents the feature of the second sample used to train the current i-th client model, represents the feature of the sample used to train the current i-th client model.

[0066] According to one embodiment of the present application, the calculation method of the feature mean of all samples corresponding to the client is as follows formula (5):

[0067]

[0068] Wherein, represents the feature mean of all samples of the i-th client, represents the feature of the a-th sample of the local training set of the i-th client model, represents the total number of samples of the i-th client.

[0069] According to an embodiment of the present application, the feature variance of all samples corresponding to the client is calculated in the following formula (6):

[0070]

[0071] wherein, denotes the feature variance of all samples of the i-th client.

[0072] In step b12, the plurality of client models after the current round of training are grouped based on the feature mean and the feature variance of all samples determined by each client, to obtain a plurality of groups.

[0073] According to an embodiment of the present application, the grouping manner comprises: inputting the feature mean and the feature variance corresponding to each client into a clustering algorithm for clustering to obtain a plurality of groups, wherein the clustering algorithm is a K-means clustering algorithm or a Gaussian mixture clustering algorithm. For example, the feature mean and the feature variance of each client are inputted into a K-means clustering algorithm to obtain K groups, K = 1, 2, … k, and the total number of client models in the k-th group is denoted as m k .

[0074] In step b13, the parameters of the plurality of client models after the current round of training in each group are aggregated to obtain a representative model of the group and the parameters of the representative model.

[0075] According to an embodiment of the present application, the manner of aggregating the parameters of the plurality of client models after the current round of training in each group is as follows:

[0076]

[0077] wherein, denotes the model parameter of the representative model of the k-th group in the t-th round, m k denotes the total number of client models after the current round of training in the k-th group, denotes the update weight parameter of the y-th client model after the current round of training corresponding to the group, denotes the parameter of the y-th client model after the current round of training corresponding to the group. The parameters of all client models in the k-th group are aggregated by the above manner to obtain the representative model of the k-th group, denoted as the representative model the parameter of the j-th position of the parameter of the representative model all positions of the trainable parameter The superscript g of each symbol is used to represent the representative model after grouping.

[0078] In step b2, inter-group aggregation is performed based on the representative model of each group to obtain an aggregated result, wherein the inter-group aggregation comprises the following steps b21, b22 and b23:

[0079] In step b21, the parameter change value of each position of each representative model relative to the corresponding position of the current server model is calculated, as shown in the following formula (8).

[0080]

[0081] wherein, represents the representative model of the kth group in the current round relative to the parameter change value of the server model at all positions, represents the parameter change value of the 0th position relative to , represents the parameter change value of the 1st position relative to , represents the parameter change value of the jth position relative to , represents the parameter change value of the Mth position relative to .

[0082] In step b22, for each position in all representative models, all parameters are obtained, and the consistency index of the parameter update direction of the position is calculated according to the number of positive parameter change values and the number of negative parameter change values at the corresponding position.

[0083] According to an embodiment of the present application, for each position in all representative models, the absolute value of the difference between the number of positive parameter change values and the number of negative parameter change values at the corresponding position is taken as the consistency index of the parameter update direction of the corresponding position. The consistency index of the parameter update direction of the corresponding position can be calculated in the same way as formula (2) of the above embodiment, that is, the parameter change values at the corresponding position in the representative model of each group are brought into formula (2), and the consistency index of the jth position in all representative models is obtained.

[0084] In step b23, the parameters at positions with consistency indexes greater than or equal to the predetermined score in all representative models are aggregated to obtain an aggregated result. In the present application, when the consistency index is greater than or equal to the predetermined score, it indicates that the parameter update directions of all representative models for the corresponding positions are mostly the same. At this time, the parameters of the positions in the representative models are referred to update the server model, which greatly improves the universality of the server model. At the same time, the parameters of the representative models are referred to update the server model, which avoids the possibility of conflict when the number of clients is too large, resulting in a large range of mask(score j ) being 0, reducing learning efficiency and other problems.

[0085] According to an embodiment of the present application, the parameters of the current server model can be updated in the same way as formula (3) of the above embodiment in step S2, and the update method is as follows:

[0086]

[0087] wherein, represents the parameter of the jth position obtained in the tth update, represents the change value of the jth position of the ith representative model in the current round relative to the current server model, represents the update weight parameter of the ith representative model in the current round.

[0088] According to an embodiment of the present application, the update weight parameter of the kth group of representative models is calculated as follows:

[0089]

[0090] wherein, is usually set to 1 / N, and N represents the total number of client models.

[0091] In step S3, each client determines the universal parameters of the client model after the current round of training based on the client model before the current round of training, the client model after the current round of training, and the server model obtained in the current round, and updates the universal parameters in each client model according to the server model in the current round to obtain the client model after the current round of update.

[0092] According to an embodiment of the present application, step S3 includes steps S31, S32, S33 and S34, wherein:

[0093] In step S31, the training change value of each position of the client model after the current round of training and the corresponding position of the client model before the current round of training is calculated.

[0094] According to one embodiment of the present application, the training change value of the parameters of the corresponding positions is calculated using the following formula (10):

[0095]

[0096] wherein Δ0 represents the training change value of the trainable parameters of all positions of the i-th client model after training, represents the trainable parameters of all positions of the i-th client model after t-1 rounds of training, 0,0 represents the training change value of the parameters of the 0-th position, Δ 0,1 represents the training change value of the parameters of the 1-st position, Δ 0,j represents the training change value of the parameters of the j-th position, Δ 0,M represents the training change value of the parameters of the M-th position.

[0097] In step S32, the aggregated change value of the parameters of each position of the updated server model and the parameters of the corresponding positions of the client model before the current round of training is calculated using the following formula (11):

[0098]

[0099] wherein Δ1 represents the aggregated change value of the trainable parameters of all positions, Δ 1,0 represents the training change value of the parameters of the 0-th position, Δ 1,1 represents the training change value of the parameters of the 1-st position, Δ 1,j represents the training change value of the parameters of the j-th position, Δ 1,M represents the training change value of the parameters of the M-th position.

[0100] In step S33, the common parameters of each client model are determined according to the training change value and the aggregated change value corresponding to each parameter.

[0101] According to one embodiment of the present application, in step S33, the parameters with the same sign of the training change value and the aggregated change value are common parameters, and the parameters with different signs are personalized parameters, wherein the sign refers to the sign indicating whether the numerical value is positive or negative. The sign of the training change value and the aggregated change value can be obtained by the following formula (12):

[0102]

[0103] wherein sign(x) is a sign function, 1 indicates that the sign of x is positive, -1 indicates that the sign of x is negative, the training change value of the parameters of the j-th position is Δ 0,j and the aggregated change value is Δ 1,j ​The sign function sign(x) is introduced to determine whether the signs of the training change value and the aggregated change value are the same.

[0104] In step S34, the general parameters in the client model after this round of training are updated according to the latest server model to obtain an updated client model. The method for updating the trained client model is as follows formula (13):

[0105]

[0106] where λ is a hyperparameter, is the jth position parameter of the ith client model.

[0107] According to another embodiment of the present application, after steps S31 and S32 are performed, in step S33, the parameters of each client model are determined to be general or personalized parameters according to the training change value and the aggregated change value corresponding to each parameter. The parameters with different signs of the training change value and the aggregated change value are personalized parameters. In step S34, the general parameters in the client model after this round of training are updated according to the latest server model, and the personalized parameters are not updated, to obtain an updated client model. The accuracy of the client model is ensured, and the generalization and personalization requirements of the model are also considered.

[0108] According to one embodiment of the present application, the final server model and the final client model are obtained by completing multiple rounds of federated training according to the above embodiments.

[0109] According to one embodiment of the present application, the client model or the server model is used for image classification prediction to obtain the classification result of the image. The image classification prediction includes: in the field of lung CT image recognition, the labels of image classification include virus infection, bacteria infection and normal; in the field of road image recognition, the labels of image classification include cat, dog, person and car.

[0110] According to one embodiment of the present application, a domain generalization federated system is provided, which includes: a server for obtaining a final server model based on the method described in the above embodiments; and a client, which is deployed with a client model updated according to the above method. In addition, according to other embodiments of the present application, after the final server model is obtained on the server, if a new client appears, the final server model can be directly deployed to the new client.

[0111] ​According to another embodiment of the present application, another domain generalization federated system is provided, comprising: a server, configured to obtain a final server model based on the method of the above-mentioned embodiment, and distribute the server model to each client; and a client, on which the obtained server model is deployed, for classification prediction of input data to be predicted by the client.

[0112] To further verify the effectiveness of the federated learning method of the present application, the inventors conducted a simulation experiment. The specific experimental process is as follows:

[0113] First, select a picture dataset: VLCS [Fang, Chen, Ye Xu, and Daniel N. Rockmore. "Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias." Proceedings of the IEEE International Conference on Computer Vision. 2013.], which contains four sub-datasets Caltech101, LabelMe, SUN09, and VOC2007. Each sub-dataset corresponds to the data of a client, and there are four clients. During training, only LabelMe, SUN09, and VOC2007 participate in training. For each client participating in training, 10% of the data is used for model training, 20% for verification, and 20% for testing. For Caltech101, which does not participate in training, it is used as a newly added client to verify the generalization ability of the final server model. Among them, AlexNet is used as the initial server model. Stochastic Gradient Descent (SGD) is used for optimization, and the learning rate is 0.01.

[0114] Second, select existing methods for comparison, including:

[0115] Method 1: FedAVG [McMahan, Brendan, et al. "Communication-efficient learning of deep networks from decentralized data." Artificial intelligence and statistics. PMLR, 2017.].

[0116] Method two: FedProx [Li, Tian, et al. "Federated optimization in heterogeneous networks." Proceedings of Machine Learning and Systems 2 (2020): 429-450.].

[0117] Method three: FedBN [Li, Xiaoxiao, et al. "Fedbn: Federated learning on non-iid features via local batch normalization." arXiv preprint arXiv:2102.07623 (2021).].

[0118] Then, the test methods in the experiment include:

[0119] Test method one: use the final server model obtained by each method to predict on the corresponding client of the Caltech101 sub-data set that did not participate in training, to test the generalization ability, and record the accuracy.

[0120] Test method two: use the final client model of each client (the client is one corresponding to the LabelMe, SUN09, VOC2007 sub-data set respectively) obtained by each method to predict on the corresponding client test data set, to test the individualization ability, record the accuracy, and finally compare the average prediction accuracy of all clients.

[0121] Finally, the experimental results are analyzed as shown in Table 1 below:

[0122] Table 1

[0123]

[0124] In the above Table 1, the first column of data is the prediction accuracy of each method in the sub-data set Caltech101 based on the test method one, the second column of data is the prediction accuracy of each method in the sub-data set LabelMe based on the test method two, the third column of data is the prediction accuracy of each method in the corresponding sub-data set SUN09 based on the test method two, the fourth column of data is the prediction accuracy of each method in the sub-data set VOC2007 based on the test method two, and the final average prediction accuracy is the average of the prediction accuracy of each method based on the above four sub-data sets. According to the prediction accuracy of each method corresponding to the sub-data set Caltech101, it can be seen that the method of the application has the best accuracy in terms of generalization ability. According to the prediction accuracy of each method corresponding to the sub-data sets LabelMe, SUN09 and VOC2007, it can be seen that the method of the application is much better than the methods one, two and three in terms of individualization ability. Therefore, the method of the application is higher than the existing methods in terms of individualization ability and generalization ability, which fully illustrates the beneficial effects of the method of the application.

[0125] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of the steps can be performed concurrently or even in a different order, as long as the desired function can be achieved.

[0126] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0127] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a magneto-optical or other optical device, a portable

[0128] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments covered by the claims. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the spirit and scope of the described embodiments. It is intended that the scope of the application should only be limited by the appended claims.

Claims

1. A federated learning method, the method comprising distributing, by a server, an initialized server model to a plurality of clients as initial client models, and completing, by the server and the clients in cooperation, multiple rounds of federated training, wherein, Each round of federated training includes: S1, training the client model of each client with its local training set to obtain a plurality of client models after this round of training and upload to the server, wherein the input data dimensions of the samples in the local training sets of different clients are the same and the label type sets are the same, the current client model of the first round of training is the initial client model, and the current client model of other rounds of training is the client model updated after the last round of training; S2, determining the consistency index of the parameter update direction at each position in the plurality of client models based on the plurality of client models uploaded in step S1, and aggregating the parameters at the corresponding positions of the plurality of client models based on the consistency index to update the parameters of the current server model and deliver to the plurality of clients; wherein when the number of clients is less than or equal to a predetermined number threshold, S2 includes: calculating the parameter change value of each position of each client model after this round of training and the corresponding position of the current server model; for all parameters at each position in all client models after this round of training, calculating the consistency index of the parameter update direction at this position according to the number of positive parameter change values and the number of negative parameter change values at the corresponding position; the consistency index of the parameter update direction is calculated as follows: wherein, an indicator of consistency of the parameters of the jth position, denotes the total number of client models after this round of training, , denotes the change value of the jth position of the ith client model after this round of training relative to the parameter at the jth position of the current server model. aggregating the parameters at the positions with consistency index greater than or equal to a predetermined score in all client models after this round of training to obtain the aggregation result; S3, determining the general parameters of the client model after the current round of training based on the client model before the current round of training, the client model after the current round of training and the server model obtained in the current round by each client, and updating the general parameters in each client model according to the server model of the current round to obtain the client model after the current round of update.

2. The method of claim 1, wherein, The parameters of the current server model are updated in step S2 as follows: wherein, denotes the parameter of the jth position obtained in the tth round of update, denotes the parameter of the jth position of the server model obtained in the (t-1)th round of update, denotes the consistency index of the parameter of the jth position, denotes the mask indicating whether the parameter of the jth position of the current server model needs to be updated, the value of the mask corresponding to the parameter with the consistency index greater than or equal to a predetermined score is set to 1, and the value of the mask corresponding to the rest of the parameters is set to 0, denotes the total number of client models after the current round of training, denotes the change value of the parameter of the jth position of the ith client model after the current round of training relative to the parameter at the jth position of the current server model, denotes the update weight parameter of the ith client model after the current round of training.

3. The method of claim 1, wherein, when the number of clients exceeds the predetermined number threshold, S2 includes: grouping the plurality of client models after this round of training and performing intra-group aggregation on each group obtained after grouping to obtain a representative model of each group; inter-group aggregation based on the representative model of each group to obtain an aggregation result, wherein inter-group aggregation includes the following steps: calculating the parameter change value of each position of each representative model and the corresponding position of the current server model; for all parameters at each position in all representative models, calculating the consistency index of the parameter update direction at this position according to the number of positive parameter change values and the number of negative parameter change values at the corresponding position; aggregating the parameters at the positions with consistency index greater than or equal to a predetermined score in all representative models to obtain the aggregation result.

4. The method of claim 3, wherein, The grouping and intra-group aggregation of each group obtained after grouping include: obtaining the feature mean and feature variance of all samples calculated by each client using the server model received in the last round to extract the features of all samples in its local training set; grouping the plurality of client models after the current round of training based on the feature mean and the feature variance of all samples determined by each client to obtain a plurality of groups; performing parameter aggregation on the parameters of the plurality of client models after the current round of training in each group to obtain a representative model of the group.

5. The method of claim 4, wherein, The grouping manner comprises: inputting the feature mean and the feature variance of all samples determined by each client into a clustering algorithm to obtain a plurality of groups, wherein the clustering algorithm is a K-means clustering algorithm or a Gaussian mixture clustering algorithm.

6. The method of claim 4, wherein, The parameter aggregation manner on the parameters of the plurality of client models after the current round of training in each group is as follows: wherein, denotes the model parameters of the representative model of the kth group in the tth round, denotes the total number of client models after this round of training of the kth group, denotes the update weight parameter of the ith client model after this round of training of the corresponding group, denotes the parameter of the ith client model after this round of training of the corresponding group.

7. The method of claim 1, wherein, Step S3 comprises: S31, calculating the training change value of each position of the parameters of the client model after the current round of training and the corresponding position of the parameters of the client model before the current round of training; S32, calculating the aggregation change value of each position of the parameters of the updated server model and the corresponding position of the parameters of the client model before the current round of training; S33, determining the general parameters of each client model according to the training change value and the aggregation change value corresponding to each parameter; S34, updating the general parameters in the client model after the current round of training according to the updated server model to obtain an updated client model.

8. The method of claim 7, wherein, In step S33, the parameters with the same sign of the training change value and the aggregation change value are general parameters, and the parameters with different signs are personalized parameters, wherein the sign refers to a sign indicating a positive number or a negative number.

9. The method according to any one of claims 1 to 8, characterized in that, The image is classified and predicted by using the client model or the server model to obtain a classification result of the image.

10. The method of claim 9, wherein, The classification prediction of the image comprises: In the field of lung CT image recognition, the labels of image classification include infection of virus, infection of bacteria and normal; or In the field of road image recognition, the labels of image classification include cat, dog, human and car.

11. A domain generalization federated system, comprising: It comprises: a server for obtaining a final server model based on the method of any one of claims 1-10, and distributing the server model to the client; a client for deploying the server model on the client for classification prediction of the input data to be predicted of the client.

12. A domain generalization federated system, comprising: It comprises: a server for obtaining a final server model based on the method of any one of claims 1-10, and distributing the server model to the client; a client for deploying the server model on the client, obtaining a final client model based on the method of any one of claims 1-10, and using the final client model for classification prediction of the input data to be predicted of the client.

13. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method of any one of claims 1-10.

14. An electronic device, comprising: It comprises: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method of any one of claims 1-10 by executing the executable instructions.

Images