Federated learning method and system for scenarios where data is not independent and identically distributed

Abstract

The present invention discloses a federated learning method and system for data non-independent and identically distributed scenarios, including multiple clients and a central server; the central server is used to divide a target data set into multiple sub-data sets in a non-independently distributed manner so that each sub-data set contains all types of data, and distributes the sub-data sets to clients; the client is used to guide the sub-data set to train the current local model based on the received sub-data set according to the current local anchor point, and update the local anchor point and local model parameters, and upload the model data to the central server according to an agreed communication method; the central server is also used to aggregate the received model data to obtain aggregated data, and transmit the aggregated data to the client according to the agreed communication method as the basis for the next round of federated learning. On the basis of ensuring the security of user data, the method improves the practicality of the federated learning system in specific scenarios, and at the same time solves the communication efficiency and statistical heterogeneity problems of the federated learning system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and information security, and in particular relates to a federated learning method and system for scenarios where data are not independent and identically distributed. Background Art

[0002] With the continuous in-depth application of new technologies such as big data, artificial intelligence, and cloud computing in various industries, global data has shown explosive growth and massive aggregation, and the value of data has become increasingly prominent. The circulation and transaction of data as a factor of production faces two key problems: right confirmation and privacy protection. Data is essentially information, which is not exclusive or exclusive, and can be owned by most people at the same time. In the digital economy era, the marginal cost of disseminating personal information is almost zero, and it can spread quickly throughout the world. This low cost makes data protection face special difficulties. At present, companies and organizations are increasingly collecting detailed information about users. On the one hand, valuable information can be extracted from these raw data belonging to different organizations, which can be used to improve the quality of products, services, and benefits through machine learning technology; on the other hand, there will be potential abuse and attacks in distributed scenarios, which poses a great challenge to data privacy and security. The traditional way of collecting data from the user end and integrating it to train machine learning models is worried that it may violate privacy.

[0003] Federated Learning (FL) provides a flexible solution that allows machine learning applications to perform distributed learning in a way that keeps private data locally, allowing multiple private data owners to participate in the collaborative training of common models without violating legal and ethical requirements. Federated learning allows different organizations and devices to jointly participate in the training and construction of machine learning models, and the data used by these organizations and devices is stored locally. Federated learning algorithms do not infringe on user privacy data while ensuring the model training process. Users can mine the value of data information locally and ultimately build and use machine learning models.

[0004] The FedAvg algorithm proposed by McMahan et al. is the most widely used basis for federated learning. It reduces communication costs by training multiple iterations locally on the client. The main challenges involved in federated learning include communication efficiency, system heterogeneity, statistical heterogeneity, and privacy. In order to reduce the communication cost in federated learning, some studies have suggested using data compression techniques such as quantization and sketching, while others have suggested using split learning. In order to solve the problem of system heterogeneity, asynchronous communication and client-side active sampling techniques have been proposed. Statistical heterogeneity is a hot topic in current federated learning research. One research trend is to adjust the global model to adapt to personalized local models for non-independent and identically distributed data, for example, by combining federated learning with assisted learning, meta-learning, multi-tasklearning, transfer learning, and knowledge distillation. However, these personalized methods usually introduce additional computational and communication overheads that may be unnecessary, and the final effect is not satisfactory. Summary of the invention

[0005] In view of the above, the purpose of the present invention is to provide a federated learning method and system for scenarios where data are not independent and identically distributed, which, on the basis of ensuring the security of user data, further improves the practicality of the federated learning system in specific scenarios, while solving the communication efficiency and statistical heterogeneity problems of the federated learning system.

[0006] To achieve the above-mentioned purpose of the invention, the embodiment provides a federated learning system for a data non-independent and identically distributed scenario, including multiple clients and a central server, each client having a communication channel with the central server;

[0007] The central server is used to divide the target data set into multiple sub-data sets in a non-independent distribution manner so that each sub-data set contains all types of data, and is also used to distribute the sub-data sets to the clients so that each client has one sub-data set;

[0008] The client is used to train the current local model based on the received sub-dataset and the current global anchor point to guide the sub-dataset, and update the local anchor point and local model parameters, and upload the model data to the central server according to the agreed communication method, wherein the local anchor point is a feature point that can represent the classification category determined according to the feature vector corresponding to the local data, and the model data includes the new local anchor point and local model parameters;

[0009] The central server is also used to aggregate the received model data to obtain aggregated data, and transmit the aggregated data to the client according to the agreed communication method as the basis for the next round of federated learning, where the aggregated data includes the global anchor points obtained by aggregating all new local anchor points and the global model parameters obtained by aggregating all local model parameters.

[0010] In one embodiment, when the central server divides the target data set into multiple sub-data sets in a non-independent distribution manner, a label-based non-independent distribution manner is adopted, including:

[0011] First, the row and column parameters of the double random matrix are set according to the number of clients and the types of data labels in the federated learning system;

[0012] Then, a double random matrix is ​​generated according to the set row parameters and column parameters;

[0013] Finally, the target data set is divided according to the generated double random matrix so that each sub-dataset contains all types of data.

[0014] In one embodiment, when the client trains the current local model based on the current global anchor point guidance sub-dataset, the loss function total loss used is:

[0015] total loss=base loss+α*anchor loss+β*triplet loss

[0016] Among them, α and β are hyperparameters, and base loss is the task loss for the classification prediction task based on the classification prediction results output by the local model;

[0017] Anchor loss is a guidance loss that guides local model training based on the feature vector corresponding to the current local anchor point and each type of data;

[0018] Triplet loss is the contrast loss between feature vectors of different categories.

[0019] In one embodiment, the anchor loss is expressed as:

[0020]

[0021] d i =distance(embedding i ,local anchor i )

[0022] Among them, i is the category index, k is the total number of categories, and y i represents the label of the i-th category, di Represents the feature vector embedding generated by the i-th type of data i With the current local anchor local anchor i The distance between them, distance() represents the distance calculation operation, including the Euclidean distance calculation operation or the cosine similarity operation, and CrossEntropyLoss() represents the cross entropy loss function;

[0023] The contrast loss triplet loss is expressed as:

[0024]

[0025] d ij =distance(embedding i ,embedding j )

[0026] Among them, j is the category index, y j represents the label of the jth category, margin represents the preset threshold, which ranges from 0 to 1, max{} represents the maximum value function, and d ij Represents the feature vector embedding generated by the i-th type of data i embedding with the feature vector generated by the j-th data j The distance between.

[0027] In one embodiment, when training a local model, the client uses a momentum optimization algorithm to update the local anchor point. The specific formula is:

[0028] local anchor i '=γ*embedding i +(1-γ)*local anchor i ,i=1,...,k

[0029] Among them, i is the category index, k is the total number of categories, and local anchor i ' represents the new local anchor point corresponding to the i-th type of data, local anchor i Indicates the current local anchor point corresponding to the i-th type of data, embedding i Represents the feature vector embedding generated by the i-th type of data i , γ represents the anchor point update momentum, which is the ratio of the number of batches of data in the local training process batchsize to the total amount of data in the local training process data size, that is,

[0030] In one embodiment, the central server aggregates all local model parameters in the following manner:

[0031]

[0032]

[0033] Where n represents the local model index, N is the total number of local models, global model represents the global model parameters, local model n Represents the nth local model parameter, local weight n Represents the aggregate weight of the nth local model parameter, local data n It indicates the data size of the sub-dataset for training the nth local model, and total data indicates the data size of the target dataset.

[0034] In one embodiment, the central server aggregates all new local anchor points in the following manner:

[0035]

[0036] Among them, global anchor represents the global anchor point, local anchor nj Represents the new local anchor point of the j-th label corresponding to the n-th local model, anchor weight nj The anchor weight of the j-th label corresponding to the n-th local model is calculated as follows:

[0037]

[0038] in, Indicates the amount of data with j-type labels in the i-th client, num label j Indicates the total amount of data of the jth label of all clients.

[0039] In one embodiment, before federated learning, the central server is further used to construct a federated learning task and send it to each client, wherein the federated learning task includes an initial local model, an initial local anchor point, and federated learning hyperparameters, wherein the federated learning hyperparameters include a gradient descent algorithm optimizer, a local training learning rate, the total number of clients participating in federated learning, the number of rounds of local training of the client per round, the total number of communication rounds, the spatial dimension of the local anchor point, the weight decay rate, and the agreed communication method; the client and the central server perform federated learning according to the federated learning task;

[0040] Among them, the number of initial local anchor points is equal to the number of label types of the target dataset;

[0041] The local model includes a feature extraction network, a projection network and an output layer, wherein the feature extraction network is used to extract the feature map of the input sample data, the projection network is used to project the feature map into the feature projection space to obtain the feature vector, and the output layer is used to perform prediction calculations based on the feature vector and classify the data to predict the results.

[0042] In one embodiment, the communication modes include four types:

[0043] In the first case, the uploaded model data includes local model parameters and new local anchor points, and the downloaded aggregated data includes global model parameters and global anchor points.

[0044] In the second case, the uploaded model data includes the new local anchor point, and the downloaded aggregated data includes the global model parameters and the global anchor point;

[0045] The third type is that the uploaded model data includes local model parameters and new local anchor points, and the downloaded aggregated data includes global anchor points;

[0046] The fourth type is that the uploaded model data includes the new local anchor point, and the downloaded aggregated data includes the global anchor point;

[0047] When the central server and the client choose the first communication mode to communicate, after training the model locally, the client uploads the local model parameters and the new local anchor point to the central server at the same time; the central server aggregates all local model parameters and the new local anchor point to obtain the global model parameters and the global anchor point, and transmits the global model parameters and the global anchor point to the client as the current local model and the current global anchor point for the next round of local training;

[0048] When the central server and the client choose the second communication mode to communicate, the client uploads the new local anchor point to the central server after training the model locally; the central server aggregates all new local anchor points to obtain the global anchor point, and transmits the historically stored global model parameters and global anchor point to the client as the current local model and current global anchor point for the next round of local training;

[0049] When the central server and the client choose the third communication mode to communicate, after the client trains the model locally, it uploads the local model parameters and the new local anchor point to the central server at the same time; the central server aggregates all local model parameters and the new local anchor point to obtain the global model parameters and the global anchor point, and transmits the global anchor point to the client as the current global anchor point for the next round of local training;

[0050] When the central server and the client choose the fourth communication mode to communicate, the client uploads the new local anchor point to the central server after training the model locally; the central server aggregates all local anchor points to obtain the global anchor point, and transmits the global anchor point to the client as the current global anchor point for the next round of local training.

[0051] To achieve the above-mentioned purpose of the invention, the embodiment further provides a federated learning method for a data non-independent and identically distributed scenario, wherein the federated learning method adopts the above-mentioned federated learning system, and the federated learning method includes:

[0052] The central server establishes a communication channel with the client, creates a federated learning task, and sends it to each client;

[0053] The central server divides the target data set into multiple sub-data sets in a non-independent distribution manner so that each sub-data set contains all types of data, and distributes the sub-data sets to the clients so that each client has one sub-data set;

[0054] Based on the received sub-dataset, the client trains the current local model according to the global anchor point as the current local anchor point, updates the local anchor point and local model parameters, and uploads the model data to the central server according to the agreed communication method, where the local anchor point is the feature point that can represent the classification category in the feature vector corresponding to each type of data, and the model data includes the new local anchor point and local model parameters;

[0055] The central server aggregates the received model data to obtain aggregated data, and transmits the aggregated data to the client according to the agreed communication method as the basis for the next round of federated learning. The aggregated data includes the global anchor points obtained by aggregating all new local anchor points and the global model parameters obtained by aggregating all local model parameters.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The central server divides the target data set into multiple sub-data sets in a non-independent distribution manner so that each sub-data set contains all types of data, and sends each sub-data set to each client, so that the sub-data set owned by each client contains all types of data and does not communicate data with other clients, forming a scenario suitable for federated learning, ensuring full utilization of the target data set in the federated learning scenario, and improving the utilization rate of the target data set and the accuracy of federated learning;

[0058] The client trains the current local model based on the local anchor point guidance sub-dataset. Since the local anchor point is a feature point that can represent the classification category extracted from the feature map corresponding to the local data, the use of local anchor point guidance and constraint training can improve the efficiency of federated learning. At the same time, the central server aggregates the local anchor points uploaded by the client, and then sends the global anchor point to each client for the next round of federated learning. In this way, the global anchor point realizes the guidance and constraint of the global data distribution.

[0059] In the process of federated learning, local anchor points that condense local data knowledge are uploaded to the central server instead of local models for global information exchange. Since the local anchor points are at least 1000 orders of magnitude lower than the local model parameters, the communication cost is greatly reduced while ensuring the accuracy of model training. In addition, the central parameter server and local clients can adaptively adjust the specific communication method according to their demands based on the communication method.

[0060] Since the local anchor point is a feature point that is closely related to the feature vector of the data and can represent the label of each category, the training method based on the guidance of the local anchor point is compatible with various existing federated aggregation algorithms, does not constitute a conflict, and has good compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 It is a flowchart of a federated learning system for a data non-independent and identically distributed scenario provided by an embodiment;

[0063] Figure 2 This is a diagram of the principle of federated learning provided by the embodiment, wherein ① the central server transmits the global model and global anchor point to the client. ② the client uses the sub-dataset to train the local model. ③ the sub-dataset outputs the prediction result after passing through the local model. and feature vector embedding. ④ Update the local anchor point using feature vector embedding. ⑤ The client transmits the local model parameters and the new local anchor point to the central server. ⑥ The central server aggregates the local model parameters and the new local anchor point using the model weight and anchor weight to obtain a new global model and global anchor point;

[0064] Figure 3 It is a flowchart of a federated learning method for a data non-independent and identically distributed scenario provided by an embodiment. DETAILED DESCRIPTION

[0065] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0066] Figure 1 It is a flowchart of a federated learning system for a data non-independent and identically distributed scenario provided by an embodiment. Figure 2 Schematic diagram of federated learning provided by the embodiment. Figure 1 and Figure 2 As shown, the federated learning system for data non-independent and identically distributed scenarios provided by the embodiment includes a central server and multiple clients, wherein the client can be any electronic device with computing power, including but not limited to smart watches, smart phones, various computers, etc. The central server can be a terminal server located in the terminal, or it can be a cloud server located in the cloud. The central server has a communication channel with each client participating in the federated learning to achieve secure communication of data and messages. The cooperation between the client and the central server realizes the federated learning task in the data non-independent and identically distributed scenario.

[0067] In an embodiment, the central server serves as the instructor of the federated learning task for the data non-independent and identically distributed scenario, and is used to divide the target data set into multiple sub-data sets in a non-independently distributed manner so that each sub-data set contains all types of data, and is also used to distribute the sub-data sets to the clients so that each client has one sub-data set.

[0068] In the existing method, a non-independent distribution method based on data is adopted, that is, all data of the target data set are randomly sampled according to the Dirichlet distribution, and the sampling results are distributed to each client. In this way, the data types contained in the sub-datasets distributed to each client are uneven. Some clients are allocated sub-datasets that contain data of all class labels, while some clients are allocated sub-datasets that do not contain data of all class labels. This will lead to poor federated learning effects of sub-datasets based on uneven data distribution and poor robustness of the obtained model.

[0069] To this end, in an embodiment, when the central server divides the target data set into multiple sub-data sets in a non-independent distribution manner, a label-based non-independent distribution method is adopted, including: first, setting the row parameters and column parameters of the double random matrix according to the number of clients and the types of data labels in the federated learning system; then, generating the double random matrix according to the set row parameters and column parameters; finally, dividing the target data set according to the generated double random matrix so that each sub-data set contains all types of data.

[0070] In the embodiment, the dimension (n, k) of the double random matrix is ​​determined according to the number of clients and the type of data tags. In probability and combinatorial mathematics, the double random matrix is ​​a square matrix of non-negative real numbers, and the sum of each row and each column is 1, that is:

[0071]

[0072] In the embodiment, the BIRKHOFF-VON NEUMANN algorithm is used to generate a double random matrix. The algorithm shows that in the double random matrix X, there are θ1,...,θ k ≥0, and permutation matrices P1,...,P k , so that X = θ1P1+…+θ k P k .x ij represents the value of the element in the i-th row and j-th column of the double random matrix, representing the probability of data assigned to the i-th client and belonging to the k-th label. n and k represent the number of clients and the type of data label, respectively. Specifically, when generating the double random matrix X, random sampling based on Dirichlet distribution generates θ1,...,θ k ≥0, Generate permutation matrices P1,...,P based on PCG-64 pseudo-random number generator k , PCG-64 is a 128-bit implementation of an O'Neill permutation congruential generator. The generated doubly random matrix X satisfies Can be used for partitioning of target datasets.

[0073] In the embodiment, the generated double random matrix X is used to partition the target data set so that each non-independent and identically distributed sub-data set contains data of all labels, which is beneficial to federated learning in non-independent and identically distributed scenarios.

[0074] In the embodiment, the central server, as the instructor of the federated learning system, is also used to construct a federated learning task and send it to each client to organize the federated learning. The federated learning task includes the initial local model, the initial global anchor point, and the federated learning hyperparameters, wherein the federated learning hyperparameters include the gradient descent algorithm optimizer, the local training learning rate, the total number of clients participating in the federated learning, the number of local training rounds of the client, the total number of communication rounds, the spatial dimension of the global anchor point, the weight decay rate, the agreed communication method, the training termination condition, etc. The client and the central server perform federated learning according to the federated learning task; the training termination condition includes reaching the total number of communication rounds or the global model convergence.

[0075] In the embodiment, the PCG-64 pseudo-random number generator is used to randomly initialize the global anchor point according to the data label type to generate an (n, k, a)-dimensional matrix. Where n represents the number of clients, k represents the label type of the target data set, a represents the global anchor point dimension, and the initial number of global anchor points is equal to the label type of the target data set. The global anchor point dimension can be set to 64, 128, or 256.

[0076] The local model includes a feature extraction network, a projection network and an output layer, wherein the feature extraction network is used to extract the feature map of the input sample data, and the projection network is used to project the feature map into a feature projection space of limited dimension to obtain the corresponding feature vector. The output layer is used to perform prediction calculations based on the feature vector and classify the prediction results. Preferably, the feature extraction network can be a feature extraction network of a Simple-CNN network, a feature extraction network of Vgg-11, a feature extraction network of ResNet-18, a feature extraction network based on CNN, and a feature extraction network of ResNet-54. The projection network can adopt an MLP network. The output layer can adopt a fully connected neural network.

[0077] The client is used to train the current local model based on the received sub-dataset and the current global anchor point guidance value dataset, and update the local anchor point and local model parameters, and upload the model data to the central server according to the agreed communication method.

[0078] In the embodiment, when the client performs the initial round of federated learning, the current global anchor point is the initial global anchor point issued by the central server; when performing other rounds of federated learning other than the initial round, the global anchor point of the current round is the global anchor point issued by the central server. The local anchor point is a feature point that can represent the classification category determined according to the feature vector corresponding to the local data, and the model data includes a new local anchor point and local model parameters.

[0079] In the embodiment, when the client trains the current local model based on the current global anchor point guidance sub-dataset, the loss function total loss used is:

[0080] total loss=base loss+α*anchor loss+β*triplet loss

[0081] Among them, α and β are hyperparameters, and the preferred value is 0.01. Base loss is the task loss for the classification prediction task based on the classification prediction result output by the local model. Specifically, it can be the classification prediction result and the true value label The cross entropy loss function is expressed as

[0082] Anchor loss is a guidance loss for guiding local model training based on the feature vector corresponding to the current global anchor point and each type of data. Preferably, the guidance loss anchor loss is expressed as:

[0083]

[0084] d i =distance(embedding i ,global anchor i )

[0085] Among them, i is the category index, k is the total number of categories, and y i represents the label of the i-th category, d i Represents the feature vector embedding generated by the i-th type of data i With the current global anchor i The distance between them can be calculated using the 1-norm. distance() represents the distance calculation operation, including the Euclidean distance calculation operation or the cosine similarity operation. CrossEntropyLoss() represents the cross entropy loss function.

[0086] Triplet loss is the contrast loss between feature vectors of different categories. Preferably, the contrast loss tripletloss is expressed as:

[0087]

[0088] d ij =distance(embedding i ,embedding j )

[0089] Among them, j is the category index, y j represents the label of the jth category, margin represents the preset threshold, which ranges from 0 to 1, max{} represents the maximum value function, and d ij Represents the feature vector e ing generated by the i-th type of data i embedding with the feature vector generated by the j-th data j The distance between.

[0090] In the embodiment, when training the local model, the client uses the loss function total loss and performs back propagation to update the local model parameters, and uses the momentum optimization algorithm to update the local anchor point. The specific formula is:

[0091] local anchor i'=γ*embedding i +(1-γ)*local anchor i ,i=1,...,k

[0092] Among them, i is the category index, k is the total number of categories, and local anchor i ' represents the new local anchor point corresponding to the i-th type of data, γ represents the anchor point update momentum, which is the ratio of the batch size of each batch of data in the local training process to the total data size in the local training process, that is, embedding i Represents the feature vector embedding generated by the i-th type of data i , local anchor i Indicates the current local anchor point corresponding to the i-th type of data. It should be noted that the current local anchor point is the global anchor point sent by the central server in the previous round, that is, the local anchor of the client is initialized to the global anchor in each communication round, denoted as local anchor i =global anchor.

[0093] In the federated learning task, the central server is also used to aggregate the received model data to obtain aggregated data, and transmit the aggregated data to the client according to the agreed communication method as the basis for the next round of federated learning. The aggregated data includes the global anchor points obtained by aggregating all new local anchor points and the global model parameters obtained by aggregating all local model parameters.

[0094] In the embodiment, the central server aggregates all local model parameters in the following manner:

[0095]

[0096]

[0097] Where n represents the local model index, N is the total number of local models, global model represents the global model parameters, local model n Represents the nth local model parameter, local weight n Represents the aggregate weight of the nth local model parameter, local data n It indicates the data size of the sub-dataset for training the nth local model, and total data indicates the data size of the target dataset.

[0098] The data volume ratio between the sub-dataset and the target data set is used as the model aggregation weight, which has a more weighted value to obtain a more accurate global model.

[0099] In the embodiment, the central server aggregates all new local anchor points in the following manner:

[0100]

[0101] Among them, global anchor represents the global anchor point, local anchor nj Represents the new local anchor point of the j-th label corresponding to the n-th local model, anchor weight nj The anchor weight of the j-th label corresponding to the n-th local model is calculated as follows:

[0102]

[0103] in, Indicates the amount of data with j-type labels in the i-th client, num label j It indicates the total data volume of the jth class label of all clients. It should be noted that after the target data set is divided, the anchor weight of each client can be constructed.

[0104] The anchor weight is determined according to the data volume of each type of label of each client and the total data volume of this type of label in the target data set. The anchor weight has a more weighted value to obtain a more accurate global anchor.

[0105] In the federated learning system, four communication methods are provided. During federated learning, the client negotiates the communication method with the central server. Subsequent federated learning and model data upload and aggregated data delivery are carried out according to the negotiated communication method.

[0106] In the embodiment, the following four communication modes are provided:

[0107] In the first method, the uploaded model data includes local model parameters and new local anchor points, and the downloaded aggregated data includes global model parameters and global anchor points. When the central server and the client choose the first communication method to communicate, after training the model locally, the client uploads the local model parameters and the new local anchor point to the central server at the same time; the central server aggregates all local model parameters and new local anchor points to obtain global model parameters and global anchor points, and downloads the global model parameters and global anchor points to the client as the current local model and current global anchor point for the next round of local training.

[0108] In the second method, the uploaded model data includes new local anchor points, and the downloaded aggregated data includes global model parameters and global anchor points. When the central server and the client choose the second communication method, the client uploads the new local anchor point to the central server after training the model locally; the central server aggregates all new local anchor points to obtain the global anchor point, and downloads the historically stored global model parameters and global anchor point to the client as the current local model and current global anchor point for the next round of local training.

[0109] In the third method, the uploaded model data includes local model parameters and new local anchor points, and the downloaded aggregated data includes global anchor points. When the central server and the client choose the third communication method to communicate, after training the model locally, the client uploads the local model parameters and new local anchor points to the central server at the same time; the central server aggregates all local model parameters and new local anchor points to obtain global model parameters and global anchor points, and downloads the global anchor point to the client as the current global anchor point for the next round of local training.

[0110] The fourth type is that the uploaded model data includes the new local anchor point, and the downloaded aggregated data includes the global anchor point. When the central server and the client choose the fourth communication method to communicate, the client uploads the new local anchor point to the central server after training the model locally; the central server aggregates all local anchor points to obtain the global anchor point, and downloads the global anchor point to the client as the current global anchor point for the next round of local training.

[0111] It should be noted that different communication methods can be used in the entire federated learning process. For example, the number of communication rounds is 100, of which the first to 30 rounds use the first communication method, that is, the communication method of downloading / uploading model parameters and anchor points, the second communication method is used in the 31st to 60th rounds, that is, the communication method of downloading model parameters and anchor points and uploading anchor points, and the fourth communication method is used in the 61st to 100th rounds, that is, the communication method of downloading / uploading anchor points. Each client in the federated learning system communicates with the central server in multiple rounds according to the agreed communication method, and a high-precision global model is achieved through an efficient federated aggregation process.

[0112] Figure 3 is a flow chart of a federated learning method for a data non-independent and identically distributed scenario provided by an embodiment. Figure 3 As shown, the embodiment further provides a federated learning method for a scenario where data is not independent and identically distributed. The federated learning method adopts the above-mentioned federated learning system, and the federated learning method includes:

[0113] Step 1: The central server establishes a communication channel with the client. The central server creates a federated learning task and sends it to each client.

[0114] Step 2: The central server divides the target data set into multiple sub-data sets in a non-independent distribution manner so that each sub-data set contains all types of data, and distributes the sub-data sets to the clients so that each client has one sub-data set;

[0115] Step 3: Based on the received sub-dataset, the client trains the current local model according to the global anchor point as the current local anchor point, updates the local anchor point and local model parameters, and uploads the model data to the central server according to the agreed communication method;

[0116] Among them, the local anchor point is a feature point in the feature vector corresponding to each type of data that can represent the classification category, and the model data includes a new local anchor point and a local model parameter;

[0117] Step 4: The central server aggregates the received model data to obtain aggregated data, and transmits the aggregated data to the client according to the agreed communication method as the basis for the next round of federated learning;

[0118] The aggregated data includes the global anchor points obtained by aggregating all new local anchor points and the global model parameters obtained by aggregating all local model parameters.

[0119] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A federated learning system for scenarios where data is not independent and identically distributed, characterized in that: It includes multiple clients and a central server, and each client has a communication channel with the central server; The central server is used to divide the target data set into multiple sub-data sets in a non-independent distribution manner so that each sub-data set contains all types of data, and is also used to distribute the sub-data sets to the clients so that each client has one sub-data set; The client is used to train the current local model based on the received sub-dataset and the current global anchor point to guide the sub-dataset, and update the local anchor point and local model parameters, and upload the model data to the central server according to the agreed communication method, wherein the local anchor point is a feature point that can represent the classification category determined according to the feature vector corresponding to the local data, and the model data includes the new local anchor point and local model parameters; The central server is also used to aggregate the received model data to obtain aggregated data, and transmit the aggregated data to the client according to the agreed communication method as the basis for the next round of federated learning, where the aggregated data includes the global anchor points obtained by aggregating all new local anchor points and the global model parameters obtained by aggregating all local model parameters.

2. The federated learning system for data non-IID scenarios according to claim 1, characterized in that: When the central server divides the target data set into multiple sub-data sets in a non-independent distribution manner, a label-based non-independent distribution method is adopted, including: First, the row and column parameters of the double random matrix are set according to the number of clients and the types of data labels in the federated learning system; Then, a double random matrix is ​​generated according to the set row parameters and column parameters; Finally, the target data set is divided according to the generated double random matrix so that each sub-dataset contains all types of data.

3. According to the federated learning system for data non-IID scenarios of claim 1, when the client trains the current local model based on the current global anchor point guiding the sub-dataset, the loss function total loss used is: total loss=base loss+α*anchor loss+β*triplet loss in, α and β are hyperparameters, and base loss is the task loss for the classification prediction task based on the classification prediction results output by the local model; Anchor loss is a guidance loss that guides local model training based on the feature vector corresponding to the current local anchor point and each type of data; Triplet loss is the contrast loss between feature vectors of different categories.

4. According to the federated learning system for non-independent and identically distributed data scenarios according to claim 3, the anchor loss is expressed as: d i =distance(embedding i ,local anchor i ) in, i is the category index, k is the total number of categories, y i represents the label of the i-th category, d i Represents the feature vector embedding generated by the i-th type of data i With the current local anchor local anchor i The distance between them, distance() represents the distance calculation operation, including the Euclidean distance calculation operation or the cosine similarity operation, and CrossEntropyLoss() represents the cross entropy loss function; The contrast loss triplet loss is expressed as: d ij =distance(embedding i ,embedding j ) Among them, j is the category index, y j represents the label of the jth category, margin represents the preset threshold, which ranges from 0 to 1, max{} represents the maximum value function, and d ij Represents the feature vector embedding generated by the i-th type of data i embedding with the feature vector generated by the j-th data j The distance between.

5. According to the federated learning system for non-independent and identically distributed data scenarios of claim 1, when training the local model, the client uses a momentum optimization algorithm to update the local anchor point. The specific formula is: local anchor i ’=γ*embedding i +(1-γ)*local anchor i ,i=1,...,k in, i is the category index, k is the total number of categories, and local anchor i ' represents the new local anchor point corresponding to the i-th type of data, local anchor i Indicates the current local anchor point corresponding to the i-th type of data, embedding i Represents the feature vector embedding generated by the i-th type of data i , γ represents the anchor point update momentum, which is the ratio of the batch size of each data in the local training process to the total data size in the local training process, that is, 6. According to the federated learning system for non-IID scenarios of data according to claim 1, the central server aggregates all local model parameters in the following manner: in, n represents the local model index, N is the total number of local models, global model represents the global model parameters, local model n Represents the nth local model parameter, local weight n Represents the aggregate weight of the nth local model parameter, local data n It indicates the data size of the sub-dataset for training the nth local model, and total data indicates the data size of the target dataset.

7. According to the federated learning system for non-IID scenarios of data as claimed in claim 1, the aggregation method adopted by the central server for aggregating all new local anchor points is: in, global anchor means global anchor point, local anchor nj Represents the new local anchor point of the j-th label corresponding to the n-th local model, anchor weight nj The anchor weight of the j-th label corresponding to the n-th local model is calculated as follows: in, Indicates the amount of data with j-type labels in the i-th client, num label j Indicates the total amount of data of the j-th label of all clients.

8. According to the federated learning system for data non-IID scenarios of claim 1, before the federated learning, the central server is also used to construct a federated learning task and send it to each client, wherein: The federated learning task includes the initial local model, the initial local anchor point, and the federated learning hyperparameters, where the federated learning hyperparameters include the gradient descent algorithm optimizer, the local training learning rate, the total number of clients participating in the federated learning, the number of local training rounds of the client, the total number of communication rounds, the spatial dimension of the local anchor point, the weight decay rate, and the agreed communication method; the client and the central server perform federated learning based on the federated learning task; Among them, the number of initial local anchor points is equal to the number of label types of the target dataset; The local model includes a feature extraction network, a projection network and an output layer, wherein the feature extraction network is used to extract the feature map of the input sample data, the projection network is used to project the feature map into the feature projection space to obtain the feature vector, and the output layer is used to perform prediction calculations based on the feature vector and classify the data to predict the results.

9. According to the federated learning system for non-IID scenarios of data in claim 1, the communication modes include four types: In the first case, the uploaded model data includes local model parameters and new local anchor points, and the downloaded aggregated data includes global model parameters and global anchor points. In the second case, the uploaded model data includes the new local anchor point, and the downloaded aggregated data includes the global model parameters and the global anchor point; The third type is that the uploaded model data includes local model parameters and new local anchor points, and the downloaded aggregated data includes global anchor points; The fourth type is that the uploaded model data includes the new local anchor point, and the downloaded aggregated data includes the global anchor point; When the central server and the client choose the first communication mode to communicate, after training the model locally, the client uploads the local model parameters and the new local anchor point to the central server at the same time; the central server aggregates all local model parameters and the new local anchor point to obtain the global model parameters and the global anchor point, and transmits the global model parameters and the global anchor point to the client as the current local model and the current global anchor point for the next round of local training; When the central server and the client choose the second communication mode to communicate, the client uploads the new local anchor point to the central server after training the model locally; the central server aggregates all new local anchor points to obtain the global anchor point, and transmits the historically stored global model parameters and global anchor point to the client as the current local model and current global anchor point for the next round of local training; When the central server and the client choose the third communication mode to communicate, after the client trains the model locally, it uploads the local model parameters and the new local anchor point to the central server at the same time; the central server aggregates all local model parameters and the new local anchor point to obtain the global model parameters and the global anchor point, and transmits the global anchor point to the client as the current global anchor point for the next round of local training; When the central server and the client choose the fourth communication mode to communicate, the client uploads the new local anchor point to the central server after training the model locally; the central server aggregates all local anchor points to obtain the global anchor point, and transmits the global anchor point to the client as the current global anchor point for the next round of local training.

10. A federated learning method for scenarios where data is not independent and identically distributed, characterized in that: The federated learning method adopts the federated learning system according to any one of claims 1 to 9, and the federated learning method includes: The central server establishes a communication channel with the client, creates a federated learning task, and sends it to each client; The central server divides the target data set into multiple sub-data sets in a non-independent distribution manner so that each sub-data set contains all types of data, and distributes the sub-data sets to the clients so that each client has one sub-data set; Based on the received sub-dataset, the client trains the current local model according to the global anchor point as the current local anchor point, updates the local anchor point and local model parameters, and uploads the model data to the central server according to the agreed communication method, where the local anchor point is the feature point that can represent the classification category in the feature vector corresponding to each type of data, and the model data includes the new local anchor point and local model parameters; The central server aggregates the received model data to obtain aggregated data, and transmits the aggregated data to the client according to the agreed communication method as the basis for the next round of federated learning. The aggregated data includes the global anchor points obtained by aggregating all new local anchor points and the global model parameters obtained by aggregating all local model parameters.

Images