A fast and efficient federated learning method for data skew

By incorporating a batch normalization layer into federated learning and dynamically selecting clients, the label skew and feature skew issues under Non-IID data conditions are resolved, improving model accuracy and convergence speed, enabling personalized design, and protecting data privacy.

CN114997374BActive Publication Date: 2024-11-26GUANGXI NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210671622.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-14
Publication Date
2024-11-26
Estimated Expiration
2042-06-14

AI Technical Summary

Technical Problem

Existing federated learning methods have failed to effectively address label and feature skew issues among clients under Non-IID data conditions, leading to decreased model accuracy and performance, and posing a risk of data privacy breaches.

Method used

By adding a batch normalization layer to the convolutional neural network on the client side and dynamically selecting high-quality clients to participate in training, the client selection is carried out by utilizing the homogeneity of label distribution and the bias of parameter vectors, and combined with global model aggregation, the impact of data heterogeneity is mitigated.

Benefits of technology

It improves the convergence speed and accuracy of the model while protecting data privacy and enabling personalized model design.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114997374B_ABST
    Figure CN114997374B_ABST
Patent Text Reader

Abstract

The present invention discloses a fast and efficient federated learning method for data skew, and proposes a federated model framework for label skew and feature skew problems in the case of Non-IID training data. When there are serious label skew and feature skew problems in the training data set, high-quality clients are dynamically selected to participate in training to alleviate the label distribution skew problem, and a batch normalization (BN) layer is added to the client local model to alleviate the feature skew problem. This greatly reduces the impact of data heterogeneity on model performance, effectively improves the model convergence speed and accuracy, and realizes model personalized design while protecting data privacy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to a fast and efficient federated learning method for data skew. Background Art

[0002] With the continuous development of Internet technology, more and more people are beginning to worry about the problem of privacy data leakage. As many countries have successively promulgated data privacy and security regulations, the centralized learning that was previously widely used in the field of machine learning is no longer applicable to the field of multi-party secure computing due to the risk of privacy leakage. In 2016, Google proposed the distributed framework of Federated Learning, which is used for multiple participants (clients) to collaboratively train a machine learning model while retaining private data locally. It has now received widespread attention.

[0003] In the standard federated learning framework FedAvg, each round of update selects a random subset of clients from all current clients to participate in training. The selected clients use local data for training and then upload the update results to the server. The server aggregates these update results and sends the global model of this round of updates to all participants. This process is repeated until the model converges. However, the standard federated learning method FedAvg is not designed for non-independent and identically distributed (Non-IID) data. If it is deployed on Non-IID samples, the model performance will be degraded. Therefore, a major challenge in federated learning is label skew and feature skew. Label skew refers to the phenomenon that the data of different clients may have different sources and preferences, and the distribution of data labels between different clients is extremely unbalanced. For example, in medical diagnosis tasks, due to the different advantages of different hospitals, the number of patients with different diseases in different hospitals varies greatly, which will cause the disease types (labels) of data from different hospitals (clients) to be unbalanced. Feature skew refers to the phenomenon that even if the data of different clients belong to the same label, they still have different forms of expression, and their data distribution in the feature space is also biased. For example, different clients use different imaging devices, which may lead to large differences in the appearance of the same type of images, which in turn leads to deviations in the distribution of data in the feature space. For federated learning methods, the greater the difference in data distribution between clients, the greater the difference in the local models updated by the clients, which may seriously affect the global model aggregation effect of federated learning. Although people have spent a lot of effort to optimize the federated model so that the model converges when Non-IID data exists, the problems of data label skew and feature skew have not been fully solved, which will greatly affect the accuracy and performance of the model. Therefore, how to select a set of high-quality clients and sample data for federated learning tasks to solve label skew and reduce the impact of feature skew on model accuracy without accessing the local data sets and training processes of multiple participants is a problem that needs to be solved.

[0004] At present, there are some methods that attempt to optimize models under Non-IID data. For example, Fraboni et al. used cluster sampling technology to perform unbiased customer sampling, proposed two aggregate sampling methods based on sample quantity and similarity, and proved through experiments that the use of cluster sampling method for node selection can make the aggregate model achieve faster and smoother convergence during training and testing. Li et al. alleviated data heterogeneity and improved the stability of global model convergence by adding proximal terms to the federated learning framework. Diao et al. proposed a federated learning framework in which the local model and the global model have different architectures to train heterogeneous local models with different computational complexities. Fernando et al. proposed a method to use generative enhancement to make up for the lack of minority group samples, increase the minority samples until they reach a balance with the majority, and thus correct the class balance problem. However, the existing federated learning methods do not fully consider the mixed non-independent and identically distributed problems of uneven category distribution in each client data set and feature tilt in sample data. How to reduce the impact of label distribution tilt and feature tilt on the effect of federated learning while ensuring model accuracy and model training convergence speed still needs further exploration and research. Summary of the invention

[0005] The present invention aims to solve the problem of client label distribution skew and feature skew in federated learning tasks involving multiple parties, and provides a fast and efficient federated learning method for data skew.

[0006] To solve the above problems, the present invention is achieved through the following technical solutions:

[0007] A fast and efficient federated learning method for data skew includes the following steps:

[0008] Step 1: Each client builds a local improved convolutional neural network model by adding a batch normalization layer after each convolutional layer and each fully connected layer of the convolutional neural network;

[0009] Step 2: The server calculates the intersection of the local label set of each client's local data set and the federal task label set. If the number of local data corresponding to the label in the intersection is greater than the minimum number of training samples required by the target model, the client corresponding to the intersection is considered to be a related client.

[0010] Step 3: The server selects the relevant clients that participate in each round of federated learning training from the relevant clients;

[0011] When screening the relevant clients participating in the first round of federated learning training, the server selects the relevant clients with a high selection probability as the relevant clients participating in the first round of training based on the selection probability of the relevant clients;

[0012] When screening the relevant clients participating in the non-first round of federated learning training, the server selects the relevant clients with large parameter vector deviations as the relevant clients participating in the non-first round of federated learning training based on the parameter vector deviations of the relevant clients;

[0013] Step 4: Each client trains the local improved convolutional neural network model using the local data set to obtain the parameter vector of the local model in this round; the parameter vector of the local model in this round includes the parameter vector of the local batch normalization layer in this round and the parameter vectors of other local layers, i.e., non-batch normalization layers in this round;

[0014] Step 5: The relevant clients participating in this round of federated learning training upload the local non-batch normalization layer parameter vector of this round to the server; the server globally averages and aggregates the local non-batch normalization layer parameter vectors of this round uploaded by all relevant clients participating in this round of federated learning training layer by layer to form the global non-batch normalization layer parameter vector of this round, and returns it to all relevant clients participating in this round of federated learning training;

[0015] Step 6: Each relevant client participating in this round of federated learning training updates the local improved convolutional neural network model using the global non-batch normalization layer parameter vector of this round returned by the server and the local batch normalization layer parameter vector of this round saved locally;

[0016] Step 7: Repeat steps 3-6 until the global non-batch normalization layer parameter vector of this round converges to the optimal global non-batch normalization layer parameter vector;

[0017] Step 8: The server broadcasts the optimal global non-batch normalization layer parameter vector to all clients, and the client uses the optimal global non-batch normalization layer parameter vector returned by the server and the final round of batch normalization layer parameter vector saved locally to obtain the final local improved convolutional neural network model;

[0018] Step 9: The client uses the final local improved convolutional neural network model to classify the images to be classified.

[0019] In the above step 3, the relevant client C k The probability of being selected P k for:

[0020]

[0021] In the formula, h k For the relevant client C k Homogeneity of label distribution, q k (y) is the relevant client C k The local label set Y k The distribution of labels y, q u(y) is the distribution of label y in the federated task label set Y; h j For the relevant client C j Homogeneity of label distribution, q j (y) is the relevant client C j The local label set Y j The distribution of labels y, q u (y) is the distribution of label y in the federated task label set Y; S kj For the relevant client C k and the associated client C j The similarity of v k,i For the relevant client C k The content embedding vector of the i-th local data, n k For client C k The local dataset D k The number of local data; v j,i For the relevant client C j The content embedding vector of the i-th local data, n j For client C j The local dataset D j The number of local data items; C k , C j ∈R, R represents the set of related clients.

[0022] In the above step 3, the relevant client C k The parameter vector deviation for:

[0023]

[0024] In the formula, θ t-1,k For the relevant client C k The parameter vector of the local non-batch normalization layer of round t-1, θ t-1 is the parameter vector of the global non-batch normalization layer in round t-1; C k ∈R, R represents the set of related clients.

[0025] In the above step 3, the number of relevant clients participating in each round of federated learning training is determined based on the given price budget and the willing price published by each relevant client, that is, the sum of the willing prices of the relevant clients participating in each round of federated learning training is less than or equal to the given budget.

[0026] Compared with the prior art, the present invention proposes a federated model framework for the label skew and feature skew problems in the Non-IID case of training data. When there are serious label skew and feature skew problems in the training data set, the label distribution skew problem is alleviated by dynamically selecting high-quality clients to participate in the training, and a batch normalization (BN) layer is added to the client local model to alleviate the feature skew problem. This greatly reduces the impact of data heterogeneity on model performance, effectively improves the model convergence speed and accuracy, and realizes personalized model design while protecting data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of a system framework for fast and efficient federated learning for data skew.

[0028] Figure 2 Flowchart of a fast and efficient federated learning method for data skew.

[0029] Figure 3 This is the structure diagram of the convolutional neural network with batch normalization layer added. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific examples.

[0031] The present invention aims at image classification tasks from different clients (devices), and each client uses federated learning to collaboratively train an image classification model without sharing privacy-sensitive data. In order to solve the serious imbalance in label distribution of training data samples from different clients and the problem of feature skew in sample data stored by local clients, the present invention proposes a method to solve the problem of non-independent and identically distributed (Non-IID) data of client label distribution skew and feature skew in federated learning tasks under the premise of privacy protection, so as to improve the model accuracy and convergence speed.

[0032] See also Figure 1 and 2 The present invention proposes a fast and efficient federated learning method for data skew, comprising the following steps:

[0033] 1. Before federal training begins

[0034] Step 1. Each client C k ∈N (N represents the client set) to construct a local improved convolutional neural network model by adding a batch normalization layer after each convolutional layer and each fully connected layer of the convolutional neural network.

[0035] In order to solve the problem of feature tilt in client sample data where the sample labels are the same but the feature representations are different. The present invention adds a batch normalization (BN) layer on the basis of the traditional convolutional neural network to alleviate the feature tilt problem before model aggregation. In the neural network, as the parameter vector is transmitted layer by layer, the eigenvalue distribution of each layer will gradually approach the upper and lower ends of the activation function output interval (activation function saturation interval), which is easy to cause the gradient to disappear. The batch normalization layer transforms and reconstructs the feature distribution so that the eigenvalue falls in the interval where the activation function is more sensitive to the input. Since slight changes in the input will cause large changes in the loss function, that is, the gradient becomes larger, adding the BN layer can avoid the gradient disappearance, and can also speed up the convergence speed. The feature distribution after the BN layer operation will be closer to the real data feature distribution, which is conducive to solving the feature tilt problem in the heterogeneous federated learning data. In the present invention, a BN layer is added after each feature extraction layer (convolutional layer and fully connected layer) of the client convolutional neural network model, that is, a BN layer is added after each convolutional layer, and a BN layer is added after each fully connected layer. The output of each feature extraction layer is normalized to make the feature distribution closer to the real data feature distribution, so as to alleviate the feature skew problem existing in the client sample data and better play the feature extraction role of the client convolutional neural network model.

[0036] Step 2: The server calculates C for each client k The intersection of the local label set of the local dataset ∈ N and the federal task label set. If the number of local data corresponding to the labels in the intersection is greater than the minimum number of training samples required by the target model, the client corresponding to the intersection is considered to be the relevant client C. k ∈R (R represents the set of related clients, R∈N).

[0037] Before starting to train the federated task, the server calculates the intersection of each client label and the federated task label set to determine the relevant clients, as follows: k Each has its corresponding local data set D k ={(X k ,Y k )}. X k is the local sample set, sample x k,i ∈X k ; Y k is the local label set, label y k,i ∈Y k ; Sample set X k For each sample x in k,i There are corresponding labels y k,i Belongs to its label set Y k ;(x k,i ,y k,i) is the i-th local data, i=1,2,...,n k , n k Represents client C k Calculate the number of local data items. k The intersection with the current federated task label set Y, if the number of local data corresponding to the labels in the intersection is greater than the minimum number of training samples required by the target model, that is, {(x k,i ,y k,i )|y k,i ∈Y k ∩Y}|>v (where v is the minimum number of training samples required by the target model), then it is considered that client C k For the relevant clients.

[0038] 2. Federation training process

[0039] 2.1. Client Selection

[0040] Step 3: The server receives the data from the relevant client C. k ∈R, select the relevant clients C that participate in each round of federated learning training k ∈M t (M t represents the set of related clients participating in the tth round of training, M t ∈R).

[0041] The present invention selects the most suitable client for participating in federated training through a client screening mechanism during each round of training. During each round of training, the server selects a subset of clients with the smallest label distribution skew from the clients that can participate in the training based on the client importance calculation method to participate in the training, so as to solve the label skew problem of extremely unbalanced label distribution between different clients.

[0042] 1) When screening relevant clients participating in the first round (t=1) of federated learning training, the server selects relevant clients with a high probability of being selected as clients participating in the first round of training based on the probability of the relevant clients being selected.

[0043] The relevant clients participating in the first round of training mainly consider two indicators: first, the homogeneity of the client label distribution and the federated task label distribution; second, the diversity of the client sample distribution. In order to consider the above two indicators at the same time, when selecting the relevant clients participating in the first round (t=1) of federated learning training, each relevant client C k ∈R needs to take advantage of its label distribution homogeneity h k , related client C j The label distribution homogeneity h j , and other related clients C j ∈R similarity Skj , calculate the relevant client C k ∈R is selected with a probability of P k :

[0044]

[0045] Related Client C k Homogeneity of label distribution h∈R k for:

[0046]

[0047] In the formula, q k (y) is the relevant client C k The local label set Y k The distribution of labels y, q u (y) is the distribution of label y in the federated task label set Y.

[0048] Related Client C j The label distribution homogeneity h j for:

[0049]

[0050] In the formula, q j (y) is the relevant client C j The local label set Y j The distribution of labels y, q u (y) is the distribution of label y in the federated task label set Y;

[0051] Related Client C k ∈R and the associated client C j ∈R similarity S kj for:

[0052]

[0053] In the formula, v k,i For the relevant client C k The content embedding vector of the i-th local data, n k For client C k The local dataset D k The number of local data; v j,i For the relevant client C j The content embedding vector of the i-th local data, n j For client C j The local dataset D j The number of local data entries.

[0054] That is, when homogeneity increases and similarity decreases, the probability of a client being selected increases, so the server can tend to select clients whose label distribution has high similarity with the federated task label distribution, while avoiding selecting clients with high similarity to other client samples.

[0055] 2) When screening relevant clients participating in non-first round (t=2, 3, ...) federated learning training, the server selects relevant clients with large parameter vector deviations as relevant clients participating in non-first round federated learning training based on the parameter vector deviations of the relevant clients.

[0056] In order to reduce the computational cost, the server dynamically selects important clients to participate in the training in each iteration after the first round of training. At this time, the importance of all samples of each client is no longer calculated, but the client with a large deviation between the local model parameter vector and the global model parameter vector is selected, because the client with a larger deviation in the parameter vector has a greater contribution to the training of the current global model. The parameter vector deviation of the client can be expressed as If client C k With greater value, there is a greater probability of being selected in the tth round of training. t-1,k Client C in round t-1 k Updated local model parameter vector, θ t-1 is the global model parameter vector updated in the t-1th round.

[0057] In addition, for federated learning training, each client C k Before model training, a willingness price b for participating in the federation task will be published. k , when selecting the relevant clients to participate in each round of training, the subsequent server will determine the number of relevant clients participating in each round of federated learning training based on the given budget and the willingness price published by each relevant client, that is, the willingness price b of the relevant clients participating in each round of federated learning training k The sum is within the given price budget B.

[0058] 2.2 Model training phase

[0059] Step 4: Each client C k ∈N uses the local data set to train the local improved convolutional neural network model to obtain the parameter vector of the local model of this round; the parameter vector of the local model of this round includes the parameter vector of the local batch normalization layer of this round and the parameter vectors of other local layers of this round, namely, the non-batch normalization layer.

[0060] The present invention dynamically selects the participating client subsets during the training process and adds a batch normalization (BN) layer to the client's local model. In each round t=1,2,...,T, client Ck ∈N (N represents the set of all clients) using the local dataset D k Perform local convolutional neural network model training. Each client uses a convolutional neural network with a BN layer added to perform local model training. The client's local improved convolutional neural network model includes input values, convolutional layers, fully connected layers, BN layers, ReLu activation functions, pooling layers, and output values, such as Figure 3 As shown. Before the first round of training, the server initializes the global model parameter vector (including the global batch normalization layer parameter vector and the global non-batch normalization layer parameter vector), and sends the initialized model parameter vector to all clients. The client takes m training samples for each training batch in the local model training process. The input value of the convolutional neural network model of the current client is X = [x (1) ,x (2) ,...,x (m) ]. Assume that a feature extraction layer (convolutional layer or fully connected layer) in the model is the lth layer, and its corresponding intermediate hidden value is in Represents the output value of the i-th neuron in layer l, and is calculated separately The mean and variance Use the obtained mean and variance to Perform normalized calculations: Where ∈ is used to prevent the use of a small positive number when the divisor is 0. In order to ensure that the original feature distribution is not lost and restore the original expressiveness of the data, the BN layer introduces two learnable parameters: the scaling factor γ and the translation factor β, which transform and reconstruct the feature distribution so that the distribution after batch normalization is more in line with the actual distribution of the data and ensure the nonlinear expressiveness of the model. The output of the i-th neuron in the BN layer is: After each neuron is calculated and output by the BN layer, these output values ​​will be operated by the activation function and continue to be passed to the next layer of the network for subsequent model training until the local model training of the client is completed.

[0061] Step 5: Relevant clients C participating in this round of federated learning training k ∈M t Upload the parameter vector of the local non-batch normalization layer of this round to the server; the server will upload all the relevant clients C participating in this round of federated learning training k ∈M t The uploaded local non-batch normalization layer parameter vectors of this round are globally averaged and aggregated layer by layer to form the global non-batch normalization layer parameter vector of this round, and returned to all relevant clients C participating in this round of federated learning training k ∈M t .

[0062] Because the statistical parameter vector of the BN layer contains local data information to a certain extent, after all clients complete local training, all clients C k ∈N (N represents the client set) will save the local batch normalization layer parameter vector of this round locally. However, due to the parameter aggregation requirement of federated learning, only the relevant clients C participating in the tth round of federated learning training need to be saved locally. k ∈M t (M t The t-th round local non-batch normalization layer parameter vector (representing the set of related clients participating in the t-th round of training) is uploaded to the server to participate in global aggregation.

[0063] In order to protect data privacy, the relevant clients participating in the training need to keep all parameter vectors of the BN layer in the convolutional neural network model locally and upload the parameter vectors of other layers to the server for global aggregation. That is, the server will perform global average aggregation on the received parameter vectors of other layers layer by layer. Where m t+1 Represents the relevant client subset M participating in the t+1th round of federated learning training t+1 The number of clients in Indicates that in the t+1 round of updates, client C k The parameter vector of the lth layer (non-BN layer) of the neural network model.

[0064] Step 6: Each client C participating in this round of federated learning training k ∈M t The local improved convolutional neural network model is updated using the global non-batch normalization layer parameter vector of this round returned by the server and the local batch normalization layer parameter vector of this round saved locally.

[0065] Step 7: Repeat steps 3-6 until the global non-batch normalization layer parameter vector of this round converges to the optimal global non-batch normalization layer parameter vector.

[0066] Repeat the above steps to select the next round of clients participating in the training. The clients participating in the federated training upload a new round of non-BN layer parameter vectors to the server. The server aggregates and updates the new round of global model parameter vectors and checks whether the current global model converges. If the model converges, the federated training ends. If the model does not converge, the next round of client selection continues training until the global model converges and returns the final global model parameter vector θ * .

[0067] The following algorithm is the federated training algorithm process with the BN layer added:

[0068]

[0069] Step 8: The server broadcasts the optimal global non-batch normalization layer parameter vector to all clients C k ∈N, client C k ∈N uses the optimal global non-batch normalization layer parameter vector returned by the server and the batch normalization layer parameter vector of the final round saved locally to obtain the final locally improved convolutional neural network model.

[0070] After the server returns the currently updated global model parameter vector, each local client combines the updated global model parameter vector with the local BN layer (the BN layer parameter vector is always kept locally) parameter vector to form a new local model. This not only solves the problems of label tilt and feature tilt, but also realizes the personalization of the client model.

[0071] Step 9. Client C k ∈N uses the final local improved convolutional neural network model to classify the images to be classified.

[0072] After the federated task training is completed, each client obtains a personalized federated learning model for the image classification task. For different images, each client can accurately classify the images.

[0073] The present invention can effectively reduce the adverse effects of label skew and feature skew problems in client data sets on model training, improve model training accuracy and convergence speed, and realize personalized design of the model while protecting privacy. The method is simple and very effective.

[0074] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A fast and efficient federated learning method for data skew, characterized by: The steps include: Step 1: Each client builds a local improved convolutional neural network model by adding a batch normalization layer after each convolutional layer and each fully connected layer of the convolutional neural network; Step 2: The server calculates the intersection of the local label set of each client's local data set and the federal task label set. If the number of local data corresponding to the label in the intersection is greater than the minimum number of training samples required by the target model, the client corresponding to the intersection is considered to be a related client. Step 3: The server selects the relevant clients that participate in each round of federated learning training from the relevant clients; When screening the relevant clients participating in the first round of federated learning training, the server selects the relevant clients with a high selection probability as the relevant clients participating in the first round of training based on the selection probability of the relevant clients; When screening the relevant clients participating in the non-first round of federated learning training, the server selects the relevant clients with large parameter vector deviations as the relevant clients participating in the non-first round of federated learning training based on the parameter vector deviations of the relevant clients; Step 4: Each client trains the local improved convolutional neural network model using the local data set to obtain the parameter vector of the local model in this round; the parameter vector of the local model in this round includes the parameter vector of the local batch normalization layer in this round and the parameter vectors of other local layers, i.e., non-batch normalization layers in this round; Step 5: The relevant clients participating in this round of federated learning training upload the local non-batch normalization layer parameter vector of this round to the server; the server globally averages and aggregates the local non-batch normalization layer parameter vectors of this round uploaded by all relevant clients participating in this round of federated learning training layer by layer to form the global non-batch normalization layer parameter vector of this round, and returns it to all relevant clients participating in this round of federated learning training; Step 6: Each relevant client participating in this round of federated learning training updates the local improved convolutional neural network model using the global non-batch normalization layer parameter vector of this round returned by the server and the local batch normalization layer parameter vector of this round saved locally; Step 7: Repeat steps 3-6 until the global non-batch normalization layer parameter vector of this round converges to the optimal global non-batch normalization layer parameter vector; Step 8: The server broadcasts the optimal global non-batch normalization layer parameter vector to all clients, and the client uses the optimal global non-batch normalization layer parameter vector returned by the server and the final round of batch normalization layer parameter vector saved locally to obtain the final local improved convolutional neural network model; Step 9: The client uses the final local improved convolutional neural network model to classify the images to be classified.

2. A fast and efficient federated learning method for data skew according to claim 1, characterized in that: In step 3, the relevant client C k The probability of being selected P k for: In the formula, h k For the relevant client C k Homogeneity of label distribution, q k (y) is the relevant client C k The local label set Y k The distribution of labels y, q u (y) is the distribution of label y in the federated task label set Y; h j For the relevant client C j Homogeneity of label distribution, q j (y) is the relevant client C j The local label set Y j The distribution of labels y, q u (y) is the distribution of label y in the federated task label set Y; S kj For the relevant client C k and the associated client C j The similarity of v k,i For the relevant client C k The content embedding vector of the i-th local data, n k For client C k The local dataset D k The number of local data; v j,i For the relevant client C j The content embedding vector of the i-th local data, n j For client C j The local dataset D j The number of local data items; C k , C j ∈R, R represents the set of related clients.

3. The fast and efficient federated learning method for data skew according to claim 1, characterized in that: In step 3, the relevant client C k The parameter vector deviation for: In the formula, θ t-1,k For the relevant client C k The parameter vector of the local non-batch normalization layer of round t-1, θ t-1 is the parameter vector of the global non-batch normalization layer in round t-1; C k ∈R, R represents the set of related clients.

4. The fast and efficient federated learning method for data skew according to claim 1, characterized in that: In step 3, the number of relevant clients participating in each round of federated learning training is determined based on the given price budget and the willing price published by each relevant client, that is, the sum of the willing prices of the relevant clients participating in each round of federated learning training is less than or equal to the given budget.