Non-IID Data Processing Method Based on Federated Learning Framework

Abstract

The present invention discloses a method for processing non-independent and identically distributed data based on a federated learning framework. The steps are as follows: On the client side, a similarity calculation method is constructed based on the Wasserstein distance to aggregate clients with similar data distributions. On the server side, an integrated model calculation method is constructed to fine-tune the global model obtained by averaging and weighting all client models. During the process of model parameter transmission, the parameter masking technology and the attention mechanism technology are respectively used to minimize the communication cost and the calculation cost of the overall system. The present invention effectively eliminates the model bias problem caused by non-independent and identically distributed data during the process of processing non-independent and identically distributed data, enhances the performance of the model, and also significantly reduces the communication overhead and energy consumption of the overall system.

Description

Technical Field

[0001] The present invention belongs to the field of communication technologies, and further relates to a method for processing non-independent and identically distributed data based on a federated learning framework in the fields of distributed and privacy protection technologies. The present invention can be used in an environment where sensor devices, independent data center institutions, and edge servers work together. On the premise of ensuring user privacy, the non-independent and identically distributed data is processed based on the federated learning framework, providing an important basis and guarantee for analyzing and learning the data information of multiple private data owners. Background Art

[0002] With the rapid development of information technologies such as mobile Internet, cloud computing, and Internet of Things, the world has entered the big data era. As a new type of production factor in the digital economy era, the importance of data has been recognized by all sectors of society. Driven by the dual needs of big data fusion applications and privacy protection, privacy-preserving computing has emerged and become an important technology for promoting the trustworthy circulation of data at home and abroad. Privacy-preserving computing can build a secure circulation paradigm where data is available but invisible, available but not storable, controllable, and measurable. Federated learning is an important technology in privacy-preserving computing, which refers to an artificial intelligence technology in which two or more participating parties jointly participate. On the premise of ensuring that the original data of each data party does not leave its local domain, a machine learning model is collaboratively constructed and used. Federated learning embodies the principles of centralized data collection and minimization, which can reduce the privacy risks and communication efficiency overhead at the system and statistical levels brought by traditional centralized machine learning and data mining methods.

[0003] The University of Electronic Science and Technology provided a method for processing non-independent and identically distributed data based on a hierarchical user training management system in its patent document "A Hierarchical User Training Management System and Method for Non-Independent and Identically Distributed Data" (application number CN 202110959654.7, publication number CN 113672684 A). The implementation steps of this method are as follows: First, users are scored by calculating the similarity between the node data distribution and the global distribution while protecting user labels; Second, the global parameter server performs reasonable grouping processing according to the user scores to make the data distribution between groups approximately uniform and solve the problem of non-independent and identically distributed data; Third, a parallelism growth curve is set according to the needs of users to transform the training from serial to parallel, ensuring the parallelism of the model while ensuring the convergence speed and accuracy of the model, and saving training time. Although this invention combines the full synchronization algorithm with the sequential federated learning algorithm, it can significantly reduce the communication load and save computing resources. However, the deficiencies of this method are still: when calculating the similarity between the node data and the global data distribution, there may be noise, resulting in incorrect grouping, and the model accuracy is limited or even endangered.

[0004] Henan University provided a method for processing non-i.i.d. data based on the clustering idea during the federated learning training process in its patent document "A Clustering Federated Learning Method and Device" (Application No. CN202210809648.8, Publication No. CN 115169582 A). The implementation steps of this method are as follows: First, obtain the feature vectors of the local data distributions of the clients, where the feature vectors are obtained by using the Deep Sets model to extract features from the local data distributions on the clients; Second, use the K-Means clustering algorithm to cluster the feature vectors of the local data distributions of all clients, so as to group the clients with similar data distributions into the same group; Third, set corresponding cluster identifiers for each group, different groups correspond to different cluster identifiers, and the clients within the same group have the same cluster identifier; Fourth, schedule the clients according to the cluster identifiers of the clients, so that the clients with the same cluster identifier jointly train a model using the federated learning algorithm. This invention can solve the problem of the decline in model accuracy caused by the heterogeneity of data distributions and can reduce the computational load of client nodes. However, the still existing deficiency of this method is that for the federated learning model after training, it only alleviates the negative impact brought by heterogeneous data on the clients and cannot well solve the model drift problem on the central server node.

[0005] Hebei University of Technology provided a method for processing non-i.i.d. data for a large number of industrial big data in its patent document "A Joint Modeling Method for Non-Independent and Identically Distributed Industrial Big Data" (Application No. CN202210250580.4, Publication No. CN114676765 A). The steps of this method are as follows: First, each local factory uploads its local non-i.i.d. data set to the central factory; Second, the central factory performs global model updates, calculates the training weights and training probabilities of each local factory, and selects the local factories that are beneficial to the optimization of the global model to send the global model parameters; Third, the selected local factories perform local updates, and select the local model parameters close to the current optimal global model from the locally deviated local factory models and upload them to the central factory; Fourth, repeat the steps of optimizing the global model and uploading the local model parameters until the model training ends. This invention can efficiently learn a large number of non-i.i.d. data existing in the industrial field through the joint modeling method. However, the still existing deficiency of this method is that joint training of a large number of industrial big data will bring non-negligible communication delays and system overheads, which will in turn lead to problems such as low computational efficiency and high computational load of local nodes. Summary of the Invention

[0006] The object of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a method for processing non-independent and identically distributed data based on a federated learning framework, which is used to solve the problem of local update bias caused by existing federated learning technologies when processing non-independent and identically distributed data, the problem of model aggregation bias introduced by differences between different clients during global model aggregation by the central server, as well as the problems of uplink and downlink communication losses and time overhead in the overall system of federated learning during the process of model parameter transmission.

[0007] The idea for achieving the object of the present invention is as follows: A method for processing non-independent and identically distributed data based on a federated learning framework is proposed in an environment based on a central server and clients. Each client updates the prediction layer parameters of the model by training its private data and sends them to the central server. The central server receives the prediction layer parameters of all client models, packages them, and sends them to each client. Each client calculates the prediction layer parameters of the client most similar to it based on the overall prediction layer parameters, and weights them to obtain synthetic parameters, so that each client model can learn from the synthetic parameters to improve the local update bias problem. Further, the central server weights the overall parameters of all client models to obtain an integrated model, so that the global model aggregated by each client model on the server can learn more generalization information from the integrated model to solve the model aggregation bias problem. Finally, during the process of parameter transmission of all models, the parameters are compressed by parameter masking, and different weighting coefficients are assigned to each client based on the attention mechanism, effectively reducing the communication and computing costs. Since only gradient information is transmitted during the overall training process, the privacy of user data is also well protected.

[0008] To achieve the above object, the specific implementation steps of the present invention are as follows:

[0009] Step 1, each client aggregates similar clients based on the Wasserstein distance to obtain its synthetic parameters:

[0010] Step 1.1, each client uses a list to store the prediction layer parameters of the model and then uploads them to the server side. The server side receives the prediction layer parameters of all client models.

[0011] Step 1.2, the server side packages all the prediction layer parameters and then sends them to each client.

[0012] Step 1.3, each client receives the prediction layer parameters of all clients and calculates the Wasserstein distance between the prediction layer parameters of each client and those of other clients according to the Wasserstein distance formula.

[0013] Step 1.4: Sort all the Wasserstein distances between each client and other clients in ascending order, and select the top M clients in the sorting to form the similar clients of this client, where M is greater than or equal to 5;

[0014] Step 1.5: Calculate the synthesis parameter of each client;

[0015] Step 2: Calculate the loss of each client based on the synthesis parameter:

[0016] Step 2.1: Use the class importance formula to calculate the ratio factor of each client's private data to each category of data samples in the entire training dataset. The entire training dataset refers to the aggregation of private data on all clients;

[0017] Step 2.2: Calculate the relative entropy distance between the synthesis parameter of each client and the current client's model parameter, and then train the current client's model through the stochastic gradient descent algorithm;

[0018] Step 3: Fine-tune the global model using the constructed server-side integrated model:

[0019] Step 3.1: The server receives the model parameters of all clients and obtains the server-side integrated model through a weighted calculation method;

[0020] Step 3.2: The server-side calculates the integration ratio of each client respectively through a ratio calculation method according to the model parameters of each client;

[0021] Step 3.3: For all data samples of each category in the entire training dataset, sample 100 data samples, randomly shuffle the order of all sampled data samples, and then pack and merge them all into an auxiliary dataset;

[0022] Step 3.4: Calculate the relative entropy distance loss between the integrated model and the global model obtained by averaging and multiplying all client models on the server side. First, divide the parameters of the integrated model and the global model by their respective squared sums, and then use the relative entropy distance formula to calculate the distance value between the parameters of the integrated model and the global model;

[0023] Step 3.5: Update the parameter values corresponding to the global model through the stochastic gradient descent algorithm;

[0024] Step 4: Optimize the transfer process of model parameters through parameter masking and attention mechanism:

[0025] Step 4.1: Each client sets a parameter masking matrix with the same size as the current client's model parameter, and sets the initial element value of this parameter masking matrix to 1;

[0026] Step 4.2: Multiply the current client model by the parameter mask matrix to obtain a new parameter matrix, retrain the new parameter matrix using the current client's private data, and update the new parameter matrix using the stochastic gradient descent algorithm. Each client uploads the trained parameter matrix to the server.

[0027] Step 4.3: The server receives all the client model parameters compressed by the parameter mask, and then assigns different weighting ratios to the model parameters of each client through the attention mechanism calculation method.

[0028] The present invention has the following advantages compared with the prior art:

[0029] First, when updating the client model, the present invention constructs a similarity calculation method based on the Wasserstein distance and aggregates clients with similar data distributions, overcoming the drawback of the prior art that when directly calculating the similarity between node data and the global data distribution during the process of dealing with non-independent and identically distributed data, there may be noise, resulting in incorrect client grouping and harm to the model accuracy. The present invention effectively converges the information most similar to the current client's data distribution during the process of dealing with non-independent and identically distributed data, enhancing the performance of the client model.

[0030] Second, during the central server aggregation stage, the present invention constructs an integrated model calculation method to fine-tune the global model obtained by averaging and weighting all client models, overcoming the drawback of the prior art that when dealing with non-independent and identically distributed data, only the influence of heterogeneous data on a single client is considered, ignoring the model heterogeneous drift problem existing on the central server, resulting in slow convergence of the aggregated global model and poor prediction accuracy. The present invention effectively guides the global model to eliminate the model bias problem by integrating different client models during the process of dealing with non-independent and identically distributed data, achieving better performance.

[0031] Third, since the present invention uses the parameter mask technology and the attention mechanism technology respectively during the model parameter transfer process to minimize the communication cost and calculation cost of the overall system, overcoming the drawbacks of the prior art such as low calculation efficiency and high calculation load of local nodes caused by large-batch data calculation during the process of dealing with non-independent and identically distributed data, the present invention can effectively reduce the communication overhead and energy consumption when dealing with a large amount of non-independent and identically distributed data, and expand the total amount of data that can be stored on each client on the premise of stable performance. Description of the Drawings

[0032] Figure 1 is a flowchart of the present invention;

[0033] Figure 2 is a structural diagram of the model used in the training of the present invention;

[0034] Figure 3 Create a visualization graph for the non - independent and identically - distributed data of the present invention. Detailed implementation manners

[0035] Refer to the attached Figure 1 and the attached Figure 2 Further describe the specific implementation steps of the present invention.

[0036] Before specific implementation, first perform non - independent and identically - distributed processing on the original data set: Since the original data sets participating in model training are all data sets that conform to independent and identically - distributed, but this does not meet the assumption of non - independent and identically - distributed among cross - client private data in the federated learning system. Therefore, the present invention divides the original data set based on the Dirichlet distribution.

[0037] The data distribution of the labeled samples in the data set can be expressed as p(x,y), written as p(x,y)=p(x|y)p(y), and then the sample distribution is efficiently divided through the label distribution p(y) of the samples. Further, assume there are K class labels and N different clients, and the samples of each class label need to be divided among different client devices according to different ratios. Given the class label distribution matrix Its row vector Represents the probability distribution vector of class k on different clients (each dimension represents the ratio of samples of class k divided into different clients), and this random vector is sampled from the Dirichlet distribution.

[0038] Refer to the attached Figure 3 Further describe the implementation steps of the non - independent and identically - distributed data of the present invention.

[0039] The present invention tests the non - independent and identically - distributed data partitioning algorithm based on the Dirichlet distribution on the CIFAR - 10 data set and performs visual presentation, where it is stipulated that the number of clients N = 20, and the parameter vector α of the Dirichlet distribution satisfies α i = 1.0, where i = 1,2,...,N.

[0040] The present invention only splits the training data set and sends it to each client as its private data, and then evaluates the performance of the global model under the test data set.

[0041] Step 1, each client aggregates similar clients based on the Wasserstein distance to obtain composite parameters.

[0042] Step 1, each client uses a list to store the prediction layer parameters of the model and then uploads them to the server side, and the server side receives the prediction layer parameters of all client models.

[0043] Step 2: The server packs all the prediction layer parameters and then sends them to each client.

[0044] Step 3: Each client receives the prediction layer parameters of all clients, calculates the Wasserstein distance, and then takes the average sum and product of the prediction layer parameters of the top five clients with the smallest values to obtain the synthetic parameters.

[0045] The formula for the Wasserstein distance is as follows:

[0046]

[0047] where d i,j represents the Wasserstein distance between the prediction layer parameters of the i-th client and the j-th client, ∑ represents the summation symbol, c represents the serial number of the client's class label, represents the total number of the client's class labels, represents the mean of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the i-th client model, represents the mean of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the j-th client model, represents calculating and taking the square of the difference between them; represents the approximate variance of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the i-th client model, represents the approximate variance of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the j-th client model, represents first calculating the difference between the arithmetic square root of and the arithmetic square root of taking the square of the difference, and then taking the arithmetic square root.

[0048] The synthetic parameter of each client is obtained by the following formula:

[0049]

[0050] where represents the synthetic parameter of the client calculated according to the structure w of the model parameters of each client, m represents the index of the similar client calculated according to the structure w of the model parameters of each client, M represents the total number of the calculated similar clients, represents the model parameters of the client with index m among the similar clients of the model parameter structure w of each client.

[0051] Step 2: Each client calculates the loss based on the synthetic parameters.

[0052] Each client model has two loss functions during the training process. The first loss function is the cross-entropy loss function constructed based on the local private data (x, y). The second loss function is to distill the model parameters of the client based on the synthetic parameters, that is, L Local Loss function.

[0053] Step 1: Each client calculates the ratio of the quantity of its private data to the quantity of data samples of each category in the entire training dataset, and then multiplies it by the class importance formula.

[0054] The class importance formula is as follows:

[0055]

[0056] Among them, represents the class importance formula, x represents the index of the total amount of data in the private data D of the k-th client, y represents the data index x corresponding to the class label in the private data D of the k-th client k in, L k represents the cross-entropy loss function, CE represents the prediction value of the model of the k-th client with the structure of model parameters w for the data index x, and and respectively represent the number of samples corresponding to the class label y in the private data D of the k-th client k and the entire training dataset D.

[0057] Step 2: The client model calculates the loss through the cross-entropy loss function and the L Local loss function based on the synthetic parameters and the current client model parameters, and then updates the model parameters through the stochastic gradient descent algorithm.

[0058] Step 3: Build an ensemble model on the server side to fine-tune the global model.

[0059] To fine-tune the global model on the server side, first the server needs to receive the model parameters sent by all clients, then the server integrates the models of all clients to obtain an ensemble model, and finally fine-tunes the global model obtained by averaging and weighting the models of all clients based on the auxiliary dataset.

[0060] The weighted calculation method is as follows:

[0061]

[0062] Among them, The integrated model on the server side, k represents the index value represented by the current client, k = 1, 2... K, and K represents the total number of clients. Represents the model parameters of the k-th client. Represents the private data D of the k-th client. k The numerical value of the total amount of data. Represents the numerical value of the total amount of data of the entire training dataset D, e. k Represents the ratio of the model parameters of the k-th client in the integrated model on the server side.

[0063] The calculation method of the above ratio is:

[0064]

[0065] Among them, sigmoid represents the normalization function, ⊙ represents the matrix multiplication operation, r k Represents the learnable parameter matrix, and λ represents the parameter used to control the degree of regularization. Represents taking The operation of taking the absolute value first and then calculating its square value.

[0066] Step 1: The models of each client correspond to different integration ratios. According to And e k The calculation formula of is multiplied and added one by one to obtain the integrated model.

[0067] Step 2: Sample 100 data samples for each category of data samples from the training dataset, then shuffle and merge them to obtain an auxiliary dataset.

[0068] Step 3: The integrated model fine-tunes the global model based on the auxiliary dataset. First, divide the parameters of the integrated model and the global model by their respective corresponding sum of squares, and then use the relative entropy distance formula to calculate the distance value between them.

[0069] The relative entropy distance mentioned above is obtained by the following formula:

[0070]

[0071] Among them, L Global Represents the relative entropy distance loss, D sam Represents the auxiliary dataset on the server side, L KL Represents the relative entropy distance. Represents the global model. Represents the integrated model. Represents taking Taking the absolute value first, then calculating the square value, and then calculating its arithmetic square root; Represents taking Calculate the square value of the absolute value, and then calculate its arithmetic square root.

[0072] Step 4, update the parameter values corresponding to the global model through the stochastic gradient descent algorithm.

[0073] Step 4, optimize the parameter transfer process of the model through parameter masking and attention mechanism.

[0074] Step 1, each client sets a parameter masking matrix equal to the size of the current client model parameters, and sets the initial value of the parameter mask to 1.

[0075] Step 2, multiply the current client model by the parameter mask to obtain a new parameter matrix, then retrain the parameter matrix using the private data of the current client, and update the parameter matrix using the stochastic gradient descent algorithm.

[0076] Step 3, each client uploads the trained parameter matrix to the server. The server receives all the parameter matrices, and then calculates the weighted ratio corresponding to each client through the attention mechanism calculation method for each client's parameter matrix. Multiply each client's parameter matrix by its corresponding weighted ratio to obtain the global model.

[0077] The attention mechanism calculation method is as follows:

[0078]

[0079] where r k represents the weighted ratio of the k-th client, attention represents the self-attention calculation operation, softmax represents the normalization exponential function, represents the model parameters of the k-th client after being compressed by the parameter mask.

[0080] Step 4, repeat the above steps to complete the training.

[0081] The following further illustrates the effect of the present invention in combination with simulation experiments:

[0082] 1. Simulation experiment conditions:

[0083] The hardware platform for the simulation experiment of the present invention is: a central server computer with a processor of Intel Xeon E3-1231V3, a main frequency of 3.6GHz, a memory of 64GB, and an NVIDIA GeForce RTX 3090 graphics card. Three client computers with a processor of Intel(R) Core(TM) i7-9700F, a main frequency of 3.0GHz, a memory of 16GB, and an NVIDIA GeForce RTX 1060 graphics card.

[0084] The software platform for the simulation experiment of the present invention is: Ubuntu 20.04 LTS, 64-bit operating system, Python 3.8, PyTorch deep learning framework (version 1.11.0), and PyCharm code writing software.

[0085] 2. Contents of the simulation experiment and analysis of its results:

[0086] In the simulation experiment of the present invention, the present invention and an existing technology (ResNet neural network classification method) are respectively used to predict three common image classification datasets (CIFAR-10 dataset, CIFAR-100 dataset, and mini-ImageNet dataset), and classification prediction results are obtained. Among them, in the simulation experiment of the present invention, the ratio of the training set to the test set is 8:2.

[0087] In the simulation experiment, the ResNet neural network classification method adopted refers to:

[0088] The image classification method based on the ResNet neural network model proposed by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun in their published paper "Deep Residual Learning for Image Recognition" ([1] Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778), simply referred to as the ResNet neural network classification method.

[0089] To verify the effect of the simulation experiment of the present invention, the accuracy rate of the global model on the test dataset is used as a quantitative evaluation index to evaluate the global models trained by the method of the present invention and other different methods.

[0090] Table 1: Prediction performance accuracy rate (%) of different methods on CIFAR-10 and CIFAR-100 datasets

[0091]

[0092] In the simulation experiment, other different methods refer to the Federated Averaging algorithm (FedAvg), Federated Proximal algorithm (FedProx), Federated Normalized Averaging algorithm (FedNova), Federated Continual Learning algorithm (FedCurv), Federated Control Variates algorithm (Scaffold), Federated Bayesian Ensemble algorithm (FedBE), Federated Dual Averaging algorithm (FedDF), and Federated Generalized Knowledge Distillation algorithm (FedGEN). α represents the degree of non - independent and identically distributed of the dataset partitioned based on the Dirichlet distribution. The smaller α is, the greater the degree of non - independent and identically distributed of the data.

[0093] Table 2: Prediction performance accuracy (%) of different methods on the mini - ImageNet dataset

[0094]

[0095] As can be seen from Table 1 and Table 2, compared with other different methods, the method of the present invention enables the model trained by the method of the present invention to achieve higher classification prediction accuracy on different datasets and different degrees of imbalance. Especially on the mini - ImageNet dataset, although its training data is complex and severely imbalanced, the global model trained by the method of the present invention still achieves excellent prediction accuracy.

[0096] The above simulation experiments show that: the present invention proposes a method for processing non - independent and identically distributed data based on the federated learning framework. Through client aggregation, server - side fine - tuning, as well as parameter masking and attention mechanism technologies, it solves the problems that may exist in the prior art when processing non - independent and identically distributed data, such as noise, model drift on the central server, and low efficiency in the face of a large amount of data. It is a more practical and efficient method for processing non - independent and identically distributed data.

Claims

1. A method for processing non - independently and identically distributed data based on a federated learning framework, characterized in that, For non-independent and identically distributed data, corresponding processing is performed respectively on the client side, the server side, and during the transmission process of model parameters based on the federated learning framework; the specific steps of this method are as follows: Step 1, each client aggregates similar clients based on the Wasserstein distance to obtain its synthetic parameters: Step 1.1, each client uses a list to store the prediction layer parameters of the model and then uploads them to the server side. The server side receives the prediction layer parameters of all client models; Step 1.2, the server side packages all the prediction layer parameters and then sends them to each client; Step 1.3, each client receives the prediction layer parameters of all clients and calculates the Wasserstein distance between the prediction layer parameters of each client and those of other clients according to the Wasserstein distance formula; Step 1.4, sort all the Wasserstein distances between each client and other clients in ascending order, and select the first M clients in the sorting to form the similar clients of this client, where M is greater than or equal to 5; Step 1.5, calculate the synthetic parameters of each client; Step 2, calculate the loss of each client based on the synthetic parameters: Step 2.1, use the class importance formula to calculate the ratio factor of the private data of each client to the data samples of each category in the entire training dataset. The entire training dataset refers to the aggregation of the private data on all clients; Step 2.2, calculate the relative entropy distance between the synthetic parameters of each client and the model parameters of the current client, and then train the current client model through the stochastic gradient descent algorithm; Step 3, fine-tune the global model using the constructed server-side integrated model: Step 3.1, the server receives the model parameters of all clients and obtains the server-side integrated model through a weighted calculation method; Step 3.2, the server side calculates the integration ratio of each client respectively through a ratio calculation method according to the model parameters of each client; Step 3.3, sample 100 data samples for each category of data samples in the entire training dataset, randomly shuffle the order of all sampled data samples, and then pack and merge them all into an auxiliary dataset; Step 3.4, calculate the relative entropy distance loss between the integrated model and the global model obtained by averaging and multiplying all client models on the server side. First, divide the parameters of the integrated model and the global model by their respective squared sums, and then use the relative entropy distance formula to calculate the distance value between the parameters of the integrated model and the global model; Step 3.5, update the parameter values corresponding to the global model through the stochastic gradient descent algorithm; Step 4, optimize the transmission process of model parameters through parameter masking and attention mechanism: Step 4.1, each client sets a parameter masking matrix with the same size as the model parameters of the current client, and sets the initial element values of this parameter masking matrix to 1; Step 4.2: Multiply the current client model by the parameter mask matrix to obtain a new parameter matrix, retrain the new parameter matrix using the current client's private data, update the new parameter matrix using the stochastic gradient descent algorithm, and each client uploads the trained parameter matrix to the server; Step 4.3: The server receives all the client model parameters compressed by the parameter mask, and then assigns different weighted ratios to the model parameters of each client through the attention mechanism calculation method.

2. The method for processing non-independent and identically distributed data based on the federated learning framework according to claim 1, wherein The Wasserstein distance formula described in Step 1.3 is as follows: Among them, d i,j represents the Wasserstein distance between the prediction layer parameters of the i-th client and the j-th client, ∑ represents the summation symbol, c represents the serial number of the client's class label, represents the total number of the client's class labels, represents the mean of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the i-th client model, represents the mean of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the j-th client model, represents calculating and the difference between them and taking the square of the difference; represents the approximate variance of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the i-th client model, represents the approximate variance of the prediction layer parameters obtained after training the data samples with the c-th class label in the entire training dataset by the j-th client model, represents first calculating the difference between the arithmetic square roots of and taking the square of the difference, and then taking the arithmetic square root.

3. The method for processing non-independent and identically distributed data based on the federated learning framework according to claim 1, wherein The synthetic parameters of each client described in Step 1.5 are obtained by the following formula: Among them, represents the synthetic parameter of the client calculated according to the structure w of the model parameters of each client, m represents the index of the neighboring client calculated according to the structure w of the model parameters of each client, and M represents the total number of the calculated neighboring clients. represents the model parameter of the client with index m among the neighboring clients of the model parameter structure w of each client.

4. The method for processing non-independent and identically distributed data based on the federated learning framework according to claim 2, wherein The class importance formula described in Step 2.1 is as follows: Among them, represents the class importance formula, x represents the index of the total amount of data in the k-th client's private data D k and y represents the data index x in the k-th client's private data D k corresponding class label, L CE represents the cross-entropy loss function, represents the predicted value of the model of the k-th client with the structure of the model parameters w for the data index x, and respectively represent the number of samples corresponding to the class label y in the k-th client's private data D k and the entire training data set D.

5. The method for processing non-independent and identically distributed data based on the federated learning framework according to claim 1, wherein The weighted calculation method described in Step 3.1 is: Among them, represents the integrated model on the server side, k represents the index value represented by the current client, k = 1, 2... K, and K represents the total number of clients. represents the model parameters of the k-th client. represents the private data D of the k-th client. k The numerical value of the total amount of data. represents the numerical value of the total amount of data of the entire training data set D, e. k represents the ratio of the model parameters of the k-th client in the integrated model on the server side.

6. The method for processing non-independent and identically distributed data based on the federated learning framework according to claim 5, wherein The ratio calculation method described in Step 3.2 is: Among them, sigmoid represents the normalization function, ⊙ represents the matrix multiplication operation, r k represents the learnable parameter matrix, λ represents the parameter used to control the degree of regularization, represents the operation of first taking the absolute value and then calculating its squared value.

7. The non-independent and identically distributed data processing method based on the federated learning framework according to claim 1, wherein The relative entropy distance described in Step 3.4 is obtained by the following formula: Among them, L Global represents the relative entropy distance loss, D sam represents the auxiliary data set on the server side, L KL represents the relative entropy distance, represents the global model, represents the ensemble model, represents taking first the absolute value and then calculating the squared value, and then calculating its arithmetic square root; represents taking first the absolute value and then calculating the squared value, and then calculating its arithmetic square root.

8. The method for processing non-independent and identically distributed data based on the federated learning framework according to claim 1, wherein The attention mechanism calculation method described in Step 4.3 is: where r k represents the weighted ratio of the k-th client, attention represents the self-attention calculation operation, and softmax represents the normalization exponential function. represents the model parameters of the k-th client after being compressed by the parameter mask.

Images