Federal learning method and system
By performing initial clustering and dynamic reclustering in federated learning, the problem of low model utility of traditional federated learning methods under Non-IID data is solved, and more efficient model convergence and accuracy improvement are achieved.
Patent Information
- Application Number
- CN202411829342.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-12
- Publication Date
- 2025-04-18
AI Technical Summary
When traditional federated learning methods process Non-IID data, the model utility is low, resulting in a decrease in global model accuracy, slow convergence speed, and poor learning performance.
By obtaining the distribution information of the participant's tag data for initialization clustering, the participant is clustered using the first clustering algorithm, the model parameters in each cluster are extracted for clustering based on the model parameter distance, and the first judgment is made. If the clustering result is split, reclustering adjustment is performed.
The accuracy and efficiency of clustering are improved, and the convergence speed and accuracy of the federated learning model are improved through dynamic reclustering adjustment, solving the problem of low model utility.
Smart Images
Figure CN119849597A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data security technology, and in particular to a federated learning method and system. Background Art
[0002] In the actual scenario of federated learning, due to the non-independent and identically distributed data between participating devices, federated learning faces the problem of low model utility. In terms of model utility, traditional machine learning usually requires data IID (Independent and Identically Distributed) distribution, while the participants of federated learning are widely distributed, resulting in differences in the region, time domain, etc. of the data collected locally by each participant, and its distribution is usually Non-IID. A large number of studies have shown that although the classic framework of federated learning FedAvg can converge on Non-IID data, the decline in the accuracy of federated learning is inevitable. The main reason for the decline in the accuracy of the global model of federated learning is the weight difference of the local model caused by Non-IID data, that is, due to the heterogeneity of the local data distribution, local models with the same initial parameters will converge to different models. In the federation process, the difference between the local model and the global model continues to increase, resulting in slow convergence and poor learning performance.
[0003] The earliest method to deal with the Non-IID problem was to create a global shared dataset that contains part of the data of each participant. However, this method deviates from the original intention of federated learning to protect user privacy. The more classic method to solve the Non-IID problem is the participant clustering method. However, most of the existing studies do not make full use of the data distribution information of the participants, resulting in unreasonable clustering, which leads to low model accuracy and ineffective improvement of model utility. Therefore, a flexible and high-utility clustering federated learning optimization algorithm is urgently needed. Summary of the invention
[0004] In view of the above existing problems, the present invention is proposed.
[0005] Therefore, the present invention provides a federated learning method and system to solve the problem that traditional methods do not fully utilize the data distribution information of participants, resulting in unreasonable clustering, which in turn leads to low model accuracy and ineffective improvement of model effectiveness.
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0007] In a first aspect, the present invention provides a federated learning method, comprising:
[0008] Obtaining distribution information of tag data of participants, and initializing clustering of the distribution information;
[0009] Based on the initialization clustering result, the participants are clustered using the first clustering algorithm to obtain outliers, and the outliers are assigned to corresponding clusters;
[0010] Extract the model parameters in each cluster, perform clustering based on the model parameter distance, and make a first judgment on the clustering results;
[0011] If the first judgment result is splitting, re-clustering adjustment is performed.
[0012] As a preferred solution of the federated learning method of the present invention, initializing clustering of the distribution information includes:
[0013] Clustering is performed based on the Wasserstein distance of the label distribution. For different distribution information, the Wasserstein distance is: for each joint distribution, the expected distance between the random variable and the joint distribution is calculated when the random variable is extracted from the joint distribution, that is, the minimum value of the expected distance under all joint distributions.
[0014] As a preferred solution of the federated learning method of the present invention, clustering the participants includes:
[0015] Clustering the participants into different clusters and outliers using a first clustering algorithm;
[0016] Outliers are randomly assigned to clusters, and in the federation process, the outliers are rejoined to the cluster with the smallest cosine distance to its model parameters.
[0017] As a preferred solution of the federated learning method of the present invention, clustering based on model parameter distance includes:
[0018] For any cluster, the model parameters of all participants in it are clustered and expressed as:
[0019]
[0020] in, For cluster g j All the participants in For cluster g j Participants c i The corresponding local model parameters, DBSCAN(·) is DBSCAN clustering.
[0021] As a preferred solution of the federated learning method of the present invention, the first judgment on the clustering result includes:
[0022] If the clustering result of the cluster is shown as one cluster, the clustering result mark is the first threshold, otherwise it is the second threshold;
[0023] If it is the second threshold, a second judgment is performed;
[0024] The second judgment includes: if the ratio of the sum of the first judgment results of the cluster to the total number of clusters is greater than a third threshold, then the first judgment result of the cluster is splitting, and re-clustering is performed.
[0025] As a preferred solution of the federated learning method described in the present invention, the re-clustering class includes:
[0026] Reconstruct the parameter distance as a sample of the second clustering algorithm, and define the model parameter matrix of the real-time extraction information of the participants as follows: for each pair of participants, calculate the absolute value of the difference between the pair of participants and other participants, sum the absolute values, and divide the sum by the number of participants other than the pair of participants;
[0027] The distance between the models of the participants is the difference between the model parameters of each pair of participants, and the Euclidean norm of the difference is taken.
[0028] As a preferred solution of the federated learning method described in the present invention, it also includes:
[0029] The original label distribution information is added for auxiliary clustering, and the Wasserstein distance matrix of the original label distribution information is constructed as follows:
[0030]
[0031] Among them, Y i ,Y j are the label distributions of participants i and j, respectively, ‖xy‖ is the distance between samples x and y, γ(x,y) is the joint distribution of x and y, ∏(Y i ,Y j ) is the distribution Y i ,Y j The joint probability distribution of
[0032] The model parameter matrix and the distance matrix are combined to obtain a weighted matrix, and the i-th row of the weighted matrix is used as the clustering sample of participant i.
[0033] In a second aspect, the present invention provides a federated learning system, comprising:
[0034] An initialization clustering module is used to obtain distribution information of tag data of participants and initialize clustering of the distribution information;
[0035] An outlier classification module, used to cluster the participants using a first clustering algorithm based on the initialization clustering result to obtain outliers, and assign the outliers to corresponding clusters;
[0036] A first judgment module is used to extract model parameters in each cluster, perform clustering based on model parameter distance, and perform a first judgment on the clustering result;
[0037] The re-clustering module is used to make re-clustering adjustments if the first judgment result is splitting.
[0038] In a third aspect, the present invention provides a computing device, comprising:
[0039] Memory and processor;
[0040] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the federated learning method are implemented.
[0041] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the federated learning method when executed by a processor.
[0042] Compared with the prior art, the present invention has the following beneficial effects: the present invention uses the initialization clustering technology to ensure rapid and accurate clustering, and uses re-clustering to achieve more accurate re-clustering. The method is more accurate and rapid, and helps the federated learning process converge faster, by performing initialization clustering of participants based on the label distribution information of the participant data; the method uses re-clustering to reconstruct the participant clusters, and then updates the global model parameters of each cluster according to the FedAvg algorithm to prevent the participants in some clusters from not being selected and causing the cluster to be unable to participate in this round of federated training; information utilization is increased, and the original data information and the federated process are used simultaneously to extract information in real time when the participants are clustered, eliminating the limitations and inaccuracies of information utilization in previous clustering federated learning methods, thereby improving the accuracy of the federated training model. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0044] Figure 1 A schematic diagram of the overall process logic of the federated learning method according to an embodiment of the present invention;
[0045] Figure 2 A schematic diagram of the architecture of a federated learning method according to an embodiment of the present invention;
[0046] Figure 3This is a comparison chart of prediction accuracy of the federated learning method described in an embodiment of the present invention. DETAILED DESCRIPTION
[0047] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0048] Example 1
[0049] Reference Figure 1-Figure 3 , as an embodiment of the present invention, provides a federated learning method, including:
[0050] S100: Obtain distribution information of tag data of participants, and initialize clustering of the distribution information;
[0051] S200: based on the initialization clustering result, clustering the participants using the first clustering algorithm to obtain outliers, and assigning the outliers to corresponding clusters;
[0052] S300: extracting model parameters from each cluster, performing clustering based on model parameter distance, and performing a first judgment on the clustering result;
[0053] S400: If the first judgment result is splitting, re-clustering adjustment is performed.
[0054] It should be noted that the federated learning method based on dynamic clustering improves the comprehensiveness of the use of user data distribution information, making the clustering results of the participants more accurate, thereby improving the effectiveness of the federated learning model; combining multiple information for clustering improves the accuracy of the federated learning model, and makes the convergence of the model more stable, reducing the decline in model performance due to data distribution heterogeneity.
[0055] In the embodiment of the present application, the above step S100 includes the following sub-step A1:
[0056] In A1: Clustering is performed based on the Wasserstein distance of the label distribution. For different distribution information, the Wasserstein distance is: for each joint distribution, the expected distance between the random variable and the joint distribution is calculated when the random variable is drawn from the joint distribution, that is, the minimum value of the expected distance under all joint distributions.
[0057] In the embodiment of the present application, the initiator initializes the prediction model and defines the parameters related to federated learning, including the maximum number of iterations, learning rate, learning termination conditions, and model parameters of the model;
[0058] Specifically, the initiator can determine the maximum number of model iterations and learning rate based on actual needs, and input them into the model as hyperparameters. This allows users to control the model's calculation time, convergence speed, and model accuracy. At the same time, the number of iterations and learning rate are not private. The initiator can send them as plain text to the participants, who will input them into their own models as hyperparameters.
[0059] It should be noted that the initiator is allowed to determine the maximum number of iterations and learning rate of the model according to actual needs, which provides flexibility and allows the federated learning process to be customized according to different application scenarios and data characteristics; by setting the maximum number of iterations, the computing resource consumption and time of model training can be controlled; the adjustment of the learning rate directly affects the convergence speed and final performance of the model. An appropriate learning rate can help the model converge to the optimal solution faster while avoiding overfitting or underfitting.
[0060] In the embodiments of the present application, Figure 2 As shown, each participant uploads the distribution information of the label data in its local data set, and the central server directly performs DBSCAN clustering based on the Wasserstein distance of the label distribution;
[0061] Specifically, for two different distributions, the Wasserstein distance between them is expressed as:
[0062] W(p,q)=inf γ～∏(p,q) E (x,y)～γ [d(x,y)]
[0063] Among them, ∏(p,q) is the joint probability distribution of distribution p and q, (x,y) is the sample point sampled from the joint distribution γ, and d(x,y) is the distance between x and y, usually using Euclidean distance, Manhattan distance, etc.
[0064] It should be noted that the Wasserstein distance between the sample label distributions of each participant is used as a metric to initialize clustering; Wasserstein distance, as an effective metric for measuring data distribution differences, enables this algorithm to effectively and quickly complete clustering based on the original data distribution information of the participants. Wasserstein distance has good adaptability to different data distributions and can handle distributions of various shapes and scales. Wasserstein distance has good robustness to outliers and noise.
[0065] In the embodiment of the present application, the above step S200 includes the following sub-steps B1-B2;
[0066] In B1: using the first clustering algorithm to cluster the participants into different clusters and outliers;
[0067] In B2: outliers are randomly assigned to clusters, and in the federation process, outliers are re-added to the cluster with the smallest cosine distance to its model parameters;
[0068] Specifically, after the initial clustering, the first clustering algorithm is the DBSCAN clustering algorithm to adaptively cluster n participants into K clusters. and possible outliers Outliers are randomly assigned to clusters and rejoined to the cluster with the smallest cosine distance to its model parameters in a formal federation process, expressed as:
[0069]
[0070] in, For participant c i The model parameters, For cluster g j The global model parameters, g * The most appropriate cluster for the participants selected based on cosine distance;
[0071] It should be noted that using the DBSCAN algorithm for initial clustering can effectively process areas of different densities, identify natural clusters and outliers, thereby improving the accuracy of clustering, randomly dividing outliers into clusters, and readjusting them according to the model parameter cosine distance during the federation process, thereby enhancing the robustness of the model to abnormal data. The DBSCAN algorithm does not require the pre-specified number of clusters and can adaptively process data of different densities. By minimizing the cosine distance of outliers during the federated learning process, the stability of the model is improved.
[0072] In the embodiment of the present application, the above step S300 includes the following sub-steps C1-C4;
[0073] In C1: For any cluster, the model parameters of all participants in it are clustered and expressed as:
[0074]
[0075] in, For cluster g j All the participants in For cluster g j Participants c iThe corresponding local model parameters, DBSCAN(·) is DBSCAN clustering.
[0076] In C2: if the clustering result of the cluster shows one cluster, the clustering result mark is the first threshold, otherwise it is the second threshold;
[0077] In C3: if it is the second threshold, a second judgment is performed;
[0078] In C4: the second judgment includes: if the ratio of the sum of the first judgment results of the cluster to the total number of clusters is greater than a third threshold, then the first judgment result of the cluster is splitting, and re-clustering is performed.
[0079] Specifically, in round t, for each cluster The information extracted locally by the participants in the training, i.e., the model parameters, is clustered by DBSCAN based on the l2 distance of the model parameters to see whether the cluster will split into multiple clusters. That is, for any cluster g j (j=1,2…,K), perform DBSCAN clustering on the model parameters of all participants, and record whether the clustering result is split;
[0080] Adaptive re-clustering determined by DBSCAN clustering, if the cluster g j The clustering result is displayed as a cluster, the first threshold is 0, the second threshold is 1, and the clustering result mark otherwise
[0081] Represents the current cluster g j There are large differences between the model parameters of the participants, that is, there are differences in the real-time extracted information, which represents the current cluster g j The clustering structure is not reasonable and needs further re-clustering and adjustment;
[0082] In the embodiment of the present application, the third threshold may be 0.6, which indicates the proportion of unreasonable clustering in the total number of clusters. A large threshold may lead to unreasonable clustering results, while a small threshold may trigger unnecessary re-clustering operations.
[0083] Specifically, the DBSCAN clustering results of all clusters are counted. Reclustering is triggered only if the clusters exceeding the third threshold are judged to be split, that is, they meet the following conditions:
[0084]
[0085] in, For cluster g j The adaptive judgment result of τ is the threshold for triggering re-clustering;
[0086] Re-clustering will be performed only when the adaptive judgment result of the cluster exceeding the threshold τ is split;
[0087] It should be noted that by performing DBSCAN clustering on the model parameters of the participants in each cluster, the clustering structure can be dynamically adjusted to adapt to changes in data distribution. The use of the DBSCAN algorithm allows the algorithm to adaptively identify the number of clusters, increasing the flexibility of clustering so that it can adapt to different data sets. It can avoid unnecessary re-clustering caused by accidental events during the execution of the algorithm or improper setting of algorithm hyperparameters, thereby improving robustness.
[0088] In the embodiment of the present application, the above step S400 includes the following sub-steps D1-D2;
[0089] In D1: reconstruct the parameter distance as a sample of the second clustering algorithm, and define the model parameter matrix of the real-time extraction information of the participants as: for each pair of participants, calculate the absolute value of the difference between the pair of participants and other participants, sum the absolute values, and divide the sum by the number of participants other than the pair of participants;
[0090] In D2: The distance between the models of the participants is the difference between the model parameters of each pair of participants, and the Euclidean norm of the difference is taken.
[0091] Specifically, consider cluster g j (j=1,2…,K) Assuming that the re-clustering is triggered in round t, the local model parameters of the participant are In most previous works, the model parameters of the neural network are usually flattened directly and used as clustering samples. However, in general, there are many parameters in the neural network model, and the dimension is too high after flattening. Commonly used clustering algorithms will encounter problems in high-dimensional clustering, mainly because the Euclidean distance loses its effect in high-dimensional space. Therefore, the parameter distance matrix is reconstructed as a sample for the second clustering algorithm, K-Means clustering. The model parameter matrix of the defined participant real-time information extraction is expressed as:
[0092]
[0093] Among them, M p , is composed of a matrix M, i and j are different participants, then the element at the (i, j)th position of the matrix represents the relative distance between the i-th participant and the j-th participant and the remaining participants' model parameters. The matrix M is directly defined as the distance between the participant model parameters, expressed as:
[0094] M i,j =‖w i -w j ‖2,i,j=1,2,…,n
[0095] Among them, w i 、w j are the model parameters of participants i and j respectively;
[0096] In the embodiment of the present application, after completing steps D1-D2 in the above step S400, the following steps D3-D4 are also included;
[0097] In D3: add the original label distribution information for auxiliary clustering, and construct the Wasserstein distance matrix of the original label distribution information as follows:
[0098]
[0099] Among them, Y i ,Y j are the label distributions of participants i and j, respectively, ‖xy‖ is the distance between samples x and y, γ(x,y) is the joint distribution of x and y, ∏(Y i ,Y j ) is the distribution Y i ,Y j The joint probability distribution of
[0100] In D4: the model parameter matrix and the distance matrix are combined to obtain a weighted matrix, and the i-th row of the weighted matrix is used as the clustering sample of participant i;
[0101] Specifically, in addition to the information extracted in real time during model training, the original label distribution information is also added for auxiliary clustering. By combining the original data distribution information with the information extracted in real time during training, the utilization of data distribution information is increased to improve the clustering effect;
[0102] The weighting matrix is expressed as:
[0103] M weighted =α1M p +α2M w
[0104] Among them, α1 and α2 are weighted coefficients of the model parameter matrix and the original information matrix, satisfying α1+α2=1, which are used to achieve different degrees of utilization of the two types of information;
[0105] It should be noted that by combining the original data distribution information with the real-time extracted information during training, the utilization of data distribution information is increased to improve the clustering effect. By reconstructing the parameter distance matrix, the differences between the participants can be captured more accurately, and the accuracy of clustering can be improved. In view of the problem that the neural network model has many parameters and high dimensions, the parameter distance matrix is constructed to avoid the direct use of Euclidean distance in high-dimensional space, solve the problems that may be encountered in high-dimensional clustering, and realize the initialization of clustering based on the original label distribution information of the DBSCAN algorithm, and the adaptive re-clustering of the real-time extracted information based on the K-Means algorithm. By combining multiple information for clustering, the comprehensiveness of the utilization of user data distribution information is improved, making the clustering results of the participants more accurate, thereby improving the utility of the federated learning model.
[0106] The above is a schematic scheme of a federated learning method of this embodiment. It should be noted that the technical scheme of the federated learning system and the technical scheme of the above-mentioned federated learning method belong to the same concept, and the details of the technical scheme of the federated learning system in this embodiment that are not described in detail can be referred to the description of the technical scheme of the above-mentioned federated learning method.
[0107] The federated learning system in this embodiment includes:
[0108] An initialization clustering module is used to obtain distribution information of tag data of participants and initialize clustering of the distribution information;
[0109] An outlier classification module, used to cluster the participants using a first clustering algorithm based on the initialization clustering result to obtain outliers, and assign the outliers to corresponding clusters;
[0110] A first judgment module is used to extract model parameters in each cluster, perform clustering based on model parameter distance, and perform a first judgment on the clustering result;
[0111] The re-clustering module is used to make re-clustering adjustments if the first judgment result is splitting.
[0112] This embodiment also provides a computing device, which is applicable to federated learning, including:
[0113] Memory and processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the federated learning method proposed in the above embodiment.
[0114] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for implementing federated learning proposed in the above embodiment is implemented.
[0115] The storage medium proposed in this embodiment and the method for implementing federated learning proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0116] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.
[0117] Example 2
[0118] Different from the first embodiment, a verification test of a federated learning method is provided to verify and illustrate the technical effects adopted in this method.
[0119] The federated classification task is performed on the FEMNIST dataset. FEMNIST is short for Federated ExtendedMNIST, which contains 62 grayscale images of handwritten numbers and letters. The 62 types of images include 10 types of numbers, 26 types of lowercase letters, and 26 types of uppercase letters. This experiment samples images of "a"-"j" to construct a 10-category dataset, and divides it into 200 participants to simulate a distributed environment. CNN, MCLR, and MLP models are used for classification.
[0120] The initiator determines the maximum number of model iterations T = 300 and the learning rate lr = 0.003, the batch size of local training is set to 10, and the sampling ratio of participants is set to 0.1, that is, 10% of the participants are sampled for training in each federated training. In addition, when initializing clustering, the parameter radius ε of the DBSCAN clustering algorithm is set to 0.5, and the minimum number of points MinPts is set to 5. The threshold ε of the DBSCAN clustering algorithm during training is set to 0.8, and the minimum number of points MinPts is set to 3.
[0121] Each participant i uploads its local dataset D i The label data y in iThe central server directly performs DBSCAN clustering based on the Wasserstein distance of the label distribution and obtains 5 initial cluster centers;
[0122] After the initial DBSCAN clustering, n = 200 participants are adaptively clustered into 5 clusters And several outliers Outliers are randomly assigned to clusters and rejoined to the cluster with the smallest cosine distance to its model parameters in a formal federation process, i.e.
[0123]
[0124] Where: Indicates outlier (participant) c i The model parameters, Represents cluster g j The global model parameters, g * represents the participant c selected based on the cosine distance i The most suitable cluster.
[0125] Every k = 10 rounds, a re-clustering decision is made to determine whether the current cluster structure is reasonable. The information extracted locally by the participants in the training, i.e., the model parameters, is clustered by DBSCAN based on the l2 distance of the model parameters to see whether the cluster will split into multiple clusters. That is, for any cluster g j (j=1,2…,K), perform DBSCAN clustering on the model parameters of all participants, and record whether the clustering result is split.
[0126] If the cluster g j The clustering result is displayed as a cluster, then the clustering result mark otherwise Represents the current cluster g j There are large differences between the model parameters of the participants, that is, there are differences in the real-time extracted information, which represents the current cluster g j The clustering structure is not reasonable and needs further re-clustering and adjustment. Count the DBSCAN clustering results of all clusters. If they meet the following conditions:
[0127]
[0128] Then re-clustering is triggered, because The regathering class was not triggered.
[0129] like Figure 3As shown, experiments on the FEMNIST data set show that the accuracy of the model corresponding to the present invention is significantly better than that of the comparison algorithm and the convergence is more stable, proving that the clustering of participants in the model corresponding to the present invention is relatively reasonable when initializing clustering, and the distribution of participants in the cluster is close to the IID distribution, so re-clustering is not triggered. The present invention quickly reaches a high accuracy rate in the early stage of training and remains stable in subsequent training, showing the characteristics of rapid convergence. Compared with other algorithms, the model of the present invention shows higher stability during training, and the accuracy fluctuates less, which helps to improve the reliability of the model and the consistency of prediction.
[0130] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A federated learning method, characterized in that: include: Obtaining distribution information of tag data of participants, and initializing clustering of the distribution information; Based on the initialization clustering result, the participants are clustered using the first clustering algorithm to obtain outliers, and the outliers are assigned to corresponding clusters; Extract the model parameters in each cluster, perform clustering based on the model parameter distance, and make a first judgment on the clustering results; If the first judgment result is splitting, re-clustering adjustment is performed.
2. The federated learning method according to claim 1, characterized in that: Initializing clustering of the distribution information includes: Clustering is performed based on the Wasserstein distance of the label distribution. For different distribution information, the Wasserstein distance is: for each joint distribution, the expected distance between the random variable and the joint distribution is calculated when the random variable is extracted from the joint distribution, that is, the minimum value of the expected distance under all joint distributions.
3. The federated learning method according to claim 2, characterized in that: Clustering the parties involves: Clustering the participants into different clusters and outliers using a first clustering algorithm; Outliers are randomly assigned to clusters, and in the federation process, the outliers are rejoined to the cluster with the smallest cosine distance to its model parameters.
4. The federated learning method according to claim 3, characterized in that: Performing clustering based on model parameter distances includes: For any cluster, the model parameters of all participants in it are clustered and expressed as: in, For cluster g j All the participants in For cluster g j Participants c i The corresponding local model parameters, DBSCAN(·) is DBSCAN clustering.
5. The federated learning method according to claim 4, characterized in that: The first judgment of the clustering results includes: If the clustering result of the cluster is shown as one cluster, the clustering result mark is the first threshold, otherwise it is the second threshold; If it is the second threshold, a second judgment is performed; The second judgment includes: if the ratio of the sum of the first judgment results of the cluster to the total number of clusters is greater than a third threshold, then the first judgment result of the cluster is splitting, and re-clustering is performed.
6. The federated learning method according to any one of claims 1 to 5, characterized in that: The re-clustering classes include: Reconstruct the parameter distance as a sample of the second clustering algorithm, and define the model parameter matrix of the real-time extraction information of the participants as follows: for each pair of participants, calculate the absolute value of the difference between the pair of participants and other participants, sum the absolute values, and divide the sum by the number of participants other than the pair of participants; The distance between the models of the participants is the difference between the model parameters of each pair of participants, and the Euclidean norm of the difference is taken.
7. The federated learning method according to claim 6, characterized in that: Also includes: The original label distribution information is added for auxiliary clustering, and the Wasserstein distance matrix of the original label distribution information is constructed as follows: Among them, Y i ,Y j are the label distributions of participants i and j, respectively, ‖xy‖ is the distance between samples x and y, γ(x,y) is the joint distribution of x and y, ∏(Y i ,Y j ) is the distribution Y i ,Y j The joint probability distribution of The model parameter matrix and the distance matrix are combined to obtain a weighted matrix, and the i-th row of the weighted matrix is used as the clustering sample of participant i.
8. A system using the federated learning method according to any one of claims 1 to 7, characterized in that: include: An initialization clustering module is used to obtain distribution information of tag data of participants and initialize clustering of the distribution information; An outlier classification module, used to cluster the participants using a first clustering algorithm based on the initialization clustering result to obtain outliers, and assign the outliers to corresponding clusters; A first judgment module is used to extract model parameters in each cluster, perform clustering based on model parameter distance, and perform a first judgment on the clustering result; The re-clustering module is used to make re-clustering adjustments if the first judgment result is splitting.
9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the federated learning method described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the federated learning method according to any one of claims 1 to 7.