Safety clustering federated learning method and system based on generalization parameters

Abstract

The invention relates to the technical field of federated learning, in particular to a safe clustering federated learning method and system based on generalization parameter.The method comprises the steps that during training of a global model based on a local data set, a client generates a noise vector with the same dimension as a to-be-uploaded parameter in the local model according to data distribution of the local data set; superposing with at least a part of parameters in the local model to obtain noise adding model parameters; the server receives noise adding model parameters of a plurality of clients, calculates the parameter similarity between any two clients and constructs a proximity matrix; through clustering and grouping aggregation of all clients, corresponding cluster-level global model parameters are generated and issued to each client in a corresponding cluster to realize global model updating. According to the method, the privacy is protected by using the noise associated with local data distribution, and clustering is performed based on the noise-added parameters, so that the noise is offset during clustering, and the model performance is not damaged while the privacy is protected.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of federated learning, and specifically relates to a secure clustering federated learning method and system based on a generalization parameter. BACKGROUND

[0002] The statements in this section merely refer to the background art of the present application and do not necessarily constitute the prior art.

[0003] In actual deployment, the client data of federated learning usually presents a non-identically distributed characteristic, and due to the differences in user habits, device types and use environments, the data distribution (such as category, feature and quantity) of different clients often has significant deviation. This data heterogeneity breaks the basic assumption of traditional machine learning on identically distributed data, resulting in a serious decline in the generalization performance of the global model obtained by directly aggregating local models, and more communication rounds are needed for convergence, which further aggravates the communication burden.

[0004] In order to deal with the above-mentioned problem of data heterogeneity, the prior art attempts not to pursue the training of a single global model, but to generate a customized model for each client. So that the client can participate in the collaborative training of the global model while also obtaining personalized local models by combining regularization, local fine-tuning, model interpolation, multi-task learning and knowledge distillation methods. Although the personalized method can improve the adaptability of the model to local data to some extent, privacy protection is insufficient. Specifically, during the personalized training process, the client usually still needs to upload model parameters (such as model weights, gradients, etc.) to the server to obtain global knowledge. These parameters may still leak sensitive information of the original training data through reverse reasoning and other attack methods, which poses a potential risk of privacy leakage.

[0005] In addition to personalized federated learning, another way is clustering federated learning, which aims to automatically divide clients with similar data distribution into the same cluster and train a cluster-level global model for each cluster. This method is between a single global model and a completely personalized model, trying to maintain the generalization ability of the model while improving the local performance. However, the existing clustering methods often need the client to upload complete model parameters when measuring the similarity of the client, which will bring high communication overhead and privacy risk. If random noise is simply added to the uploaded parameters for privacy protection, and the model parameters (after adding noise) are used for clustering, it will seriously interfere with the server's accurate judgment of the similarity of the clients, resulting in clustering failure and ultimately damaging the model performance. SUMMARY

[0006] The application provides a secure clustering federated learning method and system based on generalization parameters, which trains a more targeted and high-performance model for a client group with similar data distribution under the premise of avoiding original data leakage, thereby overcoming the defects of poor performance of a single global model and poor generalization and high cost of a completely personalized model, and achieving a balance among the three.

[0007] In order to achieve the above-mentioned purpose, the application adopts the following technical solutions: The first aspect of the application discloses a secure clustering federated learning method based on generalization parameters, comprising the following steps: The server initializes a global model and sends the global model to all participating clients; The client receives the global model, trains the global model based on the local data set to obtain a local model; during the training, the client generates a noise vector with the same dimension as the to-be-uploaded parameter in the local model according to the data distribution of the local data set, and superimposes the noise vector with at least part of the parameters in the local model to obtain a noisy model parameter, and uploads the noisy model parameter to the server; The server receives the noisy model parameters from multiple clients, calculates the parameter similarity between any two clients according to the received noisy model parameters, and constructs a proximity matrix based on the parameter similarity; The server clusters all clients according to the proximity matrix to obtain one or more client clusters, aggregates the noisy model parameters uploaded by all clients in each cluster to generate corresponding cluster-level global model parameters, and updates the global model by distributing the cluster-level global model parameters to each client in the corresponding cluster.

[0008] Further, the client generates a noise vector with the same dimension as the to-be-uploaded parameter in the local model according to the data distribution of the local data set, specifically: statistics the k class labels with the most sample quantity in the local data set, takes the sorting result of the class label as a distribution factor representing the data distribution, initializes a pseudo-random number generator using the distribution factor to obtain a noise vector conforming to a Gaussian distribution.

[0009] Further, the parameters in the local model are divided into personalized parameters and generalization parameters, the personalized parameters capture specific features in the local data, and the generalization parameters represent high-level semantic information and general representation common to clients; the generalization parameters are superimposed with the noise vector to obtain the noisy model parameter, the noisy model parameter is uploaded to the server, and the personalized parameters remain in the client and do not participate in communication.

[0010] Further, the server calculates the parameter similarity between any two clients according to the received noisy model parameters, specifically: the server calculates the Euclidean distance between the noisy model parameters of any two clients.

[0011] Furthermore, a proximity matrix is ​​constructed, specifically: based on the parameter similarity calculation results between any two clients, a symmetric proximity matrix is ​​constructed, where the diagonal elements are zero, and the off-diagonal elements reflect the differences between the parameters of the noisy model.

[0012] Furthermore, the server clusters all clients based on the proximity matrix to obtain one or more client clusters. Specifically, the server uses the obtained proximity matrix as input and uses the K-means clustering algorithm to group the N noisy model parameters. The resulting client clusters contain sets of clients with similar parameter patterns.

[0013] Furthermore, the noisy model parameters uploaded by all clients within each cluster are aggregated to generate corresponding cluster-level global model parameters. Specifically, this is achieved by calculating the average generalization parameter within each cluster. This reflects the global generalized knowledge shared by the clients of this cluster.

[0014] Furthermore, global model updates are achieved by distributing cluster-level global model parameters to each client within the corresponding cluster. Specifically, each client receives the average generalization parameter corresponding to its cluster, which is distributed by the server. Replace the generalization parameters in its local model with the received average generalization parameters. This involves reconstructing the overall model parameters to achieve a global model update.

[0015] Furthermore, during the aggregation of noisy model parameters uploaded by all clients within each cluster, the effect is enhanced through knowledge sharing, including: The server aggregates the knowledge of this group, specifically: ; The server aggregates knowledge from other groups, specifically: ; Knowledge sharing, specifically: ; The aggregation results enhanced by knowledge sharing Send to the client; in, Indicates client i Generalization parameters after adding noise This represents the number of clients in the k-th group. This represents the number of clients excluding the k-th group. Indicates the number of rounds of knowledge sharing. V It refers to the intensity of knowledge acquisition.

[0016] A second aspect of the present invention provides a secure clustering federated learning system based on generalization parameters, comprising: a server configured to: initialize a global model, and send the global model to all participating clients; a client configured to: receive the global model, train the global model based on a local data set to obtain a local model; during the training, the client generates a noise vector with the same dimension as the parameters to be uploaded in the local model according to the data distribution of the local data set, and superimposes the noise vector with at least part of the parameters in the local model to obtain a noisy model parameter, and uploads the noisy model parameter to the server; the server is further configured to: receive the noisy model parameters from the plurality of clients, calculate the parameter similarity between any two clients according to the received noisy model parameters, and construct a proximity matrix therefrom; the server is further configured to: cluster all the clients according to the proximity matrix to obtain one or more clusters of clients, and aggregate the noisy model parameters uploaded by all the clients in each cluster to generate corresponding cluster-level global model parameters, and update the global model by distributing the cluster-level global model parameters to each client in the corresponding cluster.

[0017] Compared with the prior art, the above one or more technical solutions have the following beneficial effects: 1. The noise associated with the local data distribution is used to protect privacy, and precise clustering is performed based on the noisy model parameters. Specifically, the client generates specific noise according to its own data characteristics, adds it to the model parameters, and then uploads it; the server analyzes the noisy parameters to automatically group the clients with similar data distributions into a cluster, and generates a cluster-level global model for each cluster. This makes the noise "canceled out" during clustering, thereby protecting privacy while not compromising model performance.

[0018] 2. In the server-side aggregation stage, the proximity analysis based on the Euclidean distance and the K-means clustering algorithm are used to adaptively group the generalized parameters uploaded by the clients. By constructing a proximity matrix and identifying clusters of clients with similar parameters, fine-grained grouping and aggregation of heterogeneous models is achieved.

[0019] 3. To improve the stability of the generalized parameters, the server not only performs aggregation, but also enhances its effectiveness through knowledge sharing, preventing the formation of knowledge silos, so that clients can only obtain the knowledge of the intra-group domain model, but cannot obtain more global knowledge.

[0020] 4. In the model update link, the server distributes the grouped and aggregated generalized parameters to the corresponding clients, and each client updates the shared part of the local model accordingly, while keeping the personalized parameters unchanged. This mechanism supports periodic local training and global collaborative updating, effectively balancing the model personalization needs and generalization ability, while protecting local data privacy and promoting overall model performance. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a schematic diagram of a secure clustering federated learning process based on generalization parameters provided in one or more embodiments of the present invention; Figure 2 This is a schematic diagram of a secure clustering federated learning system architecture based on generalization parameters provided in one or more embodiments of the present invention; Figure 3 This is a schematic diagram comparing the accuracy of clustering federated learning provided by one or more embodiments of the present invention (K=10); Figure 4 This is a schematic diagram comparing the accuracy of clustering federated learning provided by one or more embodiments of the present invention (K=20); Figure 5 This is a schematic diagram illustrating the effect of the FedAvg-based method provided in one or more embodiments of the present invention; Figure 6 This is a schematic diagram illustrating the effect of clustering federated learning provided by one or more embodiments of the present invention; Figure 7 This is a schematic diagram of the communication cost of clustered federated learning provided by one or more embodiments of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] As described in the background section, existing technologies attempt to generate customized models for each client to address the issue of data heterogeneity. While personalization methods can improve the model's adaptability to local data to some extent, they suffer from insufficient privacy protection. Specifically, during personalized training, the client typically still needs to upload model parameters (such as model weights and gradients) to the server to obtain global knowledge. These parameters may still leak sensitive information from the original training data through attacks such as reverse engineering, posing a potential privacy risk.

[0026] Meanwhile, the personalized method focuses too much on fitting the local data distribution of individual clients, leading to poor performance when the model faces slight drifts in data distribution or new classes, and resulting in poor generalization.

[0027] In addition, the server needs to coordinate and manage a large number of heterogeneous personalized models, greatly increasing system complexity and communication overhead. Moreover, each client additionally undertakes the task of personalized training, which also puts higher requirements on the computing and storage resources of edge devices.

[0028] In addition to personalized federated learning, another approach is clustered federated learning, which aims to automatically divide clients with similar data distributions into the same cluster and train a cluster-level global model for each cluster. This method is between a single global model and a completely personalized model, trying to improve local performance while maintaining model generalization ability.

[0029] However, existing clustering methods often require clients to upload complete model parameters when measuring client similarity, which also brings high communication overhead and privacy risks. If random noise is simply added to the uploaded parameters for privacy protection, and the model parameters (after adding noise) are used for clustering, it will seriously interfere with the server's accurate judgment of client similarity, leading to clustering failure and ultimately harming model performance.

[0030] In summary, whether it is traditional federated learning, personalized federated learning, or initial clustered federated learning, when dealing with data heterogeneity, it is difficult to balance model performance, communication efficiency, system scalability, and privacy protection strength.

[0031] Therefore, the present scheme gives a secure clustered federated learning method based on generalized parameters, which can efficiently divide clients with non-independent and identically distributed data into appropriate clusters, reflecting the implicit correlation between local generalization model weights and client underlying data distribution. Specifically, the system measures the similarity between clients through the model weights obtained by a small number of local iterative training; to reduce communication overhead, the server only requires clients to upload a portion of the weights selected by strategy; to protect privacy, noise is introduced into the uploaded parameters.

[0032] To avoid the problem of weakening the accuracy of server-side model aggregation by directly adding random noise, the present scheme proposes a noise addition mechanism based on the distribution factor generated by the client's local data: the client generates a distribution factor using its own data, thereby generating Gaussian noise and applying it to the model parameters. This mechanism ensures privacy while effectively avoiding the interference of noise on clustering performance, thereby balancing privacy protection and model performance, protecting parameter privacy while not affecting the final clustering performance.

[0033] The related terms involved in the present scheme are as follows: K-means algorithm is a widely used clustering analysis method, which is used to divide a data set into K clusters, so that the points within the cluster are as similar as possible, while the points between the clusters are as different as possible. The algorithm optimizes the position of the cluster center by iteration to minimize the sum of the square of the distance between the points within the cluster and the cluster center.

[0034] FedAvg (federated averaging algorithm) is used to solve the problem of distributed machine learning when data is scattered and cannot be stored centrally. The core idea is to let each client train the model independently on the local data, and then send the model update to the server. The server integrates the updates of each client by weighted averaging to get the global model.

[0035] In combination Figure 1 The overall process of the present scheme is introduced, including the following steps: The client trains the model on the local data set; During training, the distribution factor is determined according to the class label of the main data in the local data set, and a noise vector consistent with the dimension of the model parameter is constructed accordingly, so as to ensure that the noise generation process is implicitly related to the data distribution of the client; After training, the client extracts the fully connected parameters of the model, and adds the generated noise vector to the fully connected parameters and uploads them to the server; After receiving the noisy parameters, the server measures the similarity between the clients based on the Euclidean distance, constructs a proximity matrix and completes the clustering grouping; The server aggregates the generalization model parameters uploaded by the clients within the group to obtain the global model update shared by the group, and distributes the aggregation result to the corresponding client.

[0036] In view of the problems of model parameter leakage risk and low clustering accuracy in clustered federated learning, the present scheme can efficiently and accurately divide the clients with non-independent and identically distributed data into corresponding clusters under the condition of relying on only a small amount of model parameters, while effectively protecting user privacy and avoiding sensitive information leakage.

[0037] The federated system of the present scheme includes two roles: the client participating in federated learning and the server , the system architecture shown in Figure 2 introduces the federated learning process of the present scheme, including the following steps: Step 1: Initialize the global model on the server and broadcast it to all clients; Step 2: After receiving the global model, each client performs training based on the local data to generate a corresponding local private model, and sorts the class labels of the main data in the local data set, and the resulting sorted result is used as the distribution factor; Step 3: The client flattens the generalization parameters of the local model, superimposes noise of the same dimension, generates noisy model parameters, and sends them to the server. Step 4: The server receives the generalized model, and uses the Euclidean distance to adaptively cluster the model parameters. Specifically, the server calculates the Euclidean distance between each pair of parameters, constructs a proximity matrix, and uses the K-means algorithm to cluster it to obtain the grouping result. For each group, the generalization parameters are aggregated, and finally the obtained generalization parameters are sent to the corresponding client. Step 5: After receiving the generalization model parameters, the client replaces the local model's fully connected parameters with the obtained generalization model parameters, and completes the update of the local model. The updated model is used for the next round of training.

[0038] Steps 2-5 are iteratively executed.

[0039] This scheme uses noise associated with local data distribution to protect privacy, and performs precise clustering based on noisy model parameters. Specifically, the client generates specific noise according to its own data characteristics, adds it to the model parameters, and then uploads them; the server analyzes these noisy parameters, automatically groups clients with similar data distributions into a cluster, and generates a cluster-level global model for each cluster. This allows the noise to be "offset" during clustering, thereby protecting privacy while not compromising model performance.

[0040] The detailed process of this scheme is described below.

[0041] In step 1, the server initializes the global model and distributes it to all clients .

[0042] In step 2, the client receives the global model, trains the model using the local data set to obtain a private model , as shown in the following formula: ; where is the gradient during training, , is the learning rate.

[0043] Then, the k most frequent class labels in the local data set are counted, as shown in the following formula: ; where y is the class label of the client's local data set, and the top k class labels are sorted to obtain the distribution factor, i.e.​ .

[0044] Distribution factor It can be obtained based on a hash-based deterministic generation method, specifically: hash =SHA256( ) , and the first 4 bits of hash are taken as the random seed.

[0045] The distribution factor in the present scheme delta is used to control the disturbance strength introduced in the local clustering process or the data partition strategy, and its value directly affects the privacy protection level and global model convergence performance of the federated learning system. To achieve controllable balance between the two, delta it is not randomly generated, but derived through a deterministic hash mechanism: specifically, calculate hash SHA256 ( delta ), and take the first 4 bytes (i.e. 32 bits) of the hash value as the seed of the subsequent pseudo-random number generator (PRNG).

[0046] Considering the problems of determinism and reproducibility, the same delta value generates the same random seed in any participant or training round, thereby ensuring that the disturbance or sampling behavior is logically consistent. This avoids model oscillation caused by completely random disturbance, helps to improve the stability and convergence speed of federated clustering, and alleviates the performance degradation caused by excessive noise introduction.

[0047] Secondly, delta It can itself serve as a privacy budget (such as associated with differential privacy ) or a device heterogeneity measurement parameter. By mapping it to a deterministic seed, the system can accurately bind the privacy protection strength (determined by delta ) and the randomization operation, so that stronger privacy requirements (such as smaller correspond to greater disturbance) automatically trigger the corresponding strength of randomization without additional communication negotiation. This coupling mechanism realizes the adaptive co-optimization of privacy protection and model utility.

[0048] In summary, the hash-based deterministic seed generation mechanism not only guarantees the reproducibility and efficiency of the federated clustering process, but also builds a privacy-performance joint regulation channel driven by the distribution factor δ, effectively supporting the design goal of maintaining high model accuracy under strict privacy constraints.

[0049] In step 3, each client participating in training decouples the parameters based on its local model structure.

[0050] Assume that the model of the client is composed of layer neural networks, and its overall parameters are represented as: ; wherein i denotes the i th client.

[0051] This parameter is divided into two parts: the first L-b layers constitute the personalized parameter , which is used to capture the specific features in the local data and preserve the uniqueness of the individual data distribution; the last b layers are regarded as the generalization parameter , which is responsible for extracting the high-level semantic information and general representation shared by all clients.

[0052] To enhance the privacy security of the generalization parameter in the communication process, the client introduces a noise disturbance mechanism based on the distribution factor.

[0053] Specifically, the client initializes a pseudo-random number generator using a pre-set or agreed distribution factor , and generates a noise vector that conforms to the Gaussian distribution based on it, which has the same dimension as . Then the noise vector is added to the original generalization parameter to obtain the noisy generalization parameter: This operation can improve the anti-inference ability in the parameter transmission process without significantly affecting the model performance.

[0054] Finally, the client only uploads the noisy generalization parameter to the server, while the personalized parameter remains local and does not participate in communication, thereby achieving a balance between privacy protection and communication efficiency.

[0055] In step 4, the server successively receives the noisy generalization parameters uploaded from N clients.

[0056] To mine the similarity of models between different clients and achieve fine-grained aggregation, the server first calculates the Euclidean distance between the generalization parameters of any two clients to measure their geometric proximity in the parameter space.

[0057] For client i and client j , the distance is defined as: ; where and are the noisy generalization parameters of client i and client j , and are the noisy generalization parameters of client i and client jthe first element of the noised generalization parameter vector. k

[0058] Based on the distance calculation results between all pairs, the server constructs a symmetric proximity matrix where the diagonal elements are zero and the off-diagonal elements reflect the parameter difference. Subsequently, the server uses the proximity matrix as input and adopts the K-means clustering algorithm to adaptively group the N generalization parameters.

[0059] K-means finds the optimal clustering structure by minimizing the within-group sum of squares and finally divides the clients into K non-overlapping clusters, denoted as the grouping result where each cluster contains a set of clients with similar parameter patterns.

[0060] After completing the clustering, the server performs a group aggregation operation on the generalization parameters within each cluster to calculate the average generalization parameter of the group: where represents the number of clients in the kth group. The aggregation result

[0061] represents the global generalization knowledge shared by the group of clients. (2) The server aggregates the knowledge of other groups: ; where represents the number of clients in the kth group; (3) Knowledge sharing: ; where represents the number of rounds of knowledge sharing, V is the strength of knowledge acquisition, which is set to 0.1 in this scheme; (4) The aggregation result enhanced by knowledge sharing is issued to the clients.

[0062] Through knowledge sharing, knowledge islands are prevented, and the robustness of the model is improved. Specifically, traditional schemes (such as CFL) will only simply aggregate the models within the group and then issue them to the clients. This approach will make the clients only get the knowledge of the models within the group, and will not get more global knowledge; which is reflected in specific scenarios, for example, the recommendation algorithm will not only recommend the most interesting content to the user, but will also recommend some other information, such as news and other content.

[0063] ​​​​To solve the above problems, the scheme designs a double model (in-group-out-group) to provide global knowledge through proximal optimization (knowledge sharing) and improve the robustness of the model. Through the design of the optimization target, the in-group model is responsible for providing personalized knowledge, and the out-group model is responsible for providing global knowledge. Proximal optimization is a means of acquiring knowledge.

[0064] In step 5, the client receives the global aggregated generalization parameters corresponding to the cluster it belongs to issued by the server At this time, the client replaces the generalization part in its local model with this updated shared parameter, i.e. reconstructs its overall model parameter as where the personalized parameter remains unchanged, continuing to retain the specificity features learned locally. This "local retention, global update" strategy not only ensures the adaptability of the model to local data, but also integrates common knowledge from similar clients, which helps to improve the generalization performance and convergence speed of the model.

[0065] Subsequently, the client uses the updated complete model to perform the next round of forward propagation and backward update on the local data set, and continues to participate in subsequent training iterations. Through this periodic parameter decoupling, noisy uploading, clustering aggregation and parameter replacement mechanism, the system can effectively deal with the challenges brought by data heterogeneity among clients while protecting data privacy and reducing communication overhead, and promote federated learning to be more efficient and robust.

[0066] In existing clustering federated learning schemes, the uploaded complete model parameters of the client are relied on. This approach not only has high communication cost but also has unsatisfactory model clustering performance. In the present scheme, only the fully connected layer of the model parameters is uploaded. It is found through experiments that the fully connected layer has the greatest impact on clustering effect, so this design is chosen.

[0067] Correspondingly, the secure clustering federated learning system based on generalization parameters comprises: a server configured to initialize a global model and send the global model to all participating clients; a client configured to receive the global model, train the global model based on a local data set to obtain a local model, during the training, the client generates a noise vector with the same dimension as at least one part of the parameters in the local model according to the data distribution of the local data set, and superimposes the noise vector with the at least one part of the parameters in the local model to obtain a noisy model parameter, and uploads the noisy model parameter to the server; the server is further configured to receive the noisy model parameters from multiple clients, calculate the parameter similarity between any two clients according to the received noisy model parameters, and construct a proximity matrix based on the parameter similarity; The server is further configured to: cluster all the clients according to the proximity matrix to obtain one or more client clusters, and aggregate all the noisy model parameters uploaded by the clients in each cluster to generate corresponding cluster-level global model parameters, and realize global model updating by delivering the cluster-level global model parameters to each client in the corresponding cluster.

[0068] Experimental verification.

[0069] For the above scheme, experiments are carried out on the public data set CIFAR-100. The experiments use VGG13 model and ResNet18 model as training model, and the mode is: full parameter training. The data division simulates the non-independent and identically distributed scene, and the specific setting is that each client randomly allocates 10 categories (K=10) or 20 categories (K=20).

[0070] The comparative baseline method is FedProx, which is an optimization algorithm for federated learning, aiming to solve the model convergence problem caused by the data distribution heterogeneity between client devices. It introduces a model update term (i.e. proximal term) into the local optimization objective function to limit the local model update from deviating too far from the global model. This method enhances the training stability, especially suitable for real scenarios where device computing power is different or data is non-independent and identically distributed (Non-IID).

[0071] The accuracy comparison results are as shown in Figure 3 -allocating 10 categories (K=10) and Figure 4 -allocating 20 categories (K=20) as shown.

[0072] In addition, in order to compare the clustering performance, the clients in the above data set are divided into five groups, and the effect of the existing clustering federated learning method (CFL) is tested as shown in Figure 5 The clustering effect of the clustering federated learning scheme of the present scheme is as shown in Figure 6 .

[0073] The communication cost of the CFL scheme is 131.5M*4=526MB and 11.7M*4=46.8MB respectively, while the present scheme is 103.2M*4=412.8MB and 5.8M*4=23.2MB respectively. According to the communication cost shown in Figure 7 The present scheme reduces the communication cost by more than 15%.

[0074] It can be seen that the traditional clustering federated learning clusters the model parameters of the clients, groups the clients according to the clustering results, and then performs personalized training. The traditional method does not consider the protection of the model parameters in the clustering stage. If the model parameters are directly added with noise or encrypted, and then the model parameters (after adding noise) are clustered, the clustering effect will be significantly reduced. Therefore, the traditional clustering federated learning scheme usually prioritizes the clustering effect and does not consider privacy protection.

[0075] In the clustering federated learning, by clustering the model parameters, the clients with similar data distribution are divided into the same group for personalized training. After clustering, the clients are labeled with data tags. The noise generated by the clients with consistent data tags is consistent, and the noise added to the model parameters will not affect the final clustering effect (Euclidean distance). Therefore, the present scheme ranks the top k categories with the largest number as the distribution factor. In this way, the privacy of the parameters can be protected, and the final clustering performance will not be affected. In this way, the noise is associated with the local data distribution, so that the noise vectors of the clients in the same group will be the same, and the final clustering performance will not be affected. Therefore, the present scheme protects the parameter privacy while not affecting the final clustering performance.

[0076] The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A secure clustering federated learning method based on generalization parameters, characterized in that, Includes the following steps: The server initializes the global model and sends it to all participating clients; The client receives the global model, trains the global model based on the local dataset to obtain the local model; during training, the client generates a noise vector with the same dimension as the parameters to be uploaded in the local model according to the data distribution of the local dataset, and superimposes it with at least some of the parameters in the local model to obtain the noisy model parameters, and uploads the noisy model parameters to the server. The server receives noisy model parameters from multiple clients, calculates the parameter similarity between any two clients based on the received noisy model parameters, and constructs a proximity matrix accordingly. The server clusters all clients based on the proximity matrix to obtain one or more client clusters. It then aggregates the noisy model parameters uploaded by all clients within each cluster to generate corresponding cluster-level global model parameters. By distributing the cluster-level global model parameters to each client in the corresponding cluster, the global model is updated.

2. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, The client generates a noise vector with the same dimension as the parameters to be uploaded in the local model based on the data distribution of the local dataset. Specifically, it counts the k category labels with the most samples in the local dataset, uses the sorting result of the category labels as the distribution factor to characterize the data distribution, and uses the distribution factor to initialize the pseudo-random number generator to obtain a noise vector that conforms to a Gaussian distribution. The distribution factor is generated by a hash function, and the first few bytes of the hash value are used as the seed for the subsequent pseudo-random number generator.

3. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, The parameters in the local model are divided into personalized parameters and generalized parameters. Personalized parameters capture specific features in local data, while generalized parameters represent high-level semantic information and general representations shared across clients. The generalized parameters are superimposed with the noise vector to obtain the noisy model parameters. The noisy model parameters are uploaded to the server, while the personalized parameters are retained on the client and do not participate in communication.

4. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, Based on the received noisy model parameters, the server calculates the parameter similarity between any two clients. Specifically, the server calculates the Euclidean distance between the noisy model parameters of any two clients.

5. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, Constructing a proximity matrix involves building a symmetric proximity matrix based on the parameter similarity calculation results between any two clients. The diagonal elements are zero, and the off-diagonal elements reflect the differences between the parameters of the noisy model.

6. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, The server clusters all clients based on the proximity matrix to obtain one or more client clusters. Specifically, the server uses the obtained proximity matrix as input and uses the K-means clustering algorithm to group N noisy model parameters. The resulting client clusters contain sets of clients with similar parameter patterns.

7. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, The noisy model parameters uploaded by all clients within each cluster are aggregated to generate the corresponding cluster-level global model parameters. Specifically, this is done by calculating the average generalization parameter within each cluster. This reflects the global generalized knowledge shared by the clients of this cluster.

8. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, During the aggregation of noisy model parameters uploaded by all clients within each cluster, the effect is enhanced through knowledge sharing, including: The server aggregates the knowledge of this group, specifically: ; The server aggregates knowledge from other groups, specifically: ; Knowledge sharing, specifically: ; The aggregation results enhanced by knowledge sharing Send to the client; in, Indicates the client i Generalization parameters after adding noise This represents the number of clients in the k-th group. This represents the number of clients excluding the k-th group. Indicates the number of rounds of knowledge sharing. V It refers to the intensity of knowledge acquisition.

9. The secure clustering federated learning method based on generalization parameters as described in claim 1, characterized in that, Global model updates are achieved by distributing cluster-level global model parameters to each client within the corresponding cluster. Specifically, each client receives the average generalization parameter corresponding to its cluster, which is distributed by the server. Replace the generalization parameters in its local model with the received average generalization parameters. This involves reconstructing the overall model parameters to achieve a global model update.

10. A secure clustering federated learning system based on generalization parameters, characterized in that, include: The server is configured to initialize the global model and send it to all participating clients. The client is configured to: receive the global model, train the global model based on the local dataset to obtain the local model; during training, the client generates a noise vector with the same dimension as the parameters to be uploaded in the local model according to the data distribution of the local dataset, and superimposes it with at least some of the parameters in the local model to obtain the noisy model parameters, and uploads the noisy model parameters to the server. The server is also configured to: receive noisy model parameters from multiple clients, calculate the parameter similarity between any two clients based on the received noisy model parameters, and construct a proximity matrix accordingly; The server is also configured to: cluster all clients based on the proximity matrix to obtain one or more client clusters, aggregate the noisy model parameters uploaded by all clients in each cluster to generate corresponding cluster-level global model parameters, and update the global model by distributing the cluster-level global model parameters to each client in the corresponding cluster.

Images