Self-adaptive clustering federal learning method based on knowledge migration

Abstract

The invention provides an adaptive clustering federal learning method based on knowledge migration. In each round of training of the central server, the server calculates a similarity matrix according to client model parameters, and performs adaptive clustering grouping on the clients; performing weighted average aggregation on the client model parameters in the same group to obtain an intra-group model; through a knowledge migration strategy, the group model is optimized, knowledge sharing between different groups is promoted, and the merging and clustering process is accelerated; and after the self-adaptive clustering is completed, carrying out intra-group federation training. And the server performs federal average aggregation on the model parameters of the clients in the group, and updates the group model. Inputting local data into the updated model to obtain a prediction result; and constructing a loss function between a prediction result and a real label, and repeating the training step by taking loss minimization as a target. According to the training method provided by the invention, the problems of deviation and difficulty in convergence caused by data heterogeneity can be effectively solved, and the individuation and generalization performance of the model can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to an adaptive clustering federated learning method based on knowledge transfer. Background Art

[0002] In recent years, with the rapid development of big data and artificial intelligence, data privacy and security issues have become increasingly prominent. Traditional centralized machine learning methods require that all data be concentrated on the server for training, which can easily lead to data leakage and privacy violations. In order to solve this problem, federated learning has been proposed as a new distributed machine learning framework. It allows multiple clients to jointly train a global model without sharing the original data, thereby protecting data privacy. However, in practical applications, client data usually has the characteristics of non-independent and identically distributed (Non-IID), that is, there are differences in the data distribution of each client. This data heterogeneity will cause the performance of the global model to degrade on some clients, and may even fail to converge, which brings huge challenges to federated learning.

[0003] In order to solve the problems caused by non-IID data, researchers have proposed clustering federated learning methods at different times. In 2018, Smith et al. proposed the MOCHA algorithm, which regards federated learning as a multi-task learning problem and solves data heterogeneity through an optimization framework. Subsequently, in 2019, Sattler et al. proposed a clustering-based federated learning method CFL, which dynamically clusters clients by measuring the similarity between model updates. In 2020, Briggs et al. proposed a federated learning algorithm for non-IID data, improving performance through clustering and personalized models.

[0004] However, these methods usually require the number of clusters to be set in advance, or the computational complexity is high during the clustering process, and they lack the ability to adapt to the dynamic changes in client data distribution. At the same time, the lack of an effective knowledge sharing mechanism between different clusters may lead to insufficient generalization of the model. Summary of the invention

[0005] In view of the above shortcomings, the present invention proposes an adaptive clustering federated learning method based on knowledge transfer. The method includes: by introducing the knowledge transfer technology in multi-task learning, effective information sharing between different clusters is realized, and the generalization ability of the model is enhanced; there is no need to pre-set the number of clusters, and adaptive clustering can be completed in a few rounds according to the similarity of the clients; by aggregating the models within the clusters, the frequency of global communication is reduced, the communication overhead is reduced, and the training efficiency is improved. This innovation not only solves the model bias and convergence problems caused by data heterogeneity, but also achieves a dual improvement in model performance and communication efficiency under the premise of protecting data privacy. The process of training an adaptive clustering federated learning method based on knowledge transfer includes:

[0006] S1: System initialization, including client model and parameter initialization, similarity matrix and entity set initialization;

[0007] S2: Adaptive clustering stage, using greedy aggregation strategy and knowledge transfer technology. The greedy strategy maximizes the minimum similarity to cluster similar entities. Knowledge transfer promotes knowledge sharing between different groups and speeds up the merging clustering process until the classification of all client groups no longer changes.

[0008] S3: In-group federated learning phase, in this phase the remaining federated learning process is carried out within the group to complete the final process of minimizing the loss function.

[0009] Preferably, the process of initializing the system includes: determining the number of clients m participating in federated learning, each client randomly initializing its own model parameters It is necessary to set parameters such as client participation ratio c, local learning rate η, local training round number T, etc.; it is also necessary to initialize the similarity matrix ɑ (0) , set all elements to -1; initialize entity set E (0) .

[0010] Preferably, the S2 adaptive clustering stage includes:

[0011] S21: The server selects a fixed-size random client subset from the client to participate in the current training. The selected client M (r) Train the model on local data to generate model W i (r) Upload to the server;

[0012] S22: The server calculates the client model update difference ΔW i (r) , calculate the cosine similarity between entities and update the similarity matrix α (r) , used to merge similar entities;

[0013] S23: For all possible entity pairs (including individual clients and aggregated client groups), the server identifies and merges the entity pairs E1 and E2 that maximize the minimum similarity, and forms an independent client group according to the merging rule;

[0014] S24: Update group model parameters for each client group We further introduce knowledge transfer Prox(G new ), to achieve knowledge sharing between different groups;

[0015] S25: The server updates the global model based on the existing group model. The client can use the updated global model G in the next round of local updates. (r) ;

[0016] S26: Repeat steps S21 to S25 until no new client group is generated in N consecutive rounds.

[0017] Preferably, in S22, the client model update difference ΔW is calculated i (r) The formula is:

[0018] ΔW i (r) =W i (r) -G (r-1)

[0019] Among them, W i (r) represents the local model parameters uploaded by client i to the server in round r, G (r-1) : The global model parameters sent by the server to the client in round r-1.

[0020] Furthermore, the cosine similarity between clients The formula is:

[0021]

[0022] in, Similarity measure between client i and client j in round r, <·,·> represents the inner product (dot product) of two vectors, ∥·∥: represents the Euclidean norm (L2 norm) of the vector.

[0023] Further, construct the similarity matrix α (r) , α (r) is a symmetric matrix with elements represents the similarity between client i and client j in round r.

[0024] Preferably, each client group set in S24 has a group model, and the participating clients in each round are weighted averaged within their set, and the formula for updating the group model is:

[0025]

[0026] Among them, C t is the set of clients participating in this round of training in the current group t, n i is the number of local data samples of client i.

[0027] Furthermore, the formula for inter-group knowledge transfer is:

[0028]

[0029] Among them, G old and G new are all group model parameters before and after proximal optimization; ∥·∥ F represents the Frobenius norm; λ is the regularization parameter, g(·) is the regularization function, and the L2 norm is usually chosen.

[0030] Preferably, in S25, the global model update formula is:

[0031]

[0032] in, represents the total number of data samples of all clients in group t, Represents the total number of data samples for all clients in all groups.

[0033] Preferably, the S3 intra-group federation training process includes:

[0034] S31: In each round r, the server randomly selects a proportion c of clients from each group to participate in the training. The selected client i receives the group model of the group to which it belongs. Update the local model, and the client will update the model W i (r) Upload back to the server;

[0035] S32: Similar to step S25, group model parameter aggregation and inter-group knowledge transfer for each client group;

[0036] S33: Repeat steps S31-S32 until the model converges or the maximum number of iterations is reached, and then stop training.

[0037] The adaptive clustering federated learning method based on knowledge transfer of the present invention comprehensively solves the key challenges of federated learning in a non-independent and identically distributed data environment, improves model performance, reduces communication overhead, and enhances the applicability and practical value of the algorithm; the present invention solves the problems of model performance degradation and convergence difficulty caused by non-IID data through adaptive clustering, and improves the personalization degree and training efficiency of the model by dynamically clustering similar clients; the present invention optimizes the clustering process through similarity calculation, reduces computational complexity and communication overhead, and makes the algorithm more efficient and more suitable for practical applications; the present invention solves the problem of lack of information sharing between different clusters through a knowledge transfer mechanism, and promotes knowledge sharing by introducing regularization terms between group models, improves the generalization ability of the model, and prevents overfitting BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall flow chart of the present invention;

[0039] Figure 2 This is a diagram showing an embodiment of the step S2 adaptive clustering stage of the present invention;

[0040] Figure 3 This is a diagram showing an implementation example of the intra-group federated learning phase in step S3 of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] An adaptive clustering federated learning method based on knowledge transfer, implementing e.g. Figure 1 As shown, it includes the following steps:

[0043] Step S1, system initialization, including client model and parameter initialization, similarity matrix and entity set initialization;

[0044] Specifically, it includes two parts:

[0045] (1) Model parameter initialization: Determine the number of clients m participating in federated learning, and each client randomly initializes its own model parameters It is necessary to set the client participation ratio c and the number of clients actually participating in training in each round. in Indicates rounding down. Parameters such as local learning rate η and local training round number T are used for subsequent training.

[0046] (2) Similarity matrix entity set initialization: Initialize the similarity matrix α (0) , set all elements to -1, indicating that the similarity between all clients has not been calculated; initialize the entity set E (0) , contains all m clients, each client is an independent entity, and the entity set is used to record the group member model and the number of groups.

[0047] Step S2, implementation diagram Figure 2 As shown in the figure, in the adaptive clustering stage, greedy aggregation strategy and knowledge transfer technology are used. The greedy strategy maximizes the minimum similarity to cluster similar entities. Knowledge transfer promotes knowledge sharing between different groups and speeds up the merging clustering process until the classification of all client groups no longer changes.

[0048] Specifically, the method comprises the following steps performed in sequence:

[0049] S21: The server selects a fixed-size random client subset from the client to participate in the current training. The selected client M (r) Train the model on local data to generate model W i (r) Upload to the server.

[0050] The client will go through three steps: receiving server parameters, updating the local training model, and uploading the model locally:

[0051] (1) Each randomly selected client i receives the global model G trained in the previous round from the server (r-1) ;

[0052] (2) The client performs T rounds of local updates based on its local data and the global model. The client uses stochastic gradient descent (SGD) or other optimization algorithms to minimize the local loss function F i (G (r-1) ), denoted as And update the model parameters;

[0053] (3) The client sends its model W i (r) Upload back to the server.

[0054] Specifically, the local loss function F i Defined as:

[0055]

[0056] Among them, n i represents the number of samples of local data of client i; l(W; x ij ,y ij ) is the loss function, which measures the effect of model parameters W on samples (xij ,y ij ) prediction error.

[0057] Specifically, client i calculates the loss function F i Gradient with respect to model parameters:

[0058]

[0059] Among them, this step is used to calculate the global model parameter G (r-1) The average gradient of the model on the local data.

[0060] Specifically, the client model update formula is:

[0061]

[0062] Among them, the global model parameter G (r-1) As a starting point, the parameters are updated along the negative gradient direction of the loss function (that is, the direction in which the loss function decreases fastest), and the learning rate η controls the magnitude of the update.

[0063] In applications, clients usually perform T rounds of local updates, not just one round. This means that the client will repeat the following update steps T times locally:

[0064]

[0065] Through multiple iterations, the client can make better use of local data and improve the performance of the model on local data.

[0066] S22: The server calculates the client model update difference ΔW i (r) , calculate the cosine similarity between entities and update the similarity matrix α (r) , used to merge similar entities;

[0067] Specifically, calculate the client model update difference ΔW i (r) The formula is:

[0068] ΔW i (r) =W i (r) -G (r-1) (5)

[0069] Among them, W i (r) represents the local model parameters uploaded by client i to the server in round r, G (r-1) : The global model parameters sent by the server to the client in round r-1, ΔW i(r) It represents the update direction and magnitude of the model parameters by client i on the local data, reflecting the impact of its local data on the model.

[0070] Specifically, calculate the cosine similarity between clients The formula is:

[0071]

[0072] in, The similarity measure between client i and client j in round r, <·,·> represents the inner product (dot product) of two vectors, ||·||: represents the Euclidean norm (L2 norm) of the vector. When , it means that the directions of the two vectors are exactly the same, and the model update directions of clients i and j are consistent, indicating that their local data may have similar distributions; when When , it means that the two vectors are in opposite directions, and the model is updated in opposite directions, which may indicate that there are differences in local data; when , it means that the two vectors are orthogonal and the model update direction is irrelevant.

[0073] Specifically, construct the similarity matrix α (r) is a matrix of pairs, where the elements Represents the similarity between client i and client j in round r. For all clients i,j∈M participating in this round of training (r) ,calculate Fill the calculated similarity into the corresponding position of the matrix.

[0074] S23: For all possible entity pairs (including individual clients and aggregated client groups), the server identifies and merges the entity pairs E1 and E2 that maximize the minimum similarity, and forms an independent client group according to the merging rule;

[0075] Specifically, before merging two entities, the following two conditions must be met at the same time:

[0076] Condition 1: The entity satisfies the similarity formula:

[0077]

[0078] in, is the maximum similarity between two entities, is the minimum similarity within each entity.

[0079] Condition 2, minimum similarity threshold:

[0080]

[0081] Minimum similarity between merged entities must be ensured Greater than the preset threshold α0.

[0082] For all entity pairs that meet the above conditions (including individual clients and aggregated client groups), the following merge rules are executed: 1. When both are single clients, create a new client group including the two clients; 2. When one is a client and the other is a client group, add the client to the client group; 3. When both are client groups, merge the two client groups to form a new client group.

[0083] After the merge is complete, update the entity set E (r) and the similarity matrix α (r) , to reflect the new clustering structure.

[0084] S24: Update group model parameters for each client group We further introduce knowledge transfer Prox(G new ), to achieve knowledge sharing between different groups;

[0085] Specifically, each client group set has a group model. The participating clients in each round are weighted averaged within their set. The formula for updating the group model is:

[0086]

[0087] in, is the group model parameter of group t in the rth round of training, C t is the set of clients participating in this round of training in the current group t, n i is the number of local data samples of client i. The server performs weighted averaging on the local models of the clients in the group to obtain a new group model. The weight of each client is the number of its local data samples n i , the total weight is the total number of data samples of all clients in the group

[0088] Specifically, the formula for inter-group knowledge transfer is:

[0089]

[0090] Among them, G old and G new are all group model parameters before and after proximal optimization; ∥·∥ F represents the Frobenius norm, which is used to measure the difference between two matrices and calculate the amplitude of model parameter update; λ is the regularization parameter, g(·) is the regularization function, and the L2 norm is usually selected.

[0091] S25: The server updates the global model based on the existing group model. The client can use the updated global model G in the next round of local updates. (r) ;

[0092] Specifically, the global model update formula is:

[0093]

[0094] in, represents the total number of data samples of all clients in group t, Represents the total number of data samples for all clients in all groups.

[0095] S26: Repeat steps S21 to S25 until no new client group is generated in N consecutive rounds.

[0096] Step S3, implementation diagram Figure 3 In the intra-group federated learning phase, the remaining federated learning process is performed within the group to complete the final minimization of the loss function L all process.

[0097] Specifically, the method comprises the following steps performed in sequence:

[0098] S31: In each round r, the server randomly selects a proportion c of clients from each group to participate in the training. The selected client i receives the group model of the group to which it belongs. Update the local model, and the client will update the model W i (r) Upload back to the server;

[0099] S32: Similar to step S25, group model parameter aggregation and inter-group knowledge transfer for each client group;

[0100] S33: Repeat steps S31-S32 until the model converges or the maximum number of iterations is reached, and then stop training.

[0101] Specifically, in S3, the loss function L all The group model loss L t and knowledge transfer regularization L Prox The overall optimization objective function expression is L all =L t +L Prox .

[0102] Specifically, the group model loss L t Defined as the weighted average of all client losses in the group:

[0103]

[0104] Among them, F i (G t ) is the client i in the model parameter G t The local loss function under .

[0105] Specifically, the knowledge transfer regularization loss function L Prox for:

[0106]

[0107] Among them, l is the total number of groups, and λ is the regularization parameter that controls the strength of knowledge transfer.

[0108] Specifically, the overall optimization goal L all It is expressed as:

[0109]

[0110] in, is the total number of data samples from all clients, It is the proportion of the number of data samples in group t in the total data, which is used to weight the group loss so that the group with a large amount of data contributes more to the total loss.

[0111] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An adaptive clustering federated learning method based on knowledge transfer, characterized in that: include: When initializing the client it is considered a separate "entity"; Adaptive aggregation strategy, no need to pre-specify the number of clusters; calculate similarity based on the client's gradient update, and aggregate the most similar clients or client groups; Regularization terms are introduced between group models to promote knowledge sharing between different groups. The process of adaptive clustering federated learning based on knowledge transfer includes: S1: System initialization, including client model and parameter initialization, similarity matrix and entity set initialization; S2: Adaptive clustering stage, using greedy aggregation strategy and knowledge transfer technology. The greedy strategy maximizes the minimum similarity to cluster similar entities. Knowledge transfer promotes knowledge sharing between different groups and speeds up the merging clustering process until the classification of all client groups no longer changes. S3: In-group federated learning phase, in this phase the remaining federated learning process is carried out within the group to complete the final process of minimizing the loss function.

2. The adaptive clustering federated learning method based on knowledge transfer according to claim 1, characterized in that: In the S1 initialization phase, it is necessary to determine the number of clients m participating in federated learning and each client randomly initializes its own model parameters. It is necessary to set parameters such as client participation ratio c, local learning rate η, local training round number T, etc.; initialize the similarity matrix α (0) , set all elements to -1; initialize entity set E (0) .

3. The adaptive clustering federated learning method based on knowledge transfer according to claim 1, characterized in that: In the S2 adaptive clustering stage, 1···r rounds of training are required, and each round includes the following steps: S21: The server selects a fixed-size random client subset from the client to participate in the current training. The selected client M (r) Train the model on local data to generate model W i (r) Upload to the server; S22: The server calculates the client model update difference ΔW i (r) , calculate the cosine similarity between entities and update the similarity matrix α (r) , used to merge similar entities; S23: For all possible entity pairs (including individual clients and aggregated client groups), the server identifies and merges the entity pairs E1 and E2 that maximize the minimum similarity, and forms an independent client group according to the merging rule; S24: Update the group model parameter G of each client group t (r) , we further introduce knowledge transfer Prox(G new ), to achieve knowledge sharing between different groups; S25: The server updates the global model based on the existing group model. The client can use the updated global model G in the next round of local updates. (r) ; S26: Repeat steps S21 to S25 until no new client group is generated in N consecutive rounds.

4. The adaptive clustering federated learning method based on knowledge transfer according to claim 1, characterized in that: Federated learning training in an S3 group requires 1…r rounds of training, and each round includes the following steps: S31: In each round r, the server randomly selects a proportion c of clients from each group to participate in the training. The selected client i receives the group model G of its group. t (r-1) , update the local model, and the client will update the model W i (r) Upload back to the server; S32: group model parameter aggregation for each client group and inter-group knowledge transfer; S33: Repeat steps S31-S32 until the model converges or the maximum number of iterations is reached, and then stop training.

5. The adaptive clustering federated learning method based on knowledge transfer according to claim 3, characterized in that: S22 calculates the client model update difference ΔW i (r) The formula is: ΔW i (r) =In i (r) -G (r-1) Among them, W i (r) represents the local model parameters uploaded by client i to the server in round r, G (r-1) Represents the global model parameters sent by the server to the client in round r-1. Calculate cosine similarity between clients The formula is: in, Similarity measure between client i and client j in round r, <·,·> represents the inner product of two vectors, ||·||: represents the Euclidean norm (L2 norm) of the vector. Construct similarity matrix α (r) , α (r) is a symmetric matrix with elements represents the similarity between client i and client j in round r.

6. The adaptive clustering federated learning method based on knowledge transfer according to claim 3, characterized in that: In S24, the group model update formula is: Among them, C t is the set of clients participating in this round of training in the current group t, n i is the number of local data samples of client i. The formula for inter-group knowledge transfer is: Among them, G old and G new are the model parameters of all groups before and after proximal optimization, respectively; ||·|| F represents the Frobenius norm, which is used to measure the difference between two matrices and calculate the amplitude of model parameter update; λ is the regularization parameter, g(·) is the regularization function, and the L2 norm is usually selected.

7. The adaptive clustering federated learning method based on knowledge transfer according to claim 3, characterized in that: In S25, the global model update formula is: in, represents the total number of data samples of all clients in group t, Represents the total number of data samples for all clients in all groups.

Images