Personalized heterogeneous federal learning method based on similarity distillation

Abstract

The invention discloses a personalized heterogeneous federal learning method based on similarity distillation, and belongs to the field of data security, and the federal learning method comprises the following specific steps: Q1, selecting a plurality of groups of clients to carry out local training, and constructing a local representation; q2, the server side evaluates the representation similarity between the clients and generates a personalized prototype for each client; q3, transmitting the personalized consensus back to a local client, and training a local model by minimizing personalized integration loss; according to the method, the output of the client model can be adaptively aggregated, a plurality of different knowledge can be generated and transmitted to the corresponding model, the knowledge collaboration efficiency is improved, compared with the traditional global consensus, the knowledge composition can be adaptively adjusted to adapt to the bearing capacities of different models, independent training can be performed, aggregation is performed in a personalized manner, and the knowledge collaboration efficiency is improved. The method is suitable for the resource-limited edge device.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a personalized heterogeneous federated learning method based on similarity distillation. Background Art

[0002] The birth of federated learning has demonstrated a distributed training paradigm for machine learning models that implicitly utilizes massive user data from edge devices by exchanging model parameters, while effectively ensuring user data privacy. Due to its features of protecting user privacy and efficient use of network bandwidth, federated learning has been widely used in edge networks, recommendation systems, human activity recognition and other fields. The traditional (Vallian) federated learning algorithm FedAvg[1] and its related improvements are based on distributed collaborative training to obtain a good global model. In recent years, all-for-one has gradually transformed into one-for-all. Considering that user data participating in federated training usually presents non-independent and identically distributed characteristics, the concept of personalized federation has emerged, and all-for-one has gradually transformed into one-for-all. On the other hand, new neural network models are emerging in an endless stream. Traditional federation can only train a single architecture, which means that models with customized model architectures cannot participate in federated training. With the development of large-scale language and vision models, the diversity of model architectures and the development of parameter intellectual property rights mean that traditional federation is difficult to play a role. At the same time, the dependence on a unified model in classical methods also brings the possibility of federated reverse attacks, which will weaken the advantages of federated learning such as privacy protection and knowledge collaboration. Although there are various high-quality public data sets that can train a good single model, the diversity of massive user data and the need for personalized end-side models cannot be ignored; therefore, it is particularly important to invent a personalized heterogeneous federated learning method based on similarity distillation.

[0003] Existing personalized heterogeneous federated learning methods cannot adaptively aggregate the output of client models, reducing the efficiency of knowledge collaboration. At the same time, they cannot adapt to the carrying capacity of different models and are not suitable for resource-constrained edge devices. To this end, we propose a personalized heterogeneous federated learning method based on similarity distillation. Summary of the invention

[0004] The purpose of the present invention is to solve the defects in the prior art and propose a personalized heterogeneous federated learning method based on similarity distillation.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A personalized heterogeneous federated learning method based on similarity distillation. The specific steps of this federated learning method are as follows:

[0007] Q1. Select multiple groups of clients for local training and build local representations.

[0008] Q2, the server evaluates the representation similarity between clients and generates a personalized prototype for each client;

[0009] Q3. Pass the personalized consensus back to the local client and train the local model by minimizing the personalized ensemble loss.

[0010] Q4: Aggregate the classification header parameters uploaded by each client and fine-tune the aggregated global classification header;

[0011] Q5. Send the updated global classification head to each client and conduct the next round of local training.

[0012] As a further solution of the present invention, the specific steps of selecting multiple groups of clients for local training and constructing local representations in step Q1 are as follows:

[0013] S1.1: Set the number of clients K, local private dataset, global round number T, and local round number τ for each round of training, and initialize the heterogeneous local model for each client The server initializes the global classification header parameter φ (0) ,in, represents the k-th client heterogeneous local model, and k∈K;

[0014] S1.2: The client loads a small batch of training samples from its private dataset to train the local model, and in each iteration, updates the local model parameters using a small batch stochastic gradient descent algorithm, repeating τ rounds of training;

[0015] S1.3: After each client completes its local update, the server sends a random sample of the local dataset to each client, and the client calculates the local dataset sample D k The local representation vector s(θ k ,x), and upload it to the server together with the parameters of the classification header.

[0016] As a further solution of the present invention, the specific calculation formula for training the local model in S1.2 is as follows:

[0017]

[0018] in, represents the local model of the kth client in the tth round of training, where t∈τ; represents the local model of the kth client during the t-1th round of training; D k represents the private dataset of the kth client. In each local update iteration, a random sample of size B is extracted from the private dataset.k A small batch of data, where k∈K; η represents the learning rate.

[0019] As a further solution of the present invention, the detailed steps of the server side evaluating the representation similarity between the clients in Q2 are as follows:

[0020] S2.1: Each client starts with its private sample (x, y)∈D k Get the feature representation vector s(θ k ,x), based on the local representation vector s(θ k ,x), and calculate the average feature representation vector of samples of category c, that is, the local prototype, and then convert the calculated average feature representation vector into an aggregation weight through softmax to ensure that the sum of each weight is 1;

[0021] S2.2: Calculate the public representation x∈D of each client based on the aggregate weight k The server then sends the personalized prototype and the global header parameter φ (t-1) Transmitted to each client, where φ (t-1) represents the server-side global header parameter in the t-1th round of global training, t∈T. If it is the first round of global training, then φ (t-1) =φ (0) .

[0022] As a further solution of the present invention, the specific calculation formula of the average feature representation vector of the client in S2.1 is as follows:

[0023]

[0024] In the formula, represents the local prototype of category c computed by client k; represents the local private dataset of category c in client k; s(θ k ,x) represents the local representation vector of client k;

[0025] The specific calculation formula for the personalized prototype described in S2.2 is as follows:

[0026]

[0027] In the formula, represents the personalized prototype of client i about category c; σ represents a scaling hyperparameter used to control the degree of difference in collaborative knowledge between different clients; represents the cosine similarity between the local models output by client i and client j on the common representation x, where i∈K, j∈K; α i,jA hyperparameter representing the reference weight of client i, which is used to adjust the client's own aggregation weight.

[0028] As a further solution of the present invention, the specific steps of training the local model in Q3 are as follows:

[0029] S3.1: The server distributes the personalized prototype calculated for each client to the corresponding client. Each client receives the corresponding personalized prototype and initializes the classification header parameters according to the global header parameters sent by the server.

[0030] S3.2: A personalized ensemble loss function is constructed based on the hard loss between the predicted output of the client local model and the actual label, and the Kullback-Leibler divergence between the local representation vector and the personalized prototype. The local model is trained and updated with the goal of minimizing the personalized ensemble loss. The training and update process is repeated for τ rounds. After the update is completed, the trained local model parameters are saved on the client.

[0031] As a further solution of the present invention, the specific calculation formula of the personalized integrated loss function in S3.2 is as follows:

[0032]

[0033] In the formula, l CE (w; x, y) represents the hard loss between the predicted output of the client local model and the actual label, l CE (w; x, y) = -logp(y|x; w); KL(·,·) represents the local feature representation Personalized prototypes ; λ represents a hyperparameter used to balance the two loss components.

[0034] As a further solution of the present invention, the specific steps of aggregating the classification header parameters uploaded by each client and fine-tuning the aggregated global classification header described in Q4 are as follows:

[0035] S4.1: After the local training of each client is completed, the local classification header parameters are uploaded to the server. The server aggregates the classification header parameters of all clients according to the weights, and uses the local representation vector uploaded by the client to post-train the global classification header;

[0036] S4.2: The post-training aggregate classification head uses the local dataset sample D kThe personalized prototypes and the corresponding true labels are optimized and updated to obtain a global classification head that enhances the generalization performance of the model, and the updated global head parameters are distributed from the server to each client. In the next training round, each client will replace the classification head of the previous round with the updated global head.

[0037] As a further solution of the present invention, the specific calculation formula of weighted aggregation in S4.1 is as follows:

[0038]

[0039] The specific calculation formula for post-training described in S4.1 is as follows:

[0040]

[0041] The specific calculation formula for the optimization update described in S4.2 is as follows:

[0042]

[0043] The final client model described in S4.2 is specifically expressed as:

[0044]

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] This personalized heterogeneous federated learning method based on similarity distillation sets the number of clients, global rounds and local rounds in each round of training, then each client initializes a heterogeneous local model, the server initializes the global classification head parameters, and the client loads a small batch of training samples from its private data set to train the local model, and in each iteration, uses a small batch stochastic gradient descent algorithm to update the local model parameters. When each client completes its local update, the server sends a random sample of the local data set to each client, and the client calculates the local representation vector of the local data set sample and uploads it to the server together with the parameters of the classification head. The server calculates the representation similarity between the clients, and then converts the calculated representation similarity result into an aggregation weight through softmax. Based on the aggregation weight, the personalized prototype of each client about the public representation is calculated. Subsequently, the server transmits the personalized prototype and the global header parameters to each client, and each client receives the corresponding personalized prototype, and at the same time initializes the classification header parameters according to the global header parameters sent by the server. A personalized ensemble loss function is constructed, and the local model is trained and updated with the goal of minimizing the personalized ensemble loss. After the local training of each client is completed, the local classification head parameters are uploaded to the server. The server aggregates the classification head parameters of all clients according to the weights, and uses the local representation vector uploaded by the client to post-train the global classification head. The aggregated classification head after post-training uses the personalized prototypes of the local dataset samples and the corresponding real labels to optimize and update to obtain a global classification head that enhances the generalization performance of the model. The updated global head parameters are distributed from the server to each client, and in the next training round, each client will replace the classification head of the previous round with the updated global head. It can adaptively aggregate the output of the client model, generate multiple different knowledge and pass them to the corresponding model, improve the efficiency of knowledge collaboration, and compared with the traditional global consensus, it can adaptively adjust the knowledge structure to adapt to the carrying capacity of different models, and can be independently trained and aggregated in a personalized manner, which is suitable for resource-constrained edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0048] Figure 1 A flowchart of the personalized heterogeneous federated learning method based on similarity distillation proposed in the present invention;

[0049] Figure 2 This is an algorithm flow chart of the personalized heterogeneous federated learning method based on similarity distillation proposed in the present invention;

[0050] Figure 3A schematic diagram of a server aggregation algorithm for the personalized heterogeneous federated learning method based on similarity distillation proposed in the present invention;

[0051] Figure 4 Schematic diagram of the client local training algorithm of the personalized heterogeneous federated learning method based on similarity distillation proposed in the present invention. DETAILED DESCRIPTION

[0052] Reference Figure 1-Figure 4 , a personalized heterogeneous federated learning method based on similarity distillation, the specific steps of this federated learning method are as follows:

[0053] Select multiple groups of clients for local training and build local representations.

[0054] Specifically, the number of clients K, the local private dataset, the number of global rounds T, and the number of local rounds τ are set for each round of training, and each client initializes a heterogeneous local model The server initializes the global classification header parameter φ (0) ,in, represents the kth client heterogeneous local model, and k∈K. The client loads a small batch of training samples from its private dataset to train the local model, and in each iteration, uses a small batch stochastic gradient descent algorithm to update the local model parameters, repeating τ rounds of training. When each client completes its local update, the server sends a random sample of the local dataset to each client, and the client calculates the local dataset sample D k The local representation vector s(θ k ,x), and upload it to the server together with the parameters of the classification header.

[0055] It should be further explained that the specific calculation formula for training the local model is as follows:

[0056]

[0057] in, represents the local model of the kth client in the tth round of training, where t∈τ; represents the local model of the kth client during the t-1th round of training; D k represents the private dataset of the kth client. In each local update iteration, a random sample of size B is extracted from the private dataset. k A small batch of data, where k∈K; η represents the learning rate.

[0058] The server evaluates the representation similarity between clients and generates a personalized prototype for each client.

[0059] Specifically, each client selects its private sample (x, y)∈D kGet the feature representation vector s(θ k ,x), based on the local representation vector s(θ k ,x), and calculate the average feature representation vector of samples of category c, that is, the local prototype. Then, the calculated average feature representation vector is converted into an aggregation weight through softmax to ensure that the sum of each weight is 1. Based on the aggregation weight, the public representation x∈D of each client is calculated. k The server then sends the personalized prototype and the global header parameter φ (t-1) Transmitted to each client, where φ (t-1) represents the server-side global header parameter in the t-1th round of global training, t∈T. If it is the first round of global training, then φ (t-1) =φ (0) .

[0060] It should be further explained that the specific calculation formula of the average feature representation vector of the client is as follows:

[0061]

[0062] In the formula, represents the local prototype of category c computed by client k; represents the local private dataset of category c in client k; s(θ k ,x) represents the local representation vector of client k;

[0063] The specific calculation formula for the personalized prototype described in S2.2 is as follows:

[0064]

[0065] In the formula, represents the personalized prototype of client i about category c; σ represents a scaling hyperparameter used to control the degree of difference in collaborative knowledge between different clients; represents the cosine similarity between the local models output by client i and client j on the common representation x, where i∈K, j∈K; α i,j A hyperparameter representing the reference weight of client i, which is used to adjust the client's own aggregation weight.

[0066] The personalized consensus is passed back to the local client and the local model is trained by minimizing the personalized ensemble loss.

[0067] Specifically, the server distributes the personalized prototype calculated for each client to the corresponding client. Each client receives the corresponding personalized prototype and initializes the classification header parameters according to the global header parameters sent by the server. The personalized ensemble loss function is constructed based on the hard loss between the predicted output of the client local model and the actual label, and the Kullback-Leibler divergence between the local representation vector and the personalized prototype. The local model is trained and updated with the goal of minimizing the personalized ensemble loss. The training and updating process is repeated for τ rounds. After the update is completed, the trained local model parameters are saved on the client.

[0068] In addition, it should be noted that the specific calculation formula of the personalized integration loss function is as follows:

[0069]

[0070] In the formula, l CE (w; x, y) represents the hard loss between the predicted output of the client local model and the actual label, l CE (w; x, y) = -logp(y|x; w); KL(·,·) represents the local feature representation Personalized prototypes ; λ represents a hyperparameter used to balance the two loss components.

[0071] Aggregate the classification header parameters uploaded by each client and fine-tune the aggregated global classification header.

[0072] Specifically, after the local training of each client is completed, the local classification header parameters are uploaded to the server. The server aggregates the classification header parameters of all clients according to the weights, and uses the local representation vector uploaded by the client to post-train the global classification header. The aggregated classification header after post-training uses the local dataset sample D k The personalized prototypes and the corresponding true labels are optimized and updated to obtain a global classification head that enhances the generalization performance of the model, and the updated global head parameters are distributed from the server to each client. In the next training round, each client will replace the classification head of the previous round with the updated global head.

[0073] It should be noted that the specific calculation formula for weighted aggregation is as follows:

[0074]

[0075] The specific calculation formula for post-training is as follows:

[0076]

[0077] The specific calculation formula for optimization and update is as follows:

[0078]

[0079] The final client model is specifically expressed as:

[0080]

[0081] The updated global classification head is sent to each client and the next round of local training is carried out.

Claims

1. A personalized heterogeneous federated learning method based on similarity distillation, characterized by: The specific steps of the federated learning method are as follows: Q1. Select multiple groups of clients for local training and build local representations. Q2, the server evaluates the representation similarity between clients and generates a personalized prototype for each client; Q3. Pass the personalized consensus back to the local client and train the local model by minimizing the personalized ensemble loss. Q4: Aggregate the classification header parameters uploaded by each client and fine-tune the aggregated global classification header; Q5. Send the updated global classification head to each client and conduct the next round of local training.

2. The personalized heterogeneous federated learning method based on similarity distillation according to claim 1, characterized in that: The specific steps of selecting multiple groups of clients for local training and building local representations described in step Q1 are as follows: S1.1: Set the number of clients K, local private dataset, global round number T, and local round number τ for each round of training, and initialize the heterogeneous local model for each client The server initializes the global classification header parameter φ (0) ,in, represents the k-th client heterogeneous local model, and k∈K; S1.2: The client loads a small batch of training samples from its private dataset to train the local model, and in each iteration, updates the local model parameters using a small batch stochastic gradient descent algorithm, repeating τ rounds of training; S1.3: After each client completes its local update, the server sends a random sample of the local dataset to each client, and the client calculates the local dataset sample D k The local representation vector s(θ k ,x), and upload it to the server together with the parameters of the classification header.

3. The personalized heterogeneous federated learning method based on similarity distillation according to claim 2 is characterized in that: The specific calculation formula for training the local model described in S1.2 is as follows: in, represents the local model of the kth client in the tth round of training, where t∈τ; represents the local model of the kth client during the t-1th round of training; D k represents the private dataset of the kth client. In each local update iteration, a random sample of size B is extracted from the private dataset. k A small batch of data, where k∈K; η represents the learning rate.

4. The personalized heterogeneous federated learning method based on similarity distillation according to claim 2 is characterized in that: The detailed steps for the server to evaluate the representation similarity between clients described in Q2 are as follows: S2.1: Each client starts with its private sample (x, y)∈D k Get the feature representation vector s(θ k ,x), based on the local representation vector s(θ k ,x), and calculate the average feature representation vector of samples of category c, that is, the local prototype, and then convert the calculated average feature representation vector into an aggregation weight through softmax to ensure that the sum of each weight is 1; S2.2: Calculate the public representation x∈D of each client based on the aggregate weight k The server then sends the personalized prototype and the global header parameter φ (t-1) Transmitted to each client, where φ (t-1) represents the server-side global header parameter in the t-1th round of global training, t∈T. If it is the first round of global training, then φ (t-1) =φ (0) .

5. The personalized heterogeneous federated learning method based on similarity distillation according to claim 1, characterized in that: The specific calculation formula of the average feature representation vector of the client described in S2.1 is as follows: In the formula, represents the local prototype of category c computed by client k; represents the local private dataset of category c in client k; s(θ k ,x) represents the local representation vector of client k; The specific calculation formula for the personalized prototype described in S2.2 is as follows: In the formula, represents the personalized prototype of client i about category c; σ represents a scaling hyperparameter used to control the degree of difference in collaborative knowledge between different clients; represents the cosine similarity between the local models output by client i and client j on the common representation x, where i∈K, j∈K; α i,j A hyperparameter representing the reference weight of client i, which is used to adjust the client's own aggregation weight.

6. The personalized heterogeneous federated learning method based on similarity distillation according to claim 4 is characterized in that: The specific steps for training the local model described in Q3 are as follows: S3.1: The server distributes the personalized prototype calculated for each client to the corresponding client. Each client receives the corresponding personalized prototype and initializes the classification header parameters according to the global header parameters sent by the server. S3.2: A personalized ensemble loss function is constructed based on the hard loss between the predicted output of the client local model and the actual label, and the Kullback-Leibler divergence between the local representation vector and the personalized prototype. The local model is trained and updated with the goal of minimizing the personalized ensemble loss. The training and update process is repeated for τ rounds. After the update is completed, the trained local model parameters are saved on the client.

7. The personalized heterogeneous federated learning method based on similarity distillation according to claim 6, characterized in that: The specific calculation formula of the personalized ensemble loss function described in S3.2 is as follows: In the formula, l CE (w; x, y) represents the hard loss between the predicted output of the client local model and the actual label, l CE (w; x, y) = -logp(y|x; w); KL(·,·) represents the local feature representation With personalized prototypes ; λ represents a hyperparameter used to balance the two loss components.

8. The personalized heterogeneous federated learning method based on similarity distillation according to claim 6, characterized in that: The specific steps for aggregating the classification header parameters uploaded by each client and fine-tuning the aggregated global classification header as described in Q4 are as follows: S4.1: After the local training of each client is completed, the local classification header parameters are uploaded to the server. The server aggregates the classification header parameters of all clients according to the weights, and uses the local representation vector uploaded by the client to post-train the global classification header; S4.2: The post-training aggregate classification head uses the local dataset sample D k The personalized prototypes and the corresponding true labels are optimized and updated to obtain a global classification head that enhances the generalization performance of the model, and the updated global head parameters are distributed from the server to each client. In the next training round, each client will replace the classification head of the previous round with the updated global head.

Images