Federal learning method based on representation driving head clustering under edge device

Abstract

The invention discloses a federated learning method based on representation driving head clustering under an edge device, which extracts core features from local original data through representation learning, and only uploads data representation to a server, so that communication overhead can be reduced, core information of the original data is reserved, and accuracy of a clustering process is ensured. And the server is based on the shared layers in the representation training clusters, so that the precision of the model is ensured. According to the method, a model is decoupled into a representation generation module and a shared header, and the representation generation module is used for generating data representation; the shared head improves the generalization ability of the model by learning knowledge of different clients. The central server dynamically clusters the clients according to the representation uploaded in each round, the clustering structure can be adjusted in time according to the change of data in each round of iteration, the accuracy and adaptability of clustering are ensured, and finally, the central server uses the data representation of the clients in the cluster to train the shared header of each cluster. And the trained shared head is sent to the client, so that the generalization ability of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to federated learning and artificial intelligence technology, and in particular to a federated learning method based on representation-driven head clustering in an edge device. Background Art

[0002] Mobile edge devices, such as smartphones, tablets, IoT devices, and drones, generate a large amount of data every day, but they are not fully utilized due to privacy protection limitations. Federated learning is a distributed machine learning method that solves this problem. It allows devices to collaboratively train models without sharing raw data, thereby effectively protecting data privacy. However, traditional federated learning algorithms (such as FedAvg) have a significant problem of slowing down model convergence when facing data heterogeneity. In addition, some personalized federated learning methods adjust the model to adapt to local data, which alleviates this problem, but still faces the challenges of local model overfitting and slow model convergence. To address this challenge, some studies have proposed personalized federated learning, the core purpose of which is to train a model suitable for local data for each client while maintaining data privacy and security. Methods for personalized federated learning include model stratification, transfer learning, meta-learning methods, and knowledge distillation. However, most of these methods rely on each client to independently use its own small and specific local dataset to adjust the model, which limits the ability to learn from other client data, resulting in local model overfitting and slow model convergence. Cluster federated learning has been proposed as a new strategy to achieve cross-client knowledge sharing by clustering clients with similar data distribution. However, most existing CFL methods rely on initial clustering, which requires a lot of communication and calculation before training begins. After clustering is completed, the client is fixed in the cluster to which it belongs, which limits the system's adaptability in dynamic environments.

[0003] Although recent studies, IFCA and AICFL have achieved dynamic adjustment of clustering during federated learning training. However, the server needs to broadcast the cluster model in each iteration, and the client needs to calculate the cluster identity locally, which increases the communication cost, especially on resource-constrained edge devices, resulting in reduced model training efficiency. Another study, HiCFL, performs client clustering by calculating the cosine similarity of the fully connected layer or convolutional layer of the model, introducing the concept of model stability. Therefore, there is no need to communicate the entire model between the server and the client, and the number of communication rounds is reduced, thereby reducing communication and computing overhead. However, after the clustering stage, the clustering results will not be adjusted, and the entire model needs to be communicated for traditional FL training of each cluster, resulting in large communication overhead.

[0004] For example, patent CN114936595A discloses a model fine-tuning and head aggregation method in federated learning. The client needs to send the entire model (local representation + head) to the server, and the communication overhead is very large. The server of this technical solution globally aggregates the local representation, so the degree of personalization of the model will be greatly reduced; for example, patent CN118470412A discloses a federated target detection learning method based on representation enhancement and weighted aggregation in a cloud-edge environment. The entire client model needs to be uploaded to the server, which will greatly increase the communication overhead. At the same time, its representation adds an imbalance factor (for scenarios such as few-sample learning, the categories of few samples are unbalanced), and some strengthening operations need to be performed on the imbalance factor. In the final model aggregation, this will expose the information of the original data.

[0005] Therefore, although existing technologies and methods have proposed strategies to improve communication overhead and dynamic clustering, they still have two major shortcomings: (1) how to reduce the amount of communication while ensuring that the model accuracy is not affected; (2) how to ensure the accuracy and adaptability of clustering when there is limited data.

[0006] These issues have not yet been fully resolved and become key challenges for existing technologies. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide a federated learning method based on representation-driven head clustering on edge devices.

[0008] Technical solution: A federated learning method based on characterization-driven head clustering in an edge device of the present invention comprises the following steps:

[0009] Step 1: Generate data representation, that is, use the representation generation module on the edge device to generate the representation of the data sample; and obtain the data prototype by weighted average of the data representation. The specific method is:

[0010] Step 1.1: After each round of local training, the edge device will Decoupling into a shared header and representation generation module

[0011]

[0012] In the above formula, ○ is the splicing symbol, Represents the local model after local training is completed. and They represent the t-th round of federated learning, the local shared head and representation generation module after decoupling the local model of the k-th edge device, respectively;

[0013] Step 1.2: Edge devices use local representation generation modules Generate representations of the data samples participating in this round of training

[0014]

[0015] In the above formula, x i represents the i-th data sample, Indicates the use of the representation generation module to generate data samples x i Function of

[0016] Step 1.3: Take the weighted average of the sample representations with the same label to obtain the data prototype (That is, it refers to the content generated by local data through the representation generation module);

[0017]

[0018] In the above formula, Indicates the total number of sample data with label q participating in the training in this round;

[0019] Step 2: Cluster the clients, that is, first calculate the representation distance between the clients, and then group the clients with similar data distribution into one cluster;

[0020] Step 3: Based on the clients clustered in step 2, train the shared headers in each cluster. For client k belonging to multiple clusters, aggregate the shared headers of all its clusters and then send them to the corresponding clients (edge ​​devices). Then use the FedAvg algorithm to summarize the headers of all clusters.

[0021] Furthermore, steps 2 and 3 are executed on the central server. The performance of the central server is much stronger than that of the edge device. This can minimize the involvement of edge devices in calculations, which helps to improve the overall speed.

[0022] Furthermore, the detailed method of step 2 is:

[0023] Calculates the prototype of the data received from the client The representation distance dis between them is used to obtain the similarity between the data distributions of the two clients;

[0024]

[0025] In the above formula, dis q,r represents the representation distance between the q-label data prototype of the k-th client and the l-label data prototype of the r-th client, and ε represents the threshold hyperparameter; when the representation distance is less than ε, the two data prototypes are clustered into the same cluster in this round, and the total number of clusters after clustering is M, and the number of clients in the m-th cluster is |Cm.

[0026] Furthermore, the detailed method of step 3 is:

[0027] When the central server has completed clustering all the representations, use the intra-cluster representation and the corresponding label q to train the global shared head within the cluster

[0028]

[0029] In the above formula, η θ represents the global learning rate, R m represents the representation within the cluster, and y represents the label corresponding to the representation within the cluster;

[0030] Since an edge device k may belong to different clusters, the shared headers of different clusters are aggregated and weighted averaged to obtain a new round of shared headers of edge device k;

[0031]

[0032] In the above formula, represents the shared layer after client k aggregates and updates, M k is the set of all clusters to which client k belongs, It indicates the proportion of samples of client k in the mth cluster to the total samples, is the number of data samples of client k belonging to cluster m;

[0033] After the shared head training of each cluster is completed, the central server aggregates the shared layers of each cluster to obtain the shared layers of the global model. The formula is as follows:

[0034]

[0035] In the above formula, It represents the ratio of the number of clients in the mth cluster to the total number of clients, |Cm| is the number of clients in cluster m, and M is the total number of clusters.

[0036] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0037] (1) The present invention proposes a representation-driven head clustering algorithm designed specifically for federated learning on edge devices. This algorithm allows the server to cluster edge devices using limited information during training, enhances the model's ability to adapt to the changing distribution of edge device data, reduces the server's computational overhead, and ensures the accuracy of the clustering results.

[0038] (2) The present invention uses representations in each cluster to train the global shared head in the cluster, eliminating the need to transmit model weights. This method reduces the amount of data transmitted while maintaining model accuracy, thereby enhancing the applicability of federated learning in resource-constrained mobile edge environments.

[0039] (3) The present invention can reduce communication overhead while ensuring model accuracy. In addition, the present invention is not limited to model types and can be integrated into the federated learning framework without bringing additional burden to edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0041] Figure 2 It is a schematic diagram of the overall process of the present invention;

[0042] Figure 3 A schematic diagram of the distance between client prototypes in the first round of the embodiment;

[0043] Figure 4 This is a schematic diagram of the distance between client prototypes in the 50th round of the embodiment;

[0044] Figure 5 Schematic diagram of the distance between client prototypes in the 300th round of the embodiment

[0045] Figure 6 It is a schematic diagram of comparing the accuracy of the embodiment in CIFAR-10 dynamic and static scenes;

[0046] Figure 7 It is a schematic diagram of comparing the accuracy of the embodiment in CIFAR-100 dynamic and static scenes;

[0047] Figure 8 Schematic diagram of the analysis of different hyperparameter thresholds ε on CIFAR-10 in the embodiment;

[0048] Fig. 9 Schematic diagram of the analysis of different hyperparameter thresholds ε on CIFAR-100 in the embodiment;

[0049] Fig.10 This is a schematic diagram of the variation of the model accuracy with the threshold ε on CIFAR-10 in the embodiment;

[0050] Fig.11 Schematic diagram of the variation of model accuracy with ε on CIFAR-100 in this example. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0052] like Figure 1 and Figure 2 As shown, the federated learning method based on characterization-driven head clustering under the edge device of the present invention is characterized by comprising the following steps:

[0053] Step 1: Generate data representation, that is, use the representation generation module on the edge device to generate the representation of the data sample; and obtain the data prototype by weighted average of the data representation. The specific method is:

[0054] Step 1.1: After each round of local training, the edge device will Decoupling into a shared header and representation generation module

[0055]

[0056] In the above formula, ○ is the splicing symbol, Represents the local model after local training is completed. and They represent the t-th round of federated learning, the local shared head and representation generation module after decoupling the local model of the k-th edge device, respectively;

[0057] Step 1.2: Edge devices use local representation generation modules Generate representations of the data samples participating in this round of training

[0058]

[0059] In the above formula, x i represents the i-th data sample, Indicates the use of the representation generation module to generate data samples x i Function of

[0060] Step 1.3: Take the weighted average of the sample representations with the same label to obtain the data prototype

[0061]

[0062] In the above formula, Indicates the total number of sample data with label q participating in the training in this round;

[0063] Step 2: Cluster the clients, that is, first calculate the representation distance between the clients, and then group the clients with similar data distribution into one cluster;

[0064] Step 3: Based on the clients clustered in step 2, train the shared headers in each cluster. For client k belonging to multiple clusters, aggregate the shared headers of all its clusters and then send them to the corresponding clients. Then use the FedAvg algorithm to summarize the headers of all clusters.

[0065] The detailed processes of step 2 and step 3 of this embodiment are all executed in the central server. And the central server is based on prototype clustering, and uses the prototype to train the shared header in the cluster, and sends the cluster header to the corresponding client.

[0066] The detailed method of step 2 of this embodiment is:

[0067] Calculates the prototype of the data received from the client The representation distance dis between them is used to obtain the similarity between the data distributions of the two clients;

[0068]

[0069] In the above formula, dis q,r represents the representation distance between the q-label data prototype of the k-th client and the l-label data prototype of the r-th client, and ε represents the threshold hyperparameter; when the representation distance is less than ε, the two data prototypes can be clustered into the same cluster in this round.

[0070] The detailed method of step 3 of this embodiment is:

[0071] When the central server has completed clustering all the representations, use the intra-cluster representation and the corresponding label q to train the global shared head within the cluster

[0072]

[0073] In the above formula, η θ represents the global learning rate, R m represents the representation within the cluster, and y represents the label corresponding to the representation within the cluster;

[0074] Then, the shared headers of different clusters are aggregated and weighted averaged to obtain a new round of shared headers of edge device k;

[0075]

[0076] In the above formula, represents the shared layer after client k aggregates and updates, M k is the set of all clusters to which client k belongs, It indicates the proportion of samples of client k in the mth cluster to the total samples, is the number of data samples of client k belonging to cluster m;

[0077] After the shared head training of each cluster is completed, the central server aggregates the shared layers of each cluster. The formula is as follows:

[0078]

[0079] In the above formula, It represents the ratio of the number of clients in the mth cluster to the total number of clients, |Cm| is the number of clients in cluster m, and M is the total number of clusters.

[0080] Example 1

[0081] This embodiment algorithmizes the relevant steps in the entire technical solution, which is expressed as follows:

[0082] 1. Define and initialize parameters: Set the total number of clients to N, the number of edge devices in each round of communication to K, the total number of communication rounds to T, and the local learning rate to η l , the global learning rate is η θ , the number of communication clusters in each round is M, and the threshold for determining the similarity of data representation is ε. For the t=0 communication round, perform the following steps: For each edge device k, initialize the local model Decouple parameters into local shared headers and representation generation module Generate data representation using the representation generation module Averaging data representations to generate prototypes And upload it to the server; the server uses Euclidean distance to calculate the correlation dis between data representations; cluster edge devices based on dis, and divide edge devices with highly correlated data distribution into the same cluster M.

[0083] 2. Intra-cluster model shared head training: Use the intra-cluster representation R on each cluster m m and the corresponding label y to train the shared head within the cluster For each communication round from 1 to T, perform the following steps:

[0084] The edge device side operation is as follows: For each selected client K, the following operations are performed: Download the shared header With the local representation generation module Splice and update the local model, and use local data to update the local model Perform gradient descent update to obtain new model parameters The edge device uses the new representation generation module to generate data representations, averages them, and uploads them to the server;

[0085] The server side operation is as follows: The server is based on the new representation Re-cluster the clients and re-train the shared head in each cluster For edge device k belonging to multiple clusters, aggregate the shared headers of the clusters to which it belongs to obtain the shared header of edge device k And send it to the corresponding client.

[0086] 3. Global model aggregation: The server aggregates the shared headers of each cluster Perform aggregation to obtain the updated global shared header

[0087] Example 2

[0088] To verify the technical effect of the present invention, this embodiment conducts experiments on two publicly available real datasets that are widely used to evaluate the performance of federated learning models: CIFAR-10 and CIFAR-100. All experiments are developed using Python 3.11 and PyTorch 2.2.2 and executed on a standard computing platform equipped with NVIDIA GeForce RTX4090 GPU and 24GB RAM.

[0089] To simulate non-independent and identically distributed data, this embodiment sets up two groups of experiments on the CIFAR-10 dataset to simulate non-independent and identically distributed data; Setting 1: Each client selects two different data classes and sets the number of users to 20; Setting 2: Each client selects four different data classes and sets the number of users to 50; In addition, the number of each class is equal, and all clients have the same amount of data. In CIFAR-100, the number of clients is set to 10, each client selects ten different data classes, and each client has 3,000 images, of which every 300 images belong to the same class.

[0090] Regarding the selection of the model, this embodiment applies a 4-layer CNN model, including two convolutional layers and two fully connected layers, to perform classification tasks on the CIFAR-10 and CIFAR100 datasets.

[0091] The technical solution DRCFL of the present invention is compared with 8 popular benchmark methods, including FedAvg, LG-FedAvg, FedProto, FedGH, FedPAC, IFCA, PACFL and FedCAC, and the average accuracy and communication cost of the local model are reported. The whole optimization process adopts the stochastic gradient descent method. The experimental results show that the technical solution DRCFL of the present invention is superior to these methods in many aspects such as system reliability, stability and communication efficiency.

[0092] The experimental results of this embodiment are as follows:

[0093] (1) Representation similarity verification

[0094] The data representation obtained through the representation learning process is used as the basis for client clustering in the technical solution DRCFL of the present invention. Figures 3 to 5 As shown in Figure 2, preliminary experiments using Euclidean distance to calculate the distance between data representations of different categories in the CIFAR-10 dataset show that the data representations can accurately reflect the similarity of the original data distribution.

[0095] (2) Comparison of model performance

[0096] Precision evaluation: This embodiment tests the local model of each client multiple times, takes the average of the precision of all clients, and records multiple groups of average test precision. In terms of communication cost, the communication rounds required for model convergence are recorded, and the communication cost of each round is calculated (that is, the amount of data uploaded and downloaded by each client multiplied by the number of participating clients), and then the total communication cost is obtained. The total communication cost is calculated by multiplying the communication cost of each round by the number of communication rounds. This experiment first evaluates the model performance of different federated learning methods on the CIFAR-10 dataset. In the experimental configuration, the number of users participating in federated learning is set to 20 and 50 respectively, and the number of categories is 2 and 4 respectively. The experimental results are shown in Table 1.

[0097] Table 1

[0098]

[0099]

[0100] Table 1 Comparison of test accuracy %, number of communication rounds required for convergence, and total communication overhead of different methods on CIFAR-10. The symbol "—" indicates that the algorithm has not converged. From the above results, it can be seen that when the number of clients N=20, the technical solution DRCFL of the present invention achieves a test accuracy of 88.70%, and the communication overhead is only 10.90MB. In contrast, the single-round communication overhead of FedProto is only 56.1KB, because this method only needs to upload the local prototype and receive the global prototype from the server. However, FedProto requires 257 rounds to converge, which is much higher than the 115 rounds of the technical solution DRCFL of the present invention, and the total communication overhead is 14.1MB, which is about 29.36% higher than the technical solution DRCFL of the present invention. The FedGH method requires the client to upload the prototype and download the global prediction header, and the global prediction header needs to be broadcast in each round, resulting in its total communication overhead being five times that of DRCFL.

[0101] When N=50, the technical solution DRCFL of the present invention still maintains the lowest communication overhead, which is only 95.1MB, and achieves a test accuracy of 70.95%. In contrast, although FedPAC performs best in test accuracy, reaching 78.96%, its communication cost is much higher than DRCFL. FedPAC needs to transmit representations and models in each round, resulting in a total communication cost that is more than 500 times that of DRCFL. Regardless of how the number of clients changes, the technical solution DRCFL of the present invention always maintains the best performance in terms of communication overhead and achieves a higher test accuracy.

[0102] Table 2

[0103]

[0104]

[0105] Table 2 compares the average test accuracy %, communication rounds required for convergence, and total communication overhead on CIFAR-100. In order to evaluate the performance of the algorithm on a diverse dataset, it was tested on the CIFAR-100 dataset. The experimental results in Table 2 show that the technical solution DRCFL of the present invention achieved the highest test accuracy of 61.09%. In contrast, the test accuracy of FedCAC was 60.18%, but its total communication cost reached 21.0GB. FedGH performed better in terms of communication overhead, with a total communication cost of 1.2GB and a test accuracy of 59.74%, but its performance was limited by slow convergence speed and communication strategy. The total communication cost of the technical solution DRCFL of the present invention is 0.2GB, which is about 83.33% lower than that of FedGH.

[0106] Although the existing CFL method can usually converge within 100 rounds under the same experimental conditions, its communication cost per round is significantly higher than that of the technical solution DRCFL of the present invention, which is about 500 times that of the technical solution DRCFL of the present invention. Although these methods converge faster, their total communication cost is higher. For example, the test accuracy of PACFL is 57.20%, but its total communication cost is 53.6GB, which is about 268 times that of the technical solution DRCFL of the present invention. It can be seen that the technical solution DRCFL of the present invention can significantly reduce the communication cost while ensuring high test accuracy, and performs well on a variety of data sets.

[0107] (3) Comparison of test accuracy in static and dynamic scenarios

[0108] To simulate the dynamic changes of data distribution, this experiment initially selected 5 categories on the CIFAR-10 dataset, and set 10 clients for the first 50 rounds, with each client assigned 2 categories. In the 50th round, 10 new clients with different data distributions (non-IID, 2 / 10) were introduced. In the CIFAR-100 dataset, 50 categories were initially selected, and 10 clients were set for the first 100 rounds, with each client assigned 10 of the 50 categories; in the 100th round, 10 new clients with different data distributions (non-IID, 10 / 100) were introduced.

[0109] This example aims to simulate the integration of new clients and evaluate whether DRCFL can quickly adapt to dynamic data distribution. Figure 6 , Figure 7 As shown. In the dynamic scenario of the CIFAR-10 dataset, 10 new clients were introduced in the 50th round, resulting in a decrease in test accuracy. This decrease was attributed to the fact that the data distribution of the new clients was significantly different from the original data distribution, and the model needed to be adjusted to adapt to these changes. After 25 rounds of adaptation, the test accuracy quickly recovered and gradually approached the level in the static scenario. A similar trend was also observed in the experiment of the CIFAR-100 dataset. These results show that the technical solution DRCFL of the present invention can effectively adapt the model to the ever-changing data distribution scenario.

[0110] (4) Hyperparameter threshold analysis

[0111] This embodiment performs cross-validation and cross-validates the optimal range of hyperparameter thresholds on CIFAR-10 (non-IID, 2 / 10) and CIFAR-100 (non-IID, 10 / 100). Specifically, for the CIFAR-10 dataset, the thresholds are set to {3.0, 5.0, 5.5, 6.0, 6.5, 7.0, 16.0}; for the CIFAR-100 dataset, the thresholds are set to {5.0, 5.5, 6.0, 6.3, 6.5, 7.0, 9.0}. Figure 8-11 The effects of different thresholds on model accuracy and convergence speed are shown. The results show that the threshold has little effect on the final convergence accuracy, and the difference between the highest and lowest convergence accuracy does not exceed 1%. However, the threshold has a significant effect on the convergence speed of the model. For example, when the threshold is set at 5.5-6.5, the number of rounds required for model convergence is reduced by about 35%-56% compared to the case where the threshold is 3.0, and there is a significant fluctuation.

[0112] In summary, the present invention extracts core features from local raw data through representation learning, and only uploads data representation to the server, which can reduce communication overhead while retaining the core information of the original data and ensuring the accuracy of the clustering process. The server trains the shared layer within the cluster based on these representations to ensure the accuracy of the model. The present invention decouples the model into a representation generation module and a shared header. The representation generation module is used to generate data representations; the shared header improves the generalization ability of the model by learning the knowledge of different clients. The central server dynamically clusters the clients according to the representations uploaded in each round, and can adjust the clustering structure in time according to the changes in data in each round of iteration to ensure the accuracy and adaptability of the clustering. Finally, the central server uses the data representation of the clients in the cluster to train the shared header of each cluster, and sends the trained shared header to the client to enhance the generalization ability of the model.

Claims

1. A federated learning method based on representation-driven head clustering on edge devices, characterized in that: The following steps are involved: Step 1: Generate data representation, that is, use the representation generation module on the edge device to generate the representation of the data sample; and obtain the data prototype by weighted average of the data representation. The specific method is: Step 1.1: After each round of local training, the edge device will Decoupling into a shared header and representation generation module In the above formula, ○ is the splicing symbol, Represents the local model after local training is completed. and They represent the t-th round of federated learning, the local shared head and representation generation module after decoupling the local model of the k-th edge device, respectively; Step 1.2: Edge devices use local representation generation modules Generate representations of the data samples participating in this round of training In the above formula, x i represents the i-th data sample, Indicates the use of the representation generation module to generate data samples x i Function of Step 1.3: Take the weighted average of the sample representations with the same label to obtain the data prototype In the above formula, Indicates the total number of sample data with label q participating in the training in this round; Step 2: Cluster the clients, that is, first calculate the representation distance between the clients, and then group the clients with similar data distribution into one cluster; Step 3: Based on the clients clustered in step 2, train the shared headers in each cluster. For client k belonging to multiple clusters, aggregate the shared headers of all its clusters and then send them to the corresponding clients. Then use the FedAvg algorithm to summarize the headers of all clusters.

2. The method for federated learning based on representation-driven head clustering in edge devices according to claim 1, characterized in that: Perform steps 2 and 3 on the central server.

3. The federated learning method based on characterization-driven head clustering in the edge device according to claim 1 or 2, characterized in that: The detailed method of step 2 is: Calculates the prototype of the data received from the client The representation distance dis between them is used to obtain the similarity between the data distributions of the two clients; In the above formula, dis q,r represents the representation distance between the q-label data prototype of the k-th client and the l-label data prototype of the r-th client, and ε represents the threshold hyperparameter; when the representation distance is less than ε, the two data prototypes are clustered into the same cluster in this round. The total number of clusters after clustering is M, and the number of clients in the m-th cluster is |Cm|.

4. The method for federated learning based on representation-driven head clustering in edge devices according to claim 1, characterized in that: The detailed method of step 3 is: When the central server has completed clustering all the representations, use the intra-cluster representation and the corresponding label q to train the global shared head within the cluster In the above formula, η θ represents the global learning rate, R m represents the representation within the cluster, and y represents the label corresponding to the representation within the cluster; Then, the shared headers of different clusters are aggregated and weighted averaged to obtain a new round of shared headers of edge device k; In the above formula, represents the shared layer after client k aggregates and updates, M k is the set of all clusters to which client k belongs, It indicates the proportion of samples of client k in the mth cluster to the total samples, is the number of data samples of client k belonging to cluster m; After the shared head training of each cluster is completed, the central server aggregates the shared layers of each cluster to obtain the shared layers of the global model. The formula is as follows: In the above formula, It represents the ratio of the number of clients in the mth cluster to the total number of clients, |Cm| is the number of clients in cluster m, and M is the total number of clusters.

Images