A communication federated learning method based on distillation and double-dimensional collaborative compression
Patent Information
- Application Number
- CN202611077896.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-20
- Publication Date
- 2026-10-09
AI Technical Summary
[0041]有益效果：本发明涉及一种基于蒸馏与双维协同压缩的通信联邦学习方法，将数据分布划分为若干簇，将计算能力划分为高、低层模块，即可实现数据分布和计算能力两种语义不可通约的信号，在距离度量中严格解耦，免除了传统拼接式聚类所需的启发式平衡权重，提升了聚类质量并增强了分组的可解释性；
Smart Images
Figure CN122885555A_ABST
Abstract
Description
Technical Field
[0001] This invention specifically relates to a communication federated learning method based on distillation and two-dimensional collaborative compression. Background Technology
[0002] With the rapid development of big data and artificial intelligence, data privacy and security issues have received much attention. Federated learning, as a privacy-preserving distributed machine learning paradigm, enables multiple clients, such as mobile devices or institutions, to collaboratively train a global model without uploading their local raw data. They can collaborate by exchanging model parameters or updating them, demonstrating great potential in data-sensitive fields such as finance, healthcare, and the Internet of Things.
[0003] However, the actual deployment of federated learning faces two mutually coupled and amplified bottlenecks. The first is data heterogeneity: client data is usually non-independent and identically distributed, with significant differences in feature distribution, label distribution, and sample size. This causes the local training objective to deviate from the global optimum, leading to client drift, resulting in slow convergence, decreased accuracy, or even divergence.
[0004] Secondly, there is high communication overhead: modern deep networks contain millions to billions of parameters, and each round requires all participating clients to download the complete global model and upload local updates, which severely restricts the system's scalability in bandwidth-constrained environments.
[0005] More importantly, data heterogeneity increases the number of rounds required for convergence, thus multiplying the communication overhead, while heterogeneity-independent compression further exacerbates client drift. Summary of the Invention
[0006] Purpose of the invention: To provide a communication federated learning method based on distillation and two-dimensional collaborative compression, which solves the above-mentioned problems existing in the prior art.
[0007] Technical solution: A communication federated learning method based on distillation and two-dimensional collaborative compression, comprising the following steps:
[0008] S1. Collect data distribution statistics and computing power measurement from K groups of clients through the server, then decompose the global model using the server to obtain the feature extractor and classifier head, and divide the feature extractor into K sequential modules;
[0009] S2. The data distribution features are extracted by each group of clients based on the frozen feature extractor. The corresponding hardware computing power and data signature are calculated for each group of clients. The hardware computing power and data signature are normalized respectively. The normalized computing power and normalized signature are then fed back to the server.
[0010] S3. The server performs K-Means clustering on the normalized signature, divides the clients into several clusters, sorts them in ascending order according to the average computing power of the clients in each cluster, forms an ordered client group, and assigns a feature extraction module of corresponding depth to each client. The set obtained by mapping and assigning is sorted in ascending order according to the average computing power in the cluster, and an expert group is built.
[0011] S4. The server broadcasts the global model and the allocation information of clients to ordered client groups in a one-to-many manner. Local updates are obtained through local optimization using task loss and distillation loss. The two momentum coefficients, task momentum coefficient and distillation momentum coefficient, are asymmetrically adjusted according to the ordered client group to reduce the impact of noisy task gradients caused by highly heterogeneous local data. At the same time, the weight of the distillation gradient, which represents global consensus, is increased to suppress global model overfitting, alleviate client drift, and adapt to the ordered submodule of ordered client group allocation.
[0012] S5. The client applies a binary mask specific to the ordered client group to the local update, outputs the feature extractor update part in the local update, performs Top-K sparsification with error feedback on the feature extractor update part, obtains sparse update and uploads it to the client. The server only aggregates the updates of clients within the ordered client group for the feature extraction module of each expert group, and aggregates the updates of all participating clients for the shared classifier head to complete the global model update.
[0013] S6. Repeat steps S2 to S3 every preset readjustment period until the preset number of communication rounds is reached, and output the final global model.
[0014] Preferably, the process of using the server to decompose the global model is as follows:
[0015] First, the global model Decomposed into feature extractor and classifier head The feature extractor is then divided into K sequential modules according to depth, denoted as follows: , ..., Define K groups of clients as The Kth group of clients is responsible for training the Kth sequential module, where the classifier head is shared;
[0016] Train K sequential modules for each of the K groups of clients to obtain the capabilities of each group of clients. Based on the high and low capabilities, divide the corresponding K sequential modules into high-level modules and low-level modules.
[0017] Preferably, the data distribution features are extracted by each group of clients based on the frozen feature extractor, and the corresponding hardware computing power and data signature are calculated for each group of clients. The calculation process is as follows:
[0018] Calculate the feature mean by freezing the feature extractor. with standard deviation Through the characteristic mean with standard deviation Data signature obtained by concatenation and calculation The data signature is then normalized to obtain a unit-length vector, and the hardware computing power is independently evaluated and normalized to obtain a computing power scalar. Sign the data Computational scalar Feedback is sent to the server.
[0019] Preferably, the server performs K-Means clustering on the normalized signature as follows:
[0020] Data signature by the server Computational scalar Perform K-Means clustering to divide the clients into K clusters;
[0021] The objective is to minimize the sum of squared Euclidean distances from samples within each cluster to the center of the cluster, and to calculate the distance metric, which is used to reflect the heterogeneity of the data.
[0022] The server calculates the average computing power of each cluster, sorts the clusters in ascending order based on the average computing power, and maps them to ordered group sequences in turn. Clients with similar data are grouped into one cluster, and clients with strong average computing power are assigned to feature extraction modules with high depth.
[0023] Preferably, when dividing the client into K clusters, the sample farthest from the centroid in the largest cluster is moved to the smallest cluster;
[0024] The entire process of calculating the corresponding hardware computing power and data signature for each group of clients, and the server performing K-Means clustering on the normalized signature, is considered as one round. In each... After each round, the clustering process of the current round is re-executed using the updated data signature and computing power scalar.
[0025] Preferably, the process of asymmetrically adjusting the two momentum coefficients according to their respective ordered client groups is as follows:
[0026] The server broadcasts a one-to-many global model and ordered client group allocation information. The selected clients open task momentum buffers and distillation momentum buffers locally. Each client retains the complete global model throughout the local training process, and all module parameters participate in local updates without freezing any modules.
[0027] During the local training process, the local dataset is divided into several data samples. At the beginning of the current training round, the latest complete global model broadcast to the client is used as the global teacher model. The local model that the client is training and updating based on the local data is used as the student model. In the client's local data, the objective and correct classification label corresponding to each input sample is used as the real label. The global teacher model forward-engineers the teacher intermediate features and teacher logical values, and the local student model forward-engineers the student intermediate features and student logical values.
[0028] The task loss is calculated by cross-entropy using student logical values and true labels, and the mean squared error distillation loss is calculated using student intermediate features and teacher intermediate features. The local optimization objective is the weighted sum of the two.
[0029] Perform two independent backpropagations to obtain the task gradient and distillation gradient, which are then fed into the task momentum buffer and distillation momentum buffer, respectively, based on the momentum coefficient. Introduce specific adjustment factors for the K sequential modules. In terms of fundamental momentum Overlapping group specific adjustment factor The corresponding task momentum coefficient is calculated. and distillation momentum coefficient A relatively larger specific adjustment factor is configured for the client corresponding to the higher-level module. To suppress noisy task gradients and prioritize gradient distillation, smaller specific adjustment factors are configured for the clients corresponding to lower-level modules. To balance local learning with global alignment.
[0030] Preferably, the task momentum coefficient is obtained. and distillation momentum coefficient Based on the corresponding task momentum coefficient and distillation momentum coefficient Calculate the task loss and distillation loss, and then perform local optimization using the calculated task loss and distillation loss to update the task momentum buffer and distillation momentum buffer, resulting in a local update. The two momentum buffers are updated locally using the following formulas:
[0031] In the formula: This represents the task momentum buffer in round t. Indicates the task momentum coefficient; This represents the task momentum buffer in round t-1; This represents the task gradient in round t; Indicates the momentum coefficient of distillation; This represents the distillation momentum buffer zone in the (t-1)th round; This represents the distillation gradient in round t;
[0032] Get , The model parameters are calculated using the following formula, and the parameters of the global model are updated accordingly.
[0033] In the formula: This represents the model parameters in round t-1; This represents the learning rate.
[0034] The preferred sparse update process is as follows:
[0035] Client-side compute local full update ,in Indicates client The local model parameters obtained after local training are compared with the global model parameters distributed by the server. Subtracting them will give you the total number of parameter updates for this client in this round. ;
[0036] Apply an ordered client-group-specific binary mask to local updates on the client. This yields the updated portion of the feature extractor in the local update.
[0037] The updated feature extractor data from this round is added to the accumulated error from the previous round to obtain the feedback adjustment update. In the process of Top-K sparsification, the client discards the accumulated value of smaller gradients due to the compression of communication and stores it locally as the accumulated error of the previous round.
[0038] Adjustments and updates based on feedback Apply Top-K sparsity, globally sort by absolute value from largest to smallest, and retain the compression ratio of the largest absolute value. The sparse update to be transmitted is obtained, and the discarded part in this round is used as the update error. Local storage is performed so that the gradient signal is accumulated over multiple rounds and then transmitted selectively, thus achieving near-lossless transmission within the time window. In the local training process on the client side, the gradient signal is obtained by backpropagation calculation of the weighted sum of the task loss and distillation loss.
[0039] The server aggregates updates for clients within an ordered client group only for the feature extraction module of each expert group, and aggregates updates for all clients for the shared classifier head, thus completing precise knowledge fusion symmetrical to the client compression strategy.
[0040] Preferred: Ordered client group specific binary mask The parameter is a static tensor of the same dimension as the global model. It takes the value 1 when the parameter belongs to the module assigned to the k-th group or the shared classifier head, and takes the value 0 otherwise. The specific binary mask is uniquely determined by the sequential model decomposition and the allocation of clusters to ordered client groups, without introducing free hyperparameters that need to be manually searched.
[0041] Beneficial effects: This invention relates to a communication federated learning method based on distillation and two-dimensional collaborative compression, which divides the data distribution into several clusters and the computing power into high- and low-level modules. This enables the two semantically incommensurable signals of data distribution and computing power to be strictly decoupled in the distance metric, eliminating the heuristic balancing weights required by traditional splicing clustering, improving the clustering quality and enhancing the interpretability of grouping.
[0042] The training clients open task momentum buffers and distillation momentum buffers locally, and adjust the two momentum coefficients asymmetrically according to their ordered client groups. This suppresses noise gradients in highly heterogeneous groups while strengthening the global consensus signal, effectively alleviating client drift and making the training trajectory smoother. During updates, Top-K sparsification with error feedback is performed, achieving a reduction in communication overhead by orders of magnitude without discarding key gradient information. Attached Figure Description
[0043] Figure 1 This is an overall system block diagram of the present invention;
[0044] Figure 2 This is a diagram of the overall architecture of the present invention;
[0045] Figure 3 This is a schematic diagram illustrating the ordered group-to-module allocation relationship of the present invention;
[0046] Figure 4 This is a graph showing the variation of the test accuracy of this invention with the number of communication rounds under different datasets. Detailed Implementation
[0047] like Figures 1 to 4 As shown, this invention provides a technical solution: a communication federated learning method based on distillation and two-dimensional collaborative compression, comprising the following steps:
[0048] Step 1: Collect data distribution statistics and computing power metrics from K groups of clients via the server, and then decompose the global model using the server. The decomposition process is as follows:
[0049] First, the global model Decomposed into feature extractor and classifier head The feature extractor is then divided into K sequential modules according to depth, denoted as follows: , ..., Define K groups of clients as The Kth group of clients is responsible for training the Kth sequential module, where the classifier head is shared;
[0050] Train K sequential modules for each of the K groups of clients to obtain the capabilities of each group of clients. Based on the high and low capabilities, divide the corresponding K sequential modules into high-level modules and low-level modules.
[0051] The global model is decomposed into a feature extractor and a classifier head, and the feature extractor is divided into K sequential modules.
[0052] Step 2: Each group of clients extracts data distribution features based on the frozen feature extractor. The frozen feature extractor is a pre-trained network model on a public dataset, such as ImageNet, and the network model is such as ResNet-18. The frozen feature extractor is only used for initial coarse extraction of data signatures and does not participate in updates during federated learning. The frozen feature extractor is independent of the target feature extractor that needs to be trained and optimized in the federated model. After the data distribution feature extraction is completed, the corresponding hardware computing power and data signature are calculated for each group of clients. The calculation process for obtaining the corresponding hardware computing power and data signature is as follows:
[0053] Calculate the feature mean by freezing the feature extractor. with standard deviation Obtain the feature set calculated by each client on the representative subset, and then use the feature mean. with standard deviation Data signature obtained by concatenation and calculation Data signature The calculation formula is as follows:
[0054] ;
[0055] The data signature is then normalized to a unit-length vector to obtain a normalized signature. Finally, the hardware computing power is independently evaluated to obtain a normalized computing power scalar. The normalized computing power is obtained, and then the obtained normalized computing power and normalized signature are fed back to the server.
[0056] Step 3: The server performs K-Means clustering on the normalized signature, dividing the clients into several clusters. The clients in each cluster are sorted in ascending order according to their average computing power to form ordered client groups. Each group is assigned a feature extraction module of the corresponding depth. The assigned sets are sorted in ascending order according to their average computing power within the cluster to build an expert group.
[0057] The server performs K-Means clustering on the normalized signatures as follows:
[0058] Data signature by the server Computational scalar Perform K-Means clustering to divide the clients into K clusters;
[0059] The objective is to minimize the sum of squared Euclidean distances from samples within each cluster to the center of the cluster, and to calculate the distance metric, which is used to reflect the heterogeneity of the data.
[0060] The server calculates the average computing power of each cluster, sorts the clusters in ascending order of average computing power, and maps them to ordered group sequences in turn. Clients with similar data are grouped into one cluster, and clients with strong average computing power are assigned to feature extraction modules with high depth.
[0061] When dividing the client into K clusters, the sample farthest from the centroid in the largest cluster is moved to the smallest cluster;
[0062] The entire process of calculating the corresponding hardware computing power and data signature for each group of clients, and the server performing K-Means clustering on the normalized signature, is considered as one round. In each... After each round, the clustering process of the current round is re-executed using the updated data signature and computing power scalar.
[0063] Step 4: The server broadcasts the global model and the allocation information of clients to ordered client groups in a one-to-many manner. Local updates are obtained through local optimization using task loss and distillation loss. The task momentum coefficient and distillation momentum coefficient are asymmetrically adjusted according to the ordered client group to reduce the impact of noisy task gradients caused by highly heterogeneous local data. At the same time, the weight of the distillation gradient, which represents global consensus, is increased to suppress global model overfitting, alleviate client drift, and adapt to the ordered sub-modules of ordered client group allocation.
[0064] That is, the server broadcasts the global model and the ordered client group allocation information to one-to-many. The selected client opens a task momentum buffer and a distillation momentum buffer locally. Each client retains the complete global model throughout the local training process. All module parameters participate in local updates and no module is frozen.
[0065] During the local training process, the local dataset is divided into several data samples. At the beginning of the current training round, the latest complete global model broadcast to the client is used as the global teacher model. The local model that the client is training and updating based on the local data is used as the student model. In the client's local data, the objective and correct classification label corresponding to each input sample is used as the real label. The global teacher model forward-engineers the teacher intermediate features and teacher logical values, and the local student model forward-engineers the student intermediate features and student logical values.
[0066] The task loss is calculated by cross-entropy using student logistic values and true labels, and the mean squared error distillation loss is calculated using student intermediate features and teacher intermediate features. The local optimization objective is a weighted sum of the two, and its calculation formula is as follows:
[0067] In the formula: This represents the local composite optimization objective, which indicates the overall loss that needs to be minimized during local training on the client side. Indicates parameters; This represents the cross-entropy task loss; This represents the mean square error distillation loss;
[0068] Perform two independent backpropagations to obtain the task gradient and distillation gradient, which are then fed into the task momentum buffer and distillation momentum buffer, respectively, based on the momentum coefficient. Introduce specific adjustment factors for the K sequential modules. In terms of fundamental momentum Overlapping group specific adjustment factor The corresponding task momentum coefficient is calculated. and distillation momentum coefficient Among them, the task momentum coefficient The calculation formula is: Distillation momentum coefficient The calculation formula is: A relatively larger specific adjustment factor is configured for the client corresponding to the higher-level module. To suppress noisy task gradients and prioritize gradient distillation, smaller specific adjustment factors are configured for the clients corresponding to lower-level modules. To balance local learning and global alignment, the network module depth allocated by the ordered client group is adapted, and then the task momentum coefficient is obtained. and distillation momentum coefficient Based on the corresponding task momentum coefficient and distillation momentum coefficient Calculate the task loss and distillation loss, and then perform local optimization using the calculated task loss and distillation loss to update the task momentum buffer and distillation momentum buffer, resulting in a local update. The two momentum buffers are updated locally using the following formulas:
[0069] In the formula: This represents the task momentum buffer in round t. Indicates the task momentum coefficient; This represents the task momentum buffer in round t-1; This represents the task gradient in round t; Indicates the momentum coefficient of distillation; This represents the distillation momentum buffer zone in the (t-1)th round; This represents the distillation gradient in round t;
[0070] Get , The model parameters are calculated using the following formula, and the parameters of the global model are updated accordingly.
[0071] In the formula: This represents the model parameters in round t-1; This represents the learning rate.
[0072] Step 5: The client applies an ordered client-group-specific binary mask to the local update, outputs the feature extractor update portion of the local update, performs Top-K sparsification with error feedback on the feature extractor update portion, obtains the sparse update, and uploads it to the client. The specific process of the sparse update is as follows:
[0073] Client-side compute local full update ,in Indicates client The local model parameters obtained after local training are compared with the global model parameters distributed by the server. Subtracting them will give you the total number of parameter updates for this client in this round. ;
[0074] Apply an ordered client-group-specific binary mask to local updates on the client. The updated part of the feature extractor in the local update is obtained. The calculation formula is as follows:
[0075] In the formula: This represents a binary mask specific to an ordered client group. If the parameter position belongs to the feature extraction module assigned to the current group or to the globally shared classifier head, the corresponding position of the binary mask specific to the ordered client group is set to 1. For the remaining parameter positions, the corresponding position of the binary mask specific to the ordered client group is set to 0. ⊙ represents element-wise multiplication.
[0076] The updated feature extractor data from this round is added to the accumulated error from the previous round to obtain the feedback adjustment update. The cumulative error from the previous round is the accumulated value of smaller gradients discarded due to communication compression during Top-K sparsification, which is stored locally as the cumulative error and used to adjust and update the feedback. Apply Top-K sparsity, globally sort by absolute value from largest to smallest, and retain the compression ratio of the largest absolute value. The sparse update to be transmitted is obtained, and the discarded part in this round is used as the update error. Perform local storage, where update error The calculation formula is as follows:
[0077] In the formula: Indicates update error; This indicates feedback and adjustments / updates; Indicates sparsity; This represents the Kth sequential module; This represents the compression ratio; the gradient signal is accumulated over multiple rounds and then transmitted selectively, thus achieving near-lossless performance within the time window. In the client's local training process, the gradient signal is calculated by backpropagation of the weighted sum of the task loss and distillation loss.
[0078] The server aggregates updates for clients within an ordered client group only for the feature extraction module of each expert group, achieving precise knowledge fusion symmetrical to the client compression strategy. It also aggregates updates for all participating clients for the shared classifier head, completing the global model update.
[0079] Among them, the ordered client group specific binary mask The parameter is a static tensor of the same dimension as the global model. It takes the value 1 when the parameter belongs to the module assigned to the k-th group or the shared classifier head, and takes the value 0 otherwise. The specific binary mask is uniquely determined by the sequential model decomposition and the allocation of clusters to ordered client groups, without introducing free hyperparameters that need to be manually searched.
[0080] Step Six, every After each round, steps two and three are re-executed using the updated data signature and capability score until the preset number of communication rounds is reached, at which point the final global model is output.
[0081] Example 1
[0082] Through the above technical solution, the present invention can achieve the following working process:
[0083] The experiments were all conducted on a server equipped with an NVIDIA RTX 3090 graphics card and an Intel Xeon Gold 6330 CPU, using Python 3.8 and the PyTorch framework.
[0084] Assume a total number of communication rounds T=200, a total number of clients N=100, 10 clients are selected for training in each round, each selected client performs E=10 local training rounds, the number of clusters K=3, and the compression ratio is... Take 0.1, momentum coefficient Set the learning rate to 0.9. The value is set to 0.01. Dirichlet distribution is used to simulate data heterogeneity, with parameters... The smaller the value, the higher the heterogeneity. Our method was compared with representative methods such as FedAvg, FedProx, HeteroFL, FedRolex, FedLMT, and ClippedQuantFL on three benchmark datasets: CIFAR-10, CIFAR-100, and MNIST, under the same conditions. Experimental results show that when the parameter α=0.1, our method achieves an accuracy of approximately 61.78% on CIFAR-10, significantly outperforming all baselines. In terms of total communication, our method requires only approximately 7.933 Gbit, 12.676 Gbit, and 7.166 Gbit on CIFAR-10, CIFAR-100, and MNIST, respectively, to achieve the target accuracy, with communication ratios per round of approximately 7.08%, 8.43%, and 6.73%, respectively, all significantly lower than the optimal baseline. Ablation experiments further validated the necessity and effectiveness of the three modules—grouped dual-momentum distillation, two-dimensional collaborative compression, and dynamic clustering—and the two-stage clustering design.
[0085] Table 1. Comparison of test accuracy between this method and baseline methods.
[0086]
[0087] Table 1 compares the test accuracy of our proposed method with that of baseline methods under different datasets and varying degrees of heterogeneity. As shown in Table 1, our proposed method achieves optimal or near-optimal test accuracy under all degrees of heterogeneity. Particularly under moderate heterogeneity (α=0.1), our proposed method achieves 0.6178 on CIFAR-10, significantly outperforming the optimal baseline FedLMT's 0.5256; it achieves 0.3479 on CIFAR-100 and 0.9651 on MNIST, both the highest values among all methods. Under high heterogeneity (α=0.05), our proposed method achieves 0.4375 on CIFAR-10 and 0.9513 on MNIST, demonstrating strong robustness to data heterogeneity.
[0088] Table 2. Comparison of total communication and average communication ratio between this method and baseline methods.
[0089]
[0090] Table 2 compares the total communication volume and average communication ratio of our proposed method with those of the baseline methods when achieving the preset target accuracy. As shown in Table 2, the total communication volume required for our proposed method to achieve the target accuracy on the three datasets is significantly lower than that of all baseline methods, with average communication ratios of 7.08% (CIFAR-10), 8.43% (CIFAR-100), and 6.73% (MNIST), respectively. These are all far lower than the baseline method ClippedQuantFL (12.50%, 22.02%, and 12.44%), which has the best communication efficiency.
[0091] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and all such equivalent transformations fall within the protection scope of the present invention.
Claims
1. A communication federated learning method based on distillation and two-dimensional collaborative compression, characterized in that, Includes the following steps: S1. Collect data distribution statistics and computing power measurement from K groups of clients through the server, then decompose the global model using the server to obtain the feature extractor and classifier head, and divide the feature extractor into K sequential modules; S2. The data distribution features are extracted by each group of clients based on the frozen feature extractor. The corresponding hardware computing power and data signature are calculated for each group of clients. The hardware computing power and data signature are normalized respectively. The normalized computing power and normalized signature are then fed back to the server. S3. The server performs K-Means clustering on the normalized signature, divides the clients into several clusters, sorts them in ascending order according to the average computing power of the clients in each cluster, forms an ordered client group, and assigns a feature extraction module of corresponding depth to each client. The set obtained by mapping and assigning is sorted in ascending order according to the average computing power in the cluster, and an expert group is built. S4. The server broadcasts the global model and the allocation information of clients to ordered client groups in a one-to-many manner. Local updates are obtained through local optimization using task loss and distillation loss. The task momentum coefficient and distillation momentum coefficient are asymmetrically adjusted according to the ordered client group to reduce the impact of noisy task gradients caused by highly heterogeneous local data. At the same time, the weight of the distillation gradient, which represents global consensus, is increased to suppress global model overfitting, alleviate client drift, and adapt to the ordered sub-module of ordered client group allocation. S5. The client applies a binary mask specific to the ordered client group to the local update, outputs the feature extractor update part in the local update, performs Top-K sparsification with error feedback on the feature extractor update part, obtains sparse update and uploads it to the client. The server only aggregates the updates of clients within the ordered client group for the feature extraction module of each expert group, and aggregates the updates of all participating clients for the shared classifier head to complete the global model update. S6. Repeat steps S2 to S3 every preset readjustment period until the preset number of communication rounds is reached, and output the final global model.
2. The communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 1, characterized in that, The process of decomposing the global model using a server is as follows: First, the global model Decomposed into feature extractor and classifier head The feature extractor is then divided into K sequential modules according to depth, denoted as follows: , ..., Define K groups of clients as The Kth group of clients is responsible for training the Kth sequential module, where the classifier head is shared; Train K sequential modules for each of the K groups of clients to obtain the capabilities of each group of clients. Based on the high and low capabilities, divide the corresponding K sequential modules into high-level modules and low-level modules.
3. The communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 1, characterized in that, Each group of clients extracts data distribution features based on the frozen feature extractor, and calculates the corresponding hardware computing power and data signature for each group of clients. The calculation process is as follows: Calculate the feature mean by freezing the feature extractor. with standard deviation Through the characteristic mean with standard deviation Data signature obtained by concatenation and calculation The data signature is then normalized to obtain a unit-length vector, and the hardware computing power is independently evaluated and normalized to obtain a computing power scalar. Sign the data Computational scalar Feedback is sent to the server.
4. The communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 3, characterized in that, The server performs K-Means clustering on the normalized signature as follows: Data signature by the server Computational scalar Perform K-Means clustering to divide the clients into K clusters; The objective is to minimize the sum of squared Euclidean distances from samples within each cluster to the center of the cluster, and to calculate the distance metric, which is used to reflect the heterogeneity of the data. The server calculates the average computing power of each cluster, sorts the clusters in ascending order based on the average computing power, and maps them to ordered group sequences in turn. Clients with similar data are grouped into one cluster, and clients with strong average computing power are assigned to feature extraction modules with high depth.
5. The communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 3, characterized in that, When dividing the client into K clusters, the sample farthest from the centroid in the largest cluster is moved to the smallest cluster; The entire process of calculating the corresponding hardware computing power and data signature for each group of clients, and the server performing K-Means clustering on the normalized signature, is considered as one round. In each... After each round, the clustering process of the current round is re-executed using the updated data signature and computing power scalar.
6. The communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 5, characterized in that, The process of asymmetrically adjusting the two momentum coefficients according to their respective ordered client groups is as follows: The server broadcasts a one-to-many global model and ordered client group allocation information. The selected clients open task momentum buffers and distillation momentum buffers locally. Each client retains the complete global model throughout the local training process, and all module parameters participate in local updates without freezing any modules. During the local training process, the local dataset is divided into several data samples. At the beginning of the current training round, the latest complete global model broadcast to the client is used as the global teacher model. The local model that the client is training and updating based on the local data is used as the student model. In the client's local data, the objective and correct classification label corresponding to each input sample is used as the real label. The global teacher model forward-engineers the teacher intermediate features and teacher logical values, and the local student model forward-engineers the student intermediate features and student logical values. The task loss is calculated by cross-entropy using student logical values and true labels, and the mean squared error distillation loss is calculated using student intermediate features and teacher intermediate features. The local optimization objective is the weighted sum of the two. Perform two independent backpropagations to obtain the task gradient and distillation gradient, which are then fed into the task momentum buffer and distillation momentum buffer, respectively, based on the momentum coefficient. Introduce specific adjustment factors for the K sequential modules. In terms of fundamental momentum Overlapping group specific adjustment factor The corresponding task momentum coefficient is calculated. and distillation momentum coefficient A relatively larger specific adjustment factor is configured for the client corresponding to the higher-level module. To suppress noisy task gradients and prioritize gradient distillation, smaller specific adjustment factors are configured for the clients corresponding to lower-level modules. To balance local learning with global alignment.
7. The communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 6, characterized in that, Obtain the task momentum coefficient and distillation momentum coefficient Based on the corresponding task momentum coefficient and distillation momentum coefficient Calculate the task loss and distillation loss, and then perform local optimization using the calculated task loss and distillation loss to update the task momentum buffer and distillation momentum buffer, resulting in a local update. The two momentum buffers are updated locally using the following formulas: In the formula: This represents the task momentum buffer in round t. Indicates the task momentum coefficient; This represents the task momentum buffer in round t-1; This represents the task gradient in round t; Indicates the momentum coefficient of distillation; This represents the distillation momentum buffer zone in the (t-1)th round; This represents the distillation gradient in round t; Get , The model parameters are calculated using the following formula, and the parameters of the global model are updated accordingly. In the formula: This represents the model parameters in round t-1; This represents the learning rate.
8. The communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 5, characterized in that, The specific process of sparse update is as follows: Client-side compute local full update ,in Indicates client The local model parameters obtained after local training are compared with the global model parameters distributed by the server. Subtracting them will give you the total number of parameter updates for this client in this round. ; Apply an ordered client-group-specific binary mask to local updates on the client. This yields the updated portion of the feature extractor in the local update. The feature extractor update obtained in this round is added to the accumulated error from the previous round to obtain the feedback adjustment update. The cumulative error in the previous round is the cumulative value of the smaller gradient that was discarded due to the compression of communication when the client performs Top-K sparsification. This value is stored locally as the cumulative error. Adjustments and updates based on feedback Apply Top-K sparsity, globally sort by absolute value from largest to smallest, and retain the compression ratio of the largest absolute value. The sparse update to be transmitted is obtained, and the discarded part in this round is used as the update error. Local storage is performed so that the gradient signal is accumulated over multiple rounds and then transmitted selectively, thus achieving near-lossless transmission within the time window. In the local training process on the client side, the gradient signal is obtained by backpropagation calculation of the weighted sum of the task loss and distillation loss. The server aggregates updates for clients within an ordered client group only for the feature extraction module of each expert group, and aggregates updates for all clients for the shared classifier head, thus completing precise knowledge fusion symmetrical to the client compression strategy.
9. A communication federated learning method based on distillation and two-dimensional collaborative compression according to claim 8, characterized in that, Ordered client group specific binary mask The parameter is a static tensor of the same dimension as the global model. It takes the value 1 when the parameter belongs to the module assigned to the k-th group or the shared classifier head, and takes the value 0 otherwise. The specific binary mask is uniquely determined by the sequential model decomposition and the allocation of clusters to ordered client groups, without introducing free hyperparameters that need to be manually searched.