Adaptive heterogeneous compression method for federated knowledge distillation systems
By using an adaptive heterogeneous compression method in a federated knowledge distillation system, and dynamically selecting compression strategies, the problem of low resource utilization and optimization efficiency in existing technologies is solved, achieving efficient resource utilization and model performance balance in heterogeneous environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG UNIV OF TECH
- Filing Date
- 2026-01-15
- Publication Date
- 2026-06-12
AI Technical Summary
Existing federated learning heterogeneous compression methods cannot adapt to the differences between different clients, resulting in low resource utilization and optimization efficiency, and failing to achieve the optimal balance between communication efficiency and model performance in heterogeneous environments.
An adaptive heterogeneous compression method based on a federated knowledge distillation system is adopted. By setting a local compression strategy candidate set and a strategy selector, the most suitable compression strategy is dynamically selected. Combining multiple compression techniques such as Top-K, Random-K, and Periodic-K, and using an ε-greedy strategy for decision-making, adaptive gradient compression and knowledge distillation are achieved.
It improves resource utilization and optimization efficiency, maintains a balance between model accuracy and communication efficiency in heterogeneous environments, adapts to the differences between different clients, and reduces computational and communication overhead.
Smart Images
Figure CN122198012A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of federated learning, and more specifically relates to an adaptive heterogeneous compression method for federated knowledge distillation systems. Background Technology
[0002] Federated Learning (FL) is an emerging distributed machine learning paradigm that allows multiple clients to collaboratively learn a shared model without exposing their raw data, thus protecting data privacy. However, in practice, Federated Learning (FL) faces the challenge of system heterogeneity, where clients have varying computing power, memory, and bandwidth, resulting in different model sizes that each client can support.
[0003] To address this, researchers have explored various compression techniques, such as model parameter quantization, model pruning, and parameter compression. Parameter compression, for example, selects to upload some of the client's local model's update gradients to the server, effectively reducing the amount of communication data. Simultaneously, adaptive communication cost optimization methods have been developed, such as Adaptive Gradient Quantization (AdaGQ) and the Hybrid Communication and Energy Optimization Framework (HCEF). AdaGQ adaptively adjusts the quantization level based on gradient magnitude changes, while HCEF improves overall training efficiency by jointly optimizing communication and energy consumption. Furthermore, some studies have attempted to combine multiple compression methods, such as Hybrid Sparsification-Quantization (HSQ) strategies. However, these strategies are typically statically configured and cannot dynamically and adaptively adjust compression parameters or methods based on model training, making it difficult to achieve an optimal balance between communication efficiency and model performance in heterogeneous environments.
[0004] Existing heterogeneous compression methods mostly adopt a uniform and fixed compression scheme, ignoring the differences in model architecture and data distribution between clients. They cannot adapt to the differences in local model structure, data distribution and computing resources of different clients, resulting in low resource utilization and optimization efficiency. Summary of the Invention
[0005] To address the problem of low resource utilization and optimization efficiency in existing heterogeneous compression methods, this invention proposes an adaptive heterogeneous compression method for federated knowledge distillation systems, which adapts to the differences between different clients and improves resource utilization and optimization efficiency.
[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: This invention provides an adaptive heterogeneous compression method for a federated knowledge distillation system, the federated knowledge distillation system including several clients and a federated server, and the adaptive heterogeneous compression method for the federated knowledge distillation system includes the following steps: S1: Set and initialize the client's local compression policy candidate set and policy selector; S2: On each client, train the local model set on that client based on the client's local data to obtain the local gradient of the current training round; based on the local gradient of the current training round and the residual vector of the previous training round, obtain the gradient to be compressed. S3: Each client selects a compression strategy from the local compression strategy candidate set based on the strategy selector, and uses the strategy to compress the gradient to be compressed to obtain the compressed gradient parameters; based on the gradient to be compressed and the compressed gradient, the residual vector of the current training round is generated. S4: Upload the compressed gradient parameters to the federated server. The federated server aggregates the gradient parameters uploaded by each client to obtain the aggregated model. Based on the public dataset, knowledge distillation is performed on the aggregated model to obtain the optimized model. S5: Send the optimized model to each client; calculate the reward value for selecting the compression strategy based on the optimized model; update the strategy selector in step S3 based on the reward value; S6: Use the optimized model as the local model and repeat steps S2-S6 until the preset maximum number of training rounds is reached.
[0007] Furthermore, the candidate set of local compression strategies includes: Top-K gradient sparsity compression strategy, Random-K stochastic gradient compression strategy, and Periodic-K periodic gradient compression strategy; the expression for the candidate set of local compression strategies is:
[0008] In the formula, denoted as the local compression policy candidate set, Top-K represents the Top-K gradient sparsity compression policy, Random-K represents the Random-K stochastic gradient compression policy, and Periodic-K represents the Periodic-K periodic gradient compression policy.
[0009] Furthermore, the strategy selector employs an ε-greedy strategy for decision-making, the process of which is as follows: Step 1: Initialize utility estimates for each compression strategy in the local compression strategy candidate set; Step 2: In the current training round, select the compression strategy with the highest current utility estimate with probability 1-ε; randomly select a compression strategy from the local compression strategy candidate set with probability ε; where ε represents the preset exploration rate parameter; Step 3: After completing this round of training, calculate the reward value for selecting the compression strategy; Step 4: Update the utility estimate from Step 1 based on the reward value, using the following expression:
[0010] In the formula, This represents the current utility estimate. This represents the updated utility estimate. n Indicates the client, Indicates the compression strategy. Indicates the learning rate. Represents the reward function; Step 5: In the next training round, return to Step 2 and select a compression strategy based on the updated utility estimate.
[0011] Furthermore, the process of step S2 is as follows: The local model set up on the client is trained based on the client's local data, and the local gradient for the current round is calculated. ; Based on the local gradient of the current training epoch and the residual vector of the previous training epoch. The expression for the gradient to be compressed is obtained as follows:
[0012] In the formula, Indicates the client index. t Indicates the training rounds. Indicates the first The local gradient calculated by each client after the t-th round of local training. Indicates the first The residual vector of each client in the previous training round. This represents the gradient to be compressed in the current training round.
[0013] Furthermore, based on the gradient to be compressed and the compressed gradient, the expression for generating the residual vector of the current training round is as follows:
[0014] In the formula, This represents the residual vector for the current round. The gradients to be compressed in the current training epoch. This represents the compression gradient for the current training epoch.
[0015] Furthermore, the process of performing knowledge distillation on the aggregation model based on public datasets to obtain the optimized model is as follows: The federated server uses an aggregation model to perform forward inference on a public dataset to obtain the predicted output value of the aggregation model. Calculate the average value of the predicted output of the aggregation model to generate global soft labels; Using global soft labels as supervision signals, the aggregation model is trained by distillation. When the KL divergence between the predicted output value of the aggregation model and the global soft labels is minimized, the federated server obtains the updated parameters of the aggregation model to obtain the optimized model.
[0016] Furthermore, based on the optimization model, the process of calculating the reward value for selecting the compression strategy is as follows: The client calculates the local and global loss values for the current training round based on the optimized model. Calculate the change in overall loss based on the local and global loss values of the current training round and the previous training round; Based on the overall change in loss and time cost, the reward value for the current training round is calculated using the following expression:
[0017] In the formula, This represents the reward value for the current training round. This indicates the change in overall loss. Indicates the weighting coefficient; Indicates client Local training time, Indicates the client Reduce operation time. Indicates the client The time it takes for parameters to be uploaded to the server during communication.
[0018] Furthermore, the expression for calculating the change in overall loss is as follows:
[0019] In the formula, This indicates the change in overall loss. Indicates weight, Indicates weight, This represents the global loss value from the previous training round. This represents the local loss value from the previous training round. This represents the global loss value in the current training round. This represents the local loss value for the current training round.
[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the adaptive heterogeneous compression program of the federated knowledge distillation system is executed by the processor, the steps of the adaptive heterogeneous compression method of the federated knowledge distillation system are implemented.
[0021] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the operations performed by the adaptive heterogeneous compression method of the federated knowledge distillation system.
[0022] Compared with existing technologies, the beneficial effects of this method are: This invention provides an adaptive heterogeneous compression method for a federated knowledge distillation system, belonging to the technical field of federated learning. First, the local compression policy set and policy selector for each client are initialized. Each client trains a model based on local data and calculates the gradient to be compressed using the residual from the previous round. The policy selector selects a compression policy from local candidate policies for gradient compression, simultaneously generating the residual vector for the current training round. The compressed gradient is uploaded to the federated server for aggregation, and knowledge distillation is performed using a public dataset to obtain an optimized model. The optimized model is then distributed to each client, the reward value of the selected compression policy is calculated, and the policy selector is updated. Finally, the optimized model is used as the local model for the new round, and the above process is repeated until the preset maximum number of training rounds is reached. The heterogeneous compression method proposed in this invention can adapt to the differences between different clients, improving resource utilization and optimization efficiency. Attached Figure Description
[0023] Figure 1 A flowchart illustrating the adaptive heterogeneous compression method for federated knowledge distillation proposed in this embodiment of the invention; Figure 2 This diagram illustrates the framework of the adaptive heterogeneous compression method for federated knowledge distillation proposed in this embodiment of the invention. Figure 3 This is a schematic diagram illustrating the electronic device proposed in an embodiment of the present invention. Detailed Implementation
[0024] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.
[0025] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0026] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes an adaptive heterogeneous compression method for a federated knowledge distillation system, such as... Figure 1 The flowchart shown and Figure 2 The framework diagram shown illustrates that the federated knowledge distillation system includes several clients and a federated server. The adaptive heterogeneous compression method of the federated knowledge distillation system includes the following steps: S1: Set and initialize the client's local compression policy candidate set and policy selector; S2: On each client, train the local model set on that client based on the client's local data to obtain the local gradient of the current training round; based on the local gradient of the current training round and the residual vector of the previous training round, obtain the gradient to be compressed. S3: Each client selects a compression strategy from the local compression strategy candidate set based on the strategy selector, and uses the strategy to compress the gradient to be compressed to obtain the compressed gradient parameters; based on the gradient to be compressed and the compressed gradient, the residual vector of the current training round is generated. S4: Upload the compressed gradient parameters to the federated server. The federated server aggregates the gradient parameters uploaded by each client to obtain the aggregated model. Based on the public dataset, knowledge distillation is performed on the aggregated model to obtain the optimized model. S5: Send the optimized model to each client; calculate the reward value for selecting the compression strategy based on the optimized model; update the strategy selector in step S3 based on the reward value; S6: Use the optimized model as the local model and repeat steps S2-S6 until the preset maximum number of training rounds is reached.
[0027] To mitigate model heterogeneity caused by system heterogeneity, Federated Knowledge Distillation (FedKD) was proposed, a collaborative training mechanism combining federated learning and knowledge distillation. In FedKD, clients collaborate by exchanging knowledge, rather than the original model parameters, enabling heterogeneous models to work together.
[0028] The candidate set of local compression strategies includes: Top-K gradient sparsity compression strategy, Random-K stochastic gradient compression strategy, and Periodic-K periodic gradient compression strategy; the expression for the candidate set of local compression strategies is:
[0029] In the formula, denoted as the local compression policy candidate set, Top-K represents the Top-K gradient sparsity compression policy, Random-K represents the Random-K stochastic gradient compression policy, and Periodic-K represents the Periodic-K periodic gradient compression policy.
[0030] In this embodiment, step S1 further includes: loading all local data from all clients and generating a DataLoader; creating a corresponding local model for each client and allocating heterogeneous models according to rules.
[0031] Each client performs multi-step gradient descent training on its local model based on local data, thus obtaining the local gradient.
[0032] The Top-K gradient sparsity compression strategy involves the client selecting the K entries with the largest absolute magnitudes from the gradient vector and sending them along with their corresponding indices to the server. This method effectively preserves the most important gradient directions, resulting in good model performance. However, due to the need to sort the gradients, its computational complexity is high. ,in For the entire gradient dimension, For the front One entry.
[0033] The Random-K stochastic gradient compression strategy involves the client randomly sampling K indices from all gradient dimensions and uploading the corresponding gradient values. This method requires almost no additional computation, with a complexity of only [missing information - likely a time complexity]. However, due to its random selection scheme, the resulting model has poor performance.
[0034] The Periodic-K periodic gradient compression strategy uses a predefined period length p and a counter vector as the client. By performing a round-robin update on the dimensions, Periodic-K can ensure that all dimensions are refreshed at least once every p rounds, achieving a trade-off between the accuracy of Top-K and the low cost of Random-K.
[0035] Most existing studies employ a uniform compression scheme (e.g., fixed Top-K, Random-K, Periodic-K) across all clients during the overall training process. This fixed compression strategy ignores the differences in model architecture and data distribution among clients. The Top-K method selects the top K important elements based on gradient magnitude for transmission, which can retain key update information to the greatest extent, but it has a high computational cost and is prone to information shift when the gradient distribution is sparse or unbalanced. The Random-K method achieves communication compression by randomly selecting K elements, which has a lower computational cost and is simpler to implement, but ignores the importance of gradients, which may lead to a decrease in convergence speed. Periodic-K can guarantee that all dimensions are refreshed at least once every p rounds, but when the model suddenly needs to update some important dimensions, these dimensions may be in the "not yet reached periodicity, cannot be transmitted" stage, and therefore cannot respond immediately.
[0036] Under a unified compression strategy, the merits of different methods depend on the characteristics of the client: for clients with strong computing power and stable gradient distribution, Top-K can achieve better convergence performance; for clients with limited computing resources or high noise, Random-K is more robust; while Periodic-K achieves a trade-off between the accuracy of Top-K and the low cost of Random-K.
[0037] Significant differences in model architecture, data distribution, and computing resources among clients lead to decreased model performance and resource efficiency. This rigid, standardized model training process across clients, with its fixed compression method, results in an imbalance in the optimization process among clients. Existing methods cannot adaptively optimize based on dynamic changes in parameter distribution and model structure during training, making it difficult to balance communication efficiency and model accuracy.
[0038] Existing methods and strategies, along with parameter settings, are predetermined and fixed before training begins, lacking dynamic adaptability to model training states and system communication resources. Furthermore, these strategies do not consider the increased computational resource consumption caused by compression, failing to maintain an optimal balance between resource cost and model accuracy throughout the training cycle, ultimately limiting system performance in complex heterogeneous environments.
[0039] In real-world federated scenarios, there is significant heterogeneity among clients, with varying computational resources: some clients possess strong computational capabilities, allowing Top-K compression to preserve highly important gradients and improve model accuracy; while devices with limited computational resources can use Random-K compression to reduce local computational burden. Data distribution also varies: when data distribution is not independent and identically distributed, the distribution of important gradient information differs across clients; employing differentiated compression methods can avoid information loss and model shifts caused by a uniform strategy.
[0040] In this application, to overcome the limitations of uniform compression and static heterogeneous compression and enable each client to autonomously select the most suitable compression strategy, a multi-armed bandit (MAB) mechanism is introduced. The ε-greedy strategy algorithm in the multi-armed bandit allows the client to dynamically adjust the compression decision based on the feedback of the loss amount and computing resources.
[0041] The strategy selector uses an ε-greedy strategy for decision-making, the process of which is as follows: Step 1: Initialize utility estimates for each compression strategy in the local compression strategy candidate set; Step 2: In the current training round, select the compression strategy with the highest current utility estimate with probability 1-ε; randomly select a compression strategy from the local compression strategy candidate set with probability ε; where ε represents the preset exploration rate parameter; Step 3: After completing this round of training, calculate the reward value for selecting the compression strategy; Step 4: Update the utility estimate from Step 1 based on the reward value, using the following expression:
[0042] In the formula, This represents the current utility estimate. This represents the updated utility estimate. n Indicates the client, Indicates the compression strategy. Indicates the learning rate. Represents the reward function; Step 5: In the next training round, return to Step 2 and select a compression strategy based on the updated utility estimate.
[0043] Compared to fixed-policy methods, the advantages of using MAB (Multi-Purpose Optimization) are: strong online learning capability (MAB dynamically adjusts the policy selection probability based on the real-time "rewards" generated in each training round, allowing the model to adapt to the client in real time); a balance between exploration and exploitation (MAB can balance whether to continue using the current policy or try a better new policy); and strong robustness (MAB updates the expected return of each policy online, allowing each client to develop different policy preferences based on its own characteristics). This mechanism makes this invention particularly suitable for multi-client environments with significant differences in device performance. The ε-greedy policy, as a typical MAB algorithm, has advantages such as simple implementation, fast convergence, and low computational cost. By using probabilistic... Choose the current optimal strategy, based on probability. By randomly exploring other strategies, the system maintains continuous exploration capabilities while ensuring stable performance. This mechanism is ideal for heterogeneous clients that require rapid response and have limited resources.
[0044] Furthermore, in the Warm-up phase, during the initial training phase, to mitigate bias caused by insufficient feedback data, each client performs a fixed number of warm-up rounds for each candidate policy. In the selection phase, following the Warm-up, in each round, the client selects a candidate policy with probability... Explore, that is, randomly select a strategy. The client uses probability... To utilize this strategy, we select the one with the highest average reward (i.e., the cumulative utility Q value).
[0045] In this embodiment, step S2 is as follows: The local model set up on the client is trained based on the client's local data, and the local gradient for the current round is calculated. ; Based on the local gradient of the current training epoch and the residual vector of the previous training epoch. The expression for the gradient to be compressed is obtained as follows:
[0046] In the formula, Indicates the client index. t Indicates the training rounds. Indicates the first The local gradient calculated by each client after the t-th round of local training. Indicates the first The residual vector of each client in the previous training round. This represents the gradient to be compressed in the current training round.
[0047] The client selects a local compression strategy (Top-K, Random-K, or Periodic-K) for the current round based on the MAB selector, applies the compression strategy to select the compressed model gradient, and records the residuals of the unsent portion for use in the next round. Residual accumulation is used to maintain convergence performance.
[0048] In this embodiment, the expression for generating the residual vector of the current training round based on the gradient to be compressed and the compressed gradient is as follows:
[0049] In the formula, This represents the residual vector for the current round. The gradients to be compressed in the current training epoch. This represents the compression gradient for the current training epoch.
[0050] In this embodiment, the process of performing knowledge distillation on the aggregation model based on a public dataset to obtain an optimized model is as follows: The federated server uses an aggregation model to perform forward inference on a public dataset to obtain the predicted output value of the aggregation model. Calculate the average value of the predicted output of the aggregation model to generate global soft labels; Using global soft labels as supervision signals, the aggregation model is trained by distillation. When the KL divergence between the predicted output value of the aggregation model and the global soft labels is minimized, the federated server obtains the updated parameters of the aggregation model to obtain the optimized model.
[0051] This invention introduces a public dataset on the server side. This method enables knowledge distillation across model prototypes without accessing client-side private data. The server first collects the compressed model prediction logits uploaded by each client on public data and averages them to generate a unified global soft label. Then, using this soft label as a supervision signal, the server performs distillation training on each model prototype separately, gradually aligning the output distributions of different model prototypes by minimizing the KL divergence. Finally, the server obtains the global model parameters corresponding to each model prototype and distributes them to the clients for the next round of training.
[0052] In this embodiment, the process of calculating the reward value for selecting the compression strategy based on the optimization model is as follows: The client calculates the local and global loss values for the current training round based on the optimized model. Calculate the change in overall loss based on the local and global loss values of the current training round and the previous training round; To evaluate each strategy Quality, building for the client The reward function is calculated based on the combined loss change and time cost, using the following expression:
[0053] In the formula, This represents the reward value for the current training round. This indicates the change in overall loss. Indicates the weighting coefficient; Indicates the client Local training time, Indicates the client Reduce operation time. Indicates the client The time it takes for parameters to be uploaded to the server during communication.
[0054] In this embodiment, the expression for calculating the change in overall loss is:
[0055] In the formula, This indicates the change in overall loss. Indicates weight, Indicates weight, This represents the global loss value from the previous training round. This represents the local loss value from the previous training round. This represents the global loss value in the current training round. This represents the local loss value for the current training round.
[0056] The client calculates the local and global loss functions, and their changes compared to the previous round. It also records the local training time, compression operation time, and the time it takes to upload parameters to the server. Reward values are generated based on weight allocation. The client updates the selection probability of each compression method according to the MAB algorithm and selects the compression strategy for the next round. The core of this invention is that each client autonomously and dynamically selects a compression strategy using a selector based on an ε-greedy policy algorithm.
[0057] Federated learning systems, particularly federated knowledge distillation systems with heterogeneous client models. This invention is applicable to privacy-sensitive environments and environments with limited computing resources and communication bandwidth, such as edge computing, the Internet of Things (IoT), and mobile communication systems.
[0058] This invention introduces the concept of heterogeneous compression, integrating multiple compression technologies (such as Top-K, Random-K, and Periodic-K) to achieve differentiated compression strategy selection for clients, thereby achieving a better balance between model accuracy and communication efficiency.
[0059] This invention proposes a resource-efficient federated knowledge distillation adaptive heterogeneous compression method, effectively addressing the shortcomings of existing technologies, such as the lack of adaptability of static strategies under a unified compression strategy. This framework introduces the concept of heterogeneous compression (integrating multiple compression techniques such as Top-K, Random-K, and Periodic-K) and an adaptive selection mechanism based on a multi-armed slot machine, enabling each client to autonomously and dynamically select the most suitable compression strategy based on reward feedback. Ultimately, this invention significantly reduces communication overhead and training latency (including computational and communication latency) while maintaining model accuracy and exhibiting superior adaptability in heterogeneous federated environments.
[0060] This invention significantly reduces resource overhead while maintaining competitive model accuracy. Its adaptive compression mechanism dynamically adjusts the compression strategy based on the client's real-time model performance and the availability of computational and communication resources, thus demonstrating superior adaptability and model accuracy in heterogeneous federated environments. Compared to traditional uniform compression schemes, this invention allows clients to autonomously choose the compression method most suitable for their needs, achieving personalized optimization and ultimately achieving a better trade-off between resource efficiency and model performance.
[0061] Example 2 The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the operations performed by the adaptive heterogeneous compression method of the federated knowledge distillation system.
[0062] Example 3 This invention also proposes an electronic device, such as... Figure 3 The schematic diagram shown includes a memory 101, a processor 102, and a computer program stored on the memory 101 and running on the processor 102. When the processor 102 executes the computer program, it implements the steps of the adaptive heterogeneous compression method for the federated knowledge distillation system proposed in this embodiment.
[0063] Specifically, in this embodiment, the processor 102 may include a central processing unit (CPU) or a specific integrated circuit, or one or more integrated circuits configured to implement this embodiment. The memory 101 may include a mass storage device for data or instructions. It may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 101 may include removable or non-removable (or fixed) media. Where appropriate, the memory 101 may be internal or external to the integrated gateway disaster recovery device.
[0064] Memory 101 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform steps of the adaptive heterogeneous compression method for implementing the federated knowledge distillation system proposed in this embodiment.
[0065] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. An adaptive heterogeneous compression method for a federated knowledge distillation system, the federated knowledge distillation system comprising a plurality of clients and a federated server, characterized in that, The adaptive heterogeneous compression method for federated knowledge distillation systems includes the following steps: S1: Set and initialize the client's local compression policy candidate set and policy selector; S2: On each client, train the local model set on that client based on the client's local data to obtain the local gradient of the current training round; based on the local gradient of the current training round and the residual vector of the previous training round, obtain the gradient to be compressed. S3: Each client selects a compression strategy from the local compression strategy candidate set based on the strategy selector, and uses the strategy to compress the gradient to be compressed to obtain the compressed gradient parameters; based on the gradient to be compressed and the compressed gradient, the residual vector of the current training round is generated. S4: Upload the compressed gradient parameters to the federated server. The federated server aggregates the gradient parameters uploaded by each client to obtain the aggregated model. Based on the public dataset, knowledge distillation is performed on the aggregated model to obtain the optimized model. S5: Send the optimized model to each client; calculate the reward value for selecting the compression strategy based on the optimized model; update the strategy selector in step S3 based on the reward value; S6: Use the optimized model as the local model and repeat steps S2-S6 until the preset maximum number of training rounds is reached.
2. The adaptive heterogeneous compression method for the federated knowledge distillation system according to claim 1, characterized in that, The candidate set of local compression strategies includes: Top-K gradient sparsity compression strategy, Random-K stochastic gradient compression strategy, and Periodic-K periodic gradient compression strategy; the expression for the candidate set of local compression strategies is: In the formula, denoted as the local compression policy candidate set, Top-K represents the Top-K gradient sparsity compression policy, Random-K represents the Random-K stochastic gradient compression policy, and Periodic-K represents the Periodic-K periodic gradient compression policy.
3. The adaptive heterogeneous compression method for the federated knowledge distillation system according to claim 1, characterized in that, The strategy selector uses an ε-greedy strategy for decision-making, and the process is as follows: Step 1: Initialize utility estimates for each compression strategy in the local compression strategy candidate set; Step 2: In the current training round, select the compression strategy with the highest estimated utility value with probability 1-ε; A compression strategy is randomly selected from the local compression strategy candidate set with probability ε; where ε represents the preset exploration rate parameter. Step 3: After completing this round of training, calculate the reward value for selecting the compression strategy; Step 4: Update the utility estimate from Step 1 based on the reward value, using the following expression: In the formula, This represents the current utility estimate. This represents the updated utility estimate. n Indicates the client, Indicates the compression strategy. Indicates the learning rate. Represents the reward function; Step 5: In the next training round, return to Step 2 and select a compression strategy based on the updated utility estimate.
4. The adaptive heterogeneous compression method for the federated knowledge distillation system according to claim 1, characterized in that, The process of step S2 is as follows: The local model set up on the client is trained based on the client's local data, and the local gradient for the current round is calculated. ; Based on the local gradient of the current training epoch and the residual vector of the previous training epoch. The expression for the gradient to be compressed is obtained as follows: In the formula, Indicates the client index. t Indicates the training rounds. Indicates the first The local gradient calculated by each client after the t-th round of local training. Indicates the first The residual vector of each client in the previous training round. This represents the gradient to be compressed in the current training round.
5. The adaptive heterogeneous compression method for the federated knowledge distillation system according to claim 1, characterized in that, Based on the gradient to be compressed and the compressed gradient, the expression for generating the residual vector of the current training round is: In the formula, This represents the residual vector for the current round. The gradients to be compressed in the current training epoch. This represents the compression gradient for the current training epoch.
6. The adaptive heterogeneous compression method for the federated knowledge distillation system according to claim 1, characterized in that, The process of performing knowledge distillation on the aggregation model based on public datasets to obtain the optimized model is as follows: The federated server uses an aggregation model to perform forward inference on a public dataset to obtain the predicted output value of the aggregation model. Calculate the average value of the predicted output of the aggregation model to generate global soft labels; Using global soft labels as supervision signals, the aggregation model is trained by distillation. When the KL divergence between the predicted output value of the aggregation model and the global soft labels is minimized, the federated server obtains the updated parameters of the aggregation model to obtain the optimized model.
7. The adaptive heterogeneous compression method for the federated knowledge distillation system according to claim 1, characterized in that, Based on the optimization model, the process of calculating the reward value for selecting the compression strategy is as follows: The client calculates the local and global loss values for the current training round based on the optimized model. Calculate the change in overall loss based on the local and global loss values of the current training round and the previous training round; Based on the overall change in loss and time cost, the reward value for the current training round is calculated using the following expression: In the formula, This represents the reward value for the current training round. This indicates the change in overall loss. Indicates the weighting coefficient; Indicates the client Local training time, Indicates the client Reduce operation time. Indicates the client The time it takes for parameters to be uploaded to the server during communication.
8. The adaptive heterogeneous compression method for the federated knowledge distillation system according to claim 7, characterized in that, The expression for calculating the change in overall loss is as follows: In the formula, This indicates the change in overall loss. Indicates weight, Indicates weight, This represents the global loss value from the previous training round. This represents the local loss value from the previous training round. This represents the global loss value in the current training round. This represents the local loss value for the current training round.
9. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the adaptive heterogeneous compression program of the federated knowledge distillation system is executed by the processor, it implements the steps of the adaptive heterogeneous compression method of the federated knowledge distillation system as described in any one of claims 1-8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer stored program is executed by the processor, it performs the operations performed by the adaptive heterogeneous compression method of the federated knowledge distillation system according to any one of claims 1 to 8.