Federal compression learning method and system based on knowledge distillation

Abstract

The invention discloses a federal compression learning method and system based on knowledge distillation, and the method comprises the steps: enabling each terminal device to adaptively adjust the compression ratio of the terminal device according to a real-time network, and carrying out the compression sampling through the adjusted compression ratio; then, taking a compressed sampling value as a knowledge distillation medium between edge equipment and a cloud server of different levels, and carrying out cooperative training on heterogeneous models deployed by nodes of different levels in combination with a federated learning method and knowledge distillation; the knowledge distillation is introduced in the federated learning process, so that models with different capacities and architectures can be deployed at different hierarchy nodes, and the compressed sampling value with the internal privacy protection characteristic is used for replacing original data or proxy data as a distillation medium, so that the privacy risk caused by the knowledge distillation is eliminated from the data source. And the communication efficiency, privacy protection and computing power utilization optimization of the end-side cloud are realized.

Description

Technical Field

[0001] This invention relates to the field of transfer learning technology based on biological models, and specifically to a federated compressed learning method and system based on knowledge distillation. Background Technology

[0002] Federated compression learning suffers from a fixed compression ratio limitation and a weakest node constraint. During training, a global compression ratio must be preset and fixed, and all data must be compressed and sampled at this same ratio, making adjustments impossible based on dynamic changes in network bandwidth or device status of terminal devices. Furthermore, to run on all participating computing nodes, a uniform, smaller model architecture must be used. This prevents powerful cloud servers and some edge devices from deploying more complex and accurate models, resulting in wasted computing resources and a locked performance ceiling.

[0003] Federated compressed learning based on knowledge distillation faces privacy risks associated with data sharing during heterogeneous model building. To perform knowledge distillation across different models, each party needs to use a batch of identical distilled shared data to generate comparable raw scores. Whether directly uploading raw data or uploading processed feature maps, generated data, or other proxy data, there is a potential risk of exposing the original data information. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a federated compressed learning method and system based on knowledge distillation, which can optimize communication efficiency, privacy protection, and computing power utilization between edge and cloud environments. The specific technical solution is as follows: In a first aspect, a federated compressed learning method based on knowledge distillation is provided, wherein in a first implementable manner of the first aspect, the method includes: The compression ratio of each terminal device is adaptively adjusted according to the real-time network status of each terminal device. The terminal device compresses and samples the original data using the adjusted compression ratio to obtain the corresponding compressed sample value. The compressed sampled values ​​are used as distillation shared data, and knowledge distillation is introduced into the federated learning method to collaboratively train heterogeneous models deployed at different levels of nodes.

[0005] In conjunction with the first possible implementation of the first aspect, in the second possible implementation of the first aspect, adjusting the compression ratio of the terminal device includes: Based on the network state, a portion of elements are extracted from the pre-configured sampling base matrix to form a sampling matrix, and the original data is sampled using the sampling matrix.

[0006] In conjunction with the first feasible approach of the first aspect, in the third feasible approach of the first aspect, training is performed on heterogeneous models deployed at different levels, including: mapping compressed sampled values ​​obtained by compressed sampling to a unified dimension.

[0007] In conjunction with the third possible implementation of the first aspect, the fourth possible implementation of the first aspect maps the compressed sampled values ​​to a unified dimension, including: Based on the dimension of the compressed sampled values, a portion of the elements are extracted from the transformation base matrix to form the transformation matrix. The compressed sampled values ​​are then mapped to a unified dimension using the transformation matrix.

[0008] In conjunction with the first feasible approach of the first aspect, the fifth feasible approach of the first aspect employs a knowledge distillation federated learning method for training, including: The compressed sampled value is losslessly encrypted, and the encrypted compressed sampled value is passed to the next level node along the hierarchy.

[0009] Secondly, a federated compressed learning system based on knowledge distillation is provided, wherein in a first implementable manner of the second aspect, it includes: The sampling adjustment module is configured to adaptively adjust the compression ratio of each terminal device according to the real-time network status of each terminal device. The terminal device compresses and samples the original data using the adjusted compression ratio to obtain the corresponding compressed sampling value. The collaborative training module is configured to use the compressed sampled values ​​as distillation shared data, and to introduce knowledge distillation in the federated learning method to collaboratively train heterogeneous models deployed at different levels of nodes.

[0010] In conjunction with the first possible implementation of the second aspect, in the second possible implementation of the second aspect, the sampling adjustment module includes: An adaptive sampling unit is configured to extract a portion of elements from a pre-configured sampling base matrix based on the network state to form a sampling matrix, and then sample the original data using the sampling matrix.

[0011] In conjunction with the first implementable method of the second aspect, the third implementable method of the second aspect also includes: The linear transformation module is configured to map the compressed sampled values ​​obtained from compressed sampling to a uniform dimension.

[0012] In conjunction with the second possible implementation of the second aspect, in the fourth possible implementation of the second aspect, the linear transformation module includes: Adjust the sub-unit to extract a portion of elements from the transformation basis matrix based on the dimension of the compressed sampled values ​​to form the transformation matrix; The mapping subunit is configured to map the compressed sampled values ​​to a uniform dimension using a transformation matrix.

[0013] In conjunction with the first possible implementation of the second aspect, in the fifth possible implementation of the second aspect, the terminal device includes: The encryption unit is configured to perform lossless encryption on the compressed sampled value and transmit the encrypted compressed sampled value to the next higher level node along the hierarchy.

[0014] Beneficial Effects: Employing the knowledge distillation-based federated compressed learning method and system of this invention, terminal devices can adaptively adjust their configured compression ratio based on the real-time network status with edge devices, and then compress and sample the original data using the adjusted compression ratio, thereby optimizing communication efficiency between the end, edge, and cloud. Furthermore, knowledge distillation is introduced into the federated learning method to collaboratively train heterogeneous models deployed at different levels of nodes. By introducing knowledge distillation into federated compressed learning, models with different capacities and architectures can be deployed at different levels of nodes, breaking the model constraints in federated compressed learning. Moreover, by using compressed sample values ​​with inherent privacy protection characteristics as distillation-shared data to replace the original data with leakage risks, the privacy risks associated with knowledge distillation are eliminated at the data source. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. In all the drawings, the elements or parts are not necessarily drawn to scale.

[0016] Figure 1 A flowchart illustrating a federated compressed learning method based on knowledge distillation, provided as an embodiment of the present invention; Figure 2 A system block diagram of a federated compressed learning system based on knowledge distillation provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a knowledge distillation process provided in an embodiment of the present invention. Detailed Implementation

[0017] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0018] like Figure 1 The flowchart shown illustrates a federated compressed learning method based on knowledge distillation, which includes: Step 1: Adaptively adjust the compression ratio of each terminal device according to the real-time network status of each terminal device. The terminal device compresses and samples the original data using the adjusted compression ratio to obtain the corresponding compressed sample value. Step 2: Use the compressed sampled values ​​as distillation shared data, and introduce knowledge distillation in the federated learning method to collaboratively train heterogeneous models deployed at different levels of nodes.

[0019] Specifically, firstly, the terminal device can adaptively adjust the compression ratio based on its real-time network conditions. It then compresses and samples the collected raw data using the adjusted compression ratio, uploading the compressed sampled values ​​to the edge device. The edge device then processes and analyzes these values ​​using a deployed model. For example, when the network is congested, the terminal device can increase the compression ratio, thereby reducing the communication overhead between the terminal device and the edge device and optimizing communication efficiency.

[0020] Then, the compressed sampling values ​​obtained by the terminal device can be used as a knowledge distillation medium. Combined with knowledge distillation and federated learning methods, the models deployed on different levels of nodes in the edge-cloud can be trained to achieve efficient and secure collaborative training and knowledge transfer.

[0021] Because knowledge distillation is introduced during federated compressed learning, models of varying capacities and architectures can be deployed on different edge devices and cloud servers. This breaks the constraint that federated compressed learning must adopt a uniform, small model architecture, thus fully leveraging the computing resources and performance of cloud servers and edge devices. Furthermore, by using compressed sample values ​​with inherent privacy protection characteristics, the original data or proxy data, which poses a risk of leakage, is completely replaced as the distillation medium, eliminating the privacy risks associated with knowledge distillation at the data source. This achieves optimized communication efficiency, privacy protection, and computing power utilization across edge-cloud environments.

[0022] In this embodiment, optionally, adjusting the compression ratio of the terminal device includes: Based on the network state, a portion of elements are extracted from the pre-configured sampling base matrix to form a sampling matrix, and the original data is sampled using the sampling matrix.

[0023] Specifically, the terminal device can store sampling matrices with different compression ratios, each corresponding to a different network state. The terminal device can adaptively select the appropriate sampling matrix for compression sampling based on the current network state.

[0024] However, storing sampling matrices for different network states requires storage space on the terminal device. To avoid maintaining independent storage overhead for different compression ratios, a single sampling base matrix covering all network state scenarios can be stored on the terminal device. The terminal device can extract a subset of elements from the configured sampling base matrix based on its real-time network state to form the current sampling matrix. The collected raw data is then sampled using this sampling matrix, as shown in the following formula: ; in, To compress the sampled values, For the sampling matrix, This is the original data.

[0025] When extracting a sampling matrix from the configured sampling base matrix, you can first determine the compression ratio adapted to the terminal device based on the current network status of the terminal device, and then determine the sampling dimension corresponding to the sampling matrix based on the compression ratio. Finally, based on the sampling dimension Extract the first part from the sampling fundamental matrix. The elements of the row are used as the sampling matrix.

[0026] Thus, by adaptively adjusting the sampling dimension This enables adaptive sampling under different network conditions. Moreover, adaptive sampling can generate variable-dimensional compressed sample values, which is incompatible with the fixed-dimensional prior required by conventional compressed sensing reconstruction algorithms, further increasing the difficulty for attackers to reconstruct the data.

[0027] In this embodiment, the sampling basis matrix can be generated using a Gaussian random method. Specifically, the `generate_gaussian_measurement_matrix(M, N, seed=42)` function from the NumPy library in Python is used to set the seed parameter and generate an m-row, n-column sampling basis matrix.

[0028] Using Gaussian random methods to generate the sampling basis matrix can guarantee the reconstructability of the data with a high probability and effectively reduce the correlation between measurement data. In addition to Gaussian random methods, elements of some rows can be randomly selected from the Fourier matrix or an existing Bernoulli random matrix can be used as the sampling basis matrix.

[0029] In this embodiment, optionally, training heterogeneous models deployed at different levels can include mapping compressed sampling values ​​obtained from compressed sampling to a unified dimension.

[0030] Specifically, due to differences in network conditions among different terminal devices, the dimensions of the compressed sampled values ​​obtained by these devices also differ. Furthermore, the dimensions of the compressed sampled values ​​obtained by terminal devices vary under different network conditions. To maintain dimensionality consistency during subsequent training, edge devices can unify the dimensions of the compressed sampled values ​​obtained from all terminal devices. This way, only one model needs to be deployed at the edge device to train and analyze compressed sampled data with multiple different dimensions, avoiding the need to maintain independent storage overhead for different compression ratios.

[0031] In this embodiment, optionally, mapping the compressed sampled values ​​to a unified dimension includes: Based on the dimension of the compressed sampled values, a portion of the elements are extracted from the transformation base matrix to form the transformation matrix. The compressed sampled values ​​are then mapped to a unified dimension using the transformation matrix.

[0032] Specifically, the transformation matrices that edge devices need to be configured for different network state scenarios can be obtained through training. However, this not only requires a lot of time and resources for training, but also occupies a significant amount of storage space on the edge devices to store the transformation matrices corresponding to different network state scenarios.

[0033] Therefore, a transformation basis matrix covering all network state scenarios can be obtained through training. Edge devices can then adjust the sampling dimensions of the compressed sample values. Extract the first part from the fundamental matrix of the transformation. The elements of the column are used as a transformation matrix, and the compressed sampled values ​​obtained by sampling are mapped to a unified dimension through the transformation matrix. The specific calculation formula is as follows: .

[0034] in, These are the compressed acquisition values ​​after dimensional transformation. Let be the transformation matrix.

[0035] In this embodiment, optionally, a knowledge distillation federated learning method is used for training, including: The compressed sampled value is losslessly encrypted, and the encrypted compressed sampled value is passed to the next level node along the hierarchy.

[0036] Specifically, terminal devices, edge devices, and cloud servers can form a tree-like topology. Student models are deployed on edge devices, and teacher models are deployed on cloud servers. Terminal devices act as leaf nodes in the tree topology, edge devices as child nodes, and cloud servers as parent nodes. Leaf nodes perform lossless encryption on the compressed sampled values ​​and upload the encrypted information to their child nodes. Child nodes then upload all the encrypted information they obtain to the parent node. The parent node decrypts the encrypted information, thus aggregating all the compressed sampled values ​​and preventing their leakage.

[0037] Edge devices can deploy student models, while cloud servers can deploy teacher models. Compressed sample values ​​obtained from terminal devices can be used as an intermediary in the knowledge distillation federated learning method, facilitating knowledge distillation between student and teacher models. Without knowing the sampling matrix assigned to the terminal device, the original data collected by the terminal device cannot be reconstructed from the compressed sample values. Therefore, the privacy risks associated with knowledge distillation can be eliminated.

[0038] The knowledge distillation process between the student model and the teacher model is as follows: Figure 3As shown, the cloud server uses the decrypted compressed sampled values ​​as the logits of the common dataset passed by the teacher model to the student model for knowledge distillation. This process continues until all knowledge distillation training between the student and teacher models is complete. Then, the teacher and student models are swapped, with the original student model becoming the new teacher model and the original teacher model becoming the new student model, and knowledge distillation continues. By introducing knowledge distillation into federated learning, different levels of nodes, such as edge devices and cloud servers, can deploy models of different capacities and architectures, such as CNN, ResNet10, and ResNet18.

[0039] The loss function used in knowledge distillation is: in, For temperature parameters, The softmax activation function is used. The knowledge distillation loss coefficient, Represents the crossover loss function. This represents the predicted output of the student model. For real labels, For the parameters of the student model, This represents the Kullback-Leibler divergence loss, used to measure the difference between two probability distributions. This represents the predicted output of the teacher model. This represents the parameters of the teacher model.

[0040] The specific formula for calculating the cross-loss function is as follows: ; in, The probability distribution of the student model's predicted output. The total number of categories, which is also the output dimension. As training samples, The true labels in the training samples. For student models with respect to training samples The predicted output.

[0041] The specific formula for calculating the Kullback-Leibler divergence loss is as follows: ; in, The predicted output of the student model, i.e. , The predicted output of the teacher model, i.e. .

[0042] Experiments show that, in scenarios with uniformly distributed compression ratios of 0.25, 0.1875, 0.125, and 0.0625, the learning method of this invention reduces the edge communication overhead during the initialization phase by 37.50% compared to the federated compression learning method with a fixed high compression ratio, and by 94.79% compared to the aggregated federated learning method that requires uploading a large number of bridging samples.

[0043] Furthermore, the learning method of this invention ensures an accuracy range of 55%-65% for edge models and 66%-72% for cloud models. The overall model training process is also highly efficient and stable. Training efficiency is significantly improved; on ImageNet10, only 15 epochs are needed to achieve the accuracy required for FCL (Fixed Closed Array) in 30 epochs, doubling the convergence speed.

[0044] like Figure 2 The diagram shown illustrates a system block diagram of a federated compressed learning system based on knowledge distillation. This learning system includes: The sampling adjustment module is configured to adaptively adjust the compression ratio of each terminal device according to the real-time network status of each terminal device. The terminal device compresses and samples the original data using the adjusted compression ratio to obtain the corresponding compressed sampling value. The collaborative training module is configured to use the compressed sampled values ​​as distillation shared data, and to introduce knowledge distillation in the federated learning method to collaboratively train heterogeneous models deployed at different levels of nodes.

[0045] Specifically, the learning system includes a sampling adjustment module and a collaborative training module. The sampling adjustment module, deployed on the terminal device, adaptively adjusts the configured compression ratio based on the real-time network conditions, and then compresses and samples the collected raw data using the adjusted compression ratio. For example, when the network is congested, the sampling adjustment module can increase the compression ratio of the terminal device, thereby reducing the communication overhead between the terminal device and edge devices and optimizing communication efficiency.

[0046] The collaborative training module can use the compressed sampling values ​​obtained from the terminal device as a knowledge distillation medium. It employs a knowledge distillation federated learning method to recursively train the models deployed on edge devices and cloud servers, forming a hierarchical knowledge transfer mechanism, thereby achieving efficient and secure collaborative training and knowledge transfer.

[0047] In this embodiment, optionally, the sampling adjustment module includes: An adaptive sampling unit is configured to extract a portion of elements from a pre-configured sampling base matrix based on the network state to form a sampling matrix, and then sample the original data using the sampling matrix.

[0048] Specifically, the sampling adjustment module includes an adaptive sampling unit. This adaptive sampling unit can store only one sampling base matrix that covers all network state scenarios, and extract a portion of the elements from the configured sampling base matrix as the current sampling matrix based on its own real-time network state. The terminal device samples the collected raw data through the sampling matrix.

[0049] In this embodiment, optionally, it also includes: The linear transformation module is configured to map the compressed sampled values ​​obtained from compressed sampling to a uniform dimension.

[0050] Specifically, the learning system also includes a linear transformation module deployed at the edge device. This module maps the compressed sampled values ​​obtained from the terminal device to a unified dimension, ensuring dimensionality consistency during subsequent training. Thus, only one model needs to be deployed at the edge device to train and analyze compressed sampled data with multiple dimensions, avoiding the need for independent storage overhead for different compression ratios.

[0051] In this embodiment, optionally, the linear transformation module includes: Adjust the sub-unit to extract a portion of elements from the transformation basis matrix based on the dimension of the compressed sampled values ​​to form the transformation matrix; The mapping subunit is configured to map the compressed sampled values ​​to a uniform dimension using a transformation matrix.

[0052] Specifically, the linear transformation module includes an adjustment subunit and a mapping subunit. The adjustment subunit can be adjusted according to the sampling dimension. Extracting the first part from the transformation fundamental matrix that covers all network state scenarios obtained through learning and training. The elements of the column serve as the transformation matrix. The mapping sub-unit can use the transformation matrix to map the compressed sampled values ​​obtained from sampling to a uniform dimension.

[0053] In this embodiment, optionally, the terminal device includes: The encryption unit is configured to perform lossless encryption on the compressed sampled value and transmit the encrypted compressed sampled value to the next higher level node along the hierarchy.

[0054] Specifically, the terminal device is equipped with an encryption unit that can losslessly encrypt the compressed sampled value obtained through sampling and upload the encrypted information to the edge device. The edge device can then losslessly decrypt the encrypted information to obtain the corresponding compressed sampled value. This avoids the leakage of compressed sampled values ​​and further eliminates the privacy risks associated with knowledge distillation.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A federated compressed learning method based on knowledge distillation, characterized in that, include: The compression ratio of each terminal device is adaptively adjusted according to the real-time network status of each terminal device. The terminal device compresses and samples the original data using the adjusted compression ratio to obtain the corresponding compressed sample value. The compressed sampled values ​​are used as distillation shared data, and knowledge distillation is introduced into the federated learning method to collaboratively train heterogeneous models deployed at different levels of nodes.

2. The federated compressed learning method based on knowledge distillation according to claim 1, characterized in that, Adjusting the compression ratio of the terminal device includes: Based on the network state, a portion of elements are extracted from the pre-configured sampling base matrix to form a sampling matrix, and the original data is sampled using the sampling matrix.

3. The federated compressed learning method based on knowledge distillation according to claim 1, characterized in that, Collaborative training of heterogeneous models deployed at different levels includes: Map the compressed sampled values ​​obtained by compressed sampling to a unified dimension.

4. The federated compressed learning method based on knowledge distillation according to claim 3, characterized in that, Mapping the compressed sampled values ​​to a unified dimension includes: Based on the dimension of the compressed sampled values, a portion of the elements are extracted from the transformation base matrix to form the transformation matrix. The compressed sampled values ​​are then mapped to a unified dimension using the transformation matrix.

5. The federated compressed learning method based on knowledge distillation according to claim 1, characterized in that, Training is performed using a knowledge distillation federated learning approach, including: The compressed sampled value is losslessly encrypted, and the encrypted compressed sampled value is passed to the next level node along the hierarchy.

6. A federated compressed learning system based on knowledge distillation, characterized in that, include: The sampling adjustment module is configured to adaptively adjust the compression ratio of each terminal device according to the real-time network status of each terminal device. The terminal device compresses and samples the original data using the adjusted compression ratio to obtain the corresponding compressed sampling value. The collaborative training module is configured to use the compressed sampled values ​​as distillation shared data, and to introduce knowledge distillation in the federated learning method to collaboratively train heterogeneous models deployed at different levels of nodes.

7. The federated compressed learning system based on knowledge distillation according to claim 6, characterized in that, The sampling adjustment module includes: An adaptive sampling unit is configured to extract a portion of elements from a pre-configured sampling base matrix based on the network state to form a sampling matrix, and then sample the original data using the sampling matrix.

8. The federated compressed learning system based on knowledge distillation according to claim 6, characterized in that, Also includes: The linear transformation module is configured to map the compressed sampled values ​​obtained from compressed sampling to a uniform dimension.

9. The federated compressed learning system based on knowledge distillation according to claim 8, characterized in that, The linear transformation module includes: Adjust the sub-unit to extract a portion of elements from the transformation basis matrix based on the dimension of the compressed sampled values ​​to form the transformation matrix; The mapping subunit is configured to map the compressed sampled values ​​to a uniform dimension using a transformation matrix.

10. The federated compressed learning system based on knowledge distillation according to claim 6, characterized in that, The terminal device includes: The encryption unit is configured to perform lossless encryption on the compressed sampled value and transmit the encrypted compressed sampled value to the next higher level node along the hierarchy.

Images