Lightweight knowledge distillation method, equipment, medium and product based on double knowledge distillation strategy
By employing a teacher-student collaborative learning mechanism based on a dual knowledge distillation strategy, a lightweight Transformer network is optimized, addressing the issue of high resource consumption on edge devices inherent in traditional Transformer algorithms. This enables efficient, lightweight deployment and accurate identification.
Patent Information
- Application Number
- CN202511055818.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-21
AI Technical Summary
Traditional Transformer algorithms have high computational complexity and resource consumption, making them difficult to deploy directly on edge devices. Existing lightweight methods lack model accuracy and efficiency in resource-constrained scenarios.
We adopt a dual knowledge distillation strategy, which builds a teacher-student model by alternating freezing and fine-tuning strategies. By combining hidden distillation and Logits distillation, we form a collaborative learning mechanism to optimize the student model and jointly train the teacher model, thereby reducing the number of parameters and resource requirements.
It significantly improves the stability and recognition accuracy of the model, reduces the number of model parameters and resource requirements, and is suitable for lightweight deployment in resource-constrained environments.
Smart Images
Figure CN120996103A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of communication signal analysis and identification, and more specifically, to a lightweight knowledge distillation method, apparatus, medium, and product based on a dual knowledge distillation strategy. Background Technology
[0002] In communication scenarios, the Transformer, currently the most representative self-attention architecture, is commonly used in the field of intelligent recognition. The Transformer can effectively model the long-range dependencies of modulated signals in time series, possessing excellent feature representation capabilities and discriminative performance. This network structure consists of multiple multi-head attention modules, feedforward network layers, and positional encoding, and has demonstrated excellent classification accuracy in various communication scenarios. However, the Transformer has a computational complexity of O(L²), consuming significant resources on edge devices and making direct deployment difficult. Therefore, knowledge distillation is typically considered, using the Transformer network as the teacher model and a lightweight model as the student model.
[0003] To improve the deployment friendliness of the model, the three mainstream lightweight Transformer architectures currently include Performer, Linformer, and Reformer. Performer approximates the attention function through random feature mapping, achieving linear complexity; Linformer uses low-rank decomposition to compress the attention matrix; Reformer uses Locality Sensitive Hash (LSH) mechanism to replace the original attention calculation, while introducing invertible residual layers and block feedforward layers to save memory.
[0004] Therefore, it is necessary to increase research efforts and, based on the concept of knowledge distillation, explore new dimensions to find ways to restore the expressive power of the Transformer as much as possible while effectively compressing model parameters and memory requirements. Developing new methods and steps will play an important role in the development of resource-constrained scenarios and even the entire field of recognition. Summary of the Invention
[0005] To address the challenges of direct deployment of traditional Transformer algorithms in signal type or target recognition due to their high computational complexity, significant resource consumption on edge devices, and difficulty in direct application, this invention provides a lightweight knowledge distillation method, device, medium, and product based on a dual knowledge distillation strategy. This approach enables signal type or target recognition in resource-constrained scenarios through lightweight intelligent training and learning.
[0006] In a first aspect, the present invention provides a lightweight knowledge distillation method based on a dual knowledge distillation strategy, comprising: Build teacher and student models; The teacher and student models are trained using a dataset of input communication signals. During the training process, a freeze-fine-tuning strategy is adopted to form a teacher-student collaborative learning mechanism. At the same time, a dual knowledge distillation strategy of hidden distillation and Logits distillation is introduced.
[0007] In a preferred embodiment, a traditional Transformer network is used as the teacher model, and a lightweight Transformer network is used as the student model.
[0008] In a preferred embodiment, the lightweight Transformer network employs a Reformer, Performer, or Linformer architecture.
[0009] In a preferred embodiment, training the teacher model and the student model includes: During the initial training phase, the teacher model's parameters are frozen, and only the student model's structure is optimized. Hidden distillation is performed in the later stages of training. By gradually introducing a hidden layer alignment mechanism, some attention layers of the teacher model and the student model are jointly trained. Logits distillation is performed on the predicted outputs of the teacher and student models.
[0010] In a preferred embodiment, the loss function for hidden distillation employs a joint optimization strategy, which integrates classification cross-entropy loss, KL divergence distillation loss, and feature alignment loss. The weighting coefficients of each loss are dynamically balanced based on experience to ensure stable convergence during training.
[0011] In a preferred embodiment, the loss function for performing Logits distillation based on the predicted output is the KL divergence loss function.
[0012] In a preferred embodiment, data augmentation techniques and a hierarchical learning rate mechanism are used during training to improve the model's adaptability and generalization ability.
[0013] In a second aspect, the present invention provides an electronic device, comprising: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the instructions stored in the memory to perform the method described above.
[0014] Thirdly, the present invention provides a computer-readable storage medium for storing instructions that, when executed, enable the above-described method to be implemented.
[0015] Fourthly, the present invention provides a computer program product that, when invoked by a computer, causes the computer to execute the above-described method.
[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. Effectively improves model stability and accuracy. This invention employs a teacher-student collaborative mechanism, which not only optimizes student models but also jointly trains teacher models, enabling overall collaborative modeling. The dual knowledge distillation strategy, utilizing both hidden distillation and Logits distillation based on predicted output, achieves alignment and joint feature completion effects, playing a significant role in the later stages of training. Compared to single-channel knowledge transfer, it offers advantages in mutual verification and collaborative comparative optimization, effectively improving model stability and accuracy. It addresses the problems of high computational complexity and excessive resource consumption on edge devices, making direct deployment difficult in signal type or target recognition processes, particularly with traditional Transformer algorithms.
[0017] 2. Effectively improves recognition accuracy. Compared to traditional single-channel knowledge distillation methods, this invention achieves synergistic optimization and comparative accuracy improvement. The method was thoroughly tested on the RML2016.10A dataset. Compared to the undistilled original Transformer model, the distilled model shows an average accuracy improvement of 3.7% under different signal-to-noise ratio settings. At 20dB, the accuracy is only 1.4% lower than the full Transformer model.
[0018] 3. Effectively reduces the number of model parameters and resource requirements, making it suitable for lightweight deployment and widespread adoption. Compared to the traditional Transformer method, this invention requires fewer resources after the student model learns and is more suitable for lightweight deployment in resource-constrained environments. The method of this invention reduces the number of student model parameters by more than 60% and inference time by approximately 40%. Attached Figure Description
[0019] Figure 1 A flowchart of a lightweight knowledge distillation method based on a dual knowledge distillation strategy is provided for an embodiment of the present invention.
[0020] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0023] Example like Figure 1 As shown, this embodiment of the invention provides a lightweight knowledge distillation method based on a dual knowledge distillation strategy, comprising the following steps: S100, Build the teacher model and student model; in this embodiment, a traditional Transformer network is used as the teacher model, and a lightweight Transformer network is used as the student model. The lightweight Transformer network adopts a Reformer structure, a Performer structure, or a Linformer structure, and other lightweight Transformer structures can also be used. Considering the advantages in accuracy, memory overhead, and convergence speed, the Reformer structure is preferred as the basic structure for lightweight deployment of the student model.
[0024] S200: Input a communication signal dataset to train the teacher and student models. Taking modulation recognition as an example, a communication signal dataset is constructed using communication signals of different modulation types (such as 11 types including 8PSK, QAM16, and GFSK from the RML2016a public dataset). The training data in the communication signal dataset adopts the IQ dual-channel data format, that is, each sample contains two channels, I (in-phase) and Q (quadrature), the signal sequence length is 128 sampling points, and the data dimension is [batch_size, 2, 128]. Before being fed into the model, the training data undergoes standardized preprocessing, including amplitude normalization to ensure numerical stability, and the addition of Gaussian white noise and time shift to improve the model's generalization ability. After inputting the model, taking the teacher model as an example, the communication signal in IQ dual-channel data format is processed as a complete dual-channel data stream. The Transformer network simultaneously learns the joint feature representation of the I and Q channels, with dimensions [batch_size, 2, 128]. The first channel is the I-channel signal (in-phase signal), and the second channel is the Q-channel signal (orthogonal signal). After processing by the Transformer network, a soft label output distribution based on the complete IQ signal features is obtained. During training, a freeze-fine-tuning alternating strategy is adopted to form a teacher-student collaborative learning mechanism, while simultaneously introducing a dual knowledge distillation strategy of hidden distillation and logits distillation. The specific training process includes: S201: During the initial training phase, the teacher model's parameters are frozen, and only the student model's structure is optimized. In S202, hidden distillation is performed in the later stages of training. By gradually introducing a hidden layer alignment mechanism, some attention layers of the teacher and student models are jointly trained. The alignment mechanism refers to the teacher and student models selecting multiple hidden representations during training. By constructing a loss function, the intermediate outputs of the student model are guided to align with the corresponding layer representations of the teacher model, thereby improving the overall collaborative modeling effect. The loss function for hidden distillation employs a joint optimization strategy, integrating classification cross-entropy loss, KL divergence distillation loss, and feature alignment loss. The weighting coefficients of each loss are dynamically balanced based on experience to ensure stable convergence during training. These losses are only preferred in this embodiment; in practical applications, they can be extended to other loss functions.
[0025] S203. For the predicted outputs of the teacher and student models, Logits distillation based on the predicted output is performed, which is the traditional Soft Label guidance mechanism. The loss function for Logits distillation based on the predicted output is the KL divergence loss function, which forces the student model output distribution to converge with the teacher model output, enabling it to learn the probability distribution features between categories, thereby forming a smoother and more generalized classification boundary.
[0026] During the training process described above, data augmentation techniques and a hierarchical learning rate mechanism are employed to improve the model's adaptability and generalization ability. In this embodiment, the data augmentation strategy includes signal-to-noise ratio perturbation and random perturbation of IQ features, thereby simulating channel changes in real-world communication scenarios; the hierarchical learning rate mechanism sets optimization rates according to the model structure levels, enabling robust extraction of lower-level features and more flexible adaptation of higher-level semantics.
[0027] As described above, this invention employs a teacher-student collaborative mechanism, which not only optimizes the student model but also jointly trains the teacher model, enabling overall collaborative modeling. The use of both hidden distillation and Logits distillation based on predicted output—a dual knowledge distillation approach—achieves alignment and joint feature completion effects, playing a significant role in the later stages of training. Compared to single-channel knowledge transfer, it offers advantages in mutual verification and collaborative comparative optimization, effectively improving the model's stability and accuracy.
[0028] Compared to traditional single-channel knowledge distillation methods, this invention achieves synergistic optimization and comparative accuracy improvement. The method was thoroughly tested on the RML2016.10A dataset. Compared to the undistilled original Transformer model, the distilled model shows an average accuracy improvement of 3.7% under different signal-to-noise ratio settings. At 20dB, the accuracy is only 1.4% lower than the full Transformer model.
[0029] Compared to the traditional Transformer method, this invention requires fewer resources after the student model is optimized through learning, making it more suitable for lightweight deployments in resource-constrained environments. The method of this invention reduces the number of student model parameters by more than 60% and inference time by approximately 40%.
[0030] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the lightweight knowledge distillation method based on a dual knowledge distillation strategy provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 2 As shown, the electronic device may include: At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 2 The example used is the connection between the processor and memory via a bus. The bus... Figure 2 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 2 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.
[0031] In this embodiment of the invention, the memory stores instructions that can be executed by at least one processor. By executing the instructions stored in the memory, at least one processor can execute a lightweight knowledge distillation method based on a dual knowledge distillation strategy as described above.
[0032] The processor is the control center of the device. It can connect to various parts of the control equipment through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the various functions and data processing of the device as a whole.
[0033] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.
[0034] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the lightweight knowledge distillation method based on a dual knowledge distillation strategy disclosed in the embodiments of this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.
[0035] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.
[0036] By designing and programming the processor, the code corresponding to the lightweight knowledge distillation method based on a dual knowledge distillation strategy described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the method described in the foregoing embodiments during runtime. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.
[0037] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a lightweight knowledge distillation method based on a dual knowledge distillation strategy as described above.
[0038] In some alternative embodiments, the present invention also provides a lightweight knowledge distillation method based on a dual knowledge distillation strategy, which can also be implemented as a program product including program code that, when the program product is run on a device, causes the control device to perform the steps in the lightweight knowledge distillation method based on a dual knowledge distillation strategy according to various exemplary embodiments of the present invention as described above.
[0039] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0040] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0041] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0042] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0043] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0044] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0045] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A lightweight knowledge distillation method based on a dual knowledge distillation strategy, characterized in that, include: Build teacher and student models; The teacher and student models are trained using a dataset of input communication signals. During the training process, a freeze-fine-tuning strategy is adopted to form a teacher-student collaborative learning mechanism. At the same time, a dual knowledge distillation strategy of hidden distillation and Logits distillation is introduced.
2. The lightweight knowledge distillation method based on a dual knowledge distillation strategy according to claim 1, characterized in that, A traditional Transformer network is used as the teacher model, and a lightweight Transformer network is used as the student model.
3. The lightweight knowledge distillation method based on a dual knowledge distillation strategy according to claim 2, characterized in that, The lightweight Transformer network adopts a Reformer, Performer, or Linformer structure.
4. The lightweight knowledge distillation method based on a dual knowledge distillation strategy according to claim 1, characterized in that, The training of the teacher model and student model includes: During the initial training phase, the teacher model's parameters are frozen, and only the student model's structure is optimized. Hidden distillation is performed in the later stages of training. By gradually introducing a hidden layer alignment mechanism, some attention layers of the teacher model and the student model are jointly trained. Logits distillation is performed on the predicted outputs of the teacher and student models.
5. The lightweight knowledge distillation method based on a dual knowledge distillation strategy according to claim 4, characterized in that, The loss function for hidden distillation adopts a joint optimization strategy, which integrates classification cross-entropy loss, KL divergence distillation loss and feature alignment loss. The weighting coefficients of each loss are dynamically balanced based on experience to ensure stable convergence during training.
6. The lightweight knowledge distillation method based on a dual knowledge distillation strategy according to claim 4, characterized in that, The loss function used for Logits distillation based on the predicted output is the KL divergence loss function.
7. The lightweight knowledge distillation method based on a dual knowledge distillation strategy according to claim 1, characterized in that, During training, data augmentation techniques and hierarchical learning rate mechanisms are used to improve the model's adaptability and generalization ability.
8. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which executes the instructions stored in the memory to perform the method as described in any one of claims 1-7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store instructions that, when executed, cause the method as described in any one of claims 1-7 to be implemented.
10. A computer program product, characterized in that, When the computer program product is invoked by a computer, it causes the computer to perform the method as described in any one of claims 1-7.