Knowledge distillation model lightweight method based on frequency information guidance
By introducing frequency information during knowledge distillation, selecting important high-frequency and low-frequency components, and combining spatial domain and frequency domain distillation losses, the problems of insufficient identification of feature importance and model overfitting in traditional knowledge distillation methods are solved, and the efficient learning and detection performance of student models are improved.
Patent Information
- Application Number
- CN202411710924.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-21
AI Technical Summary
Traditional knowledge distillation methods cannot effectively distinguish important frequency components in the characteristics, resulting in some key features being ignored, and student models may overfit the teacher model, lacking adaptability to noise and distribution changes.
Using a lightweight method of knowledge distillation model based on frequency information, we use frequency domain transformation of the characteristic map of the teacher model, select important high-frequency and low-frequency components, and combine spatial domain distillation loss with frequency domain distillation loss to form the final distillation loss function to guide the learning of the student model.
It significantly improves the detection accuracy and robustness of the student model, making it perform better in small object detection and complex background separation tasks, has stronger generalization capabilities, and accelerates the convergence speed of the model, realizing the lightweight and efficient model.
Smart Images
Figure CN119672489A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning, and in particular to a lightweight method for a knowledge distillation model guided by frequency information. Background Art
[0002] Knowledge Distillation for Object Detection has become a research hotspot in the field of deep learning in recent years. As a key task in computer vision, object detection has been widely used in autonomous driving, security monitoring, smart devices and other fields. However, object detection models usually have a large number of parameters and a large amount of computation, so their application is limited in resource-constrained scenarios. Knowledge distillation is an effective model compression technology that transfers knowledge from a high-performance teacher model to a lightweight student model, so that the student model can achieve detection performance close to that of the teacher model at a lower computational complexity. The object detection knowledge distillation method is mainly dedicated to solving how to effectively transfer feature information, thereby achieving model compression and performance improvement.
[0003] However, in the traditional knowledge distillation process, knowledge transfer mainly relies on soft labels or intermediate layer features, which cannot distinguish the important frequency components in the features, resulting in some key features being ignored. In addition, in traditional distillation, the student model may overfit the output or features of the teacher model and lack adaptability to noise and distribution changes. Summary of the invention
[0004] The purpose of the present invention is to provide a lightweight method for knowledge distillation model guided by frequency information to solve the problems in the above-mentioned background technology.
[0005] To achieve the above object, the technical solution adopted by the present invention is:
[0006] A lightweight method for a knowledge distillation model guided by frequency information includes the following steps: preprocessing the acquired data, including adjusting the image size, data enhancement, and image normalization;
[0007] The preprocessed data is subjected to frequency information distillation. First, the image is input into the teacher model to generate a feature map containing rich spatial and semantic information. Then, the generated feature map is transformed into the frequency domain to obtain the expression of the feature map in the frequency space. Finally, the important frequency components are selected to identify the important high-frequency and low-frequency components of the target detection task.
[0008] Guided by frequency information, the student model learns the features of the teacher model. In order to ensure that the student model can effectively imitate the frequency information of the teacher model, the spatial domain distillation loss and the frequency domain distillation loss are combined to form the final distillation loss function.
[0009] Furthermore, in the process of inputting the image into the teacher model to generate a feature map containing rich spatial and semantic information, the feature map is a high-level feature map in the teacher model containing complete target and background information, which is obtained by the following formula,
[0010] F teacher =TeacherModel(X)
[0011] Where X is the input image, F teacher is the feature map of the teacher model.
[0012] Furthermore, in the process of frequency domain transformation of the generated feature map, the feature map is converted to the frequency domain by fast Fourier transform, and the formula is expressed as follows:
[0013]
[0014] in, It is the feature map of the teacher model in the frequency domain.
[0015] Furthermore, in the process of selecting important frequency components, a differential guided simulated annealing algorithm is used to select important frequency components, which specifically includes the following steps:
[0016] Initialize the parameters of the differential guided simulated annealing algorithm, including the initial temperature T0, the cooling coefficient α, the differential evolution parameter and the number of iterations, and set the set of initial solutions, each solution representing a selection scheme for a set of frequency components;
[0017] Define the objective function to evaluate the quality of each solution. The objective function is defined by the following formula:
[0018]
[0019] in, is the performance score of the selected frequency component on the target detection task, is the sum of the energies of the selected frequency components, and λ is a trade-off coefficient used to control the balance between performance and energy;
[0020] Differential mutation operation, in each iteration, each frequency component of the current solution is differentially mutated to generate a new candidate solution. The new solution is generated by linearly combining the differences between multiple solutions. The formula is as follows:
[0021]
[0022] in, It is a new solution. is the current solution, is the best solution in the current group, is a randomly selected solution, and F is a scaling factor;
[0023] Use simulated annealing algorithm to receive new solutions to control the convergence of the algorithm;
[0024] When the maximum number of iterations is reached or the temperature drops to a certain threshold, the algorithm terminates and outputs the final frequency component;
[0025] The final frequency component is the frequency component that makes the best contribution to the target detection task by obtaining the optimal result on the objective function, and the frequency component is used to guide the student model to imitate the characteristics of the teacher model in the frequency space.
[0026] Furthermore, in the process of receiving a new solution using the simulated annealing algorithm, it is necessary to calculate the difference between the new solution and the current solution. The objective function is as follows:
[0027]
[0028] Among them, Δf is the difference between the new solution and the current solution, For the new solution, is the current solution. If the new solution is better than the current solution, the new solution is accepted. If the new solution is not better than the current solution, the new solution is accepted with probability P=exp(-Δf / T), where T is the current temperature. The temperature T is reduced after each iteration to control the search range.
[0029] Furthermore, the spatial domain distillation loss matches the spatial feature maps of the teacher model and the student model to ensure that the student model can learn the overall features of the teacher model in the spatial domain. The formula is as follows:
[0030]
[0031] Among them, L spatial is the spatial loss between the feature map of the teacher model and the feature map of the student model, F teacher is the feature map of the teacher model, F student is the feature map of the student model.
[0032] Furthermore, the frequency domain distillation loss matches the feature maps of the teacher model and the student model in the frequency domain, so that the student model imitates the feature expression of the teacher model in the frequency space. The formula is as follows:
[0033]
[0034] Among them, L freq is the frequency domain loss, is the characteristic map of the teacher model in the frequency domain, is the feature map of the student model in the frequency domain, I(f) is the set of important frequency components, and the loss is calculated only on the said frequency to reduce the impact of irrelevant information.
[0035] Furthermore, the spatial domain distillation loss and the frequency domain distillation loss are combined to form the final distillation loss function. The weight parameters α and β are adjusted to balance the influence of the spatial domain distillation loss and the frequency domain distillation loss. The formula is as follows:
[0036] L total =αL spatial +βL freq
[0037] Among them, L total is the overall distillation loss, α and β are weight parameters used to balance the weights of spatial domain and frequency domain losses.
[0038] Beneficial effects:
[0039] (1) The present invention introduces frequency information during the distillation process, so that the student model can focus on the key frequency components of the feature map in the teacher model, especially the learning of high-frequency details and low-frequency background information, thereby effectively improving the detection accuracy of the student model and making it perform better in tasks such as small target detection and complex background separation.
[0040] (2) By selectively retaining the frequency components that contribute most to the task, the student model of the present invention has stronger robustness in complex scenarios and diverse environmental conditions. For example, it is more stable when dealing with occlusion, noise interference and illumination changes. In addition, by effectively transmitting the core information of the teacher model through frequency guidance, the student model can learn important spatial and frequency features more quickly, thereby significantly accelerating the convergence speed of the model and reducing training time. On this basis, this method achieves the lightweight and high efficiency of the model, so that the student model maintains a lightweight structure while still having detection performance close to that of the teacher model. It is particularly suitable for environments with limited computing resources such as embedded devices and mobile terminals.
[0041] (3) Thanks to the guidance of frequency information, the student model can maintain good detection results on unseen data distributions and different data sets, showing strong generalization ability. This method provides an innovative and practical solution for model compression and performance improvement in target detection. While achieving a balance between detection accuracy and computational efficiency, it greatly expands the applicability and versatility of the model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is the overall flow chart of the present invention;
[0043] Figure 2is a frequency domain image in an embodiment of the present invention;
[0044] Figure 3 It is a line graph showing the changing trends of the spatial domain loss, frequency domain loss, and overall distillation loss with training iterations in an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0046] Example
[0047] A lightweight method for knowledge distillation model guided by frequency information, such as Figure 1 As shown, the following steps are included:
[0048] S100, preprocessing the acquired data, including adjusting the image size, data enhancement and image normalization.
[0049] In the object detection task, image preprocessing is an essential step in the training process. Among them, resizing (scaling) the collected images is the key to ensure that the input data meets the requirements of the model. Object detection models usually require fixed-size input images for batch processing. In the preprocessing stage, all images need to be resized to a uniform size to meet the input requirements of the model. In addition, in order to avoid image distortion, a scaling method that maintains the aspect ratio is usually used. If the resized image cannot fully meet the requirements of the model, methods such as padding or center cropping can be selected to ensure that the objects in the image are not truncated. The padding operation usually uses zero padding or mean padding so that the image can reach the desired size while maintaining the integrity of the object.
[0050] Data enhancement is an important means to improve the generalization ability of the model, especially when the amount of data is limited. Common data enhancement methods include: random horizontal flipping, random cropping, color jittering, rotation and scaling, adding noise, etc. Random horizontal flipping refers to randomly flipping the image left and right to increase the robustness of the model to targets in different directions. The bounding box will also be adjusted accordingly to adapt to the flipped image. Random cropping is to randomly crop a certain proportion of the area from the image to retain the integrity of the target object and increase the adaptability of the model to different scales. The cropped bounding box needs to be recalculated to ensure accurate positioning. Color jittering makes the model adapt to different lighting conditions by randomly changing parameters such as brightness, contrast, saturation and hue of the image. Rotation and scaling help the model recognize targets at different viewing angles and distances while ensuring the correct positioning of the bounding box. Adding noise simulates noise interference in reality by adding a small amount of Gaussian noise or salt and pepper noise to the image to improve the model's anti-interference ability.
[0051] Image normalization is also an important part of data preprocessing. Normalization scales the pixel values of the image from [0, 255] to [0, 1] or [-1, 1] so that the distribution of the input data is consistent with the data distribution of the pre-trained model. The typical formula for normalization is:
[0052]
[0053] S200 performs frequency information distillation on the preprocessed data. First, the image is input into the teacher model to generate a feature map containing rich spatial and semantic information. Then, the generated feature map is transformed in the frequency domain to obtain the expression of the feature map in the frequency space. Finally, important frequency components are selected to identify the important high-frequency and low-frequency components of the target detection task.
[0054] S300, guided by frequency information, allows the student model to learn the features of the teacher model. In order to ensure that the student model can effectively imitate the frequency information of the teacher model, the spatial domain distillation loss and the frequency domain distillation loss are combined to form the final distillation loss function.
[0055] Specifically, the lightweight method of the knowledge distillation model guided by frequency information in this embodiment introduces frequency information to guide the learning of the student model during the distillation process, extracts different frequency components from the teacher model by frequency analysis, selects the frequency components that are most valuable to the task, and guides the student model to learn these frequency information first, thereby significantly improving the feature learning effect of the student model. Specifically, the intermediate features of the teacher model are first analyzed in the frequency domain to identify the key information in the high-frequency details and low-frequency semantic components. The frequency domain loss is added to the distillation loss function so that the student model can accurately fit the feature distribution of the teacher model in the frequency space, retaining important detail information and maintaining the consistency of global semantics. Through this guidance based on frequency information, the student model not only has strong robustness and generalization ability, but also can effectively reduce the transmission of redundant information, and improve the compression effect and computational efficiency of the model. Of course, this method can be used not only in target detection tasks, but also in tasks such as image classification and speech recognition.
[0056] Preferably, in the process of inputting the image into the teacher model to generate a feature map containing rich spatial and semantic information, the feature map is a high-level feature map containing complete target and background information output by a certain layer in the teacher model, which is obtained by the following formula,
[0057] F teacher =TeacherModel(X)
[0058] Where X is the input image, F teacher is the feature map of the teacher model.
[0059] In the process of frequency domain transformation of the generated feature map, the feature map can be converted to the frequency domain by discrete Fourier transform (DFT) or fast Fourier transform (FFT). The fast Fourier transform (FFT) formula is expressed as follows:
[0060]
[0061] in, It is the feature map of the teacher model in the frequency domain, which contains all frequency components. Through this operation, the feature map can be converted from the spatial domain to the frequency domain, which is convenient for analyzing the characteristics of different frequency components.
[0062] In the frequency domain, the different frequency components of the feature map are analyzed to identify the high-frequency and low-frequency components that are most important for the target detection task. Usually, the high-frequency components contain detail information (such as edges and textures), while the low-frequency components contain global information (such as background and contours). In the process of knowledge distillation, the selection of important frequency components is a key step to ensure that the student model can learn the most representative features of the teacher model. Therefore, it is necessary to select important frequency components.
[0063] In this embodiment, the Differential Guided Simulated Annealing (DGSA) algorithm is used to select important frequency components. DGSA is an optimization algorithm that can strike a balance between global search and local search and is suitable for efficiently finding the frequency components that contribute most to the task in a complex frequency space. Specifically, the following steps are included:
[0064] Initialize the parameters of the differential guided simulated annealing algorithm, including the initial temperature T0, the cooling coefficient α, the differential evolution parameters (such as the crossover rate CR and the scaling factor F) and the number of iterations, and set the set of initial solutions, each of which represents a selection scheme for a set of frequency components.
[0065] Define the objective function to evaluate the quality of each solution. The objective function can be calculated based on the contribution of each frequency component to the detection performance. The objective function is defined by the following formula:
[0066]
[0067] in, is the performance score of the selected frequency component on the target detection task, is the total energy of the selected frequency components, λ is the trade-off coefficient used to control the balance between performance and energy,
[0068] Differential mutation operation, in each iteration, each frequency component of the current solution is differentially mutated to generate a new candidate solution. The new solution is generated by linearly combining the differences between multiple solutions. The formula is as follows:
[0069]
[0070] in, It is a new solution. is the current solution, is the best solution in the current group, is a randomly chosen solution, F is a scaling factor,
[0071] Use the simulated annealing algorithm to receive the new solution to control the convergence of the algorithm and calculate the difference between the new solution and the current solution. The objective function is as follows:
[0072]
[0073] Among them, Δf is the difference between the new solution and the current solution, For the new solution, For the current solution, if the new solution is better than the current solution (i.e., Δf<0), the new solution is accepted; if the new solution is not better than the current solution, the new solution is accepted with probability P=exp(-Δf / T), where T is the current temperature. The temperature T is lowered after each iteration to control the search range.
[0074] When the maximum number of iterations is reached or the temperature drops to a certain threshold, the algorithm terminates and outputs the final selected frequency component;
[0075] The final frequency component that obtains the best result on the objective function is the frequency component that makes the best contribution to the target detection task, that is, the frequency that makes the most contribution to the target detection task. The frequency component is used to guide the student model to imitate the characteristics of the teacher model in the frequency space.
[0076] Preferably, the spatial domain distillation loss ensures that the student model can learn the overall characteristics of the teacher model in the spatial domain by matching the spatial feature maps of the teacher model and the student model. The formula is as follows:
[0077]
[0078] Among them, L spatial is the spatial loss between the feature map of the teacher model and the feature map of the student model, F teacher is the feature map of the teacher model, F student is the feature map of the student model.
[0079] The frequency domain distillation loss matches the feature maps of the teacher model and the student model in the frequency domain, so that the student model imitates the feature expression of the teacher model in the frequency space. The formula is as follows:
[0080]
[0081] Among them, L freq is the frequency domain loss, is the characteristic map of the teacher model in the frequency domain, is the feature map of the student model in the frequency domain, I(f) is the set of important frequency components, and the loss is calculated only on the said frequency to reduce the impact of irrelevant information.
[0082] The spatial domain distillation loss and the frequency domain distillation loss are combined to form the final distillation loss function, and the rectification loss function balances the influence of the spatial domain distillation loss and the frequency domain distillation loss by adjusting the weight parameters α and β. The formula is as follows:
[0083] L total =αL spatial +βL freq
[0084] Among them, L total is the overall distillation loss, α and β are weight parameters used to balance the weights of spatial domain and frequency domain losses.
[0085] In this embodiment, three different groups of teacher models and student models are selected to combine and apply the frequency information-guided knowledge distillation model lightweight method of the present invention to evaluate the effect of the method on different model architectures.
[0086] Three experimental model combinations:
[0087] Model group 1: ResNet-50 (teacher) -> MobileNetV2 (student)
[0088] Teacher model: ResNet-50 is a deep convolutional neural network with strong feature extraction capabilities, especially in complex scenarios.
[0089] Student model: MobileNetV2 is a lightweight network suitable for resource-constrained devices with fewer parameters, but it has certain limitations in feature extraction.
[0090] Distillation goal: Through frequency guidance, MobileNetV2 can maintain better details and global information when learning the features of ResNet-50, so as to improve the performance of the small model in the detection task.
[0091] Model Group 2: ResNet-101 (Teacher) -> ShuffleNet (Student)
[0092] Teacher model: ResNet-101 has deeper layers than ResNet-50 and has stronger feature extraction capabilities, which can capture details in complex scenes.
[0093] Student model: ShuffleNet is an extremely lightweight network with high speed but limited feature expression capabilities.
[0094] Distillation goal: Guided by frequency information, let ShuffleNet learn the key frequency components in ResNet-101, especially the high-frequency details, to improve its ability to recognize small targets and edge information.
[0095] Model Group 3: EfficientNet-B3 (Teacher) -> MobileNetV3 (Student)
[0096] Teacher Model: EfficientNet-B3 is an efficient model architecture with high parameter efficiency and excellent performance on detection tasks.
[0097] Student model: MobileNetV3 is structurally optimized and suitable for use on mobile devices and embedded devices, but it has certain limitations when identifying complex features.
[0098] Distillation goal: Use frequency information to guide MobileNetV3 to effectively learn the global and detailed information of EfficientNet-B3 to improve detection accuracy and robustness in mobile scenarios
[0099] The experimental results are shown in Table 1. The results mainly include comparative analysis of mean average precision (mAP) improvement, inference speed, and model parameter quantity.
[0100] Table 1
[0101]
[0102] In this experiment, the frequency domain image is Figure 2 As shown in the figure, the bright spot in the center represents low-frequency information, and the dark area around it represents high-frequency information. The low-frequency components correspond to the overall structure and background of the image, while the high-frequency components mainly contain the edges and detail features of the image. This image can help analyze the distribution of high-frequency and low-frequency components during the knowledge distillation process, making it easier to select frequency components that are beneficial to the detection task.
[0103] In this experiment, the spatial domain loss, frequency domain loss, and overall distillation loss change with training iterations as shown below: Figure 3 As shown in the figure, it can be seen that all three curves show a gradually decreasing trend, indicating that as the training progresses, the model is gradually optimized in the two feature spaces of space and frequency, and the overall loss continues to decrease and gradually converges. The spatial domain loss is high at the beginning of training, and then drops rapidly, indicating that the model can capture the overall spatial characteristics of the image more stably. The frequency domain loss shows a certain fluctuation, which may be due to the complexity of the frequency characteristics. The model needs more iterations to learn and adapt, but it shows a downward trend overall, indicating that the model gradually grasps the high-frequency and low-frequency information of the image.
[0104] The overall distillation loss is the sum of the spatial domain loss and the frequency domain loss. It drops rapidly in the first 20 epochs, indicating that the model quickly learns spatial and frequency information in the early stages. From the 50th epoch, the overall loss gradually stabilizes, indicating that the model gradually converges and remains stable in the later stages of training, achieving good optimization results.
[0105] The experiments in this embodiment show that the lightweight method of knowledge distillation model guided by frequency information is very effective in promoting the model to learn important spatial and frequency features, which can not only speed up the convergence speed, but also help the model to achieve better recognition accuracy in detail features and edge detection. The present invention can effectively improve the detection ability and generalization performance of the model, so that the model can fully play its role in different feature spaces.
[0106] Although the embodiments of the present invention are described in the specification, these embodiments are only for reference and should not limit the protection scope of the present invention. Various omissions, substitutions and changes within the scope of the present invention should be included in the protection scope of the present invention.
Claims
1. A lightweight method for knowledge distillation model based on frequency information guidance, characterized by: The following steps are included: Preprocess the acquired data, including image resizing, data augmentation, and image normalization; The preprocessed data is subjected to frequency information distillation. First, the image is input into the teacher model to generate a feature map containing rich spatial and semantic information. Then, the generated feature map is transformed into the frequency domain to obtain the expression of the feature map in the frequency space. Finally, the important frequency components are selected to identify the important high-frequency and low-frequency components of the target detection task. Guided by frequency information, the student model learns the features of the teacher model. In order to ensure that the student model can effectively imitate the frequency information of the teacher model, the spatial domain distillation loss and the frequency domain distillation loss are combined to form the final distillation loss function.
2. The lightweight method of knowledge distillation model based on frequency information guidance according to claim 1 is characterized by: In the process of inputting an image into a teacher model to generate a feature map containing rich spatial and semantic information, the feature map is a high-level feature map containing complete target and background information in the teacher model, which is obtained by the following formula, F teacher =TeacherModel(X) Where X is the input image, F teacher is the feature map of the teacher model.
3. The lightweight method of knowledge distillation model based on frequency information guidance according to claim 1 is characterized by: In the process of frequency domain transformation of the generated feature map, the feature map is converted to the frequency domain through fast Fourier transform. The formula is expressed as follows: in, It is the feature map of the teacher model in the frequency domain.
4. The lightweight method of knowledge distillation model based on frequency information guidance according to claim 1 is characterized by: In the process of selecting important frequency components, the differential guided simulated annealing algorithm is used to select important frequency components. The following steps are included: Initialize the parameters of the differential guided simulated annealing algorithm, including the initial temperature T0, the cooling coefficient α, the differential evolution parameter and the number of iterations, and set the set of initial solutions, each solution representing a selection scheme for a set of frequency components; Define the objective function to evaluate the quality of each solution. The objective function is defined by the following formula: in, is the performance score of the selected frequency component on the target detection task, is the sum of the energies of the selected frequency components, and λ is a trade-off coefficient used to control the balance between performance and energy; Differential mutation operation, in each iteration, each frequency component of the current solution is differentially mutated to generate a new candidate solution. The new solution is generated by linearly combining the differences between multiple solutions. The formula is as follows: in, It is a new solution. is the current solution, is the best solution in the current group, is a randomly selected solution, and F is a scaling factor; Use simulated annealing algorithm to receive new solutions to control the convergence of the algorithm; When the maximum number of iterations is reached or the temperature drops to a certain threshold, the algorithm terminates and outputs the final frequency component; The final frequency component is the frequency component that makes the best contribution to the target detection task by obtaining the optimal result on the objective function, and the frequency component is used to guide the student model to imitate the characteristics of the teacher model in the frequency space.
5. The lightweight method of knowledge distillation model based on frequency information guidance according to claim 4 is characterized in that: When receiving a new solution using the simulated annealing algorithm, it is necessary to calculate the difference between the new solution and the current solution. The objective function is as follows: Among them, Δf is the difference between the new solution and the current solution, For the new solution, is the current solution. If the new solution is better than the current solution, the new solution is accepted. If the new solution is not better than the current solution, the new solution is accepted with probability P=exp(-Δf / T), where T is the current temperature. The temperature T is reduced after each iteration to control the search range.
6. The lightweight method of knowledge distillation model based on frequency information guidance according to claim 1 is characterized by: The spatial domain distillation loss matches the spatial feature maps of the teacher model and the student model to ensure that the student model can learn the overall features of the teacher model in the spatial domain. The formula is as follows: Among them, L spatial is the spatial loss between the feature map of the teacher model and the feature map of the student model, F teacher is the feature map of the teacher model, F student is the feature map of the student model.
7. The lightweight method of knowledge distillation model based on frequency information guidance according to claim 1 is characterized by: The frequency domain distillation loss matches the feature maps of the teacher model and the student model in the frequency domain, so that the student model imitates the feature expression of the teacher model in the frequency space. The formula is as follows: Among them, L freq is the frequency domain loss, is the characteristic map of the teacher model in the frequency domain, is the feature map of the student model in the frequency domain, I(f) is the set of important frequency components, and the loss is calculated only on the said frequency to reduce the impact of irrelevant information.
8. The lightweight method of knowledge distillation model based on frequency information guidance according to claim 1, characterized in that: The spatial domain distillation loss and the frequency domain distillation loss are combined to form the final distillation loss function. The weight parameters α and β are adjusted to balance the effects of the spatial domain distillation loss and the frequency domain distillation loss. The formula is as follows: L total =αL spatial +βL freq Among them, L total is the overall distillation loss, α and β are weight parameters used to balance the weights of spatial domain and frequency domain losses.