Knowledge distillation method based on frequency domain optimization

Abstract

The invention relates to a knowledge distillation method based on frequency domain optimization, and the method comprises the steps: 1, carrying out the interaction with an intermediate frequency range through frequency prompt, and enabling the process to be guided and supervised by a teacher task loss function; and 2, carrying out multiplication operation on the fixed component and the frequency band of the teacher model by utilizing the frequency prompt in the step 1, thereby identifying key points in the frequency band. According to the method, the application of dense prediction tasks is optimized, the student model can more effectively learn key features in the teacher model by dynamically integrating the information of the frequency domain, and the effect and efficiency of model training are improved. The method has wide applicability, and a more efficient learning framework can be provided in different tasks.

Description

Technical Field

[0001] The present invention relates to a knowledge distillation method based on frequency domain optimization, and belongs to the technical field of machine learning models. Background Art

[0002] In the pursuit of major technological breakthroughs, deep learning models have undergone significant architectural deepening and expansion in recent years, especially in many cutting-edge research works. However, as the scale and complexity of models increase, it becomes increasingly difficult and inefficient to deploy these models on resource-constrained edge devices. To address this challenge, knowledge distillation technology has emerged as a promising solution.

[0003] Knowledge distillation is a model compression technology in the field of machine learning. Its core concept is to transfer the knowledge content of a large and complex teacher model to a more compact student model. This process mainly trains the student model to reproduce the prediction distribution of the teacher model, rather than just learning the labels of the training data itself. This method aims to significantly reduce the size and complexity of the model while maintaining the accuracy of the prediction, thereby enhancing the computational efficiency of the model and its practicality in resource-constrained environments.

[0004] Knowledge distillation faces unique challenges in various computer vision tasks, especially in dense prediction tasks, especially the need to achieve refined distillation at the feature level. Current methods attempt to improve performance through spatial-level distillation losses, but in the spatial domain of the teacher model, sequential downsampling introduces interference, which prevents the student model from identifying and imitating specific useful information, resulting in reduced accuracy.

[0005] In-depth analysis shows that the downsampling operation eliminates high-frequency details in the frequency domain, exposing underlying patterns that are difficult to directly identify from pixel values. Although this inspires the possibility of using frequency signals for knowledge distillation, it also brings two major challenges: first, the information provided by the low-frequency band of the teacher model is relatively general. If the student is forced to imitate all low-frequency pixels, it may focus on simple samples with less information; second, although the high-frequency band provides rich detail signals, which helps to enhance the robustness and generalization ability of the model, it also introduces noise and affects the overall performance. Therefore, how to accurately locate valuable points of interest (PoIs) in low-frequency and high-frequency bands has become a key problem that needs to be solved in current knowledge distillation technology. Summary of the invention

[0006] In order to overcome the shortcomings of existing research, the present invention provides a knowledge distillation method based on frequency domain optimization.

[0007] A knowledge distillation method based on frequency domain optimization includes the following steps:

[0008] Step 1: Interact with the intermediate frequency range through frequency cues, and this process is guided and supervised by the teacher task loss function; in this process, the intermediate representation is optimized through the guidance of frequency cues to better match the output pattern expected by the teacher model.

[0009] An orthogonal wavelet basis is defined by scaling and translating the mother function Φ(t):

[0010]

[0011] in is the set of all integers, factors of Keeping a constant norm independent of the scale s, the variables s and d scale and dilute the generating function Φ to generate L 2 In wavelet space, to create a frequency representation, a discrete wavelet transform (DWT) ξ is applied to decompose each channel by bands as follows:

[0012] B i =ξ(x)

[0013] Where l is the decomposition level, the feature map Can be divided into four band, and B1 = {LL, HL, LH, HH}, where LL represents the tensor corresponding to the low frequency band, and the others are high frequency bands.

[0014] To learn the teacher’s knowledge, the invented method simulates the tensor pixel by pixel. (t) ∈R C×H×W and Represent the feature maps of the teacher and student networks respectively, and the imitation of the high-frequency band can be completed in the following way:

[0015]

[0016] a k ∈ξ(F (t) ),b k ∈ξ(φ(F (s) ))

[0017] Where L is the number of frequency bands, φ is a linear projection layer used to adapt F(s) to the same resolution as F(t), the student model studies F(s) to the same resolution as F(t) for low-frequency imitation, and learns a regularity from high frequencies through significant patterns of high-frequency textures, edges, and noise (including fine textures), distilling the performance of the student model to a level close to that of the teacher model;

[0018] Step 2: Convert the feature maps of the student model and the teacher model to the frequency domain. Use the frequency cue obtained from step 1 to perform a fixed multiplication operation with the frequency band of the teacher model to identify the points of interest within the band. Combine the spatial gating mechanism of the teacher model with the gating mechanism of the student model to jointly determine a channel-level position-aware weight to adjust the feature representation of the student model to make it closer to the learning result of the teacher model.

[0019] The step 2 specifically includes:

[0020] The present invention introduces a learnable frequency prompt P∈R B×T×C To pass T pixels imitation principle in B frequency bands of C channels, and fine-tune the teacher model, the frequency prompt can interact with the band, understand the manifold embedded in the frequency space, and use matrix multiplication to calculate the mutual information between the prompt P and the frequency pixel R(t) in the teacher band

[0021] M=P×R(t)

[0022] In order to deal with the task loss L finetune Supervised and supported stochastic gradient descent;

[0023]

[0024] The mutual information M is converted into a probability space as a mask, and the symbol σ represents the sigmoid function. represents element-wise multiplication;

[0025] After collecting all B mask frequency bands, we perform an inverse discrete wavelet transform ξ to obtain a new feature map in the spatial domain:

[0026]

[0027] And the new feature map F (t) is sent back to the teacher model. The fine-tuning loss can be viewed as an observation of the mask quality and is minimized to force the frequency cue to focus on the substantial pixels of the band.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] According to recent research, frequency domain analysis has been widely used in a variety of computer tasks including image classification, image generation, and image super-resolution. Early studies have shown that in the frequency domain, the phase component mainly captures the high-level semantic information of the original signal, while the amplitude component retains the low-level statistical characteristics. Therefore, compared with the raw pixel values ​​in the spatial domain, the frequency representation makes the underlying image patterns more intuitive and visible. In this regard, wavelet analysis stands out as a particularly effective tool in image processing because it can capture multi-scale frequency domain information in a compact form. Compared with other frequency analysis methods such as Fourier analysis, wavelet analysis provides a more comprehensive perspective. With wavelet analysis, our method is optimized for dense prediction tasks and significantly outperforms the distillation effect of image patterns by directly distilling the raw pixel values ​​in the spatial domain.

[0030] The present invention proposes a knowledge distillation method based on frequency domain optimization to eliminate unfavorable information in the frequency band. This method dynamically integrates the information in the frequency domain, so that the student model can more effectively learn the key features of the teacher model, thereby improving the effect and efficiency of model training. At the same time, this method has a wide range of applicability and can provide a more efficient learning framework in different tasks.

[0031] Frequency Knowledge Distillation specifically involves the flexible absorption and utilization of semantic frequency context in the teacher model. According to the specific needs and characteristics of the task, the feature map can be refined and adjusted through the frequency prompt mechanism, thereby improving the performance and efficiency of the student model. In addition, this method is particularly optimized for the application of intensive prediction tasks. It is worth mentioning that this method not only performs well in a variety of benchmark tests, significantly improves the accuracy of the model, but also enhances the robustness of the model. The present invention also verifies its applicability and generalization ability on large-scale visual models. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is an architectural diagram of the knowledge distillation method based on frequency domain optimization of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] A knowledge distillation method based on frequency domain optimization includes the following steps:

[0036] Step 1: Interact with the intermediate frequency range through frequency cues, and this process is guided and supervised by the teacher task loss function.

[0037] An orthogonal wavelet basis is defined by scaling and translating the mother function Φ(t):

[0038]

[0039] in is the set of all integers, factors of The variables s and d scale and dilute the generating function Φ to generate L 2 Wavelet in space. To create a frequency representation, a discrete wavelet transform (DWT) ξ is applied to decompose each channel by bands as follows:

[0040] B i =ξ(x)

[0041] Where l is the decomposition level. Feature map Can be divided into four Band. And B1={LL,HL,LH,HH}. Among them, LL represents the tensor corresponding to the low frequency band, and the others are the high frequency bands.

[0042] To learn the teacher’s knowledge, the invented method simulates the tensor pixel by pixel. (t) ∈R C×H×W and Represent the feature maps of the teacher and student networks respectively, and the imitation of the high-frequency band can be completed in the following way:

[0043]

[0044] a k ∈ξ(F (t) ),b k ∈ξ(φ(F (s) ))

[0045] Where L is the number of frequency bands and φ is a linear projection layer that adapts F(s) to the same resolution as F(t). The student model studies F(s) to the same resolution as F(t) for low-frequency imitation and learns general rules from high frequencies through significant patterns of high-frequency textures, edges, and noise (including fine textures). Finally, the performance of the student model is distilled to a point close to that of the teacher model.

[0046] Step 2: Using the frequency cues from step 1, the fixed components are multiplied by the frequency band of the teacher model to identify the key points in the frequency band.

[0047] Invent a learnable frequency cue P∈R B×T×C To pass T pixels imitation principle in B frequency bands of C channels, and it will fine-tune the teacher model. This method can interact frequency cues with bands and better understand the manifold embedded in the frequency space. First, the mutual information between the cue P and the frequency pixels R(t) in the teacher band is calculated by matrix multiplication

[0048] M=P×R(t)

[0049] Then, in order to compare with the task loss L finetune Supervised and supported stochastic gradient descent.

[0050]

[0051] The mutual information M is converted into a probability space as a mask. The symbol σ represents the sigmoid function, Represents element-wise multiplication.

[0052] After collecting all B mask frequency bands, we perform inverse discrete wavelet transform (IDWT) ξ on them to obtain a new feature map in the spatial domain:

[0053]

[0054] And the new feature map F (t) is sent back to the teacher model. The fine-tuning loss can be viewed as an observation of the mask quality and is minimized to force the frequency cue to focus on the substantial pixels of the band.

[0055] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A knowledge distillation method based on frequency domain optimization, characterized in that: The following steps are involved: Step 1: Interact with the intermediate frequency range through frequency cues, guided and supervised by the teacher task loss function; An orthogonal wavelet basis is defined by scaling and translating the mother function Φ(t): in is the set of all integers, factors of Keeping a constant norm independent of the scale s, the variables s and d scale and dilute the mother function Φ to generate wavelets in L2 space, creating a frequency representation, and the discrete wavelet transform ξ is applied to band decompose each channel through Φ as follows: B i =ξ(x) Where l is the decomposition level, the feature map Divided into four Band, and B1 = {LL, HL, LH, HH}, where LL represents the tensor corresponding to the low-frequency band, and the others are high-frequency bands, F (t) ∈R C×H×W and Represent the feature maps of the teacher and student networks respectively, and the imitation of the high-frequency band can be completed in the following way: a k ∈ξ(F (t) ),b k ∈ξ(φ(F (s) )) Where L is the number of frequency bands, φ is a linear projection layer used to adapt F(s) to the same resolution as F(t), the student model studies F(s) to the same resolution as F(t) for low-frequency imitation, learns a law from high frequencies through significant patterns of high-frequency textures, edges, and noise, and distills the performance of the student model to a level close to that of the teacher model; Step 2: Convert the feature maps of the student model and the teacher model to the frequency domain. Use the frequency cue obtained from step 1 to perform a fixed multiplication operation with the frequency band of the teacher model to identify the points of interest within the band. Combine the spatial gating mechanism of the teacher model with the gating mechanism of the student model to determine a channel-level position-aware weight, so as to adjust the feature representation of the student model to make it close to the learning result of the teacher model.

2. The knowledge distillation method based on frequency domain optimization according to claim 1, characterized in that: The step 2 specifically includes: Introducing a learnable frequency cue P∈R B×T×C To pass the T pixel imitation principle in B frequency bands of C channels, and fine-tune the teacher model, interact the frequency prompt with the band, understand the manifold embedded in the frequency space, and use matrix multiplication to calculate the mutual information between the prompt P and the frequency pixel R(t) in the teacher band M=P×R(t) In order to deal with the task loss L finetune Supervised and supported stochastic gradient descent; The mutual information M is converted into a probability space as a mask, and the symbol σ represents the sigmoid function. represents element-wise multiplication; After collecting all B mask frequency bands, we perform an inverse discrete wavelet transform ξ to obtain a new feature map in the spatial domain: And the new feature map F (t) Return the teacher model.

Images