Frequency Decoupling Knowledge Distillation Method and System Based on Wavelet Decomposition

Abstract

This invention relates to the fields of deep learning, model compression, and computer vision, and provides a frequency-decoupled knowledge distillation method and system based on wavelet decomposition. The method includes: acquiring input samples and obtaining teacher and student logits respectively; performing wavelet decomposition on the teacher and / or student logits to obtain low-frequency and high-frequency components; performing adaptive soft-threshold denoising processing based on statistical features on the high-frequency components; performing inverse wavelet transform on the low-frequency components and the denoised high-frequency components to obtain the reconstructed logit; normalizing the reconstructed logit and the original logit respectively, and performing weighted fusion based on dynamic fusion coefficients to obtain the target logit; constructing a knowledge distillation loss based on the teacher and student target logits, and training the student model by combining the supervised loss of the real labels. This invention can decouple the structural information and high-frequency perturbations in the logit, improving the stability and efficiency of knowledge distillation.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, model compression, and computer vision, and in particular to a frequency decoupling knowledge distillation method and system based on wavelet decomposition. Background Technology

[0002] With the widespread application of deep neural networks (DNNs) in computer vision tasks such as image classification, object detection, and fine-grained recognition, the model size is constantly expanding and the network structure is becoming increasingly complex. Although this has significantly improved model performance, it has also brought about higher computational costs and storage overhead, limiting its deployment and application in mobile terminals and edge devices.

[0003] To address the aforementioned issues, knowledge distillation (KD) has been widely studied as an important model compression technique. This method transfers knowledge from a high-performance teacher model to a lightweight student model, enabling the student model to achieve performance close to that of the teacher model while maintaining low computational complexity. Traditional knowledge distillation methods mainly include feature-based distillation and logit-based distillation.

[0004] Feature-based distillation methods typically achieve knowledge transfer by aligning intermediate layer features between the teacher and student models. While achieving good performance, they often rely on complex network architectures and introduce high computational and storage overhead. Furthermore, their applicability is poor when the teacher and student model structures are inconsistent. In contrast, Logit-based distillation methods directly act on the model's output layer, achieving knowledge transfer by minimizing the difference between the output probability distributions of the teacher and student models. They offer advantages such as simple structure, low computational cost, and applicability to heterogeneous network structures, thus attracting increasing attention.

[0005] However, traditional Logit-based knowledge distillation methods typically directly align the output Logit values ​​of the teacher and student models, which presents the following technical problems: (1) There is significant high-frequency perturbation information in Logit. This type of information usually comes from the instability of model prediction or noise interference. It is easy to be mistakenly transmitted as effective knowledge during the distillation process, thereby reducing the distillation effect. (2) The Logit numerical distribution has strong scale inconsistency, and the Logit amplitudes of different samples or different models vary greatly, which can easily lead to instability in the training process. (3) Traditional methods do not explicitly distinguish between structural semantic information and noise perturbation information in Logit, resulting in effective knowledge being coupled with noise information, which limits the ability of student models to learn discriminative knowledge from teacher models.

[0006] To address the aforementioned issues, existing technologies have attempted to mitigate Logit scale differences through standardization methods (such as Z-score normalization). However, these methods only adjust the values ​​at the numerical level and fail to fundamentally distinguish the different frequency components in the Logit, thus still being unable to effectively suppress the interference of high-frequency noise on knowledge transfer.

[0007] Therefore, there is an urgent need for a knowledge distillation method that can decouple Logit at the frequency level and effectively suppress high-frequency disturbance information in order to improve the stability of the distillation process and the efficiency of knowledge transfer. Summary of the Invention

[0008] The purpose of this invention is to solve at least one technical problem in the background art and to provide a frequency decoupling knowledge distillation method and system based on wavelet decomposition.

[0009] To achieve the above objectives, this invention provides a frequency decoupling knowledge distillation method based on wavelet decomposition, comprising: Obtain input samples and input them into the teacher model and student model respectively to obtain the teacher Logit output by the teacher model and the student Logit output by the student model; Wavelet decomposition is performed on the teacher's Logit and / or the student's Logit, decomposing the Logit into low-frequency components representing structural semantic information and high-frequency components representing local change information; An adaptive soft threshold denoising process is performed on the high-frequency components to obtain denoised high-frequency components, wherein the adaptive soft threshold denoising process determines the denoising threshold based on the statistical characteristics of the high-frequency components. Perform an inverse wavelet transform based on the low-frequency components and the denoised high-frequency components to obtain the reconstructed Logit; Normalization is performed on the reconstructed Logit and the teacher Logit and / or student Logit that serve as the original Logit, respectively. Based on the fusion coefficient that changes dynamically with the training phase, the normalized original Logit and the normalized reconstructed Logit are weighted and fused to obtain the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, respectively. The knowledge distillation loss is constructed based on the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, and the student model is trained by combining the supervised loss corresponding to the real label of the input sample.

[0010] According to one aspect of the invention, the wavelet decomposition employs Haar wavelet decomposition, comprising: The Logit to be decomposed is divided into even-position sub-vectors and odd-position sub-vectors according to the index; The sum of the even-position sub-vectors and the odd-position sub-vectors is taken as the low-frequency component, and the difference between the two is taken as the high-frequency component.

[0011] According to one aspect of the present invention, performing adaptive soft thresholding denoising on the high-frequency components includes: Obtain the absolute value distribution statistics of the high-frequency components; Determine the adaptive threshold based on the absolute value distribution statistics; The high-frequency components are subjected to soft threshold shrinkage using the adaptive threshold.

[0012] According to one aspect of the invention, the adaptive threshold is determined by the following factors: The maximum value of the absolute value of the high-frequency component; The median of the absolute values ​​of the high-frequency components; Weighting parameters used to adjust the contribution of the maximum value and the median value; A dynamic decay factor used to characterize changes during the training phase.

[0013] According to one aspect of the present invention, the soft threshold shrinkage process is as follows: setting elements in the high-frequency components whose absolute values ​​are lower than the adaptive threshold to zero or attenuating them to zero, and retaining elements whose absolute values ​​are higher than the adaptive threshold or shrinking them toward zero.

[0014] According to one aspect of the invention, the fusion coefficient changes dynamically with the training phase. In the early stage of training, the fusion weight of the normalized original Logit is greater than the fusion weight of the normalized reconstructed Logit, and in the later stage of training, the fusion weight of the normalized reconstructed Logit gradually increases.

[0015] According to one aspect of the invention, the knowledge distillation loss is constructed by: temperature scaling of the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, respectively, and calculating the probability distributions, then calculating the Kullback-Leibler divergence between the two probability distributions, and multiplying the divergence by the square of the temperature parameter.

[0016] According to one aspect of the invention, the supervised loss is the cross-entropy loss between the student model output and the true label, and the total training loss is obtained by weighted summation of the knowledge distillation loss and the supervised loss.

[0017] To achieve the above objectives, the present invention also provides a frequency decoupling knowledge distillation system based on wavelet decomposition, comprising: The Logit acquisition module acquires input samples and inputs the input samples into the teacher model and student model respectively to obtain the teacher Logit output by the teacher model and the student Logit output by the student model. The frequency decomposition module performs wavelet decomposition on the teacher's Logit and / or the student's Logit, decomposing the Logit into low-frequency components representing structural semantic information and high-frequency components representing local change information. The denoising module performs adaptive soft threshold denoising on the high-frequency components to obtain denoised high-frequency components, wherein the adaptive soft threshold denoising determines the denoising threshold based on the statistical characteristics of the high-frequency components. The reconstruction module performs an inverse wavelet transform based on the low-frequency components and the denoised high-frequency components to obtain the reconstructed Logit. The normalization and fusion module performs normalization processing on the reconstructed Logit and the teacher Logit and / or student Logit that serve as the original Logit, respectively. Based on the fusion coefficient that changes dynamically with the training phase, the normalized original Logit and the normalized reconstructed Logit are weighted and fused to obtain the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, respectively. The training module constructs a knowledge distillation loss based on the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, and trains the student model by combining the supervised loss corresponding to the real labels of the input samples.

[0018] To achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the frequency decoupling knowledge distillation method based on wavelet decomposition as described above.

[0019] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the frequency decoupling knowledge distillation method based on wavelet decomposition as described above.

[0020] According to the present invention, the Logit of the teacher model and the student model is decomposed into the frequency domain for structured processing, thereby achieving explicit decoupling of structural semantic information and high-frequency perturbation information in the Logit. Haar wavelet decomposition is used to decompose the Logit into low-frequency and high-frequency components, and an adaptive soft thresholding denoising mechanism is used to suppress high-frequency noise. Simultaneously, normalization processing is combined to eliminate Logit scale differences. Finally, a dynamic fusion mechanism adaptively adjusts the weight ratio of the original Logit and the reconstructed Logit at different training stages. This significantly improves the stability of the knowledge distillation process and the efficiency of knowledge transfer without changing the network structure, enabling the student model to effectively learn the discriminative knowledge of the teacher model and suppress noise interference. It is applicable to various visual task scenarios such as image classification, object detection, fine-grained recognition, and heterogeneous network knowledge transfer. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a frequency decoupling knowledge distillation method based on wavelet decomposition according to an embodiment of the present invention is shown in the schematic diagram. Figure 2 A schematic diagram illustrating the Logit wavelet decomposition process according to Embodiment 1 of the present invention; Figure 3 This diagram illustrates the high-frequency component adaptive soft threshold denoising process according to Embodiment 1 of the present invention. Figure 4 The diagram illustrates the Logit reconstruction and dynamic fusion process according to Embodiment 1 of the present invention. Detailed Implementation

[0022] The invention will now be discussed with reference to exemplary embodiments. It should be understood that the described embodiments are merely intended to enable those skilled in the art to better understand and thus implement the invention, and are not intended to imply any limitation on the scope of the invention.

[0023] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment".

[0024] Figure 1 The flowchart schematically illustrates a frequency decoupling knowledge distillation method based on wavelet decomposition according to an embodiment of the present invention. Figure 1 As shown, in this embodiment, the frequency decoupling knowledge distillation method based on wavelet decomposition includes: Obtain the input sample and input the input sample into the teacher model and student model respectively to obtain the teacher Logit output by the teacher model and the student Logit output by the student model. Perform wavelet decomposition on the teacher's Logit and / or the student's Logit, decomposing the Logit into low-frequency components representing structural semantic information and high-frequency components representing local change information; Adaptive soft threshold denoising is performed on the high-frequency components to obtain the denoised high-frequency components, wherein the adaptive soft threshold denoising is based on the statistical characteristics of the high-frequency components to determine the denoising threshold. The inverse wavelet transform is performed on the low-frequency components and the denoised high-frequency components to obtain the reconstructed Logit. Normalization is performed on the reconstructed Logit and the teacher Logit and / or student Logit that serve as the original Logit, respectively. Based on the fusion coefficient that changes dynamically with the training phase, the normalized original Logit and the normalized reconstructed Logit are weighted and fused to obtain the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, respectively. A knowledge distillation loss is constructed based on the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, and the student model is trained by combining the supervised loss corresponding to the real labels of the input samples.

[0025] In this embodiment, knowledge distillation (KD) is a model compression technique that transfers the knowledge learned by a high-performance teacher model to a lightweight student model, thereby reducing computational costs while maintaining model performance as much as possible.

[0026] Logit refers to the unnormalized score vector output by a neural network in a classification task. It is usually used as the input to the softmax function to represent the prediction tendency of each category.

[0027] Wavelet decomposition is a mathematical method that decomposes a signal into components of different frequencies, where low-frequency components typically represent the overall structural information of the signal, while high-frequency components represent local variations or disturbances.

[0028] The Haar wavelet decomposition described below is a computationally simple and efficient wavelet decomposition method that achieves frequency decomposition of a signal by dividing it into even and odd parts and calculating their sum and difference.

[0029] Soft thresholding is a commonly used signal denoising method that reduces noise by setting a threshold to suppress components with small amplitudes in the signal.

[0030] Normalization refers to the standardization of the numerical distribution of data to reduce scale differences between different samples or different models and improve training stability.

[0031] Furthermore, according to one embodiment of the present invention, the wavelet decomposition employs Haar wavelet decomposition, including: The Logit to be decomposed is divided into even-position sub-vectors and odd-position sub-vectors according to the index; The sum of the even-position sub-vectors and the odd-position sub-vectors is taken as the low-frequency component, and the difference between the two is taken as the high-frequency component.

[0032] In this embodiment, Haar wavelet decomposition only involves addition, subtraction, and division. Constant scaling eliminates the need for complex convolution or filter bank operations, resulting in low computational complexity. This makes it suitable for embedding in knowledge distillation training processes without significantly increasing training overhead.

[0033] By dividing Logit into sub-vectors based on the parity of the index, the Logit vectors output by the model can be manipulated directly without changing the network structure of the teacher and student models. This approach can be directly embedded into existing knowledge distillation frameworks and is suitable for knowledge transfer between heterogeneous networks.

[0034] The sum of the even-position subvectors and the odd-position subvectors Scaling reflects the overall trend of Logit, corresponding to the structural semantic information in the low-frequency components; the difference between the two is... The scaling reflects the local changes between scores of adjacent categories, corresponding to perturbation information in high-frequency components. This decomposition method can explicitly separate globally stable discrimination knowledge from locally unstable noise perturbations, providing a foundation for subsequent adaptive soft-thresholding denoising.

[0035] Haar wavelet decomposition is an orthogonal normalization transform, and its inverse transform can be achieved through the same sum and difference operations. The original Logit is accurately reconstructed by scaling. This orthogonality ensures energy conservation and numerical stability before and after decomposition, enabling the low-frequency and high-frequency components to be restored to the original Logit without distortion, laying the mathematical foundation for subsequent selective denoising and accurate reconstruction.

[0036] Furthermore, according to one embodiment of the present invention, adaptive soft thresholding denoising processing is performed on the high-frequency components, including: Obtain the absolute value distribution statistics of high-frequency components; Determine the adaptive threshold based on the absolute value distribution statistic; Soft threshold shrinkage is performed on high-frequency components using an adaptive threshold.

[0037] In this embodiment, by obtaining the distribution statistics of the absolute values ​​of high-frequency components, the amplitude characteristics of the perturbation information in the high-frequency components can be objectively quantified, avoiding the problems of over-suppression (excessive denoising and loss of effective information) or under-suppression (noise residue) that may be caused by using a fixed threshold, so that the denoising process can adapt to the noise distribution differences of different samples and different training stages.

[0038] An adaptive threshold is determined by the distribution statistics of the absolute values ​​of high-frequency components, so that the threshold reflects the actual amplitude distribution characteristics of the current high-frequency components, avoiding the subjectivity and blindness of manually setting a fixed threshold, and improving the objectivity and adaptability of the denoising process.

[0039] Elements with absolute values ​​below the threshold are set to zero or attenuated to directly suppress low-amplitude noise; elements with absolute values ​​above the threshold are retained or reduced towards zero to reduce disturbances while retaining high-amplitude effective discrimination information, avoiding signal abrupt changes and oscillations caused by hard thresholds, and ensuring the smoothness and stability of the reconstructed Logit.

[0040] Furthermore, according to one embodiment of the present invention, the adaptive threshold is determined by the following factors: The maximum absolute value of the high-frequency component; The median of the absolute values ​​of high-frequency components; Weighting parameters used to adjust the contribution of the maximum value and the median value; A dynamic decay factor used to characterize changes during the training phase.

[0041] In this implementation, the maximum value reflects the upper limit of the amplitude of extreme disturbances in the high-frequency components, and can capture significant anomalous fluctuations; the median reflects the central tendency of the disturbance distribution, is insensitive to outliers, and can characterize the general noise level. Combining the two and adjusting the contribution ratio through weighting parameters ensures that the threshold can respond to significant anomalies while adapting to the overall distribution, avoiding threshold bias caused by a single statistic.

[0042] The dynamic decay factor changes with the training process, making the threshold adaptive at different training stages. By introducing a dynamic decay factor that represents changes in the training stage, the threshold can be dynamically adjusted according to the model's convergence state, avoiding over-denoising that may occur in the early stages of training or noise residue that may occur in the later stages of training due to a fixed threshold, thus achieving synergistic optimization of denoising intensity and model training progress.

[0043] By using a multi-statistic fusion and dynamic adjustment mechanism, the threshold does not depend on a single empirical value or fixed assumption. It can adapt to different datasets, different model structures and different training stages of noise characteristics, thereby enhancing the stability and generalization performance of the method in practical applications.

[0044] Furthermore, according to one embodiment of the present invention, the soft threshold shrinkage process is as follows: elements in the high-frequency components whose absolute value is lower than the adaptive threshold are set to zero or decayed to zero, while elements whose absolute value is higher than the adaptive threshold are retained or reduced towards zero.

[0045] In this implementation, elements with absolute values ​​below a threshold are set to zero or attenuated to zero, directly eliminating low-amplitude noise disturbances; elements with absolute values ​​above a threshold are retained or reduced towards zero, reducing high-amplitude disturbances while avoiding the complete deletion of potentially valid discriminative information. This selective processing makes the denoising process highly targeted, unlike the coarse processing methods of uniform scaling or global truncation.

[0046] The soft thresholding function is continuous at the threshold point, preventing the high-frequency components after denoising from exhibiting the jumps and discontinuities caused by hard thresholding. This continuity ensures a smooth and natural numerical distribution of the reconstructed Logit after inverse wavelet transform, avoiding the introduction of new artifacts or oscillations due to signal abrupt changes and improving the numerical stability of the distillation training process.

[0047] Instead of setting elements directly to zero, elements above the threshold are reduced towards zero, preserving their symbol information and amplitude characteristics after threshold adjustment. This ensures that the reconstructed Logit can still reflect the presence or absence of class discriminative power and its positive or negative direction, thus ensuring that the denoised Logit still has effective knowledge transfer capabilities during knowledge distillation.

[0048] Furthermore, according to one embodiment of the present invention, the normalization process includes: calculating the mean and standard deviation of the Logit to be normalized, subtracting the mean from the Logit and then dividing by the standard deviation.

[0049] Furthermore, according to one embodiment of the present invention, the fusion coefficient changes dynamically with the training phase. In the early stage of training, the fusion weight of the normalized original Logit is greater than the fusion weight of the normalized reconstructed Logit, and in the later stage of training, the fusion weight of the normalized reconstructed Logit gradually increases.

[0050] In this implementation, the model parameters have not yet converged in the early stage of training, and the student model prediction is unstable. At this time, retaining a high proportion of the original Logit information can be used to perform stable supervision by utilizing the complete discriminative structure of the original output. In the later stage of training, the model gradually converges, and the noise characteristics in the high-frequency components tend to stabilize. Increasing the proportion of reconstructed Logit can make fuller use of the high-quality structured semantic information after frequency decoupling and denoising, thereby improving distillation efficiency.

[0051] By dynamically adjusting the fusion coefficients, the model smoothly transitions from relying on the original Logit to relying on the reconstructed Logit, avoiding optimization oscillations caused by sudden changes in information sources during training. This gradual mechanism makes the distillation loss surface change more gently, which is beneficial for the optimizer to converge stably and reduces the sensitivity to hyperparameters such as the initial values ​​of the fusion coefficients and the learning rate.

[0052] In the early stages of training, the student model needs to learn basic category discrimination boundaries. The complete frequency information in the original Logit helps to establish basic discrimination ability. In the later stages of training, the student model needs to learn the discrimination details of the teacher model in a more refined way. The reconstructed Logit after denoising can provide purer structured semantic information and reduce the obstacles of noise interference to fine knowledge transfer.

[0053] Furthermore, according to one embodiment of the present invention, the knowledge distillation loss is constructed by performing temperature scaling on the target Logit corresponding to the teacher model and the target Logit corresponding to the student model respectively and calculating the probability distribution, then calculating the Kullback-Leibler divergence between the two probability distributions, and multiplying the divergence by the square of the temperature parameter.

[0054] In this implementation, when the temperature parameter T is greater than 1, the probability distribution of the softmax output tends to flatten, and categories with low confidence initially acquire non-zero probability values, allowing the inter-category similarity relationships implied in the teacher model to be explicitly expressed. The student model can learn the teacher model's tendency to discriminate against non-target categories, obtaining richer supervision signals than hard labels, and improving its ability to distinguish samples near category boundaries.

[0055] Kullback-Leibler divergence measures the information difference between two probability distributions, allowing the output probability distribution of the student model to approximate the output probability distribution of the teacher model in an information-theoretical sense. Compared to directly aligning Logit values, probability distribution alignment eliminates the influence of differences in the output scales of different models, making the distillation objective more stable and independent of model structure.

[0056] The temperature-scaled softmax gradient decays as temperature increases. Multiplying by the square of the temperature parameter can compensate for this decay effect, ensuring that the distillation loss still maintains an appropriate gradient contribution under temperature regulation, maintaining the numerical balance between distillation loss and supervision loss, and avoiding adverse effects of temperature parameter changes on optimization dynamics.

[0057] Furthermore, according to one embodiment of the present invention, the supervision loss is the cross-entropy loss between the student model output and the true label, and the total training loss is obtained by weighted summation of the knowledge distillation loss and the supervision loss.

[0058] In this implementation, cross-entropy loss directly measures the difference between the student model output and the true label, ensuring that the student model absorbs the knowledge of the teacher model without deviating from the distribution pattern of the real data, avoiding overfitting the prediction bias of the teacher model, and ensuring that the student model has an independent basis for discrimination.

[0059] By combining knowledge distillation loss and supervised loss through weighted summation, the training objective simultaneously considers two optimization directions: distillation loss guides the student model to learn the structured knowledge and inter-category relationships from the teacher model, while supervised loss ensures the student model's accurate prediction of the true labels. The two work synergistically to avoid optimization bias caused by a single loss method.

[0060] The balance coefficient in the weighted summation can adjust the relative contribution ratio of knowledge distillation loss and supervision loss to the overall training objective, enabling the training process to balance teacher knowledge transfer and real label fitting according to specific task requirements, thereby enhancing the method's adaptability to different distillation scenarios.

[0061] According to the above-described scheme of the present invention, the present invention achieves explicit decoupling of structural semantic information and high-frequency perturbation information in the Logit by decomposing the Logit of the teacher model and the student model into the frequency domain for structured processing; it decomposes the Logit into low-frequency and high-frequency components using Haar wavelet decomposition, and suppresses high-frequency noise through an adaptive soft thresholding denoising mechanism, while eliminating Logit scale differences through normalization processing; finally, it adaptively adjusts the weight ratio of the original Logit and the reconstructed Logit at different training stages through a dynamic fusion mechanism, thereby significantly improving the stability of the knowledge distillation process and the efficiency of knowledge transfer without changing the network structure, enabling the student model to effectively learn the discriminative knowledge of the teacher model and suppress noise interference. It is applicable to various visual task scenarios such as image classification, object detection, fine-grained recognition, and heterogeneous network knowledge transfer.

[0062] Furthermore, to achieve the above objectives, the present invention also provides a frequency decoupling knowledge distillation system based on wavelet decomposition, comprising: The Logit acquisition module acquires input samples and inputs them into the teacher model and student model respectively, to obtain the teacher Logit output by the teacher model and the student Logit output by the student model. The frequency decomposition module performs wavelet decomposition on the teacher's Logit and / or the student's Logit, decomposing the Logit into low-frequency components representing structural semantic information and high-frequency components representing local change information. The denoising module performs adaptive soft threshold denoising on the high-frequency components to obtain the denoised high-frequency components. The adaptive soft threshold denoising process determines the denoising threshold based on the statistical characteristics of the high-frequency components. The reconstruction module performs an inverse wavelet transform based on the low-frequency components and the denoised high-frequency components to obtain the reconstructed Logit. The normalization and fusion module performs normalization processing on the reconstructed Logit and the teacher Logit and / or student Logit as the original Logit, respectively. Based on the fusion coefficient that changes dynamically with the training phase, the normalized original Logit and the normalized reconstructed Logit are weighted and fused to obtain the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, respectively. The training module constructs a knowledge distillation loss based on the target Logit of the teacher model and the target Logit of the student model, and combines it with the supervised loss corresponding to the real labels of the input samples to train the student model.

[0063] The frequency decoupling knowledge distillation system based on wavelet decomposition according to the present invention can realize the frequency decoupling knowledge distillation method based on wavelet decomposition. The specific process steps are as described above and will not be repeated here.

[0064] Furthermore, to achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the frequency decoupling knowledge distillation method based on wavelet decomposition as described above.

[0065] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the frequency decoupling knowledge distillation method based on wavelet decomposition as described above.

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of the invention and are only used to explain the invention. They do not limit the scope of protection of the invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0067] Example 1 This embodiment addresses the problems of large Logit scale variations, strong high-frequency perturbation interference, and coupling of structural semantic information with noise information in traditional Logit-based knowledge distillation methods. It proposes a frequency-decoupled knowledge distillation method based on wavelet decomposition. This method can be embedded as an independent Logit processing module into existing knowledge distillation frameworks without altering the network structures of the teacher and student models. Specifically, it includes the following steps: Step 1: Obtain the input samples and input them into the teacher model and student model respectively to obtain the teacher Logit output by the teacher model and the student Logit output by the student model.

[0068] Let the input sample be X, and the teacher model be denoted as X. The student model is denoted as The total number of categories is K. Then the teacher Logit output by the teacher model is denoted as... The student's Logit output by the student model is denoted as Specifically, it is expressed as: ; in, That is, both the teacher model and the student model output a Logit vector of length K, where each element in the vector corresponds to the unnormalized score of a category.

[0069] Step 2, as follows Figure 2 As shown, wavelet decomposition is performed on the teacher's Logit and / or the student's Logit to decouple the original Logit into low-frequency and high-frequency components.

[0070] Let Z be the Logit vector to be processed, which can be the teacher's Logit. Student Logit Alternatively, the two can be processed independently. First, the Logit vector Z is divided into even-position subvectors and odd-position subvectors according to the parity of the index, denoted as follows: and Then, Haar wavelet decomposition was used to decompose Logit to obtain the low-frequency components. and high frequency components The calculation method is as follows: ; Among them, low-frequency components It mainly represents the overall structural semantic information in Logit, reflecting the relatively stable global relationships between categories; high-frequency components It mainly represents local change information in Logit, which usually includes detailed changes near the class boundary and high-frequency perturbation information introduced by unstable predictions of the model.

[0071] Step 3, as follows Figure 3 As shown, adaptive soft threshold denoising is performed on high-frequency components to reduce the interference of high-frequency disturbances on the distillation process. Figure 3 The horizontal axis of both the high-frequency curve with medium noise and the curve after denoising is the index of the high-frequency component, and the vertical axis is the amplitude of the high-frequency component.

[0072] Let the high-frequency component be The denoised high-frequency components are denoted as In this embodiment, a soft threshold function is used to process high-frequency components, and its expression is as follows: ; in, For symbolic functions, Represents the absolute value of the high-frequency component. An adaptive threshold is used. When the absolute value of a high-frequency component at a certain position is less than the threshold... When the positional component is compressed to zero or near zero; when its absolute value is greater than a threshold When the threshold is exceeded, the portion exceeding the threshold is retained, thereby suppressing low-amplitude noise disturbances and retaining high-amplitude effective information.

[0073] In this embodiment, the adaptive threshold The value is determined based on the absolute value statistics of high-frequency components and the dynamic determination during the training phase. Let the maximum value of the absolute value of the high-frequency components be... The median is Then the threshold It can be represented as: ; in, and These are weighting parameters used to adjust the contributions of different statistics. This is the dynamic decay factor used to characterize the current training phase. `max(·)` represents taking the maximum absolute value of the high-frequency components, and `median(·)` represents taking the median absolute value of the high-frequency components. By simultaneously utilizing the maximum and median values, this invention can take into account both extreme perturbations and overall distribution trends, making the threshold more adaptive and thus improving the stability of the denoising effect.

[0074] Step 4, as follows Figure 4 As shown, inverse wavelet reconstruction is performed based on the low-frequency components and the denoised high-frequency components to obtain the reconstructed Logit.

[0075] Let the low-frequency component be The high-frequency components after noise reduction are Then, the even-position components and odd-position components are reconstructed by inverse Haar wavelet decomposition, denoted as follows: and The calculation method is as follows: ; In obtaining and Then, the original Logit vectors are interleaved according to their index order to obtain the reconstructed Logit, denoted as . The reconstructed Logit retains the structural semantic information in the low-frequency components while suppressing high-frequency noise disturbances to some extent.

[0076] Step 5: Normalize the reconstructed Logit and the original Logit to reduce training instability caused by differences in Logit scale between different samples or different models.

[0077] Let the refactored Logit be Then its normalization result is denoted as The calculation method is as follows: ; in, This represents the mean of the reconstructed Logit. This represents the standard deviation of the refactored Logit.

[0078] Accordingly, the same normalization process is performed on the original Logit z to obtain the normalized original Logit. Its expression is: ; The above normalization process can effectively eliminate Logit amplitude differences and improve numerical stability during distillation training.

[0079] Step 6, as follows Figure 4 As shown, based on the fusion coefficients corresponding to the training phase, the normalized original Logit and the normalized reconstructed Logit are weighted and fused to obtain the target Logit for distillation.

[0080] Let the current training step number be t, and the fusion coefficient be denoted as . The fusion coefficient changes dynamically with the training process and can be expressed as follows: ; in, This represents the initial value of the fusion coefficient. Indicates the growth rate of the fusion coefficient. This represents the upper limit of the fusion coefficient, min(·) means taking the minimum value of each term within the parentheses, and t is the current training step number. Let be the fusion coefficient corresponding to the t-th training step, and its value shall not exceed 100%. This dynamic design allows the model to retain more stable information from the original Logit in the early stages of training, while gradually increasing the proportion of the reconstructed Logit after frequency decoupling and denoising in the later stages of training.

[0081] Based on the fusion coefficient, the final goal is Logit Represented as: ; in, The original Logit after normalization. This is the normalized refactored Logit.

[0082] Step 7: Construct the knowledge distillation loss based on the target Logit corresponding to the teacher model and the target Logit corresponding to the student model.

[0083] Let the final objective of the teacher model be Logit. The final goal of the student model is Logit. Before constructing the distillation loss, the target Logit is first processed using temperature-scaled softmax. Let the temperature parameter be T, then the probability distributions of the teacher model and the student model are denoted as follows: and The probabilities of its Kth category are expressed as follows: ; Where exp(·) represents the natural exponential function, k represents the class index corresponding to the current calculated probability, m represents the class index in the denominator summation process, and both k and m are integers between 1 and K, where K represents the total number of classes. After obtaining the temperature-scaled probability distributions corresponding to the teacher and student models, the knowledge distillation loss is constructed using Kullback-Leibler divergence, denoted as . Its form of expression is: ; in, Indicates the Kullback-Leibler divergence. This is the compensation factor under temperature scaling. and These represent the complete probability distributions of the teacher model and the student model obtained by temperature scaling, respectively.

[0084] Step 8: Train the student model by combining the supervised loss corresponding to the real labels of the input samples.

[0085] Let the true label of the input sample be y, and the predicted probability of the student model be... The supervision loss is the cross-entropy loss, denoted as... Its expression is: ; Ultimately, total training losses It is obtained by weighted summation of knowledge distillation loss and supervision loss, and is expressed as: ; Here, β is the balance coefficient, used to adjust the weight between teacher supervision and real label supervision.

[0086] Step 9: Update the student model parameters using the total training loss, and repeat steps 101 to 108 until training ends, obtaining the distilled student model.

[0087] This embodiment explicitly decouples the original Logit into low-frequency structural information and high-frequency perturbation information, and suppresses high-frequency noise through an adaptive soft threshold mechanism. At the same time, it combines normalization processing and dynamic fusion strategies to enhance the transferability of teachers' knowledge and improve the stability and accuracy of the knowledge distillation process.

[0088] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method implementation, and will not be repeated here.

[0090] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0091] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs.

[0092] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0093] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to conventional solutions, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the sending / receiving methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0094] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this application.

[0095] It should be understood that the sequence number of each step in the invention and its embodiments does not absolutely imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Claims

1. A frequency decoupling knowledge distillation method based on wavelet decomposition, characterized in that, include: Obtain input samples and input them into the teacher model and student model respectively to obtain the teacher Logit output by the teacher model and the student Logit output by the student model; Wavelet decomposition is performed on the teacher's Logit and / or the student's Logit, decomposing the Logit into low-frequency components representing structural semantic information and high-frequency components representing local change information; An adaptive soft threshold denoising process is performed on the high-frequency components to obtain denoised high-frequency components, wherein the adaptive soft threshold denoising process determines the denoising threshold based on the statistical characteristics of the high-frequency components. Perform an inverse wavelet transform based on the low-frequency components and the denoised high-frequency components to obtain the reconstructed Logit; Normalization is performed on the reconstructed Logit and the teacher Logit and / or student Logit that serve as the original Logit, respectively. Based on the fusion coefficient that changes dynamically with the training phase, the normalized original Logit and the normalized reconstructed Logit are weighted and fused to obtain the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, respectively. The knowledge distillation loss is constructed based on the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, and the student model is trained by combining the supervised loss corresponding to the real label of the input sample.

2. The frequency decoupling knowledge distillation method based on wavelet decomposition according to claim 1, characterized in that, The wavelet decomposition employs Haar wavelet decomposition, including: The Logit to be decomposed is divided into even-position sub-vectors and odd-position sub-vectors according to the index; The sum of the even-position sub-vectors and the odd-position sub-vectors is taken as the low-frequency component, and the difference between the two is taken as the high-frequency component.

3. The frequency decoupling knowledge distillation method based on wavelet decomposition according to claim 1, characterized in that, Perform adaptive soft thresholding denoising on the high-frequency components, including: Obtain the absolute value distribution statistics of the high-frequency components; Determine the adaptive threshold based on the absolute value distribution statistics; The high-frequency components are subjected to soft threshold shrinkage using the adaptive threshold.

4. The frequency decoupling knowledge distillation method based on wavelet decomposition according to claim 3, characterized in that, The adaptive threshold is determined by the following factors: The maximum value of the absolute value of the high-frequency component; The median of the absolute values ​​of the high-frequency components; Weight parameters used to adjust the contribution of the maximum value and the median value; A dynamic decay factor used to characterize changes during the training phase.

5. The frequency decoupling knowledge distillation method based on wavelet decomposition according to claim 3, characterized in that, The soft threshold shrinkage process is as follows: elements in the high-frequency components whose absolute value is lower than the adaptive threshold are set to zero or decayed to zero, while elements whose absolute value is higher than the adaptive threshold are retained or reduced towards zero.

6. The frequency decoupling knowledge distillation method based on wavelet decomposition according to claim 1, characterized in that, The fusion coefficient changes dynamically with the training phase. In the early stage of training, the fusion weight of the normalized original Logit is greater than the fusion weight of the normalized reconstructed Logit. In the later stage of training, the fusion weight of the normalized reconstructed Logit gradually increases.

7. The frequency decoupling knowledge distillation method based on wavelet decomposition according to claim 1, characterized in that, The knowledge distillation loss is constructed as follows: temperature scaling is applied to the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, and the probability distribution is calculated. Then, the Kullback-Leibler divergence between the two probability distributions is calculated, and the divergence is multiplied by the square of the temperature parameter.

8. The frequency decoupling knowledge distillation method based on wavelet decomposition according to claim 1, characterized in that, The supervised loss is the cross-entropy loss between the student model output and the true label, and the total training loss is obtained by weighted summation of the knowledge distillation loss and the supervised loss.

9. A frequency decoupling knowledge distillation system based on wavelet decomposition, characterized in that, include: The Logit acquisition module acquires input samples and inputs the input samples into the teacher model and student model respectively to obtain the teacher Logit output by the teacher model and the student Logit output by the student model. The frequency decomposition module performs wavelet decomposition on the teacher's Logit and / or the student's Logit, decomposing the Logit into low-frequency components representing structural semantic information and high-frequency components representing local change information. The denoising module performs adaptive soft threshold denoising on the high-frequency components to obtain denoised high-frequency components, wherein the adaptive soft threshold denoising determines the denoising threshold based on the statistical characteristics of the high-frequency components. The reconstruction module performs an inverse wavelet transform based on the low-frequency components and the denoised high-frequency components to obtain the reconstructed Logit. The normalization and fusion module performs normalization processing on the reconstructed Logit and the teacher Logit and / or student Logit that serve as the original Logit, respectively. Based on the fusion coefficient that changes dynamically with the training phase, the normalized original Logit and the normalized reconstructed Logit are weighted and fused to obtain the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, respectively. The training module constructs a knowledge distillation loss based on the target Logit corresponding to the teacher model and the target Logit corresponding to the student model, and trains the student model by combining the supervised loss corresponding to the real labels of the input samples.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the frequency decoupling knowledge distillation method based on wavelet decomposition as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the frequency decoupling knowledge distillation method based on wavelet decomposition as described in any one of claims 1-8.

Images