Interactive knowledge distillation algorithm based on lightweight model

Abstract

The invention discloses an interactive knowledge distillation algorithm based on a lightweight model, and the algorithm comprises the following specific steps: inputting an initial data set as a main supervision task, initializing an online distillation network, carrying out the data enhancement operation of the initial data set, and carrying out the data enhancement operation of the initial data set according to the network structure characteristics. And performing module division on the online distillation network, inputting a data set after data enhancement into each network block of the online distillation network, performing back propagation by using gradient information of a loss function through a chain derivation rule, and repeatedly executing the steps 4 to 5 until the loss function value converges or reaches a preset training period, and continuous iterative optimization is carried out. The problem of insufficient neural network performance is solved, so that the model can improve the network performance as much as possible under the condition of not changing the size of the model.

Description

Technical Field

[0001] The present invention relates to the design and application field of a neural network training model algorithm, and more specifically to an interactive knowledge distillation algorithm based on a lightweight model. Background Art

[0002] With the rapid development of deep learning technology, neural networks have been widely used in various fields, including image recognition, speech processing, natural language understanding, etc. By imitating the processing mechanism of the human brain, neural network models can learn complex patterns and features from large amounts of data. However, with the increasing complexity of model structures and the continuous increase in the scale of training data, improving the training efficiency and performance of neural networks has become one of the research hotspots and challenges. High-performance networks are usually large in size, which makes them difficult to deploy in resource-constrained environments. On the contrary, lightweight networks that are easy to deploy often have unsatisfactory performance. Therefore, in resource-constrained environments, it is particularly important to improve the accuracy of small models.

[0003] Chinese Patent: 201910382276.3 Mobile intelligent interactive learning system and learning method for rail transit knowledge. This patent document discloses an intelligent interactive learning system and a learning method, which uses data stream technology to achieve rapid real-time transmission of information, making the mobile terminal lightweight to meet the user's needs for fragmented learning anytime and anywhere. At the same time, the user may also input an interactive control signal to obtain the desired learning resources or AR three-dimensional model to achieve interactive learning of knowledge on the mobile terminal.

[0004] Chinese Patent: 202410515382.5 Interactive segmentation method, device, equipment and storage medium. The learning system disclosed in this patent document provides a data basis for guiding the model to learn granularity control capabilities through mask granularity pairs of multiple granularities, so that the model can quickly master the granularity control capabilities. By monitoring the user's area selection position and area selection granularity in the preset area, the interactive segmentation model after controllable training can output a single segmentation mask according to the user's click position and quantitative segmentation granularity, so that the user can accurately control the segmentation granularity of the model and better solve the problem of spatial ambiguity.

[0005] In order to solve the problem of insufficient performance of neural networks and enable the model to improve network performance as much as possible without changing its own size, the distillation method proposed in this technical solution is used to train a lightweight neural network model in order to achieve efficient deployment and operation in a resource-constrained environment. The online distillation network is first divided into modules according to the characteristics of the network structure, and then the data set after data enhancement is input into each network block of the online distillation network. The logit output is obtained between each network block and the other network block through a branch structure, and an interactive distillation method is constructed to perform cross-level KD distillation operations to realize the transfer of cross-layer knowledge. Then, by realizing logit distillation between backbone networks and cross entropy between labels, the model's learning ability for complex features is enhanced and the overall performance is improved. Summary of the invention

[0006] The present invention discloses an interactive knowledge distillation algorithm based on a lightweight model, the main purpose of which is to overcome the above-mentioned deficiencies and shortcomings of the prior art.

[0007] The technical solution adopted by the present invention is as follows:

[0008] An interactive knowledge distillation algorithm based on a lightweight model, the specific steps of the algorithm are as follows:

[0009] Step 1: Input the initial dataset as the main supervision task;

[0010] Step 2: Initialize the online distillation network and perform data augmentation on the initial dataset;

[0011] Step 3: Divide the online distillation network into modules according to the characteristics of the network structure;

[0012] Step 4: Input the data-augmented dataset into each network block of the online distillation network. In each network, each network block obtains logit outputs from the other network block through a branching structure, and constructs an interactive distillation method to perform cross-level KD distillation operations to achieve cross-monster knowledge transfer; obtain the deep feature output and logit output of the network, build an adaptive logic fusion module, calculate the importance score of each network logit output and calculate the weighted fusion logit output, and use logit distillation to achieve interactive knowledge transfer between networks; through logit distillation between backbone networks and cross entropy between labels, enhance the model's learning ability for complex features and improve overall performance;

[0013] Step 5: Use the gradient information of the loss function to perform back propagation through the chain rule;

[0014] Step 6: Repeat steps 4 to 5 until the loss function value converges or reaches the preset training cycle. Through continuous iterative optimization, the model gradually adapts to the characteristics of the training data and improves its performance in practical applications.

[0015] Furthermore, the main supervision task in step 1 is the original prediction process of the network, which is specifically the following steps:

[0016] A. For the original supervised task, when considering a dataset X with a label space Y = {1, 2, ..., n}, each input sample x∈X is labeled y∈Y, where n represents the total number of categories;

[0017] B. The goal is to train a network that maps input x to its predicted class probability distribution;

[0018] C. The network consists of a feature extractor and a linear classifier W;

[0019] D. During the processing, First, encode the input x into a feature embedding vector Here d is the dimension of the embedding;

[0020] E. Linear matrix W∈R d×n Convert the feature vector into a class logit distribution;

[0021] F. For each category i, the probability of the category is calculated by combining the Softmax function with the temperature parameter τ, which is specifically expressed as:

[0022] Among them, z i The input sample x is passed through the network feature extractor The predicted value of the i-th category in the logit output obtained after the linear classifier W, τ is the temperature parameter, and n is the total number of categories.

[0023] Furthermore, the data augmentation operation in step 2 is to rotate the original data set to generate a transformation result about the original image, specifically in the following form:

[0024] A. First, define m rotation operations for the original image Including non-rotation C1(x), rotation of 90 degrees C2(x), rotation of 180 degrees C3(x), and rotation of 270 degrees C4(x);

[0025] B. The neural network model outputs an m-dimensional probability distribution for the rotated image x, which reflects the predicted confidence of various rotation types;

[0026] C. The category probability distribution of the transformed image is calculated by the following Softmax formula:

[0027]

[0028] Among them, z i (C k (x)) represents the input sample x through the network feature extractor The logit output obtained after processing with the linear classifier W corresponds to C k The predicted value of the i-th category of the transformed image x, τ is the temperature parameter used to adjust the smoothness of the probability distribution.

[0029] Furthermore, each network block in step 4 constructs an interactive distillation method with the other network block to perform a cross-level distillation operation, and the specific steps are as follows:

[0030] A. Training goal: Use the branch structure to make the network block of network S1 guide the low-level network block of network S2, and vice versa, so as to achieve effective transfer of deep knowledge to shallow layers, and enhance the probability distribution of self-supervision generated by the lth network block of network S1. Used to guide the first l-1 network blocks of network S2, forming a bottom-up guidance mechanism;

[0031] B. This process is implemented by measuring the difference between the two logical output distributions through the KL divergence loss function, which is specifically expressed as:

[0032]

[0033] Among them, D KL represents the KL divergence between probability distributions, and τ is a temperature parameter used to adjust the degree of loss smoothing.

[0034] Furthermore, in step 4, the adaptive logic fusion module combines the feature output and logit output of the deep network block 3 of the online distillation network, calculates the importance score of each network output and performs weighted fusion. The specific steps are as follows:

[0035] During the network forward propagation process, extract the feature output of the online distillation network And the corresponding logit output The feature outputs of the two networks are superimposed in units of channel dimensions to fully retain the feature information in each network:

[0036]

[0037] Then, the convolution regulator is used to adjust f combine Processing is performed to reduce the channel dimension and extract a more compact feature representation: f extracted=Conv(f combine ),

[0038] Among them, Conv represents the convolution regulator, and the adaptive logic fusion module uses the fully connected layer to extracted Analyze and get the importance score of each network. These importance scores are normalized by Softmax to get W = Softmax (FC attention (f extracted )),

[0039] According to the calculated importance score, the logit outputs of the two networks are weighted and fused to generate the final fused logit output q fusion =W1·l l +W2·l2

[0040] Where W = [W1, W2] is a two-dimensional vector representing the importance scores of the two network output feature maps;

[0041] The logit output of the adaptive logic fusion module guides the online distillation network, which itself is constrained by the knowledge from the main supervision task. The loss L of the adaptive logic fusion module A For: L A =L CE (q fusion , Y);

[0042] Among them, Y (One-hot) represents the true label of the main supervision task, and L CE represents the cross entropy loss.

[0043] Furthermore, the loss calculation in step 4 is specifically expressed as:

[0044] For the main supervision task, the cross entropy loss L N for:

[0045] in, Represents the logit output of the backbone network, L CE is the cross entropy loss function, and Y represents the true label;

[0046] The cross entropy loss of the auxiliary branch for the main task can be expressed as:

[0047]

[0048] Among them, C k (x) is the kth self-supervised enhancement transformation applied to input sample x;

[0049] For deep network blocks, they are constrained by the logit output from the adaptive logic fusion module, which is expressed as the loss L ALM :

[0050] Among them, q3(C k (x); τ) is the logit output of the deep network block, D KL represents KL divergence;

[0051] For the self-supervised auxiliary enhancement module loss The proposed method adopts an interactive distillation mechanism to allow auxiliary branches at different levels of the network to learn and guide each other. This process is implemented by measuring the difference between the two logistic output distributions through the KL divergence loss function, which is specifically expressed as:

[0052]

[0053] For the mutual learning mechanism between backbone networks, it allows networks S1 and S2 to guide and optimize each other through each other’s prediction results. The loss is further expressed as:

[0054]

[0055] Finally, knowledge distillation is used to train and optimize the network model, and the overall loss adopted is:

[0056] L total =L N +L aux +λL ALM +L N_KL +L A

[0057] Among them, L N represents the cross entropy loss of the backbone network for the main task, ensuring that the network can accurately learn the mapping from raw data to target labels; L aux represents the auxiliary branch self-supervised enhancement task loss, including the cross entropy loss of the main task and the cross entropy loss of the data enhancement dataset, which promotes the network to enhance feature expression through auxiliary tasks and self-supervised learning; L ALM Introduced by the adaptive logic fusion module, it dynamically adjusts the contribution of the final logic output by considering the relative importance of different auxiliary branches, enhancing the deep knowledge exchange between networks; N-KL represents the mutual learning loss between backbone networks, and measures the similarity of prediction results between backbone networks through the KL divergence function, promoting knowledge sharing and mutual learning between networks; L A represents the loss of the adaptive logic fusion module; λ is a hyperparameter set artificially.

[0058] It can be seen from the above description and explanation of the present invention that, compared with the prior art, the advantages of the present invention are:

[0059] Advantage 1: The present invention divides the online distillation network into modules based on the characteristics of the network structure, and gradually deepens the level of extraction from the network features. This hierarchical design helps to refine the feature extraction process and optimize the learning efficiency and output accuracy of each layer.

[0060] Advantage 2: The present invention improves the model's ability to understand input data and provides more hidden knowledge, performs rotation transformation to generate a series of preprocessed images as network input, and improves the model's ability to recognize images from different angles and perspectives.

[0061] Advantage 3: The present invention enhances the model's ability to learn complex features and improves overall performance by implementing logit distillation between backbone networks and cross entropy between labels.

[0062] Advantage 4: The gradient calculated by the present invention is used to update the weights of the network to reduce prediction errors and optimize network performance. Through continuous iterative optimization, the model gradually adapts to the characteristics of the training data and improves its performance in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a structural framework diagram of the present invention. DETAILED DESCRIPTION

[0064] The specific implementation of the present invention will be further described and illustrated below with reference to the accompanying drawings.

[0065] like Figure 1 As shown, an interactive knowledge distillation algorithm based on a lightweight model, the specific steps of the algorithm are as follows:

[0066] Step 1: Input the initial dataset as the main supervision task;

[0067] Step 2: Initialize the online distillation network and perform data augmentation on the initial dataset;

[0068] Step 3: Divide the online distillation network into modules according to the characteristics of the network structure;

[0069] Step 4: Input the data-augmented dataset into each network block of the online distillation network. In each network, each network block obtains logit outputs from the other network block through a branching structure, and constructs an interactive distillation method to perform cross-level KD distillation operations to achieve cross-monster knowledge transfer; obtain the deep feature output and logit output of the network, build an adaptive logic fusion module, calculate the importance score of each network logit output and calculate the weighted fusion logit output, and use logit distillation to achieve interactive knowledge transfer between networks; through logit distillation between backbone networks and cross entropy between labels, enhance the model's learning ability for complex features and improve overall performance;

[0070] Step 5: Use the gradient information of the loss function to perform back propagation through the chain rule;

[0071] Step 6: Repeat steps 4 to 5 until the loss function value converges or reaches the preset training cycle. Through continuous iterative optimization, the model gradually adapts to the characteristics of the training data and improves its performance in practical applications.

[0072] Furthermore, the main supervision task in step 1 is the original prediction process of the network, which is specifically the following steps:

[0073] A. For the original supervised task, when considering a dataset X with a label space Y = {1, 2, ..., n}, each input sample x∈X is labeled y∈Y, where n represents the total number of categories;

[0074] B. The goal is to train a network that maps input x to its predicted class probability distribution;

[0075] C. The network consists of a feature extractor and a linear classifier W;

[0076] D. During the processing, First, encode the input x into a feature embedding vector Here d is the dimension of the embedding;

[0077] E. Linear matrix W∈R d×n Convert the feature vector into a class logit distribution;

[0078] F. For each category i, the probability of the category is calculated by combining the Softmax function with the temperature parameter τ, which is specifically expressed as:

[0079] Among them, z i The input sample x is passed through the network feature extractor The predicted value of the i-th category in the logit output obtained after the linear classifier W, τ is the temperature parameter, and n is the total number of categories.

[0080] Furthermore, the data augmentation operation in step 2 is to rotate the original data set to generate a transformation result about the original image, specifically in the following form:

[0081] A. First, define m rotation operations for the original image Including non-rotation C1(x), rotation of 90 degrees C2(x), rotation of 180 degrees C3(x), and rotation of 270 degrees C4(x);

[0082] B. The neural network model outputs an m-dimensional probability distribution for the rotated image x, which reflects the predicted confidence of various rotation types;

[0083] C. The category probability distribution of the transformed image is calculated by the following Softmax formula:

[0084]

[0085] Among them, z i (C k (x)) represents the input sample x through the network feature extractor The logit output obtained after processing with the linear classifier W corresponds to C k The predicted value of the i-th category of the transformed image x, τ is the temperature parameter used to adjust the smoothness of the probability distribution.

[0086] Furthermore, each network block in step 4 constructs an interactive distillation method with the other network block to perform a cross-level distillation operation, and the specific steps are as follows:

[0087] A. Training goal: Use the branch structure to make the network block of network S1 guide the low-level network block of network S2, and vice versa, so as to achieve effective transfer of deep knowledge to shallow layers, and enhance the probability distribution of self-supervision generated by the lth network block of network S1. Used to guide the first l-1 network blocks of network S2, forming a bottom-up guidance mechanism;

[0088] B. This process is implemented by measuring the difference between the two logical output distributions through the KL divergence loss function, which is specifically expressed as:

[0089]

[0090] Among them, D KL represents the KL divergence between probability distributions, and τ is a temperature parameter used to adjust the degree of loss smoothing.

[0091] Furthermore, in step 4, the adaptive logic fusion module combines the feature output and logit output of the deep network block 3 of the online distillation network, calculates the importance score of each network output and performs weighted fusion. The specific steps are as follows:

[0092] During the network forward propagation process, extract the feature output of the online distillation network And the corresponding logit output The feature outputs of the two networks are superimposed in units of channel dimensions to fully retain the feature information in each network:

[0093]

[0094] Then, the convolution regulator is used to adjust f combine Processing is performed to reduce the channel dimension and extract a more compact feature representation: f extracted =Conv(f combine ),

[0095] Among them, Conv represents the convolution regulator, and the adaptive logic fusion module uses the fully connected layer to extracted Analyze and get the importance score of each network. These importance scores are normalized by Softmax to get W = Softmax (FC attention (f extracted )),

[0096] According to the calculated importance score, the logit outputs of the two networks are weighted and fused to generate the final fused logit output q fusion =W1·l l +W2·l2

[0097] Where W = [W1, W2] is a two-dimensional vector representing the importance scores of the two network output feature maps;

[0098] The logit output of the adaptive logic fusion module guides the online distillation network, which itself is constrained by the knowledge from the main supervision task. The loss L of the adaptive logic fusion module A For: L A =L CE (q fusion , Y);

[0099] Among them, Y (One-hot) represents the true label of the main supervision task, and L CE represents the cross entropy loss.

[0100] Furthermore, the loss calculation in step 4 is specifically expressed as:

[0101] For the main supervision task, the cross entropy loss LN for:

[0102] in, Represents the logit output of the backbone network, L CE is the cross entropy loss function, and Y represents the true label;

[0103] The cross entropy loss of the auxiliary branch for the main task can be expressed as:

[0104]

[0105] Among them, C k (x) is the kth self-supervised enhancement transformation applied to input sample x;

[0106] For deep network blocks, they are constrained by the logit output from the adaptive logic fusion module, which is expressed as the loss L ALM :

[0107] Among them, q3(C k (x); τ) is the logit output of the deep network block, D KL represents KL divergence;

[0108] For the self-supervised auxiliary enhancement module loss The proposed method adopts an interactive distillation mechanism to allow auxiliary branches at different levels of the network to learn and guide each other. This process is implemented by measuring the difference between the two logistic output distributions through the KL divergence loss function, which is specifically expressed as:

[0109]

[0110] For the mutual learning mechanism between backbone networks, it allows networks S1 and S2 to guide and optimize each other through each other’s prediction results. The loss is further expressed as:

[0111]

[0112] Finally, knowledge distillation is used to train and optimize the network model, and the overall loss adopted is:

[0113] L total =L N +L aux +λL ALM +L N_KL +L A

[0114] Among them, L N represents the cross entropy loss of the backbone network for the main task, ensuring that the network can accurately learn the mapping from raw data to target labels; Laux represents the auxiliary branch self-supervised enhancement task loss, including the cross entropy loss of the main task and the cross entropy loss of the data enhancement dataset, which promotes the network to enhance feature expression through auxiliary tasks and self-supervised learning; L ALM Introduced by the adaptive logic fusion module, it dynamically adjusts the contribution of the final logic output by considering the relative importance of different auxiliary branches, enhancing the deep knowledge exchange between networks; N-KL represents the mutual learning loss between backbone networks, and measures the similarity of prediction results between backbone networks through the KL divergence function, promoting knowledge sharing and mutual learning between networks; L A represents the loss of the adaptive logic fusion module; λ is a hyperparameter set artificially.

[0115] Example:

[0116] Take the MobileNetV2 network model as an example, and combine the interactive distillation method and the adaptive logic fusion module to illustrate:

[0117] The online distillation network model is MobileNetV2, which has a network parameter volume of approximately 3.4M. The purpose is to ensure that the model can run smoothly on resource-constrained devices while maintaining high performance. The model is initialized for training and distillation processes in subsequent steps.

[0118] like Figure 1 As shown in the figure, the input of the network is the original image. In order to improve the model's ability to understand the input data and provide more hidden knowledge, a rotation transformation is performed to generate a series of pre-processed images as network input. This step helps improve the model's ability to recognize images from different angles and perspectives.

[0119] For each peer student module in the network, we first determine whether to perform the distillation process. If distillation is performed, we select a lower-level network block in the other network for knowledge transfer and distillation operations based on the depth of the network block.

[0120] At the end of the MobileNetV2 network, the feature output and logit output of the last network block of the same level network are received and processed by the adaptive logic fusion module. The feature output is reduced in dimension and scored for importance through the fusion strategy. Based on the scoring results, the logit is weighted and fused to form the final fused logit output.

[0121] The KL divergence between the deep network block logit output and the fused logit output is calculated. In addition, the mutual KL divergence between the output logits of the backbone network and the cross entropy between the labels are also included. By combining these losses, the final loss function is formed for back propagation and network parameter update.

[0122] Data augmentation, forward propagation, distillation, adaptive logic fusion, loss calculation, and backpropagation steps are repeated until the set number of training iterations is completed. After training, the model is evaluated, and the number of parameters and size of the model remain unchanged, while the performance is improved.

[0123] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited to this. Any non-substantial improvement of the present invention using this concept should be regarded as an infringement of the protection scope of the present invention.

Claims

1. An interactive knowledge distillation algorithm based on a lightweight model, characterized by: The specific steps of the algorithm are as follows: Step 1: Input the initial dataset as the main supervision task; Step 2: Initialize the online distillation network and perform data augmentation on the initial dataset; Step 3: Divide the online distillation network into modules according to the characteristics of the network structure; Step 4: Input the data-augmented dataset into each network block of the online distillation network. In each network, each network block obtains logit outputs from the other network block through a branching structure, and constructs an interactive distillation method to perform cross-level KD distillation operations to achieve cross-monster knowledge transfer; obtain the deep feature output and logit output of the network, build an adaptive logic fusion module, calculate the importance score of each network logit output and calculate the weighted fusion logit output, and use logit distillation to achieve interactive knowledge transfer between networks; through logit distillation between backbone networks and cross entropy between labels, enhance the model's learning ability for complex features and improve overall performance; Step 5: Use the gradient information of the loss function to perform back propagation through the chain rule; Step 6: Repeat steps 4 to 5 until the loss function value converges or reaches the preset training cycle. Through continuous iterative optimization, the model gradually adapts to the characteristics of the training data and improves its performance in practical applications.

2. The interactive knowledge distillation algorithm based on a lightweight model according to claim 1, characterized in that: The main supervision task in step 1 is the original prediction process of the network, which is specifically the following steps: A. For the original supervised task, when considering a dataset X with a label space Y = {1, 2, ..., n}, each input sample x∈X is labeled y∈Y, where n represents the total number of categories; B. The goal is to train a network that maps input x to its predicted class probability distribution; C. The network consists of a feature extractor and a linear classifier W; D. During the processing, First, encode the input x into a feature embedding vector Here d is the dimension of the embedding; E. Linear matrix W∈R d×n Convert the feature vector into a class logit distribution; F. For each category i, the probability of the category is calculated by combining the Softmax function with the temperature parameter τ, which is specifically expressed as: Among them, z i The input sample x is passed through the network feature extractor The predicted value of the i-th category in the logit output obtained after the linear classifier W, τ is the temperature parameter, and n is the total number of categories.

3. The interactive knowledge distillation algorithm based on a lightweight model according to claim 1, characterized in that: The data enhancement operation in step 2 is to rotate the original data set to generate a transformation result about the original image, specifically in the following form: A. First, define m rotation operations for the original image Including non-rotation C1(x), rotation of 90 degrees C2(x), rotation of 180 degrees C3(x), and rotation of 270 degrees C4(x); B. The neural network model outputs an m-dimensional probability distribution for the rotated image x, which reflects the predicted confidence of various rotation types; C. The category probability distribution of the transformed image is calculated by the following Softmax formula: Among them, z i (C k (x)) represents the input sample x through the network feature extractor The logit output obtained after processing with the linear classifier W corresponds to C k The predicted value of the i-th category of the transformed image x, τ is the temperature parameter used to adjust the smoothness of the probability distribution.

4. The interactive knowledge distillation algorithm based on a lightweight model according to claim 1, characterized in that: In step 4, each network block and the other network block construct an interactive distillation mode to perform a cross-level distillation operation, and the specific steps are as follows: A. Training goal: Use the branch structure to make the network block of network S1 guide the low-level network block of network S2, and vice versa, so as to achieve effective transfer of deep knowledge to shallow layers, and enhance the probability distribution of self-supervision generated by the lth network block of network S1. Used to guide the first l-1 network blocks of network S2, forming a bottom-up guidance mechanism; B. This process is implemented by measuring the difference between the two logical output distributions through the KL divergence loss function, which is specifically expressed as: Among them, D KL represents the KL divergence between probability distributions, and τ is a temperature parameter used to adjust the degree of loss smoothing.

5. The interactive knowledge distillation algorithm based on a lightweight model according to claim 4, characterized in that: In step 4, the adaptive logic fusion module combines the feature output and logit output of the deep network block 3 of the online distillation network, calculates the importance score of each network output and performs weighted fusion. The specific steps are as follows: During the network forward propagation process, extract the feature output of the online distillation network And the corresponding logit output The feature outputs of the two networks are superimposed in units of channel dimensions to fully retain the feature information in each network: Then, the convolution regulator is used to adjust f combine Processing is performed to reduce the channel dimension and extract a more compact feature representation: f extracted =Conv(f combine ), Among them, Conv represents the convolution regulator, and the adaptive logic fusion module uses the fully connected layer to extracted Analyze and get the importance score of each network. These importance scores are normalized by Softmax to get W = Softmax (FC attention (f extracted )), According to the calculated importance score, the logit outputs of the two networks are weighted and fused to generate the final fused logit output q fusion =W1·l1+W2·l2 Where W = [W1, W2] is a two-dimensional vector representing the importance scores of the two network output feature maps; The logit output of the adaptive logic fusion module guides the online distillation network, which itself is constrained by the knowledge from the main supervision task. The loss L of the adaptive logic fusion module A For: L A =L CE (q fusion ,Y); Among them, Y (One-hot) represents the true label of the main supervision task, and L CE represents the cross entropy loss.

6. The interactive knowledge distillation algorithm based on a lightweight model according to claim 5, characterized in that: The loss calculation in step 4 is specifically expressed as: For the main supervision task, the cross entropy loss L N for: in, represents the logit output of the backbone network, L CE is the cross entropy loss function, and Y represents the true label; The cross entropy loss of the auxiliary branch for the main task can be expressed as: Among them, C k (x) is the kth self-supervised enhancement transformation applied to input sample x; For deep network blocks, they are constrained by the logit output from the adaptive logic fusion module, which is expressed as the loss L ALM : Among them, q3(C k (x); τ) is the logit output of the deep network block, D KL represents KL divergence; For the self-supervised auxiliary enhancement module loss The proposed method adopts an interactive distillation mechanism to allow auxiliary branches at different levels of the network to learn and guide each other. This process is implemented by measuring the difference between the two logistic output distributions through the KL divergence loss function, which is specifically expressed as: For the mutual learning mechanism between backbone networks, it allows networks S1 and S2 to guide and optimize each other through each other’s prediction results. The loss is further expressed as: Finally, knowledge distillation is used to train and optimize the network model, and the overall loss adopted is: THE total =L N +L aux +λL ALM +L N_KL +L A Among them, L N represents the cross entropy loss of the backbone network for the main task, ensuring that the network can accurately learn the mapping from raw data to target labels; L aux represents the auxiliary branch self-supervised enhancement task loss, including the cross entropy loss of the main task and the cross entropy loss of the data enhancement dataset, which promotes the network to enhance feature expression through auxiliary tasks and self-supervised learning; L ALM Introduced by the adaptive logic fusion module, it dynamically adjusts the contribution of the final logic output by considering the relative importance of different auxiliary branches, enhancing the deep knowledge exchange between networks; N-KL represents the mutual learning loss between backbone networks, and measures the similarity of prediction results between backbone networks through the KL divergence function, promoting knowledge sharing and mutual learning between networks; L A represents the loss of the adaptive logic fusion module; λ is a hyperparameter set artificially.

Images