A knowledge distillation method of an isomorphic teacher-student model

Abstract

This invention discloses a knowledge distillation method for isomorphic teacher-student models. It constructs distillation modules containing encoders and decoders for each pair of target feature layers between the teacher and student models. The encoders reduce the dimensionality of high-dimensional feature data output from the corresponding target feature layers in the teacher model to obtain latent feature data. The decoders increase the dimensionality of low-dimensional feature data output from the corresponding target feature layers in the student model to obtain reconstructed feature data. Based on the latent and reconstructed feature data of each pair of target feature layers, and combined with the task loss function of the student model, a total loss function is constructed. Finally, the total loss function is used to perform backpropagation training on the student model to update and adjust the parameters of each intermediate feature layer within the student model. This knowledge distillation method can achieve channel alignment of intermediate feature layers between the teacher and student models.

Description

Technical Field

[0001] This invention relates to artificial intelligence technology, and more particularly to a knowledge distillation method for isomorphic teacher-student models. Background Technology

[0002] Knowledge distillation is a model compression method based on knowledge transfer. Its core idea is to transfer the "knowledge" of a complex teacher model to a lightweight student model, so that the student model can maintain most of the performance of the teacher model while having a smaller size and faster inference speed.

[0003] The isomorphic teacher-student model refers to a teacher model and a student model that have the same network hierarchical structure, but the number of channels between their corresponding intermediate feature layers is different. For example, a student model is formed by lightweight processing such as channel pruning, width factor scaling, or automated channel search on the teacher model.

[0004] For the isomorphic teacher-student model, since the number of channels between the teacher model and the student model is different, channel alignment cannot be achieved during knowledge distillation, which affects the effect of knowledge distillation and makes it difficult for the student model to achieve the performance of the teacher model. Summary of the Invention

[0005] To address the shortcomings of the existing technology, this invention provides a knowledge distillation method for isomorphic teacher-student models, which can achieve channel alignment of intermediate feature layers between the teacher model and the student model, thereby improving the knowledge distillation effect.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solution: A knowledge distillation method for an isomorphic teacher-student model, comprising a teacher model and a student model, wherein the teacher model and the student model have the same network hierarchical structure, but at least one pair of intermediate feature layers with the same hierarchical position, and the teacher model and the student model have different numbers of channels in this at least one pair of intermediate feature layers; the knowledge distillation method includes the following steps: Step 10: Determine the pairs of intermediate feature layers that have the same hierarchical position but different number of channels between the teacher model and the student model as the target feature layers; Step 20: Construct a corresponding distillation module for each pair of target feature layers. Each distillation module consists of an encoder corresponding to the teacher model and a decoder corresponding to the student model. Each encoder is used to perform channel-based dimensionality reduction and compression on the high-dimensional feature data output by the corresponding target feature layer in the teacher model, so that it forms latent feature data with the same number of channels as the corresponding target feature layer in the student model. Each decoder is used to perform channel-based dimensionality up-reconstruction on the low-dimensional feature data output by the corresponding target feature layer in the student model, so that it forms reconstructed feature data with the same number of channels as the corresponding target feature layer in the teacher model. Step 30: Perform forward propagation inference on the teacher model and the student model, and obtain the high-dimensional feature data output by each target feature layer in the teacher model, and obtain the low-dimensional feature data output by each target feature layer in the student model; Step 40: Input each high-dimensional feature data into the corresponding encoder and obtain the latent feature data output by each encoder corresponding to each target feature layer in the teacher model. Input each low-dimensional feature data into the corresponding decoder and obtain the reconstructed feature data output by each decoder corresponding to each target feature layer in the student model. Step 50: Based on the latent feature data and reconstructed feature data of each pair of target feature layers, and in conjunction with the task loss function of the student model, construct the total loss function; Step 60: Use the total loss function to perform backpropagation training on the student model to update and adjust the parameters of each intermediate feature layer in the student model, so that the performance of the student model is close to that of the teacher model.

[0007] Furthermore, in step 10, the steps for determining the pairs of intermediate feature layers that have the same hierarchical position but different number of channels between the teacher model and the student model as the target feature layers are as follows: Step 11: Parse the configuration files of the teacher model and the student model, or traverse the network hierarchy structure of the teacher model and the student model to obtain the feature layer channel list of the teacher model and the student model respectively; Step 12: Based on the feature layer channel lists of the teacher model and the student model, compare the number of channels in the intermediate feature layers at the same level to determine whether the number of channels in the pair of intermediate feature layers is the same. This will identify the pairs of intermediate feature layers that are at the same level but have different numbers of channels, and these will be used as the target feature layers.

[0008] Furthermore, both the encoder and decoder are formed by stacking multiple layers of residual blocks, and the encoder and decoder corresponding to the same pair of target feature layers are structurally mirror symmetrical.

[0009] Furthermore, based on the number of channels in the same pair of target feature layers of the teacher model and student model, the number of residual blocks of the encoder and decoder corresponding to the pair of target feature layers is determined, and the number of channels of each residual block is determined by linear interpolation.

[0010] Furthermore, the number of residual blocks for the encoder and decoder within the same distillation module is as follows: in, and These represent the number of channels in the same pair of target feature layers for the teacher model and the student model, respectively. To find the maximum value function, To find the minimum value function, It is a rounding function; The number of channels in the i-th layer residual block of the encoder is as follows: The number of channels in the i-th layer residual block within the decoder is as follows: in, This is the floor function.

[0011] Furthermore, each residual block within the encoder includes a dimensionality reduction main path and a skip connection path. The dimensionality reduction main path includes a main convolutional layer, a batch normalization layer, and an activation function layer connected in sequence. The number of convolutional filters in the main convolutional layer of each residual block decreases in gradient. The skip connection path has a pre-set aligned convolutional layer, and the number of convolutional kernels between the aligned convolutional layer and the last main convolutional layer is the same. Each encoder processes the corresponding high-dimensional feature data sequentially through its internal residual blocks. When the high-dimensional feature data passes through a certain residual block, the dimensionality reduction main path of that residual block sequentially performs convolution, batch normalization, and nonlinear activation on the high-dimensional feature data and outputs the main feature data. The skip connection path of that residual block performs convolution on the high-dimensional feature data and outputs skip feature data. The skip feature data is then added to the main feature data and output.

[0012] Furthermore, each residual block within the decoder includes an upscaling main path and a skip connection path. The upscaling main path includes a main convolutional layer, a batch normalization layer, and an activation function layer connected in sequence. The number of convolutional kernels in the main convolutional layer of each residual block increases progressively. The skip connection path has a pre-set aligned convolutional layer, and the number of convolutional kernels between the aligned convolutional layer and the last main convolutional layer is the same. Each decoder processes the corresponding low-dimensional feature data sequentially through its internal residual blocks. When the low-dimensional feature data passes through a certain residual block, the upscaling main path of that residual block sequentially performs convolution, batch normalization, and nonlinear activation on the low-dimensional feature data and outputs the main feature data. The skip connection path of that residual block performs convolution on the low-dimensional feature data and outputs skip feature data. The skip feature data is then added to the main feature data and output.

[0013] Furthermore, in step 50, the steps for constructing the total loss function based on the latent feature data and reconstructed feature data of each pair of target feature layers, and in conjunction with the task loss function of the student model, are as follows: Step 51: Select an appropriate task loss function based on the functional tasks of the student model; Step 52: For each target feature layer in the teacher model, calculate the corresponding distillation loss value based on its corresponding high-dimensional feature data and latent feature data; and for each target feature layer in the student model, calculate the corresponding reconstruction loss value based on its corresponding low-dimensional feature data and reconstructed feature data. Step 53: Calculate the total distillation loss value based on the distillation loss value of each target feature layer in the teacher model, and calculate the total reconstruction loss value based on the reconstruction loss value of each target feature layer in the student model. Step 54: Based on the total distillation loss value and the total reconstruction loss value, and in conjunction with the selected task loss function, construct the total loss function.

[0014] Furthermore, the distillation loss value for each target feature layer within the teacher model is as follows: The reconstruction loss value for each target feature layer in the student model is as follows: in, The high-dimensional feature data output by each target feature layer within the teacher model. The low-dimensional feature data output by each intermediate feature layer in the student model. For the dimension reduction compression operation of the encoder, For the dimension-up reconstruction operation of the decoder, For normalization processing; The total distillation loss value of the teacher model is as follows: The total distillation loss of the student model is as follows: in, and These are the distillation loss value and reconstruction loss value of the i-th target feature layer in the teacher model and student model, respectively; The total loss function is as follows: in, Let be the task loss function of the student model. and These are the weighting coefficients for the distillation loss value and the reconstruction loss value, respectively, and N is the total number of layers in the target feature layer.

[0015] Furthermore, in step 30, forward propagation inference is performed on the teacher model and the student model, and in step 60, backpropagation training is performed on the student model, using the same training dataset.

[0016] The present invention has the following beneficial effects: The knowledge distillation method of the present invention introduces residual convolutional autoencoders into the field of feature distillation. By constructing a nonlinear mapping that includes encoders and decoders, the high-dimensional feature data of the teacher model is compressed layer by layer to a low-dimensional latent space that matches the student model, and the low-dimensional feature data of the student model is reconstructed back to the original high-dimensional space that matches the teacher model. This fundamentally solves the channel mismatch problem between the teacher model and the student model, enabling channel alignment of each intermediate feature layer between the teacher model and the student model. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the steps of the knowledge distillation method provided by this invention.

[0018] Figure 2 This is a flowchart of step 10 in the knowledge distillation method provided by the present invention.

[0019] Figure 3 This is a flowchart illustrating the steps of step 50 in the knowledge distillation method provided by the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and embodiments, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0021] In the description of this invention, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0022] Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature. In the description of this invention, "multiple" means two or more, unless otherwise explicitly specified.

[0023] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," and "setting," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] Example 1 like Figure 1 As shown, a knowledge distillation method for an isomorphic teacher-student model includes the following steps: Step 10: Identify the pairs of intermediate feature layers that have the same hierarchical position but different number of channels between the teacher model and the student model as the target feature layers.

[0025] In step 10, the isomorphic teacher-student model includes the teacher model and the student model. The teacher model and the student model have the same network hierarchy structure, but there is at least one pair of intermediate feature layers with the same hierarchical position. The teacher model and the student model have different numbers of channels on this at least one pair of intermediate feature layers. That is, on this at least one pair of intermediate feature layers, the teacher model has a first number of channels, and the student model has a second number of channels, where the first number of channels is greater than the second number of channels.

[0026] The student model can be formed by lightweight processing of the teacher model, such as channel pruning, width factor scaling, or automated channel search. The intermediate feature layer refers to the basic unit in the network responsible for generating intermediate feature maps, and may include, but is not limited to, convolutional layers, batch normalization layers, activation function layers, pooling layers, fully connected layers, residual connection layers, or PANet output layers of the teacher and student models, depending on the specific architecture of different network models.

[0027] Specifically, in step 10, the steps for determining the pairs of intermediate feature layers that have the same hierarchical position but different number of channels between the teacher model and the student model as the target feature layers are as follows: Step 11: Parse the configuration files of the teacher model and the student model, or traverse the network hierarchy structure of the teacher model and the student model to obtain the feature layer channel list of the teacher model and the student model respectively.

[0028] In step 11, the feature layer channel list of the teacher model records the name and number of channels of each intermediate feature layer in the teacher model. Similarly, the feature layer channel list of the student model records the name and number of channels of each intermediate feature layer in the student model.

[0029] Step 12: Based on the feature layer channel lists of the teacher model and the student model, compare the number of channels in the intermediate feature layers at the same level to determine whether the number of channels in the pair of intermediate feature layers is the same. This will identify the pairs of intermediate feature layers that are at the same level but have different numbers of channels, and these will be used as the target feature layers.

[0030] In step 12, the hierarchical position matching of intermediate feature layers between the teacher model and the student model is mainly based on the naming rules of each functional layer within the network model. For example, in the network model, the name of each functional layer is model.layer-nmope, where model is the model name, layer-nm indicates that the functional layer is the m-th sub-module in the n-th layer of the network model (some complex models may use two or more hierarchical structures), and ope indicates the type of operation performed by the functional layer, such as convolution operation (conv) or batch normalization (bn). In the feature layer channel lists of the teacher model and the student model, as long as the same "layer-nmope" is indexed, intermediate feature layers at the same hierarchical position can be matched between the two.

[0031] Step 20: Construct a corresponding distillation module for each pair of target feature layers. Each distillation module consists of an encoder corresponding to the teacher model and a decoder corresponding to the student model. Each encoder is used to perform channel-based dimensionality reduction and compression on the high-dimensional feature data output by the corresponding target feature layer in the teacher model, so that it forms potential feature data with the same number of channels as the corresponding target feature layer in the student model. Each decoder is used to perform channel-based dimensionality up-reconstruction on the low-dimensional feature data output by the corresponding target feature layer in the student model, so that it forms reconstructed feature data with the same number of channels as the corresponding target feature layer in the teacher model.

[0032] In step 20, both the encoder and decoder are formed by stacking multiple layers of residual blocks, and their structures are mirror-symmetrical with respect to the encoder and decoder corresponding to the same pair of target feature layers. That is, for the encoder and decoder corresponding to the same pair of target feature layers, the number of residual blocks is the same, but the number of channels in each layer of residual blocks is mirror-reversed. For example, the encoder has three residual blocks: the first layer has 256 channels, the second layer has 128 channels, and the third layer has 64 channels. Similarly, the decoder also has three residual blocks: the first layer has 64 channels, the second layer has 128 channels, and the third layer has 256 channels.

[0033] Specifically, the number of residual blocks of the encoder and decoder corresponding to the same pair of target feature layers can be determined based on the number of channels in the teacher model and student model on the same pair of target feature layers, and the number of channels in each residual block can be determined by linear interpolation.

[0034] Since the number of channels between each target feature layer in the teacher model may not be the same (they may be completely the same, partially the same, or completely different, depending on the original configuration of the teacher model), and the number of channels between each target feature layer in the student model may also not be the same (they may be completely the same, partially the same, or completely different, depending on the lightweight processing effect of the student model), the difference in the number of channels in each pair of target feature layers in the teacher model and the student model will naturally be different. Each pair of target feature layers needs to be configured with a different distillation module for knowledge distillation.

[0035] The number of residual blocks for the encoder and decoder within the same distillation module is as follows: in, and These represent the number of channels in the same pair of target feature layers for the teacher model and the student model, respectively. To find the maximum value function, To find the minimum value function, It is a rounding function; The number of channels in the i-th layer residual block of the encoder is as follows: The number of channels in the i-th layer residual block within the decoder is as follows: in, This is the floor function.

[0036] Each residual block within the encoder includes a dimensionality reduction main path and a skip connection path. The dimensionality reduction main path comprises a main convolutional layer, a batch normalization layer, and an activation function layer connected in sequence. The number of convolutional filters in the main convolutional layers of each residual block decreases in gradient. Each skip connection path contains a pre-set aligned convolutional layer, and the number of convolutional filters between the aligned convolutional layer and the last main convolutional layer is the same. Each encoder processes its corresponding high-dimensional feature data sequentially through each residual block within it. When the high-dimensional feature data passes through a certain residual block, the dimensionality reduction main path of that residual block sequentially performs convolution operations, batch normalization, and nonlinear activation on the high-dimensional feature data, outputting the main feature data. The skip connection path of that residual block performs convolution operations on the high-dimensional feature data, outputting skip feature data. The skip feature data is then added to the main feature data and output as the high-dimensional feature data for the next residual block or the latent feature data of the encoder.

[0037] The encoder's channel dimensionality reduction effect is mainly achieved by utilizing the gradient decrease in the number of convolutional filters between residual blocks in its internal dimensionality reduction main path. In other words, the number of channels in each residual block depends on the number of convolutional filters in its main convolutional layer, with one convolutional filter corresponding to one channel. Regardless of the number of channels in the input feature data, the number of channels in the output feature data of each residual block is the same as the number of convolutional filters. The skip connection path is responsible for aligning and adding the input and output feature data using the aligned convolutional layers before and after the corresponding dimensionality reduction main path.

[0038] The encoder gradually compresses the number of channels in the high-dimensional feature data by using multiple residual blocks with a gradient decreasing number of internal convolutional filters. This achieves smooth compression of the feature data, avoids excessive loss of feature data, and allows the potential feature data to retain the original features of the high-dimensional feature data to the greatest extent.

[0039] Preferably, the convolutional filter within the aligned convolutional layer uses a 1*1 convolutional kernel to avoid changing the spatial dimensions of the feature data.

[0040] Similarly, each residual block in the decoder includes an upscaling main path and a skip connection path. The upscaling main path includes a main convolutional layer, a batch normalization layer, and an activation function layer connected in sequence. The number of convolutional filters in the main convolutional layer of each residual block increases in a gradient. The skip connection path has a pre-set alignment convolutional layer, and the number of convolutional filters between the alignment convolutional layer and the last main convolutional layer is the same. Each decoder processes the corresponding low-dimensional feature data sequentially through each residual block within it. When the low-dimensional feature data passes through a certain residual block, the upscaling main path of that residual block performs convolution, batch normalization, and nonlinear activation on the low-dimensional feature data sequentially, and then outputs the main feature data. The skip connection path of that residual block performs convolution on the low-dimensional feature data and outputs the skip feature data. The skip feature data is then added to the main feature data and output as the low-dimensional feature data of the next residual block or the reconstructed feature data of the decoder.

[0041] The decoder's channel upscaling effect is primarily achieved by leveraging the gradient increase in the number of convolutional filters between residual blocks in its internal upscaling main path. In other words, the number of channels in each residual block depends on the number of convolutional filters in its main convolutional layer, with one convolutional filter corresponding to one channel. Regardless of the number of channels in the input feature data, the number of channels in the output feature data of each residual block is the same as the number of convolutional filters. The skip connection path is responsible for aligning and adding the input and output feature data using the aligned convolutional layers before and after the corresponding upscaling main path.

[0042] The decoder gradually increases the number of channels in the low-dimensional feature data through multiple residual blocks with an increasing gradient of the number of internal convolutional filters. This enables smooth reconstruction of the feature data, preserving the previous feature data while focusing on generating new features at the current scale to gradually approximate the high-dimensional feature data, thereby restoring all the details of the teacher's features with high quality.

[0043] Preferably, the convolutional filter within the aligned convolutional layer uses a 1*1 convolutional kernel to avoid changing the spatial dimensions of the feature data.

[0044] Step 30: Perform forward propagation inference on the teacher model and the student model, and obtain the high-dimensional feature data output by each target feature layer in the teacher model, and obtain the low-dimensional feature data output by each target feature layer in the student model.

[0045] In step 30, the teacher model and the student model use the same training dataset for forward propagation inference to avoid the dimensionality reduction distillation and dimensionality increase reconstruction of data features being affected by the differences in the training dataset.

[0046] Step 40: Input each high-dimensional feature data into the corresponding encoder and obtain the latent feature data output by each encoder corresponding to each target feature layer in the teacher model. Input each low-dimensional feature data into the corresponding decoder and obtain the reconstructed feature data output by each decoder corresponding to each target feature layer in the student model.

[0047] In step 40, the internal operating mechanism of the encoder and decoder is described in step 20.

[0048] Step 50: Based on the latent feature data and reconstructed feature data of each pair of target feature layers, and in conjunction with the task loss function of the student model, construct the total loss function.

[0049] Specifically, in step 50, the steps for constructing the total loss function based on the latent feature data and reconstructed feature data of each pair of target feature layers, and in conjunction with the task loss function of the student model, are as follows: Step 51: Select an appropriate task loss function based on the functional tasks of the student model.

[0050] In step 51, the task loss function refers to the original objective function that the student model needs to optimize in order to complete its functional task (such as image detection, image classification, image segmentation, etc.).

[0051] For image classification tasks, commonly used task loss functions include cross-entropy loss function, log loss function, or KL divergence function; for image detection tasks, commonly used task loss functions include focus loss function, variable focus loss function, or cross-union loss function; for image segmentation tasks, commonly used task loss functions include Dice loss function, etc.; these will not be listed here.

[0052] Step 52: For each target feature layer in the teacher model, calculate the corresponding distillation loss value based on its corresponding high-dimensional feature data and latent feature data; and for each target feature layer in the student model, calculate the corresponding reconstruction loss value based on its corresponding low-dimensional feature data and reconstructed feature data.

[0053] In step 52, the distillation loss value for each target feature layer within the teacher model is as follows: The reconstruction loss value for each target feature layer in the student model is as follows: in, The high-dimensional feature data output by each target feature layer within the teacher model. The low-dimensional feature data output by each intermediate feature layer in the student model. For the dimension reduction compression operation of the encoder, For the dimension-up reconstruction operation of the decoder, This is for normalization purposes.

[0054] Step 53: Calculate the total distillation loss value based on the distillation loss value of each target feature layer in the teacher model, and calculate the total reconstruction loss value based on the reconstruction loss value of each target feature layer in the student model.

[0055] In step 53, the total distillation loss value of the teacher model is as follows: The total distillation loss of the student model is as follows: in, and These are the distillation loss value and reconstruction loss value of the i-th target feature layer in the teacher model and student model, respectively, where N is the total number of target feature layers.

[0056] Step 54: Based on the total distillation loss value and the total reconstruction loss value, and in conjunction with the selected task loss function, construct the total loss function.

[0057] In step 54, the total loss function is as follows: in, Let be the task loss function of the student model. and These are the weighting coefficients for the distillation loss value and the reconstruction loss value, respectively.

[0058] If the distillation loss value and the reconstruction loss value are weighted using a weighted average method, then If the distillation loss value and the reconstruction loss value are weighted using a weighted linear combination, then The weighting coefficient and Specifically, this can be determined through grid search or validation set evaluation. In one example, In another example, , .

[0059] Step 60: Use the total loss function to perform backpropagation training on the student model to update and adjust the parameters of each intermediate feature layer in the student model, so that the performance of the student model is close to that of the teacher model.

[0060] In step 60, the backpropagation training refers to calculating the gradient of the total loss function with respect to the weight parameters and bias parameters of each intermediate feature layer, starting from the model output layer and working backward layer by layer. The parameters are then updated along the gradient direction using an optimizer (such as gradient descent, Adam, etc.) to gradually reduce the loss.

[0061] Preferably, forward propagation inference is performed on the teacher model and the student model in step 30, and backward propagation training is performed on the student model in step 60, using the same training dataset.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention and not to limit them. Although the embodiments of the present invention have been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the embodiments of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge distillation method of an isomorphic teacher-student model, the isomorphic teacher-student model comprising a teacher model and a student model, the teacher model and the student model having the same network layer structure, but having at least one pair of intermediate feature layers with the same layer position, the teacher model and the student model having different numbers of channels at the at least one pair of intermediate feature layers; characterized in that, The knowledge distillation method includes the following steps: Step 10: Determine the pairs of intermediate feature layers that have the same hierarchical position but different number of channels between the teacher model and the student model as the target feature layers; Step 20: Construct a corresponding distillation module for each pair of target feature layers. Each distillation module consists of an encoder corresponding to the teacher model and a decoder corresponding to the student model. Each encoder is used to perform channel-based dimensionality reduction and compression on the high-dimensional feature data output by the corresponding target feature layer in the teacher model, so as to form latent feature data with the same number of channels as the corresponding target feature layer in the student model. Each decoder is used to perform channel-based dimensionality up-reconstruction on the low-dimensional feature data output by the corresponding target feature layer in the student model, so as to form reconstructed feature data with the same number of channels as the corresponding target feature layer in the teacher model. Step 30: Perform forward propagation inference on the teacher model and the student model, and obtain the high-dimensional feature data output by each target feature layer in the teacher model, and obtain the low-dimensional feature data output by each target feature layer in the student model; Step 40: Input each high-dimensional feature data into the corresponding encoder and obtain the latent feature data output by each encoder corresponding to each target feature layer in the teacher model. Input each low-dimensional feature data into the corresponding decoder and obtain the reconstructed feature data output by each decoder corresponding to each target feature layer in the student model. Step 50: Based on the latent feature data and reconstructed feature data of each pair of target feature layers, and in conjunction with the task loss function of the student model, construct the total loss function; Step 60: Use the total loss function to perform backpropagation training on the student model to update and adjust the parameters of each intermediate feature layer in the student model, so that the performance of the student model is close to that of the teacher model.

2. The knowledge distillation method of claim 1, wherein, In step 10, the steps for determining the pairs of intermediate feature layers that have the same hierarchical position but different number of channels between the teacher model and the student model as the target feature layers are as follows: Step 11: Parse the configuration files of the teacher model and the student model, or traverse the network hierarchy structure of the teacher model and the student model to obtain the feature layer channel list of the teacher model and the student model respectively; Step 12: Based on the feature layer channel lists of the teacher model and the student model, compare the number of channels in the intermediate feature layers at the same level to determine whether the number of channels in the pair of intermediate feature layers is the same. This will identify the pairs of intermediate feature layers that are at the same level but have different numbers of channels, and these will be used as the target feature layers.

3. The knowledge distillation method of claim 1, wherein, Both the encoder and decoder are formed by stacking multiple layers of residual blocks, and the encoder and decoder corresponding to the same pair of target feature layers are structurally mirror symmetrical.

4. The knowledge distillation method of claim 3, wherein, Based on the number of channels in the same pair of target feature layers of the teacher model and student model, the number of residual blocks of the encoder and decoder corresponding to the pair of target feature layers is determined, and the number of channels of each residual block is determined by linear interpolation.

5. The knowledge distillation method of claim 4, wherein, The number of residual blocks for the encoder and decoder within the same distillation module is as follows: wherein, and are the number of channels of the teacher model and the student model on the same pair of target feature layers, respectively, is a max function, is a min function, is a ceiling function; The number of channels in the i-th layer residual block of the encoder is as follows: The number of channels in the i-th layer residual block within the decoder is as follows: in, This is the floor function.

6. The knowledge distillation method according to claim 4, characterized in that, Each residual block within the encoder includes a dimensionality reduction main path and a skip connection path. The dimensionality reduction main path includes a main convolutional layer, a batch normalization layer, and an activation function layer connected in sequence. The number of convolutional filters in the main convolutional layers of each residual block decreases in gradient. Each skip connection path has a pre-set aligned convolutional layer, and the number of convolutional kernels between the aligned convolutional layer and the last main convolutional layer is the same. Each encoder processes the corresponding high-dimensional feature data sequentially through its internal residual blocks. When the high-dimensional feature data passes through a certain residual block, the dimensionality reduction main path of that residual block sequentially performs convolution, batch normalization, and nonlinear activation on the high-dimensional feature data and outputs the main feature data. The skip connection path of that residual block performs convolution on the high-dimensional feature data and outputs skip feature data. The skip feature data is then added to the main feature data and output.

7. The knowledge distillation method according to claim 4, characterized in that, Each residual block within the decoder includes an upscaling main path and a skip connection path. The upscaling main path comprises a main convolutional layer, a batch normalization layer, and an activation function layer connected in sequence. The number of convolutional kernels in the main convolutional layer of each residual block increases progressively. Each skip connection path contains a pre-set aligned convolutional layer, with the same number of convolutional kernels between the aligned convolutional layer and the last main convolutional layer. Each decoder processes the corresponding low-dimensional feature data sequentially through its internal residual blocks. When the low-dimensional feature data passes through a certain residual block, the upscaling main path of that residual block sequentially performs convolution, batch normalization, and nonlinear activation on the low-dimensional feature data before outputting the main feature data. The skip connection path of that residual block performs convolution on the low-dimensional feature data before outputting skip feature data, and then adds the skip feature data to the main feature data before outputting the result.

8. The knowledge distillation method according to claim 1, characterized in that, In step 50, the steps for constructing the total loss function based on the latent feature data and reconstructed feature data of each pair of target feature layers, and in conjunction with the task loss function of the student model, are as follows: Step 51: Select an appropriate task loss function based on the functional tasks of the student model; Step 52: For each target feature layer in the teacher model, calculate the corresponding distillation loss value based on its corresponding high-dimensional feature data and latent feature data; and for each target feature layer in the student model, calculate the corresponding reconstruction loss value based on its corresponding low-dimensional feature data and reconstructed feature data. Step 53: Calculate the total distillation loss value based on the distillation loss value of each target feature layer in the teacher model, and calculate the total reconstruction loss value based on the reconstruction loss value of each target feature layer in the student model. Step 54: Based on the total distillation loss value and the total reconstruction loss value, and in conjunction with the selected task loss function, construct the total loss function.

9. The knowledge distillation method according to claim 8, characterized in that, The distillation loss values ​​for each target feature layer within the teacher model are as follows: The reconstruction loss value for each target feature layer in the student model is as follows: in, The high-dimensional feature data output by each target feature layer within the teacher model. The low-dimensional feature data output by each intermediate feature layer in the student model. For the dimension reduction compression operation of the encoder, For the dimension-up reconstruction operation of the decoder, For normalization processing; The total distillation loss value of the teacher model is as follows: The total distillation loss of the student model is as follows: in, and These are the distillation loss value and reconstruction loss value of the i-th target feature layer in the teacher model and student model, respectively, where N is the total number of target feature layers; The total loss function is as follows: in, Let be the task loss function of the student model. and These are the weighting coefficients for the distillation loss value and the reconstruction loss value, respectively.

10. The knowledge distillation method according to claim 1, characterized in that, In step 30, forward propagation inference is performed on the teacher model and the student model, and in step 60, backpropagation training is performed on the student model, using the same training dataset.

Images