Knowledge distillation method and device based on multi-scale feature mapping and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a knowledge distillation method based on multi-scale feature mapping. The method comprises the following steps: respectively inputting training images into a teacher network and a student network to obtain feature mappings XT and XS; determining a first feature mapping set through a teacher network based on the XT and N first scales; determining a second scale from the first scales; determining a first similarity between the first feature mapping classes of the second scale; determining a second feature mapping set through the student network based on the XS and the N first scales; determining a feature mapping loss value between the first feature mapping and the second feature mapping of the second scale; determining a second similarity between second feature mapping classes of a second scale; determining an inter-class relationship loss value according to the first similarity and the second similarity; and optimizing parameters of the student network according to the weight, the feature mapping loss value and the inter-class relationship loss value. According to the invention, the accuracy of the trained student network is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a knowledge distillation method, apparatus, device, and storage medium based on multi-scale feature mapping. Background Art

[0002] A large, complex image recognition network (called the teacher network) has more convolutional layers and neurons than a small, simple network (called the student network). This allows the teacher network to extract richer features from images when recognizing them, resulting in more accurate image recognition.

[0003] However, due to the large capacity of the teacher network, which requires a correspondingly high level of storage and computing resources, it is not suitable for deployment on terminal devices with limited storage and computing power. Due to the small capacity of the student network, it is more suitable for deployment on terminal devices with limited storage and computing power. For student networks deployed on terminal devices, improving their accuracy is a challenge that needs to be addressed. Summary of the Invention

[0004] In view of the above problems, the embodiments of the present application provide a knowledge distillation method, apparatus, device and storage medium based on multi-scale feature mapping, which are used to solve the problem of low accuracy of the trained student network in the prior art.

[0005] According to one aspect of an embodiment of the present application, a knowledge distillation method based on multi-scale feature mapping is provided, the method comprising: S101: inputting a training image into a teacher network, obtaining a feature map X output by a two-dimensional convolutional layer in an output layer of the teacher network T , wherein the teacher network is a trained image recognition network, and the penultimate layer and the last layer in the output layer of the teacher network are the two-dimensional convolution layer and the adaptive average pooling layer respectively; S102: based on the feature map X Tand N first scales, determining a first feature map set through the adaptive average pooling layer in the teacher network, wherein the first feature map set includes the first feature maps of the N first scales, and the first feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the teacher network identifying the training image; S103: determining a second scale of the feature map for training the student network in the nth training stage from the N first scales, wherein the student network is an image recognition network to be trained, and the last two layers of the output layer of the student network have the same structure as the last two layers of the output layer of the teacher network, and the initial value of n is 1; S104: determining a first similarity between different categories of the first feature map of the second scale according to the feature matrix corresponding to different categories in the first feature map of the second scale; S105: inputting the training image into the student network to obtain a feature map X output by the two-dimensional convolution layer in the output layer of the student network S ; S106: Based on the feature map X S and the N first scales, determining a second feature map set through an adaptive average pooling layer in the student network, wherein the second feature map set includes the second feature maps of the N first scales, and the second feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the student network identifying the training image; S107: determining a feature map loss value between the first feature map of the second scale and the second feature map of the second scale; S108: determining a second similarity between different categories of the second feature map of the second scale according to the feature matrices corresponding to different categories in the second feature map of the second scale; S109: determining an inter-class similarity between the teacher network and the student network according to the first similarity and the second similarity. Relationship loss value; S110: Determine the weight corresponding to the second scale according to the feature mapping loss value; S111: Determine the total loss value of the nth training stage according to the weight, the feature mapping loss value and the inter-class relationship loss value; S112: Optimize the bias parameters and weight parameters of the student network according to the total loss value; S113: If the number of training times of the nth training stage does not reach the preset number threshold corresponding to the nth training stage, repeat S105 to S112; if the number of training times of the nth training stage has reached the preset number threshold corresponding to the nth training stage, go to S114; S114: If there is a first scale that has not been selected among the N first scales, update n to n+1, and go to S103.

[0006] In an optional manner, when n is 1, S103 includes: selecting a first scale with the smallest scale from the N first scales as the second scale; when n is greater than 1, S103 includes: selecting a first scale that has not been selected from the N first scales, and using the first scale selected this time and the first scale used as the second scale in the (n-1)th training stage as the second scale together as the second scale in the nth training stage.

[0007] In an optional manner, the selecting an unselected first scale from the N first scales includes: selecting a first scale with the smallest scale from the unselected first scales among the N first scales.

[0008] In an optional manner, a total of N training stages are included, the preset number thresholds corresponding to the first N-1 training stages are all first thresholds, and the preset number threshold corresponding to the Nth training stage is a second threshold, wherein the second threshold is greater than the first threshold.

[0009] In an optional manner, the S111 includes: using the formula Determine the total loss value for the nth training stage Among them, weight k is the weight corresponding to the kth second scale among the n second scales, λ1 and λ2 are both preset values, is the feature map loss value corresponding to the k-th second scale, is the inter-class relationship loss value corresponding to the k-th second scale.

[0010] In an optional manner, the S107 includes: if the teacher network and the student network are isomorphic networks, then by formula Determine the feature map loss value corresponding to the nth second scale Among them, C is the number of preset categories, w n and h n are the nth second scale, is the value of the first feature map at the (i, j) position at the l channel of the nth second scale, is the value of the second feature map at the (i, j) position of the l channel of the nth second scale; if the teacher network and the student network are heterogeneous networks, then the formula Determine the feature map loss value corresponding to the nth second scale in,

[0011] In an optional manner, the feature map loss value is negatively correlated with the weight.

[0012] According to another aspect of an embodiment of the present application, a knowledge distillation device based on multi-scale feature mapping is provided, comprising: a first input module for inputting a training image into a teacher network, obtaining a feature map X output by a two-dimensional convolutional layer in an output layer of the teacher network; T , wherein the teacher network is a trained image recognition network, and the penultimate layer and the last layer in the output layer of the teacher network are the two-dimensional convolution layer and the adaptive average pooling layer respectively; the first determination module is used to determine the feature map X based on the feature map X. T and N first scales, determining a first feature map set through the adaptive average pooling layer in the teacher network, wherein the first feature map set includes the first feature maps of the N first scales, and the first feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the teacher network identifying the training image; a second determination module is used to determine a second scale of the feature map for training the student network in the nth training stage from the N first scales, wherein the student network is an image recognition network to be trained, and the last two layers of the output layer of the student network have the same structure as the last two layers of the output layer of the teacher network, and the initial value of n is 1; a third determination module is used to determine a first similarity between different categories of the first feature map of the second scale according to the feature matrix corresponding to different categories in the first feature map of the second scale; a second input module is used to input the training image into the student network to obtain a feature map X output by the two-dimensional convolution layer in the output layer of the student network. S ; A fourth determination module, configured to determine based on the feature map X Sand the N first scales, determining a second feature map set through an adaptive average pooling layer in the student network, wherein the second feature map set includes the second feature maps of the N first scales, the second feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the student network identifying the training image, determining a feature map loss value between the first feature map of the second scale and the second feature map of the second scale, determining a second similarity between different categories of the second feature map of the second scale according to the feature matrices corresponding to different categories in the second feature map of the second scale, determining an inter-class relationship loss value between the teacher network and the student network according to the first similarity and the second similarity, determining a weight corresponding to the second scale according to the feature map loss value, and determining a total loss value of the nth training stage according to the weight, the feature map loss value, and the inter-class relationship loss value; an optimization module for optimizing the bias parameters and weight parameters of the student network according to the total loss value; and an update module for updating n to n+1 if there is a first scale that has not been selected among the N first scales.

[0013] According to another aspect of an embodiment of the present application, a knowledge distillation device based on multi-scale feature mapping is provided, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the knowledge distillation method based on multi-scale feature mapping as described above.

[0014] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the knowledge distillation method based on multi-scale feature mapping as described above is implemented.

[0015] In an embodiment of the present application, the penultimate layer and the last layer in the output layers of the student network and the teacher network are set as a two-dimensional convolution layer and an adaptive average pooling layer, respectively, and the student network is trained based on the feature maps output by the two-dimensional convolution layers in the student network and the teacher network. Since the feature maps output by the two-dimensional convolution layers include more knowledge than the feature maps output by the traditional fully connected layers, when training the student network in the present application, the total amount of knowledge transferred from the teacher network to the student network can be increased, thereby improving the accuracy of the trained student network.

[0016] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0018] Figure 1 Shown is a schematic diagram of the output layer in an existing image recognition network and a schematic diagram of the output layer in a teacher network provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a process for a knowledge distillation method based on multi-scale feature mapping provided in an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of a knowledge distillation method based on multi-scale feature mapping provided in an embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram of the structure of a knowledge distillation device based on multi-scale feature mapping provided in an embodiment of the present application is shown;

[0022] Figure 5 A structural schematic diagram of a knowledge distillation device based on multi-scale feature mapping provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0024] Knowledge distillation is a new method for reducing model size without sacrificing accuracy. It improves the accuracy of a student network by transferring knowledge from a pre-trained teacher network. Based on the location of the transferred knowledge, knowledge distillation can be generally categorized into logit distillation and feature distillation. Logit distillation, due to its computational efficiency and ability to handle heterogeneous knowledge, is widely applicable to various distillation scenarios. Traditional logit distillation primarily focuses on the probability distribution of the teacher network's final output (i.e., the distribution of the model's confidence that the input image belongs to each category).

[0025] Since the total amount of knowledge contained in the probability distribution output by the teacher network is fixed, ideally, the maximum amount of knowledge learned by the student network during logit distillation is the knowledge contained in the probability distribution ultimately output by the teacher network. Convolutional neural networks are commonly used deep learning models for knowledge distillation. The output layer of this convolutional neural network compresses the high-dimensional feature maps extracted by the hidden layer through global pooling and tiling operations, and then performs classification through a linear layer and softmax. Although global pooling further refines the extracted features, the significant compression of the feature maps results in a significant amount of fuzzy discarding of the knowledge in the feature maps, resulting in a low amount of knowledge transferred in the final logit distillation. However, if the amount of knowledge transferred in logit distillation is too low, the student network will also have low accuracy.

[0026] Based on the above considerations, in order to increase the total amount of knowledge transferred in logit distillation and thus improve the accuracy of the student network, this application proposes a knowledge distillation method based on multi-scale feature mapping, by modifying the penultimate layer and the last layer in the output layer of the teacher network and the student network into a two-dimensional convolution layer and an adaptive average pooling layer, respectively. Then, the training images are input into the trained teacher network and the student network to be trained, respectively, and the feature maps X output by the two-dimensional convolution layer in the teacher network and the student network are obtained. T and feature map X S , then based on the feature map X T , the adaptive average pooling layer in the teacher network is used to determine the first feature maps of N first scales, based on the feature map X S , use the adaptive average pooling layer in the student network to determine the second feature maps of N first scales, determine the feature map loss value between the first feature map and the second feature map of each scale respectively, determine the first similarity and the second similarity between the first feature map and the second feature map in different categories respectively, determine the inter-class relationship loss value between the teacher network and the student network according to the first similarity and the second similarity, and determine the feature map loss value corresponding to each scale to determine the weight corresponding to each scale, and optimize the parameters of the student network according to the weight, the inter-class relationship loss value and the feature map loss value.

[0027] Figure 1 The schematic diagram of the output layer in the existing image recognition network and the schematic diagram of the output layer in the teacher network provided by the embodiment of the present application are shown. Figure 1As shown, (a) is a schematic diagram of the output layer in an existing image recognition network, and (b) is a schematic diagram of the output layer in a teacher network. The network architecture of the teacher network adopts a convolutional neural network in which the penultimate and last layers in the output layer are modified. The penultimate and last layers are a two-dimensional convolutional layer and an adaptive average pooling layer, respectively. In the embodiment of the present application, the two-dimensional convolutional layer reuses the weights and biases of the original fully connected layer. Among them, the two-dimensional convolution can be regarded as a fully connected operation on each channel of the feature map, which is responsible for mapping it to the total number of categories.

[0028] Figure 2 A flow chart of a knowledge distillation method based on multi-scale feature mapping provided in an embodiment of the present application is shown, and the method is executed by a terminal device, which may be a terminal device including one or more processors, such as a touch-screen mobile phone, a smart phone, a tablet computer, a portable electronic device or other electronic device. The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, which is not limited here. The one or more processors included in the terminal device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs, which are not limited here.

[0029] Figure 3 FIG1 shows a schematic diagram of the knowledge distillation method based on multi-scale feature mapping provided by an embodiment of the present application. Figure 2 and Figure 3 As shown, the method includes the following steps:

[0030] Step 201: Input the training image into the teacher network to obtain the feature map X output by the two-dimensional convolution layer in the output layer of the teacher network T .

[0031] The training images are used to train the student network. To improve the training effect of the student network, the training images are preprocessed images, such as those obtained by cropping and normalizing the original training images. Alternatively, the training images can be cropped to 32x32 pixels with 4 pixels of padding around the edges, horizontally flipped with a 50% probability, and normalized.

[0032] The teacher network is a trained image recognition network. In the embodiment of the present application, the output layers of the teacher network and the student network are reconstructed mainly to obtain feature maps of different scales in order to increase the total amount of knowledge transferred in the subsequent distillation process. The fully connected layer in the output layer of the network is changed to a convolutional layer and the weight parameters of the fully connected layer are reused, mainly because the fully connected layer cannot process two-dimensional feature maps, and the two-dimensional convolution with a 1x1 convolution kernel, 0 padding, and a step size of 1 can be approximately regarded as a fully connected linear operation, ensuring that the final output probability distribution remains unchanged. By exchanging the penultimate pooling layer and the convolution layer, it can be ensured that the scale of the feature map obtained by the convolution layer is large enough and contains sufficient uncompressed knowledge information.

[0033] Step 202: Based on the feature map X T and N first scales, the first feature map set is determined by an adaptive average pooling layer in the teacher network.

[0034] The first feature map set includes N first feature maps of a first scale, each of which includes a feature matrix corresponding to the probabilities of the training images belonging to various categories, obtained by the teacher network recognizing the training images. N is a positive integer greater than 2, and the N first scales can be set as needed. For example, if N is 4, the four first scales can be a 2×2 scale, a 4×4 scale, an 8×8 scale, and a 16×16 scale, respectively. Where R represents the real number in the matrix, C is the number of output data channels of the two-dimensional convolutional layer, that is, the number of classifications, and W T and H T Represents the size of the first feature map in a channel.

[0035] For example, for X T Perform adaptive average pooling operation to obtain The feature maps of three different scales have a scale distribution of [2, 4, 8]. It can be seen that For X T ontology.

[0036] Step 203: Determine a second scale of the feature map for training the student network in the nth training stage from the N first scales.

[0037] The student network is an image recognition network to be trained, and the last two layers of the output layer of the student network have the same structure as the last two layers of the output layer of the teacher network. n is a positive integer, and its initial value is 1. In this step, specifically, one or more first scales are selected from the N first scales as the second scale.

[0038] In the embodiments of the present application, preferably, in the order of the N first scales from small to large, the second scale for training the feature map of the student network in the first training stage is the first first scale, the second scale for the feature map of the student network in the second training stage is the first first scale and the second first scale, and so on. The second scale for the feature map of the student network in the nth training stage is the first n first scales.

[0039] To improve the training efficiency, it is also possible that the second scale for training the feature map of the student network in the first training stage is the first first scale, the second scale for the feature map of the student network in the second training stage is the first first scale to the third first scale, the second scale for the feature map of the student network in the third training stage is the first first scale to the fifth first scale, and so on. When n is greater than 1, the second scale for the feature map of the student network in the nth training stage is the first 2n - 1 first scales.

[0040] In some embodiments, the second scale for training the feature map of the student network in each training stage may also have only one first scale. For example, in the order of the N first scales from small to large, the second scale for the feature map of the student network in the nth training stage is the nth first scale.

[0041] Step 204: Determine the first similarity between different categories in the first feature map of the second scale according to the feature matrices corresponding to different categories in the first feature map of the second scale.

[0042] In the embodiments of the present application, in the order of the N first scales from small to large, the second scale for the feature map of the student network in the nth training stage is the first n first scales. In this step, the first similarity between the first feature map of the nth second scale in the u-th class and the v-th class can be determined by the following formula (1)

[0043]

[0044] The first similarity of each class in the first feature map of the nth second scale is:

[0045] [[ID=二十六]]

[0046] where 1 ≤ u < v ≤ C represents not considering the similarity relationship between the same classes and the repeated similarity relationship between classes.

[0047] Calculate the inter-class similarity loss of the three scales of the teacher network, mainly calculate the average similarity of each class inside the feature maps of the three scales of the teacher network. For X ∈ R 100×8×8, there are 100 classes. Without repeated calculation and self-calculation, the inter-class similarity of the feature maps of the teacher and student networks at three different scales can be obtained by cosine similarity. sim∈R 4950 .

[0048] Step 205: Input the training image into the student network to obtain the feature map X output by the two-dimensional convolution layer in the output layer of the student network S .

[0049] The training image is the same training image as the training image input into the teacher network in step 201 .

[0050] Step 206: Based on the feature map X S and N first scales, and the second set of feature maps is determined by an adaptive average pooling layer in the student network.

[0051] The second feature map set includes N second feature maps of the first scale, and the second feature map includes a feature matrix corresponding to the probability that the training image obtained by the student network recognizes the training image belongs to each category. W S and H S Represents the second feature map size in a channel.

[0052] For example, for X S Perform adaptive average pooling operation to obtain The feature maps of three different scales have a scale distribution of [2, 4, 8]. It can be seen that For X S ontology.

[0053] Step 207: Determine a feature map loss value between the first feature map at the second scale and the second feature map at the second scale.

[0054] In the implementation of this application, the second scale of the feature map used for the student network in the nth training stage is taken as the first n first scales as an example. If n is greater than 1, the feature map loss value between the first feature map and the second feature map of each second scale is determined separately in this step.

[0055] For homogeneous networks, since the networks share similar structural characteristics, feature maps have similar distributions and ranges, allowing for direct comparison. However, for heterogeneous networks, due to differences in feature extraction methods between networks, the ranges and distributions of the resulting feature maps vary, necessitating additional normalization. Therefore, feature maps at various scales are normalized, depending on whether the student and teacher networks are identical.

[0056] Specifically, if the teacher network and the student network are isomorphic networks, the feature map loss value corresponding to the nth second scale is determined by the following formula (3):

[0057]

[0058] Among them, w n and h n are the nth second scale, is the value of the first feature map at the (i, j) position at the l channel of the nth second scale, is the value of the second feature map at the (i, j) position at the l channel of the nth second scale.

[0059] If the teacher network and the student network are heterogeneous networks, the feature map loss value corresponding to the nth second scale is determined by the following formula (4):

[0060]

[0061] in, φ can be approximated as an L2 normalization function.

[0062] Calculate the feature map loss of the three scales of the teacher and student networks. The main method is to calculate the difference between the feature maps of each scale of the teacher and student networks through MSE Loss to obtain the loss of the three scales. L fea ∈R 1 .

[0063] Step 208: determining a second similarity between different categories of the second feature map at the second scale according to the feature matrices corresponding to different categories in the second feature map at the second scale.

[0064] In the embodiment of the present application, according to the order of N first scales from small to large, the second scale of the feature map used for the student network in the nth training stage is the first n first scales. In this step, the second similarity between the second feature map of the nth second scale and the uth class and the vth class can be determined by the following formula (5):

[0065]

[0066] The second similarity of each class of the second feature map of the nth second scale for:

[0067]

[0068] Among them, 1 ≤ u < v ≤ C represents not considering the similarity relationships between the same categories and duplicate similarity relationships among categories.

[0069] Calculate the inter-class similarity loss of the three scales of the student network, mainly by calculating the average similarity of each class within the feature maps of the three scales of the student network. For X ∈ R 100×8×8 , there are 100 classes. Under the conditions of not calculating repeatedly and not calculating itself, the inter-class similarity of the feature maps of the teacher and student networks at three different scales can be obtained through cosine similarity

[0070] Step 209: Determine the inter-class relationship loss value between the teacher network and the student network according to the first similarity and the second similarity.

[0071] Among them, the inter-class relationship loss value of the nth second scale can be determined by the following formula (7)

[0072]

[0073] The inter-class similarity loss of each scale can be determined through the above formula '

[0074] Step 210: Determine the weight corresponding to the second scale.

[0075] If the number of the second scales of the feature maps used for the student network in the nth training stage is greater than 1, then in this step, the weights corresponding to each second scale are determined respectively. It can be understood that the sum of the weights corresponding to all the second scales is 1. Here, taking the second scales of the feature maps used for the student network in the nth training stage as the first n first scales as an example for introduction, the weight weight corresponding to the kth second scale among the n second scales of the feature maps used for the student network in the nth training stage can be determined by the following formula (8) k :

[0076]

[0077] τ is a hyperparameter used to control the difference size of the inputs of each scale so as to calibrate the finally obtained weight coefficients. is the input value of the Softmax function corresponding to the kth second scale, corresponding to the aforementioned feature map loss value. It can be seen from formula (8) that the feature map loss value corresponding to the second scale is negatively correlated with its corresponding weight.

[0078] Figure 3 In, D i(x) corresponds to the difference between the teacher and student network outputs of sample x at scale i. The multi-scale difference and adaptive weight correspond to the difference loss and corresponding weight ratio between the teacher network feature map and the student network at each scale, respectively.

[0079] Step 211: Determine the total loss value of the nth training stage based on the weight, feature map loss value, and inter-class relationship loss value.

[0080] Here, we take the example of the first n first scales as the second scale of the feature map used for the student network in the nth training stage. At the beginning, only the smallest scale loss that is easiest to train is used. As the student network gradually masters this part, large-scale losses with higher training difficulty are gradually added to assist the student network training. When the training process reaches a certain stage, losses of all scales will be used. When n is greater than 1, each training stage will introduce a new first scale as the second scale compared to the previous training stage, that is, each training stage will introduce a new first scale loss. All losses involved in the nth training stage as follows:

[0081]

[0082] Among them, λ1 is a hyperparameter used to balance the cross entropy loss and multi-scale loss, and λ2 is a hyperparameter used to balance the feature mapping loss and inter-class relationship loss.

[0083] The total loss value of the nth training stage is determined by the following formula (10):

[0084]

[0085] Among them, L CE It is the cross entropy loss commonly used in multi-classification tasks.

[0086] Step 212: Optimize the bias parameters and weight parameters of the student network according to the total loss value.

[0087] Among them, according to Backpropagation optimizes the bias parameters and weight parameters of the student network to complete the training of the student network in the current round in the nth training stage.

[0088] Step 213: Determine whether the number of training times in the nth training stage reaches the preset number threshold corresponding to the nth training stage. If yes, go to step 214; if not, go to step 205.

[0089] The prediction threshold for each training stage is greater than 1, meaning that multiple rounds of iterative training are required for the student network in each training stage. For example, if the student network needs to be trained for 240 rounds, the threshold for the number of predictions for the first and second training stages can be set to 30, and the threshold for the number of predictions for the third training stage can be set to 180.

[0090] Step 214: Determine whether there is a first scale that has not been selected among the N first scales. If so, go to step 215; if not, end this process.

[0091] Step 215: Update n to n+1 and go to step 203.

[0092] In steps 213 to 215, if the number of training times of the nth training stage does not reach the preset number threshold corresponding to the nth training stage, the same training images as those input to the teacher network in step 201 continue to be input to the student network for the next round of training in the nth training stage.

[0093] If the number of training times in the nth training stage reaches the preset number threshold corresponding to the nth training stage, the training of the nth training stage will no longer be carried out, and it will be further determined whether all N first scales have been selected as the second scale. If all N first scales have been selected, it means that all scales have been used for training (including training in the current training stage and training in the previous training stage), and the training can be terminated to obtain a trained student network; if there is a first scale that has not been selected, but the number of training times in this training stage has reached the preset number threshold, in order to avoid overfitting and save resources, the training of this training stage is terminated and the training of the next training stage is started, and the second scale of the feature map used to train the student network in the next training stage is reselected.

[0094] In order to improve the accuracy of the trained student network, in the embodiment of the present application, the student network is iteratively trained using the training image set. When executing step 201, a training image is selected from the training image set and input into the teacher network. In step 214, if all N first scales have been selected, the next training image is selected from the training image set and executed. Figure 1 In the provided embodiment, the above steps are repeated until all training images in the training image set are selected and used to train the student network.

[0095] In an embodiment of the present application, the penultimate layer and the last layer in the output layers of the student network and the teacher network are set as a two-dimensional convolution layer and an adaptive average pooling layer, respectively, and the student network is trained based on the feature maps output by the two-dimensional convolution layers in the student network and the teacher network. Since the feature maps output by the two-dimensional convolution layers include more knowledge than the feature maps output by the traditional fully connected layers, when training the student network in the present application, the total amount of knowledge transferred from the teacher network to the student network can be increased, thereby improving the accuracy of the trained student network.

[0096] From the following formula (11), we can see that the feature mapping loss of the teacher network and the student network at a large scale is always greater than or equal to the feature mapping loss at a small scale, which means that the fitting is more difficult.

[0097]

[0098] Here, r corresponds to the degree of downsampling. Represents the value corresponding to the i-th position in the original feature map of the teacher network, Represents the value corresponding to the i-th position in the original feature map of the student network. i Then the corresponding

[0099] Therefore, in the embodiment of the present application, preferably, in the order of N first scales from small to large, the second scale of the feature map used for the student network in the nth training stage is the first n first scales, that is, the student network is subjected to a scale-by-scale progressive training method, which makes full use of the reasoning information of the teacher network and improves the efficiency of knowledge transfer. In addition, when determining the weights corresponding to each second scale, the present application takes into account the performance difference between the teacher network and the student network, and adjusts the proportion of knowledge of different second scales according to the performance difference between the teacher network and the student network, thereby giving the student network enough learning space, rather than rigidly learning the reasoning method of the teacher network, thereby improving the generalization ability of the trained student network. Moreover, the present application further improves the total amount of knowledge transferred from the teacher network to the student network by determining the multi-scale inter-class relationship loss value, that is, by fitting the relationship between the training images in each category in the teacher network and the student network, thereby further improving the accuracy of the trained student network.

[0100] Furthermore, compared to traditional feature distillation methods, the embodiments of this application do not need to consider the size differences of the intermediate layer feature maps of the heterogeneous teacher-student network combination, which reduces the complexity of the model while avoiding the dilution and waste of knowledge during the feature alignment process. Moreover, this application can be used as an external framework and effectively applied to various logit distillation methods.

[0101] In order to verify that the accuracy of the trained student network can be improved by using the knowledge distillation method based on multi-scale feature mapping provided by the present application, the following will be explained in conjunction with relevant experimental data. The dataset used in this experiment is the CIFAR100 dataset. This dataset covers 100 different categories, each of which contains 500 training set samples and 100 test set samples. The entire dataset contains a total of 50,000 training samples and 10,000 test set samples.

[0102] In terms of implementation details, in order to ensure the diversity of the experiments, five groups of teacher-student networks with the same architecture (ResNet32x4-ResNet8x4, WideResNet40-2-WideResNet16-2, ResNet56-ResNet20, ResNet110-ResNet32 and

[0103] WideResNet40-2-WideResNet40-1) and 3 groups of teacher-student networks with different architectures (ResNet32x4-ShuffleNetv1, ResNet32x4-ShuffleNetv2, WideResNet40-2-ShuffleNetv1).

[0104] In terms of parameter setting, all experiments were completed using the Pytorch deep learning framework. Each student network was trained for 240 epochs based on the use of stochastic gradient descent optimizer to optimize the model parameters. Except for the ShuffleNetV1 / V2 model, whose initial learning rate was set to 0.01, the initial learning rate of other student networks was set to 0.05 and reduced by 10 times at 150, 180, and 210 epochs. In addition, the batch size of the model was set to 64, the momentum of the stochastic gradient descent optimizer was set to 0.9, and the weight decay was set to 5x10. -4 The scale of the feature maps introduced at each stage is set to [2, 4, 8], and a feature map of a new scale is introduced every 30 epochs.

[0105] The experimental results are shown in Table 1 below.

[0106] Table 1: Experimental results on the CIFAR100 dataset

[0107]

[0108]

[0109] The teacher network and the student network have the same architecture. ↑ represents the improvement of this method compared to the CAT-KD method.

[0110] Table 2: Experimental results on the CIFAR100 dataset

[0111] Teacher ResNet32×4 ResNet32×4 WRN40-2 Acc 79.42 79.42 75.61 Student ShuffleNetV1 ShuffleNetV2 ShuffleNetV1 Acc 70.5 71.82 70.5 KD 74.07 74.45 74.83 CRD 75.11 75.65 76.05 OFD 75.98 76.82 75.85 FitNet 73.59 73.54 73.73 AT 71.73 72.73 73.32 RKD 72.28 73.21 72.21 ReviewKD 77.45 77.78 77.14 CAT-KD 78.26 78.41 77.35 Ours 78.47 78.91 77.62 ↑ +0.21 +0.50 +0.27

[0112] The teacher network and student network have different architectures. ↑ represents the improvement of this method compared to the CAT-KD method.

[0113] Figure 4 The schematic diagram of the structure of the knowledge distillation device based on multi-scale feature mapping provided by the embodiment of the present application is shown. Figure 4 As shown, the knowledge distillation device 300 based on multi-scale feature mapping includes a first input module 301, a first determination module 302, a second determination module 303, a third determination module 304, a second input module 305, a fourth determination module 306, an optimization module 307 and an update module 308.

[0114] The first input module 301 is used to input the training image into the teacher network, and obtain the feature map X output by the two-dimensional convolution layer in the output layer of the teacher network. T , where the teacher network is a trained image recognition network, and the penultimate and last layers in the output layer of the teacher network are a two-dimensional convolutional layer and an adaptive average pooling layer, respectively.

[0115] The first determining module 302 is used to determine the feature map X T and N first scales, and determining a first feature map set through an adaptive average pooling layer in the teacher network, wherein the first feature map set includes N first feature maps of the first scale, and the first feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the teacher network recognizing the training image.

[0116] The second determination module 303 is used to determine the second scale of the feature map used to train the student network in the nth training stage from N first scales, wherein the student network is an image recognition network to be trained, and the last two layers of the output layer of the student network are the same as the last two layers of the output layer of the teacher network, and the initial value of n is 1.

[0117] The third determining module 304 is configured to determine a first similarity between different categories of the first feature map at the second scale according to feature matrices corresponding to different categories in the first feature map at the second scale.

[0118] The second input module 305 is used to input the training image into the student network to obtain the feature map X output by the two-dimensional convolution layer in the output layer of the student network.S .

[0119] The fourth determining module 306 is used to determine the S and N first scales, determining a second feature map set through an adaptive average pooling layer in the student network, wherein the second feature map set includes N second feature maps of the first scale, the second feature map includes a feature matrix corresponding to the probability that the training image obtained by the student network recognizes the training image belongs to each category, determining a feature map loss value between the first feature map of the second scale and the second feature map of the second scale, determining a second similarity between different categories of the second feature map of the second scale according to the feature matrix corresponding to different categories in the second feature map of the second scale, determining an inter-class relationship loss value between the teacher network and the student network according to the first similarity and the second similarity, determining a weight corresponding to the second scale according to the feature map loss value, and determining a total loss value of the nth training stage according to the weight, the feature map loss value and the inter-class relationship loss value.

[0120] The optimization module 307 is used to optimize the bias parameters and weight parameters of the student network according to the total loss value.

[0121] The updating module 308 is configured to update n to n+1 if there is a first scale among the N first scales that has not been selected.

[0122] The knowledge distillation device 300 based on multi-scale feature mapping provided in this embodiment is used to execute the technical solution of the knowledge distillation method based on multi-scale feature mapping in the aforementioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0123] It is worth noting that the knowledge distillation device 300 based on multi-scale feature mapping provided in this embodiment also includes other modules for executing the various steps of the above-mentioned knowledge distillation method embodiment based on multi-scale feature mapping, which will not be described one by one here.

[0124] Figure 5 A structural schematic diagram of a knowledge distillation device based on multi-scale feature mapping provided in an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the knowledge distillation device based on multi-scale feature mapping.

[0125] like Figure 5 As shown, the knowledge distillation device 400 based on multi-scale feature mapping may include: a processor 402 and a memory 404.

[0126] The memory 404 is used to store a computer program 406. The memory 404 may include a high-speed RAM memory, or may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The computer program 406 may include computer-executable instructions.

[0127] The processor 402 is configured to execute the computer program 406 to implement the above-mentioned embodiment of the knowledge distillation method based on multi-scale feature mapping.

[0128] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the multi-scale feature mapping-based knowledge distillation apparatus 400 may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0129] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned knowledge distillation method embodiment based on multi-scale feature mapping.

[0130] An embodiment of the present application provides a computer program that can be executed by a processor to implement the above-mentioned knowledge distillation method embodiment based on multi-scale feature mapping.

[0131] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned knowledge distillation method embodiment based on multi-scale feature mapping.

[0132] In the several embodiments provided in this application, if any function is implemented in the form of a software function module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or other electronic device) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store computer program code.

[0133] The algorithm or demonstration provided here are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages can be utilized to realize the content of the present application described here, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of the present application.

[0134] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims that list several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

[0135] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A knowledge distillation method based on multi-scale feature mapping, characterized in that: The method comprises: S101: Input the training image into the teacher network to obtain the feature map X output by the two-dimensional convolution layer in the output layer of the teacher network T , wherein the teacher network is a trained image recognition network, and the penultimate layer and the last layer in the output layer of the teacher network are the two-dimensional convolution layer and the adaptive average pooling layer respectively; S102: Based on the feature map X T and N first scales, determining a first feature map set by the adaptive average pooling layer in the teacher network, wherein the first feature map set includes the first feature maps of the N first scales, and the first feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the teacher network identifying the training image; S103: Determining a second scale of a feature map for training a student network in an nth training stage from the N first scales, wherein the student network is an image recognition network to be trained, and the last two layers of the output layer of the student network have the same structure as the last two layers of the output layer of the teacher network, and an initial value of n is 1; S104: determining a first similarity between different categories of the first feature map at the second scale according to feature matrices corresponding to different categories in the first feature map at the second scale; S105: Input the training image into the student network to obtain the feature map X output by the two-dimensional convolution layer in the output layer of the student network S ; S106: Based on the feature map X S and the N first scales, determining a second feature map set through an adaptive average pooling layer in the student network, wherein the second feature map set includes second feature maps of the N first scales, and the second feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the student network identifying the training image; S107: Determine a feature map loss value between the first feature map of the second scale and the second feature map of the second scale; S108: determining a second similarity between different categories of the second feature map at the second scale according to feature matrices corresponding to different categories in the second feature map at the second scale; S109: Determine an inter-class relationship loss value between the teacher network and the student network according to the first similarity and the second similarity; S110: Determine a weight corresponding to the second scale according to the feature map loss value; S111: Determine the total loss value of the nth training stage according to the weight, the feature map loss value and the inter-class relationship loss value; S112: Optimizing the bias parameters and weight parameters of the student network according to the total loss value; S113: If the number of training times of the nth training stage has not reached the preset number threshold corresponding to the nth training stage, then repeating S105 to S112; if the number of training times of the nth training stage has reached the preset number threshold corresponding to the nth training stage, then going to S114; S114: If there is a first scale that has not been selected among the N first scales, update n to n+1 and go to S103.

2. The method according to claim 1, characterized in that When n is 1, the S103 includes: selecting a first scale with the smallest scale from the N first scales as a second scale; When n is greater than 1, S103 includes: selecting an unselected first scale from the N first scales, and using the selected first scale and the first scale of the (n-1)th training stage as the second scale together as the second scale of the nth training stage.

3. The method according to claim 2, characterized in that The selecting a first scale that has not been selected from the N first scales includes: A first scale with the smallest scale is selected from the unselected first scales among the N first scales.

4. The method according to claim 3, characterized in that There are N training stages in total, the preset number thresholds corresponding to the first N-1 training stages are all first thresholds, and the preset number threshold corresponding to the Nth training stage is a second threshold, wherein the second threshold is greater than the first threshold.

5. The method according to claim 3, characterized in that The S111 includes: By formula Determine the total loss value for the nth training stage Among them, weight k is the weight corresponding to the kth second scale among the n second scales, λ1 and λ2 are both preset values, is the feature map loss value corresponding to the k-th second scale, is the inter-class relationship loss value corresponding to the k-th second scale.

6. The method according to claim 1, characterized in that The S107 includes: If the teacher network and the student network are isomorphic networks, then the formula Determine the feature map loss value corresponding to the nth second scale Among them, C is the number of preset categories, w n and h n are the nth second scale, is the value of the first feature map at the (i, j) position at the l channel of the nth second scale, is the value of the second feature map at the (i, j) position at the l channel of the nth second scale; If the teacher network and the student network are heterogeneous networks, then the formula Determine the feature map loss value corresponding to the nth second scale in, 7. The method according to claim 1, characterized in that The feature map loss value is negatively correlated with the weight.

8. A knowledge distillation device based on multi-scale feature mapping, characterized in that: The device comprises: The first input module is used to input the training image into the teacher network and obtain the feature map X output by the two-dimensional convolution layer in the output layer of the teacher network. T , wherein the teacher network is a trained image recognition network, and the penultimate layer and the last layer in the output layer of the teacher network are the two-dimensional convolution layer and the adaptive average pooling layer respectively; The first determining module is configured to determine the feature map X based on the feature map X. T and N first scales, determining a first feature map set by the adaptive average pooling layer in the teacher network, wherein the first feature map set includes the first feature maps of the N first scales, and the first feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the teacher network identifying the training image; a second determining module, configured to determine, from the N first scales, a second scale of a feature map for training a student network in an nth training stage, wherein the student network is an image recognition network to be trained, and the last two layers of the output layer of the student network have the same structure as the last two layers of the output layer of the teacher network, and an initial value of n is 1; a third determining module, configured to determine a first similarity between different categories of the first feature map at the second scale based on feature matrices corresponding to different categories in the first feature map at the second scale; The second input module is used to input the training image into the student network to obtain the feature map X output by the two-dimensional convolution layer in the output layer of the student network. S ; The fourth determining module is configured to determine the S and the N first scales, determining a second feature map set through an adaptive average pooling layer in the student network, wherein the second feature map set includes the N second feature maps of the first scales, the second feature map includes a feature matrix corresponding to the probability that the training image belongs to each category obtained by the student network identifying the training image, determining a feature map loss value between the first feature map of the second scale and the second feature map of the second scale, determining a second similarity between different categories of the second feature map of the second scale according to the feature matrices corresponding to different categories in the second feature map of the second scale, determining an inter-class relationship loss value between the teacher network and the student network according to the first similarity and the second similarity, determining a weight corresponding to the second scale according to the feature map loss value, and determining a total loss value of the nth training stage according to the weight, the feature map loss value, and the inter-class relationship loss value; An optimization module, configured to optimize the bias parameters and weight parameters of the student network according to the total loss value; The updating module is configured to update n to n+1 if there is a first scale that has not been selected among the N first scales.

9. A knowledge distillation device based on multi-scale feature mapping, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the knowledge distillation method based on multi-scale feature mapping according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the knowledge distillation method based on multi-scale feature mapping described in any one of claims 1 to 7 is implemented.

Images