Multi-branch alignment knowledge distillation method

Abstract

The invention discloses a multi-branch alignment knowledge distillation method, and relates to the technical field of computer vision and deep learning. According to the method, when the space dimension of the student features is smaller than the space dimension of the teacher features, in order to learn the teacher features more comprehensively, MBAKD uses up-sampling operation to enable the student features to be aligned to the teacher features; in the channel dimension, the MBAKD aims to establish a plurality of mapping branches between the teacher features and the student features, so that the students can be aligned with the teacher features from different angles; according to the multi-branch design, a flexible bridge is added between student features and teacher features, students can understand teacher knowledge more easily, and excessive confidence of teachers can be relieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and specifically to a multi-branch alignment knowledge distillation method. Background Art

[0002] Deep neural networks (DNNs) have made significant progress in the field of computer vision image recognition. However, DNNs have large number of parameters and are difficult to deploy, which often leads to huge storage and computing overhead. To solve this problem, Hinton et al. (2015) [1] The knowledge distillation (KD) method is proposed. The KD method transfers the knowledge of the teacher model to the student model by minimizing the output difference between the teacher model and the student model. The KD method is simple and easy to implement and is widely used in various image recognition tasks. [2] .

[0003] The KD method enables students to learn from the final output of the teacher model, ignoring the intermediate learning process. Later research focused on the student's learning process, that is, feature-based knowledge distillation. [3] For example, ReviewKD [4] Through the knowledge review mechanism, students can review the knowledge they have learned during the learning process, that is, integrate shallow intermediate features. Because shallow and deep features contain different semantic information and spatial information, ReviewKD can enable students to learn richer information. [5] In each round of training, we randomly select the middle layers of the teacher model and the student model and establish a mapping to transfer knowledge. When the teacher model has more layers and the student model has fewer layers, the middle layers of the teacher model will be skipped during distillation. Therefore, random guidance can help students learn more comprehensive teacher knowledge. [6] Cross-layer knowledge transfer is achieved through semantic calibration, because the semantics of layers in different neural networks may be different, and the semantic mismatch of artificial layer associations will lead to performance degradation caused by negative regularization. To solve this problem, SemCKD automatically assigns the appropriate intermediate layers of the teacher model to the intermediate layers of the student model through the attention mechanism. [7] A single projection layer is used to transform student features and reuse the pre-trained teacher classifier for student reasoning. However, we found that simply aligning student features with teacher features limits the student's ability to learn freely. Therefore, how to design a suitable projection layer is a problem worth exploring.

[0004] References are as follows:

[0005] [1]Hinton E G,Vinyals O,Dean J.Distilling the Knowledge in a NeuralNetwork.[J].CoRR,2015,abs / 1503.02531.

[0006] [2]Hu,C.,Li,X.,Liu,D.,Chen,X.,Wang,J.,Liu,X.:Teacher-studentarchitecturefor knowledge learning:A survey.arXiv preprint arXiv:2210.17332(2022).

[0007] [3]Gou,J.,Yu,B.,Maybank,S.J.,Tao,D.:Knowledge distillation:Asurvey.International Journal ofComputerVision 129(6),1789–1819(2021).

[0008] [4]Chen,P.,Liu,S.,Zhao,H.,Jia,J.:Distilling knowledge via knowledgereview.In:Proceedings of the IEEE / CVF Conference on Computer Vision andPatternRecognition,pp.5008–5017(2021).

[0009] [5]Hu,C.,Li,X.,Liu,D.,Chen,X.,Wang,J.,Liu,X.:Teacher-studentarchitecturefor knowledge learning:A survey.arXiv preprint arXiv:2210.17332(2022).

[0010] [6]Ahn, S., Hu, SX, Damianou, A., Lawrence, ND, Dai, Z.: Variational informationdistillation for knowledge transfer. In: Proceedings of the IEEE / CVFConference on ComputerVision and Pattern Recognition, pp.9163–9171(2019).

[0011] [7]Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., Chen, C.: Knowledge dis-tillation with the reused teacher classifier. In: Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pp. 11933–11942 (2022).

[0012] During the research, the inventors found that: (1) There is a large gap in the learning ability of the teacher and student network models, and forcing the student network model to imitate the output of the teacher network model will result in poor distillation effect. (2) Using a projection layer to simply map the features of the student network model can narrow the feature gap between the teacher and the student, but the single mapping method limits the free learning ability of the student network model.

[0013] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0014] The purpose of the present invention is to provide a multi-branch alignment knowledge distillation method to solve the technical problems raised in the background technology.

[0015] To achieve the above object, the present invention provides the following technical solution: a multi-branch alignment knowledge distillation method, comprising at least the following steps:

[0016] S1: Send the RGB image to the four-layer feature encoder of the student network model for feature extraction;

[0017] S2: Send the RGB image to the four-layer feature encoder of the teacher network model for feature extraction;

[0018] S3: Get the feature information extracted by the last layer of the student network;

[0019] S4: Obtain the feature information extracted by the last layer of the teacher network;

[0020] S5: Use the nearest neighbor upsampling operation to align the feature information extracted by the last layer of the student network with the features of the last layer of the teacher network in the spatial dimension;

[0021] S6: After the alignment in the spatial dimension is completed, the student network model is aligned again with the last layer of features of the teacher model through a multi-branch projection layer, the loss between the features of the student model and the features of the teacher model is calculated, and the predicted output of the teacher model is used.

[0022] Furthermore, the encoding stage in S1 uses a convolutional neural network as the backbone network of the student network model to extract features of the RGB image, and the feature map is reduced layer by layer with the depth of the convolutional neural network, and the convolutional neural network is a small network.

[0023] Furthermore, in the encoding stage in S2, a convolutional neural network is used as the backbone network of the teacher network model to extract features of the RGB image, and the feature map is reduced layer by layer with the depth of the convolutional neural network, and the convolutional neural network is a large network.

[0024] Furthermore, the S5 at least includes the following steps:

[0025] The last layer feature f of the student network model s The size of (w s ,h s ) is upsampled to the last layer feature f of the teacher network model t The size (w t ,h t ), for the coordinates of the last layer of features of the output teacher network model (x t ,y t ), the corresponding coordinates of the last layer of features of the input student network model (x s ,y s ) is calculated as follows:

[0026]

[0027]

[0028] Then, output the last layer feature map f of the teacher network model t (x t ,y t ) is equal to the last layer feature map f of the input student network model s (x s ,y s ), that is, f t (x t ,yt )=f s (x s ,y s ).

[0029] Further, the S6 at least includes the following steps:

[0030] S600: The last layer of features of the student network model after the spatial dimension alignment in S5 is recorded as Its dimension is [b s , c s ,h s , w s ], the last layer of the teacher network model is recorded as Its dimension is [b t , c t ,h t , w t ], where h s =h t , w s =w t , b s =b t The multi-branch projection layer consists of three branches. Branch 1 consists of 1×1 convolution, 3×3 convolution, and 1×1 convolution. Each convolution is followed by batch normalization and Relu activation. Branch 2 consists of 3×3 convolution, 1×1 convolution, 1×1 convolution, and 3×3 convolution. Each convolution is followed by batch normalization and Relu activation. Branch 3 consists of 3×3 convolution, 1×1 convolution, 1×1 convolution, 3×3 convolution, 1×1 convolution, 1×1 convolution, and 3×3 convolution. Each convolution is followed by batch normalization and Relu activation.

[0031] S601: After the last layer of features of the student network model passes through three branches, the three branches are fused and the output of the three branches is recorded as f s1 、f s2 、f s3 , whose dimension is [b t , c t ,h t , w t ], the fused features are recorded as The calculation formula is as follows:

[0032]

[0033] S602: The loss is calculated by combining the last layer of features of the student network model after fusion with the last layer of features of the teacher network model. The loss formula uses mean square error loss, which is as follows:

[0034]

[0035] Where n represents the number of samples, i represents the i-th sample, and f t Represents the last layer of features of the teacher network model, Represents the last layer of features of the student network model after multi-branch fusion;

[0036] S603: Finally, the features of the student network model after multi-branch fusion are predicted using the predictor of the teacher network model.

[0037] Furthermore, the calculation process of S600 includes at least the following steps:

[0038] First is the calculation of branch 1 Bottleneck: the last layer of features of the student network model after spatial alignment After the first branch of the multi-branch projection layer, the dimension [b s , c s ,h s , w s ] is converted to After batch normalization and ReLU activation, the 3×3 convolution operation changes its dimension from Convert to The dimension remains unchanged. After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [b t , c t ,h t , w t ], and the calculation of branch one is completed after batch normalization and activation.

[0039] Then comes the calculation of branch 2 Sandglass: the last layer of features of the student network model after spatial alignment After the second branch of the multi-branch projection layer, the dimension of the network is changed from [b s , c s ,h s , w s ] is converted to [b t , c t ,h t , w t ], after batch normalization and Relu activation, the 1×1 convolution operation changes its dimension from [b t , c t ,h t , w t ] is converted to After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [bt , c t ,h t , w t ], after batch normalization and Relu activation, the 3×3 convolution operation changes its dimension from [b t , c t ,h t , w t ] is converted to [b t , c t ,h t , w t ], after batch normalization and Relu activation, the calculation of branch 2 is completed.

[0040] Finally, the calculation of branch three Sandglass+: the last layer of features of the student network model after spatial alignment After the third branch of the multi-branch projection layer, the dimension of the network is changed from [b s , c s ,h s , w s ] is converted to [b t , c t ,h t , w t ], after batch normalization and Relu activation, the 1×1 convolution operation changes its dimension from [b t , c t ,h t , w t ] is converted to After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [b t , c t ,h t , w t ], after batch normalization and Relu activation, the 3×3 convolution operation changes its dimension from [b t , c t ,h t , w t ] is converted to [b t , c t ,h t , w t ], after batch normalization and Relu activation, the 1×1 convolution operation changes its dimension from [b t , c t ,h t , w t ] is converted to After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [bt , c t ,h t , w t ], after batch normalization and Relu activation, the 3×3 convolution operation changes its dimension from [b t , c t ,h t , w t ] is converted to [b t , c t ,h t , w t ], and complete the calculation of branch three after batch normalization and ReLU activation.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] When the dimension of the student feature space is smaller than that of the teacher feature space, in order to learn the teacher features more comprehensively, MBAKD uses an upsampling operation to align the student features with the teacher features; in the channel dimension, the goal of MBAKD is to establish multiple mapping branches between the teacher features and the student features, so that the student can align with the teacher features from different angles; wherein, the multi-branch design adds a "flexible" bridge between the student features and the teacher features, making it easier for the student to understand the teacher's knowledge and also able to mitigate the teacher's over-confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 It is the innovative idea diagram of the present invention;

[0045] Figure 2 A diagram of a multi-branch alignment knowledge distillation method of the present invention;

[0046] Figure 3 It is the spatial dimension alignment diagram of the present invention;

[0047] Figure 4 It is a multi-branch channel alignment diagram of the present invention;

[0048] Figure 5 It is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0050] Traditional Chinese education uses the phrase "follow others at every step" to criticize the teaching method in which students strictly imitate the teacher's behavior. Traditional Chinese education opposes "following others at every step" because in this case, students may be overly dependent on the teacher's thinking and unable to flexibly form a better and more effective understanding of new knowledge based on their own knowledge, resulting in students' understanding of knowledge being difficult to surpass the teacher. Therefore, students should not only accept the knowledge imparted by teachers, but more importantly, cultivate the ability to think about problems from multiple perspectives, such as Figure 1 In knowledge distillation, we design a multi-branch projection layer to allow student features to learn teacher features from multiple perspectives.

[0051] Based on the above concept, a multi-branch alignment knowledge distillation method is proposed as follows: Figure 2 .

[0052] Input the RGB image to the teacher network model and the student network model at the same time, align the last layer of features of the student network model with the last layer of features of the teacher network model in the spatial dimension using the nearest neighbor upsampling operation, and then align them in the channel through the multi-branch projection layer, and calculate the loss of the aligned student network model features and teacher network model features, and finally use the predictor of the teacher network model for prediction, so as to improve the distillation effect. The specific implementation steps are as follows:

[0053] The technical route is briefly described as follows:

[0054] The student network model is based on a convolutional neural network (small) and the teacher network model is based on a convolutional neural network (large). The RGB image is input to both the teacher network model and the student network model, and features are extracted through convolution operations. The features extracted by the student network model are aligned with the features of the teacher network model in terms of spatial and channel dimensions. The nearest neighbor interpolation upsampling operation is used for alignment in the spatial dimension, and the multi-branch module is used for alignment in the channel dimension. After alignment, the loss of the student network model features and the teacher network model features is calculated, and finally the predictor output of the teacher network model is used.

[0055] The nearest neighbor interpolation upsampling operation is briefly described as follows:

[0056] The features extracted by the student network model are usually small in width and height, resulting in insufficient feature information extraction, which limits the learning of features by the student network model. In order to allow the student network model to extract more comprehensive features, the nearest neighbor interpolation upsampling operation is used to align the student network model with the features of the teacher network model.

[0057] Specific structure such as Figure 3 As shown: The feature maps extracted by the student network model are usually smaller, and they match the feature maps extracted by the larger teacher network model by padding values ​​around them.

[0058] The multi-branch channel alignment is briefly described as follows:

[0059] In order to make the features extracted by the student network model more generalizable, multi-branch channel alignment is used to transform the features three times and finally fuse them.

[0060] Specific structure such as Figure 4 As shown, the multi-branch channel alignment has three branches, namely Bottleneck, Sandglass, and Sandglass+.

[0061] Bottleneck is more economical in terms of parameter quantity. It consists of a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer. Each convolution operation is accompanied by batch normalization and Relu activation. The first 1×1 convolutional layer operation converts the feature channel dimension of the student network model to half of the feature channel dimension of the teacher network model. The channel dimension of the 3×3 convolutional layer operation remains unchanged. The second 1×1 convolutional layer operation converts the student network model features to the dimension of the teacher network model feature channel. Sandglass retains more feature information. The first 3×3 convolutional layer operation keeps the feature channel dimension of the student network model unchanged. The first 1×1 convolutional layer converts the feature dimension of the student network model to half of the dimension of the feature channel of the teacher network model. The second 1×1 convolutional layer converts the feature dimension of the student network model to the dimension of the feature channel of the teacher network model. The second 3×3 The convolution layer operation keeps the dimension of the feature channel of the student network model unchanged; Sandglass+ stacks Sandglass to extract features at a deeper level. The first 3×3 convolution layer operation keeps the dimension of the feature channel of the student network model unchanged. The first 1×1 convolution layer converts the dimension of the feature channel of the student network model to half the dimension of the feature channel of the teacher network model. The second 1×1 convolution layer converts the dimension of the feature channel of the student network model to the dimension of the feature channel of the teacher network model. The second 3×3 convolution layer operation keeps the dimension of the feature channel of the student network model unchanged. The third 1×1 convolution layer converts the dimension of the feature channel of the student network model to half the dimension of the feature channel of the teacher network model. The fourth 1×1 convolution layer converts the dimension of the feature channel of the student network model to the dimension of the feature channel of the teacher network model. The third 3×3 convolution layer operation keeps the dimension of the feature channel of the student network model unchanged. Finally, the outputs of the three branches are averaged to achieve more stable output and more generalized features.

[0062] The loss function is briefly described as follows:

[0063] The loss function uses the mean square error loss function, which is sensitive to outliers (outliers). The formula is as follows:

[0064]

[0065] Where n represents the number of samples, i represents the i-th sample, and f t Represents the last layer of features of the teacher network model, Represents the last layer of features of the student network model after multi-branch fusion.

[0066] In summary, the present invention aims to solve the problem that the learning ability gap between the teacher and student network models is large, which leads to poor distillation effect in the knowledge transfer process, and narrows the gap between the student model features and the teacher model features from the spatial dimension and the channel dimension. Specifically, the last layer of features of the student network model is aligned with the last layer of features of the teacher network model through an upsampling operation, and the last layer of features of the student network model and the last layer of features of the teacher network model are further aligned using the multi-branch channel alignment (MBCA) module, and then the loss of the student network features and the teacher network features is calculated, and finally the teacher model prediction is used for output.

[0067] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A multi-branch alignment knowledge distillation method, characterized by: At least the following steps are included: S1: Send the RGB image to the four-layer feature encoder of the student network model for feature extraction; S2: Send the RGB image to the four-layer feature encoder of the teacher network model for feature extraction; S3: Get the feature information extracted by the last layer of the student network; S4: Obtain the feature information extracted by the last layer of the teacher network; S5: Use the nearest neighbor upsampling operation to align the feature information extracted by the last layer of the student network with the features of the last layer of the teacher network in the spatial dimension; S6: After the alignment in the spatial dimension is completed, the student network model is aligned again with the last layer of features of the teacher model through a multi-branch projection layer, the loss between the features of the student model and the features of the teacher model is calculated, and the predicted output of the teacher model is used.

2. A multi-branch alignment knowledge distillation method according to claim 1, characterized in that: The encoding stage in S1 uses a convolutional neural network as the backbone network of the student network model to extract features of RGB images. The feature map is reduced layer by layer with the depth of the convolutional neural network. The convolutional neural network is a small network.

3. A multi-branch alignment knowledge distillation method according to claim 1, characterized in that: The encoding stage in S2 uses a convolutional neural network as the backbone network of the teacher network model to extract features of RGB images. The feature map is reduced layer by layer with the depth of the convolutional neural network, and the convolutional neural network is a large network.

4. A multi-branch alignment knowledge distillation method according to claim 1, characterized in that: The S5 at least comprises the following steps: The last layer feature f of the student network model s The size of (w s ,h s ) is upsampled to the last layer feature f of the teacher network model t The size (w t ,h t ), for the coordinates of the last layer of features of the output teacher network model (x t ,y t ), the corresponding coordinates of the last layer of features of the input student network model (x s ,y s ) is calculated as follows: Then, output the last layer feature map f of the teacher network model t (x t ,y t ) is equal to the last layer feature map f of the input student network model s (x s ,y s ), that is, f t (x t ,y t ) = f s (x s ,y s ).

5. A multi-branch alignment knowledge distillation method according to claim 4, characterized in that: The S6 at least comprises the following steps: S600: The last layer of features of the student network model after the spatial dimension alignment in S5 is recorded as Its dimension is [b s ,c s ,h s ,w s ], the last layer of the teacher network model is recorded as Its dimension is [b t ,c t ,h t ,w t ], where h s =h t , w s =w t , b s =b t The multi-branch projection layer consists of three branches. Branch 1 consists of 1×1 convolution, 3×3 convolution, and 1×1 convolution. Each convolution is followed by batch normalization and Relu activation. Branch 2 consists of 3×3 convolution, 1×1 convolution, 1×1 convolution, and 3×3 convolution. Each convolution is followed by batch normalization and Relu activation. Branch 3 consists of 3×3 convolution, 1×1 convolution, 1×1 convolution, 3×3 convolution, 1×1 convolution, 1×1 convolution, and 3×3 convolution. Each convolution is followed by batch normalization and Relu activation. S601: After the last layer of features of the student network model passes through three branches, the three branches are fused and the output of the three branches is recorded as f s1 、f s2 、f s3 , whose dimension is [b t ,c t ,h t ,w t ], the fused features are recorded as The calculation formula is as follows: S602: The loss is calculated by combining the last layer of features of the student network model after fusion with the last layer of features of the teacher network model. The loss formula uses mean square error loss, which is as follows: Where n represents the number of samples, i represents the i-th sample, and f t Represents the last layer of features of the teacher network model, Represents the last layer of features of the student network model after multi-branch fusion; S603: Finally, the features of the student network model after multi-branch fusion are predicted using the predictor of the teacher network model.

6. A multi-branch alignment knowledge distillation method according to claim 1, characterized in that: The calculation process of S600 at least includes the following steps: First is the calculation of branch 1 Bottleneck: the last layer of features of the student network model after spatial alignment After the first branch of the multi-branch projection layer, the dimension [b s ,c s ,h s ,w s ] is converted to After batch normalization and ReLU activation, the 3×3 convolution operation changes its dimension from Convert to The dimension remains unchanged. After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [b t ,c t ,h t ,w t ], and the calculation of branch one is completed after batch normalization and activation. Then comes the calculation of branch 2 Sandglass: the last layer of features of the student network model after spatial alignment After the second branch of the multi-branch projection layer, the dimension of the network is changed from [b s ,c s ,h s ,w s ] is converted to [b t ,c t ,h t ,w t ], after batch normalization and Relu activation, the 1×1 convolution operation changes its dimension from [b t ,c t ,h t ,w t ] is converted to After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [b t ,c t ,h t ,w t ], after batch normalization and Relu activation, the 3×3 convolution operation changes its dimension from [b t ,c t ,h t ,w t ] is converted to [b t ,c t ,h t ,w t ], after batch normalization and Relu activation, the calculation of branch 2 is completed. Finally, the calculation of branch three Sandglass+: the last layer of features of the student network model after spatial alignment After the third branch of the multi-branch projection layer, the dimension of the network is changed from [b s ,c s ,h s ,w s ] is converted to [b t ,c t ,h t ,w t ], after batch normalization and Relu activation, the 1×1 convolution operation changes its dimension from [b t ,c t ,h t ,w t ] is converted to After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [b t ,c t ,h t ,w t ], after batch normalization and Relu activation, the 3×3 convolution operation changes its dimension from [b t ,c t ,h t ,w t ] is converted to [b t ,c t ,h t ,w t ], after batch normalization and Relu activation, the 1×1 convolution operation changes its dimension from [b t ,c t ,h t ,w t ] is converted to After batch normalization and ReLU activation, the 1×1 convolution operation changes its dimension from Convert to [b t ,c t ,g t ,w t ], after batch normalization and Relu activation, the 3×3 convolution operation changes its dimension from [b t ,c t ,h t ,w t ] is converted to [b t ,c t ,h t ,w t ], and complete the calculation of branch three after batch normalization and ReLU activation.

Images