A knowledge distillation method for multi-teacher model consistency

Abstract

The present invention discloses a method for knowledge distillation of consistency of multiple teacher models. The method comprises: building a knowledge distillation architecture of consistency of multiple teacher models including a core teacher model, an auxiliary teacher model and a student model, constructing a training image set for target detection and inputting it into the architecture for training, obtaining a multi-teacher head and neck network consistency knowledge distillation module and a training loss function of a student model as the final loss function of the student model, until the loss function converges, obtaining a trained student model, and then performing target detection on the image. The method of the present invention improves the classification and positioning prediction consistency of the target detection model for complex scene detection by fusing the head and neck network consistency knowledge and the neck network consistency knowledge of multiple teacher models, and alleviates the adverse effects of the knowledge bias of a single teacher model.

Description

Technical Field

[0001] The present invention relates to a knowledge distillation method, relates to the field of computer vision, and specifically to a multi-teacher model consistency knowledge distillation method. Background Art

[0002] When making predictions, object detection models may experience inconsistencies between classification and location predictions, resulting in reduced detection accuracy. There are two types of inconsistent predictions: 1. Accurate location predictions, where the predicted box has a high IoU with the ground-truth box, but the category prediction is incorrect; 2. Correct classification and a high score for the predicted box, but the IoU with the ground-truth box is low. Existing methods address this issue through knowledge distillation, typically using a single teacher model to teach the student. However, due to uncertainties in the training process, detection results from teacher models with the same architecture can vary. These differences are transferred to the student model during knowledge distillation, affecting the distillation process and ultimately the object detection performance in the image. Summary of the Invention

[0003] In order to solve the problems existing in the background technology, the present invention provides a multi-teacher model consistency knowledge distillation method.

[0004] The technical solution adopted in the present invention is:

[0005] The multi-teacher model consistency knowledge distillation method of the present invention includes:

[0006] Step S1: Build a multi-teacher model consistency knowledge distillation architecture including a core teacher model, several auxiliary teacher models and a student model, construct a training image set for target detection and input it into the multi-teacher model consistency knowledge distillation architecture for training, obtain the multi-teacher head and neck network consistency knowledge distillation module, and use it together with the training loss function of the student model as the final loss function of the student model until the final loss function converges to obtain the trained student model.

[0007] Step S2: Input the image to be detected into the trained student model, and after processing, output the target detection result of the image and display it on the monitor.

[0008] In the step S1, the multi-teacher model consistency knowledge distillation architecture also includes a feature fusion module, which connects the outputs of the core teacher model and the teacher neck network of each auxiliary teacher model in the channel dimension; the core teacher model, each auxiliary teacher model and the student model have the same structure, including a backbone network, a neck network and a head network connected in sequence, and the training image set includes several training images. Each time the same training image is input into the core teacher model, each auxiliary teacher model and the student model, the classification output and positioning output of the core teacher model, the classification output and positioning output of each auxiliary teacher model and the classification output and positioning output of the student model are output after processing. At the same time, the outputs of the neck networks of the core teacher model and each auxiliary teacher model are processed by the feature fusion module to output a fusion feature map, and the fusion feature map is processed by the head network of the core teacher model to output a fusion classification output and a fusion positioning output; the teacher head network consistency knowledge is constructed according to the classification output and positioning output of each auxiliary teacher model and the core teacher model, and the student head network is constructed according to the classification output and positioning output of the student model. The multi-teacher head network consistency knowledge is constructed based on the classification and localization outputs of the core teacher model and each auxiliary teacher model. The multi-teacher head network consistency knowledge distillation module is constructed by combining the teacher head network consistency knowledge of each auxiliary teacher model, the teacher head network consistency knowledge of the core teacher model, the student head network consistency knowledge of the student model, the multi-teacher head network consistency knowledge fusion weights, and the multi-teacher head network consistency knowledge. A core teacher neck network consistency knowledge distillation module is constructed based on the neck network output, classification output, and localization output of the core teacher model, as well as the neck network output of the student model. An auxiliary teacher neck network consistency knowledge distillation module is constructed based on the fused feature map, fused classification output, fused localization output, and the neck network output of the student model. A multi-teacher neck network consistency knowledge distillation module is constructed based on the core and auxiliary teacher neck network consistency knowledge distillation modules. The multi-teacher head network consistency knowledge distillation module, the multi-teacher neck network consistency knowledge distillation module, and the student model's training loss function are combined as the final loss function of the student model. Both the teacher and student models are object detection models, and the backbone network depth of the teacher model is higher than that of the student model.

[0009] The feature fusion module is as follows:

[0010] f ama = M ama ( f c )

[0011] f c=Concat ( f j ), j =1, 2, ..., J

[0012] M ama ( f c )= W A ( f c )+ W C ( f c )· W G ( f c ))

[0013] W A ( f c )= Conv 2 d ( ReLU ( Conv 2 d ( f c )))

[0014] W C ( f c )= MLP ( AvgPool ( f c ))+ MLP ( MaxPool ( f c ))

[0015] W G ( f c )= MLP ( Softmax ( Conv 2 d ( f c ))× f c )

[0016] in, f ama Represents the fused feature map; f c represents the connection feature graph; Concat( ) represents the channel dimension connection operation; M ama ( ) represents the feature module; f j Indicates the j The output of the neck network of the teacher model, J Indicates the total number of core teacher models and auxiliary teacher models; W A ( ), W C ( )and W G ( ) represent the first, second and third features respectively; Conv 2 d ( ) represents a two-dimensional convolution operation; ReLU ( ) represents the activation operation of the rectified linear unit (Rectified Linear Unit); MLP ( ) represents a multilayer perceptron; AvgPool ( ) represents the average pooling operation; MaxPool ( ) represents the maximum pooling operation; Softmax ( ) represents a vector activation operation.

[0017] The multi-teacher head network consistency knowledge distillation module L MTCD as follows:

[0018]

[0019] in, λ 1 and λ 2 represent the first and second hyperparameters for balancing the loss respectively; L D and L A They represent the multi-teacher model association knowledge distillation loss function and the multi-teacher model angle knowledge distillation loss function respectively; J represents the total number of teacher models; M and N Represents the total number of detection categories and the total number of true boxes respectively; w n,j Indicates the j The first teacher model n The multi-teacher consistent knowledge fusion weight of the ground truth frame; CM m,n S Represents the student model m Detection categories,n Head network consistency knowledge under real frames, CM j,m,n T Indicates the j The first teacher model m Detection categories, n Head network consistency knowledge under real frames; CM n Indicates the n The head network consistency knowledge matrix under the real frame, CM 1. CM 2 and CM 3 respectively represent the n The head network consistency knowledge matrix under the ground truth frame CM n The three head networks in the same row have consistent knowledge; l δ ( ) represents the smooth loss Huber function; ψ A ( ) represents the angle relationship calculation function; CM 1,n ～T 、 CM 2,n ～T and CM 3,n ～T They represent the first n The three fused consistent knowledge in the same row in the fused consistent knowledge matrix of the ground-truth box, CM 1,n S 、 CM 2,n S and CM 3,n S Represents the first n The three head network consistency knowledge in the same row in the head network consistency knowledge matrix of the ground truth box; CM 1 CM 2 CM 3 means the n The head network consistency knowledge matrix under the ground truth frame CM n The angular relationship between the three head network consistency knowledge in the same row; e 12 and e 32 denote the first and second unit vectors respectively.

[0020] The head network consistency knowledge of the teacher model and the student model is as follows:

[0021]

[0022] in, CM m,n Represents the first m Detection categories, n Head network consistency knowledge under real frames; P Represents the total set of head network output indexes of the teacher model or student model, A n and B n Respectively represent n The head network of the teacher model or the student model under the real frame outputs the first and second index sets; k m,b ´ cls Indicates the m The first detection category of the head network b The classification output corresponding to the positioning output is k m " cls Indicates the m The average value of the classification output under the detection categories, k n,b ´ loc Indicates the n The ground truth box and the head network b The intersection-over-combination ratio of the positioning outputs, k n " loc Indicates the n The average of the intersection-over-union (IoU) of the ground-truth boxes and the positioning output of the head network; T 1 indicates softening temperature; k n,p ´ loc Indicates the n The total set of ground truth boxes and head network output indices P The p The intersection-over-combination ratio of the positioning outputs, k p ´ loc Represents the total set of true boxes and head network output indexes P The p The intersection-over-union ratio set of positioning outputs; TopK ( ) indicates before selection K The index of the element; k n,a ´ loc Indicates the n The ground-truth box and the first index set of the head network output A nThe a The intersection-over-union ratio of the positioning outputs.

[0023] The multi-teacher consistent knowledge fusion weights are as follows:

[0024]

[0025] in, w n,j Indicates the j The first teacher model n The multi-teacher consistent knowledge fusion weight of the ground truth frame; Softmax ( ) represents vector activation operation; w n,j c and w n,j h Respectively represent j The first teacher model n The first and second weight parameters in the multi-teacher consistency knowledge fusion weight of the ground truth frame; k b,j ´ cls Indicates the j The ground truth box of the teacher model and the second index set of the head network output B n The b The classification output set corresponding to the positioning output, k n,b,j ´ cls Indicates the j The first teacher model n The ground-truth box and the head network output second index set B n The b The classification output set corresponding to the positioning output; k b,j ´ loc Indicates the j The ground truth box of the teacher model and the second index set of the head network output B n The b The intersection-over-union ratio set of positioning outputs, k n,b,j ´ loc Indicates the j The first teacher model n The ground-truth box and the head network output second index set B n The b The intersection-and-union ratio set of positioning outputs; ( )´ represents the averaging operation.

[0026] The fusion consistency knowledge is as follows:

[0027]

[0028] in, CM m,n ～T Indicates the m The teacher model's n The fusion consistency knowledge of the ground truth boxes; P 1. P 2. ... P j 、…、 P J Respectively represent the first, second, ..., j , ..., J The total set of head network output indexes output by the teacher model, J Represents the total number of teacher models.

[0029] The multi-teacher neck network consistency knowledge distillation module L MTFPD as follows:

[0030]

[0031] in, λ 3 represents the third hyperparameter for balancing loss; L CT and L AT denote the core and auxiliary teacher neck network consistency knowledge distillation modules respectively; w A represents the adaptive weight; L Det AT represents the loss of training the head network that feeds the fused feature map into the core teacher model; L Det S Represents the training loss function of the student model. The final loss function of the student model L = L MTCD + L MTFPD + L Det S .

[0032] The core and auxiliary teacher neck network consistency knowledge distillation modules are as follows:

[0033]

[0034] in,I Represents the core teacher model CT or assistant teacher model AT , L I represents the core or auxiliary teacher neck network consistency knowledge distillation module; C 、 H and W Represent the core teacher model CT The feature maps output by the neck network of the student model and the number of channels, height, and width of the fused feature maps; Mask h,w cls,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the h Height and w Classification feature mask at width position, Mask h,w loc,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the h Height and w Position feature mask at width position; f c,h,w ,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the c Channel, h Height and w The response value of the width position, f c,h,w ,S Represents the student model S The feature map of the neck network output is in the c Channel, h Height and w Response value of width position; Softmax ( ) represents vector activation operation; k h,w,m ´ cls,I Represents the teacher model I In the h Height and w The width position positioning output or the fusion positioning output corresponding to the m The classification output or fusion classification output of each detection category, k h,w,n ´ loc,I Represents the teacher model I In the h Height and wThe positioning output of the width position or the fusion positioning output and the n The intersection-over-union (IoU) of the ground-truth boxes.

[0035] The multi-teacher model consistency knowledge distillation system of the present invention includes:

[0036] The data acquisition module acquires images for target detection.

[0037] The model construction module builds a multi-teacher model consistency knowledge distillation architecture including a core teacher model, several auxiliary teacher models, and a student model, and trains the student model.

[0038] The target detection display module uses the trained student model to process the target detection image, outputs the target detection result of the image and displays it on the display.

[0039] The electronic device of the present invention comprises: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described above.

[0040] The readable storage medium of the present invention stores program data thereon, and when the program data is executed by a processor, the method described above is implemented.

[0041] The present invention first calculates the teacher and student head network consistency knowledge and fusion consistency knowledge based on the head network classification output and the head network positioning output, realizes the quantification of the model prediction consistency, and constructs the multi-teacher consistency knowledge fusion weight based on the teacher head network classification output and the teacher head network positioning output. Then, based on the fusion consistency knowledge, the teacher head network consistency knowledge, the student head network consistency knowledge and the multi-teacher consistency knowledge fusion weight, constructs the multi-teacher head network consistency knowledge distillation module to realize the full fusion and utilization of the head network knowledge of the multi-teacher model; secondly, the core teacher model and the auxiliary teacher model are selected, and the fusion feature map is obtained through the feature fusion module, and the fusion feature map is integrated. The combined feature map is input into the core teacher model for training to obtain the fused head network classification output and the fused head network positioning output, so as to realize the fusion of the teacher model neck network knowledge. Then, based on the teacher neck network output, teacher head network classification output and teacher head network positioning output of the core teacher model, a core teacher neck network consistency knowledge distillation module is constructed. Based on the teacher model fused feature map, fused head network classification output and fused head network positioning output, an auxiliary teacher neck network consistency knowledge distillation module is constructed. Finally, based on the core teacher neck network consistency knowledge distillation module and the auxiliary teacher neck network consistency knowledge distillation module, a multi-teacher neck network consistency knowledge distillation module is constructed.

[0042] The beneficial effects of the present invention are:

[0043] The method of the present invention improves the classification and positioning prediction consistency of the target detection model for complex scene detection by fusing the head network consistency knowledge and neck network consistency knowledge of multiple teacher models, and alleviates the adverse effects of the knowledge bias of a single teacher model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the method of the present invention;

[0045] Figure 2 Schematic diagram of a knowledge distillation architecture for consistency among multiple teacher models according to an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of inconsistent predictions, where Figure 3 (a) is a schematic diagram of the first inconsistent prediction. Figure 3 (b) is a schematic diagram of the second inconsistent prediction;

[0047] Figure 4 This is a comparison chart of the impact of teacher model detection differences on knowledge distillation effects, where: Figure 4 (a) is a schematic diagram of the teacher model’s error detection. Figure 4 (b) is a schematic diagram of the correct detection of the teacher model. Figure 4 (c) is a schematic diagram of the incorrect prediction results of the student model that inherits the incorrectly detected teacher model. Figure 4 (d) is a schematic diagram of the correct prediction results of the student model that inherits the correctly detected teacher model;

[0048] Figure 5 This is a comparison chart of the effects of the student model on the artifact detection dataset before and after distillation, where: Figure 5 (a) is a schematic diagram of the detection results of the initial student model before distillation. Figure 5 (b) is a schematic diagram of the detection results of the initial student model after distillation. DETAILED DESCRIPTION

[0049] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0050] The present invention was tested on a self-built artifact detection dataset containing six artifact types, numbered 1-6. Artifact 1 is the first fixed block, Artifact 2 is the first connecting block, Artifact 3 is the first flange, Artifact 4 is the second fixed block, Artifact 5 is the second connecting block, and Artifact 6 is the second flange. The proposed multi-teacher model consistency knowledge distillation method was implemented on this artifact dataset to address the issue of inconsistent predictions. Figure 3 It shows the prediction inconsistency problem that occurs when using RetinaNet-R18 detection, such as Figure 3 As shown in (a), this is the first type of prediction inconsistency problem. Its positioning prediction is accurate, that is, the intersection of the predicted box and the true box is relatively high, but the category prediction is wrong. It can be seen that the position of the predicted box of the workpiece and the target true box are basically coincident in the figure, but the model mistakenly predicts the 6th type workpiece as the 3rd type workpiece; Figure 3 (b) shows the second type of prediction inconsistency problem. The classification of the prediction box is correct and the score is high, but the intersection-union ratio between the prediction box and the true box is low. It can be seen that the category prediction of the workpiece in the figure is correct, and the classification score reaches 0.69, but there is a large difference between the position of the prediction box and the target true box. The intersection-union ratio between the two is only 0.46, which is low.

[0051] like Figure 1 As shown, the multi-teacher model consistency knowledge distillation method of the present invention specifically includes the following steps:

[0052] First, a multi-teacher model consistency knowledge distillation architecture is built, which includes a core teacher model, several auxiliary teacher models, and a student model. Both the teacher model and the student model are target detection models, and the backbone network depth of the teacher model is higher than that of the student model. Figure 2 As shown in Figure 2, the multi-teacher model consistency knowledge distillation architecture also includes a feature fusion module. The feature fusion module connects the outputs of the teacher neck network of the core teacher model and each auxiliary teacher model in the channel dimension. The feature fusion module is as follows:

[0053] f ama = M ama ( f c )

[0054] f c =Concat ( f j ), j =1, 2, ..., J

[0055] M ama ( f c )= W A ( f c )+ W C ( f c )· WG ( f c ))

[0056] W A ( f c )= Conv 2 d ( ReLU ( Conv 2 d ( f c )))

[0057] W C ( f c )= MLP ( AvgPool ( f c ))+ MLP ( MaxPool ( f c ))

[0058] W G ( f c )= MLP ( Softmax ( Conv 2 d ( f c ))× f c )

[0059] MLP ( f c )= Conv 2 d ( ReLU ( LN ( Conv 2 d ( f c ))))

[0060] in, f ama Represents the fused feature map; f c represents the connection feature graph; Concat ( ) represents the channel dimension connection operation; M ama ( ) represents the feature module; f j Indicates the jThe output of the neck network of the teacher model, J Indicates the total number of core teacher models and auxiliary teacher models; W A ( ), W C ( )and W G ( ) represent the first, second and third features respectively; Conv 2 d ( ) represents a two-dimensional convolution operation using a 1×1 convolution kernel; ReLU ( ) represents the activation operation of the rectified linear unit (RectifiedLinear Unit); MLP ( ) represents a multi-layer perceptron, LN ( ) represents the layer normalization operation; AvgPool ( ) represents the average pooling operation; MaxPool ( ) represents the maximum pooling operation; Softmax ( ) represents a vector activation operation.

[0061] The structures of the core teacher model, each auxiliary teacher model and the student model are the same, including a backbone network, a neck network and a head network connected in sequence. The training image set includes several training images. Each time, the same training image is input into the core teacher model, each auxiliary teacher model and the student model. After processing, the classification output and positioning output of the core teacher model, the classification output and positioning output of each auxiliary teacher model and the classification output and positioning output of the student model are output. At the same time, the outputs of the neck networks of the core teacher model and each auxiliary teacher model are processed by the feature fusion module to output a fusion feature map. The fusion feature map is then processed by the head network of the core teacher model to output a fusion classification output and a fusion positioning output. The teacher head network consistency knowledge is constructed based on the classification output and positioning output of each auxiliary teacher model and the core teacher model, and the student head network consistency knowledge is constructed based on the classification output and positioning output of the student model, as follows:

[0062]

[0063] in, CM m,n Represents the first m Detection categories, n Head network consistency knowledge under real frames; P Represents the total set of head network output indexes of the teacher model or student model, A n andB n Respectively represent n The head network of the teacher model or student model under the real frame outputs the first and second index sets, P Include A n , A n Include B n ; k m,b ´ cls Indicates the m The first detection category of the head network b The classification output corresponding to the positioning output is k m " cls Indicates the m The average value of the classification output under the detection categories, k n,b ´ loc Indicates the n The ground truth box and the head network b The intersection-over-combination ratio of the positioning outputs, k n " loc Indicates the n The average of the intersection-over-union (IoU) of the ground-truth boxes and the positioning output of the head network; T 1 indicates softening temperature; k n,p ´ loc Indicates the n The total set of ground truth boxes and head network output indices P The p The intersection-over-combination ratio of the positioning outputs, k p ´ loc Represents the total set of true boxes and head network output indexes P The p The intersection-over-union ratio set of positioning outputs; TopK ( ) indicates before selection K The index of the element; k n,a ´ loc Indicates the n The ground-truth box and the first index set of the head network output A n The a The intersection-over-union ratio of the positioning outputs is calculated. Based on the classification output and positioning output of the core teacher model and each auxiliary teacher model, the multi-teacher consistency knowledge fusion weight and fusion consistency knowledge are constructed. The multi-teacher consistency knowledge fusion weight is as follows:

[0064]

[0065] in, w n,j Indicates the j The first teacher model n The multi-teacher consistent knowledge fusion weight of the ground truth frame; Softmax ( ) represents vector activation operation; w n,j c and w n,j h Respectively represent j The first teacher model n The first and second weight parameters in the multi-teacher consistency knowledge fusion weight of the ground truth frame; k b,j ´ cls Indicates the j The ground truth box of the teacher model and the second index set of the head network output B n The b The classification output set corresponding to the positioning output, k n,b,j ´ cls Indicates the j The first teacher model n The ground-truth box and the head network output second index set B n The b The classification output set corresponding to the positioning output; k b,j ´ loc Indicates the j The ground truth box of the teacher model and the second index set of the head network output B n The b The intersection-over-union ratio set of positioning outputs, k n,b,j ´ loc Indicates the j The first teacher model n The ground-truth box and the head network output second index set B n The b The intersection-and-union ratio set of positioning outputs; ( )´ represents the averaging operation.

[0066] The fusion consistency knowledge is as follows:

[0067]

[0068] in, CM m,n～T Indicates the m The teacher model's n The fusion consistency knowledge of the ground truth boxes; P 1. P 2. ... P j 、…、 P J Represent the first, second, ..., j , ..., J The total set of head network output indexes output by the teacher model, J Represents the total number of teacher models.

[0069] The teacher head network consistency knowledge of each auxiliary teacher model, the teacher head network consistency knowledge of the core teacher model, the student head network consistency knowledge of the student model, the multi-teacher consistency knowledge fusion weight and the fusion consistency knowledge are jointly constructed into a multi-teacher head network consistency knowledge distillation module L MTCD , as follows:

[0070]

[0071] in, λ 1 and λ 2 represent the first and second hyperparameters for balancing the loss respectively; L D and L A They represent the multi-teacher model association knowledge distillation loss function and the multi-teacher model angle knowledge distillation loss function respectively; J represents the total number of teacher models; M and N Represents the total number of detection categories and the total number of true boxes respectively; w n,j Indicates the j The first teacher model n The multi-teacher consistent knowledge fusion weight of the ground truth frame; CM m,n S Represents the student model m Detection categories, n Head network consistency knowledge under real frames, CM j,m,n T Indicates the j The first teacher model m Detection categories, n Head network consistency knowledge under real frames; CM n Indicates the nThe head network consistency knowledge matrix under the real frame, CM 1. CM 2 and CM 3 respectively represent the n The head network consistency knowledge matrix under the ground truth frame CM n The three head networks in the same row have consistent knowledge; l δ ( ) represents the smooth loss Huber function; ψ A ( ) represents the angle relationship calculation function; CM 1,n ～T 、 CM 2,n ～T and CM 3,n ～T They represent the first n The three fused consistent knowledge in the same row in the fused consistent knowledge matrix of the ground-truth box, CM 1,n S 、 CM 2,n S and CM 3,n S Represents the first n The three head network consistency knowledge in the same row in the head network consistency knowledge matrix of the ground truth box; CM 1 CM 2 CM 3 means the n The head network consistency knowledge matrix under the ground truth frame CM n The angular relationship between the three head network consistency knowledge in the same row; e 12 and e 32 Represent the first and second unit vectors, respectively, and their directions are from CM 1 to CM 2 and from CM 3 to CM 2.

[0072] The core teacher neck network consistency knowledge distillation module is constructed based on the output, classification output, and positioning output of the core teacher model's neck network and the output of the student model's neck network. The auxiliary teacher neck network consistency knowledge distillation module is constructed based on the fused feature map, fused classification output, fused positioning output, and the output of the student model's neck network. The details are as follows:

[0073]

[0074] in, I Represents the core teacher model CT or assistant teacher model AT , L I represents the core or auxiliary teacher neck network consistency knowledge distillation module; C 、 H and W Represent the core teacher model CT The feature maps output by the neck network of the student model and the number of channels, height, and width of the fused feature maps; Mask h,w cls,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the h Height and w Classification feature mask at width position, Mask h,w loc,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the h Height and w Position feature mask at width position; f c,h,w ,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the c Channel, h Height and w The response value of the width position, f c,h,w ,S Represents the student model S The feature map of the neck network output is in the c Channel, h Height and w Response value of width position; Softmax ( ) represents vector activation operation; k h,w,m ´ cls,I Represents the teacher model I In the h Height and w The width position positioning output or the fusion positioning output corresponding to the m The classification output or fusion classification output of each detection category, k h,w,n ´ loc,I Represents the teacher model I In theh Height and w The positioning output of the width position or the fusion positioning output and the n The intersection-over-union (IoU) of the ground-truth boxes.

[0075] Constructing a multi-teacher neck network consistency knowledge distillation module based on the core and auxiliary teacher neck network consistency knowledge distillation module L MTFPD ,as follows:

[0076]

[0077] in, λ 3 represents the third hyperparameter for balancing loss; L CT and L AT denote the core and auxiliary teacher neck network consistency knowledge distillation modules respectively; w A represents the adaptive weight; L Det AT represents the loss of training the head network that feeds the fused feature map into the core teacher model; L Det S Represents the training loss function of the student model.

[0078] The constructed target detection training image set is input into the multi-teacher model consistency knowledge distillation architecture for training, and the multi-teacher head and neck network consistency knowledge distillation module is obtained. The training loss function of the student model is used together with the student model's final loss function. The final loss function of the student model is L = L MTCD + L MTFPD + L Det S , until the final loss function converges and the trained student model is obtained.

[0079] The present invention performs inheritance verification, such as Figure 4 (a) and Figure 4As shown in (b), the detection results of the same artifact by RetinaNet-R101 teacher model 1 and RetinaNet-R101 teacher model 2 obtained using two different training seeds are shown. It can be seen that RetinaNet-R101 teacher model 1 mistakenly detects artifact 6 as artifact 3, while RetinaNet-R101 teacher model 2 can correctly identify the artifact. The above two teacher models correspond to RetinaNet-R18 student model 1 and RetinaNet-R18 student model 2, respectively. The knowledge distillation of the student models of the above two teacher models is performed with the same random number seed, as shown in the figure. Figure 4 (c) and Figure 4 As shown in (d), it can be seen that the RetinaNet-R18 student model 1 inherits the incorrect prediction results of its teacher model, while the RetinaNet-R18 student model 2 is able to correctly identify artifact 6.

[0080] After training is completed, the image to be detected is input into the trained student model, and the target detection result of the image is output after processing and displayed on the display. In the specific implementation of the present invention, the target detection RetinaNet-R101 model is used as the teacher model, and the target detection RetinaNet-R18 model is used as the student model. The multi-teacher model consistency knowledge distillation method of the present invention is used to obtain the distilled student model. At the same time, the existing target detection task balanced distillation method TBD (Task-balanced distillation for object detection) is used for comparison. Evaluation is performed on a self-built workpiece detection dataset. During the evaluation, the evaluation results are judged based on the detection accuracy evaluation index mAP (mean Average Precision) and the prediction consistency index SPCC (Spearman Correlation Coefficient). The mean average precision mAP is the average of the average precision over all categories. For multi-category target detection tasks, the average precision of each category is usually calculated and then averaged. The higher the value of the mean average precision, the better the detection performance of the model. The prediction consistency index SPCC is used to test the correlation between the prediction box classification and the positioning ranking. The higher the value, the higher the model prediction consistency. The evaluation results are shown in Table 1. It can be seen that the method of the present invention can effectively improve the detection accuracy and prediction consistency of the student model, and is better than the existing target detection task balanced distillation method TBD.

[0081] Table 1

[0082]

[0083] like Figure 5As shown in Figure 2, the comparison of the detection effects of the student model before and after some distillation is shown. Figure 5 (a) shows the detection result of the initial student model, as shown in Figure 5 (b) shows the detection results of the student model after distillation. The numbers in the upper left corner of the prediction box represent the artifact category and classification confidence, respectively. After distillation using our method, the student model reduces false positives and missed detections, and improves prediction consistency.

[0084] The present invention also designs a multi-teacher model consistency knowledge distillation system, which includes a data acquisition module, a model construction module and a target detection display module. The data acquisition module acquires images for target detection; the model construction module builds a multi-teacher model consistency knowledge distillation architecture including a core teacher model, several auxiliary teacher models and a student model and trains the student model; the target detection display module uses the trained student model to process the target detection image, outputs the target detection result of the image and displays it on the display.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0086] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented using various computer languages. The present application is described based on the flowcharts of the methods, systems, and computer program products of the embodiments of the present application.

[0087] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concepts. Therefore, the present invention is intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0088] Obviously, those skilled in the art may make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the equivalent technology of the present invention, the present application is intended to include these modifications and variations.

Claims

1. A knowledge distillation method for multi-teacher model consistency, characterized by: include: Step S1: Build a multi-teacher model consistency knowledge distillation architecture including a core teacher model, several auxiliary teacher models, and a student model. Construct a training image set for target detection and input it into the multi-teacher model consistency knowledge distillation architecture for training. Obtain a multi-teacher head and neck network consistency knowledge distillation module, and use it together with the student model's training loss function as the student model's final loss function until the final loss function converges to obtain a trained student model. Step S2: Input the image to be detected into the trained student model, and after processing, output the target detection result of the image and display it on the display; In the step S1, the multi-teacher model consistency knowledge distillation architecture also includes a feature fusion module, which connects the outputs of the core teacher model and the teacher neck network of each auxiliary teacher model in the channel dimension; the core teacher model, each auxiliary teacher model and the student model have the same structure, including a backbone network, a neck network and a head network connected in sequence, and the training image set includes several training images. Each time the same training image is input into the core teacher model, each auxiliary teacher model and the student model, the classification output and positioning output of the core teacher model, the classification output and positioning output of each auxiliary teacher model and the classification output and positioning output of the student model are output after processing. At the same time, the outputs of the neck networks of the core teacher model and each auxiliary teacher model are processed by the feature fusion module to output a fusion feature map, and the fusion feature map is processed by the head network of the core teacher model to output a fusion classification output and a fusion positioning output; the teacher head network consistency knowledge is constructed according to the classification output and positioning output of each auxiliary teacher model and the core teacher model, and the student head network is constructed according to the classification output and positioning output of the student model. Network consistency knowledge, multi-teacher consistency knowledge fusion weights and fusion consistency knowledge are constructed according to the classification output and positioning output of the core teacher model and each auxiliary teacher model, and the teacher head network consistency knowledge of each auxiliary teacher model, the teacher head network consistency knowledge of the core teacher model, the student head network consistency knowledge of the student model, the multi-teacher consistency knowledge fusion weights and fusion consistency knowledge are jointly constructed into a multi-teacher head network consistency knowledge distillation module; according to the output, classification output and positioning output of the neck network of the core teacher model and the output of the neck network of the student model, a core teacher neck network consistency knowledge distillation module is constructed; according to the fusion feature map, fusion classification output and fusion positioning output and the output of the neck network of the student model, an auxiliary teacher neck network consistency knowledge distillation module is constructed; based on the core and auxiliary teacher neck network consistency knowledge distillation modules, a multi-teacher neck network consistency knowledge distillation module is constructed; the multi-teacher head network consistency knowledge distillation module, the multi-teacher neck network consistency knowledge distillation module and the training loss function of the student model are jointly used as the final loss function of the student model.

2. The multi-teacher model consistency knowledge distillation method according to claim 1, characterized in that: The feature fusion module is as follows: f ama = M ama ( f c ) f c =Concat ( f j )， j =1、2、…、 J M ama ( f c )= W A ( f c )+ W C ( f c )· W G ( f c )) W A ( f c )= Conv 2 d ( ReLU ( Conv 2 d ( f c ))) W C ( f c )= MLP ( AvgPool ( f c ))+ MLP ( MaxPool ( f c )) W G ( f c )= MLP ( Softmax ( Conv 2 d ( f c ))× f c ) in, f ama Represents the fused feature map; f c represents the connection feature graph; Concat ( ) represents the channel dimension connection operation; M ama () indicates feature module; f j Indicates the j The output of the neck network of the teacher model, J Indicates the total number of core teacher models and auxiliary teacher models; W A (), W C ()and W G () denote the first, second and third features respectively; Conv 2 d ( ) represents a two-dimensional convolution operation; ReLU ( ) represents the activation operation of the rectified linear unit; MLP ( ) represents a multilayer perceptron; AvgPool () represents the average pooling operation; MaxPool () represents the maximum pooling operation; Softmax ( ) represents a vector activation operation.

3. The multi-teacher model consistency knowledge distillation method according to claim 1, characterized in that: The multi-teacher head network consistency knowledge distillation module L MTCD as follows: in, λ 1 and λ 2 denotes the first and second hyperparameters respectively; L D and L A They represent the multi-teacher model association knowledge distillation loss function and the multi-teacher model angle knowledge distillation loss function respectively; J represents the total number of teacher models; M and N Represents the total number of detection categories and the total number of true boxes respectively; w n,j Indicates the j The first teacher model n The multi-teacher consistent knowledge fusion weight of the ground truth frame; CM m,n S Represents the student model m Detection categories, n Head network consistency knowledge under real frames, CM j,m,n T Indicates the j The first teacher model m Detection categories, n Head network consistency knowledge under real frames; CM n Indicates the n The head network consistency knowledge matrix under the real frame, CM 1. CM 2 and CM 3 respectively represent the n The head network consistency knowledge matrix under the ground truth frame CM n The three head networks in the same row have consistent knowledge; l δ ( ) represents the smooth loss Huber function; ψ A ( ) represents the angle relationship calculation function; CM 1,n ～T 、 CM 2,n ～T and CM 3,n ～T They represent the first n The three fused consistent knowledge in the same row in the fused consistent knowledge matrix of the ground-truth box, CM 1,n S 、 CM 2,n S and CM 3,n S Represents the first n The three head network consistency knowledge in the same row in the head network consistency knowledge matrix of the ground truth box; CM 1 CM 2 CM 3 means the n The head network consistency knowledge matrix under the ground truth frame CM n The angular relationship between the three head network consistency knowledge in the same row; e 12 and e 32 denote the first and second unit vectors respectively.

4. The multi-teacher model consistency knowledge distillation method according to claim 1, characterized in that: The head network consistency knowledge of the teacher model and the student model is as follows: in, CM m,n Represents the first m Detection categories, n Head network consistency knowledge under real frames; P Represents the total set of head network output indexes of the teacher model or student model, A n and B n Respectively represent n The head network of the teacher model or the student model under the real frame outputs the first and second index sets; k m,b ´ cls Indicates the m The first detection category of the head network b The classification output corresponding to the positioning output is k m " cls Indicates the m The average value of the classification output under the detection categories, k n,b ´ loc Indicates the n The ground truth box and the head network b The intersection-over-combination ratio of the positioning outputs, k n " loc Indicates the n The average of the intersection-over-union (IoU) of the ground-truth boxes and the positioning output of the head network; T 1 indicates softening temperature; k n,p ´ loc Indicates the n The total set of ground truth boxes and head network output indices P The p The intersection-over-combination ratio of the positioning outputs, k p ´ loc Represents the total set of true boxes and head network output indexes P The p The intersection-over-union ratio set of positioning outputs; TopK ( ) indicates before selection K The index of the element; k n,a ´ loc Indicates the n The ground-truth box and the first index set of the head network output A n The a The intersection-over-union ratio of the positioning outputs.

5. The multi-teacher model consistency knowledge distillation method according to claim 1, characterized in that: The multi-teacher consistent knowledge fusion weights are as follows: in, w n,j Indicates the j The first teacher model n The multi-teacher consistent knowledge fusion weight of the ground truth frame; Softmax ( ) represents vector activation operation; w n,j c and w n,j h Respectively represent j The first teacher model n The first and second weight parameters in the multi-teacher consistency knowledge fusion weight of the ground truth frame; k b,j ´ cls Indicates the j The ground truth box of the teacher model and the second index set of the head network output B n The b The classification output set corresponding to the positioning output, k n,b,j ´ cls Indicates the j The first teacher model n The ground-truth box and the head network output second index set B n The b The classification output set corresponding to the positioning output; k b,j ´ loc Indicates the j The ground truth box of the teacher model and the second index set of the head network output B n The b The intersection-over-union ratio set of positioning outputs, k n,b,j ´ loc Indicates the j The first teacher model n The ground-truth box and the head network output second index set B n The b The intersection-and-union ratio set of positioning outputs; ( )´ represents the averaging operation.

6. The method for knowledge distillation based on multi-teacher model consistency according to claim 4, characterized in that: The fusion consistency knowledge is as follows: in, CM m,n ～T Indicates the m The teacher model's n The fusion consistency knowledge of the ground truth boxes; P 1. P 2. ... P j 、…、 P J Respectively represent the first, second, ..., j , ..., J The total set of head network output indexes output by the teacher model, J Represents the total number of teacher models.

7. The multi-teacher model consistency knowledge distillation method according to claim 1, characterized in that: The multi-teacher neck network consistency knowledge distillation module L MTFPD as follows: in, λ 3 represents the third hyperparameter; L CT and L AT denote the core and auxiliary teacher neck network consistency knowledge distillation modules respectively; w A represents the adaptive weight; L Det AT represents the loss of training the head network that feeds the fused feature map into the core teacher model; L Det S Represents the training loss function of the student model.

8. The multi-teacher model consistency knowledge distillation method according to claim 1, characterized in that: The core and auxiliary teacher neck network consistency knowledge distillation modules are as follows: in, I Represents the core teacher model CT or assistant teacher model AT , L I represents the core or auxiliary teacher neck network consistency knowledge distillation module; C 、 H and W Represent the core teacher model CT The feature maps output by the neck network of the student model and the number of channels, height, and width of the fused feature maps; Mask h,w cls,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the h Height and w Classification feature mask at width position, Mask h,w loc,I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the h Height and w Position feature mask at width position; f c,h,w I Represents the core teacher model CT The feature map or fusion feature map output by the neck network is in the c Channel, h Height and w The response value of the width position, f c,h,w S Represents the student model S The feature map of the neck network output is in the c Channel, h Height and w Response value of width position; Softmax ( ) represents vector activation operation; k h,w,m ´ cls,I Represents the teacher model I In the h Height and w The width position positioning output or the fusion positioning output corresponding to the m The classification output or fusion classification output of each detection category, k h,w,n ´ loc,I Represents the teacher model I In the h Height and w The positioning output of the width position or the fusion positioning output and the n The intersection-over-union (IoU) of the ground-truth boxes.

9. A multi-teacher model consistency knowledge distillation system applicable to the method according to any one of claims 1 to 8, characterized in that: include: Data acquisition module, acquires images for target detection; The model construction module builds a multi-teacher model consistency knowledge distillation architecture consisting of a core teacher model, several auxiliary teacher models, and a student model, and trains the student model; The target detection display module uses the trained student model to process the target detection image, outputs the target detection result of the image and displays it on the display.

Images