Object Detection Method Based on Self-Distillation and Self-Attention

Abstract

The present invention proposes an object detection method based on self-distillation and self-attention, abandons the traditional two-stage knowledge distillation algorithm, uses a one-stage self-distillation algorithm to extract the prosperous features of the detection target using mixed images, introduces the recently emerging Transformer architecture, combines the local features of the convolutional neural network with the global features of the Transformer, and directly uses a three-level threshold on Faster-RCNN to screen out high-level candidate boxes, and finally performs accurate object detection.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and computer vision technology, and in particular relates to a target detection method based on self-distillation and self-attention. Background Art

[0002] Among the many technical fields of computer vision, target detection is also a very basic task. Image segmentation, object tracking, key point detection, etc. usually rely on target detection. When performing target detection, since the number, size and posture of objects in each image are different, that is, unstructured output, this is very different from image classification, and objects are often occluded and truncated, so object detection technology is also very challenging. Since its birth, it has always been one of the focus areas of researchers. Because target detection has a wide range of applications and huge commercial value, academia and industry are constantly exploring new target detection algorithms. In recent years, with the help of deep learning, convolutional neural networks and the new transformer architecture, major breakthroughs have been made in the field of computer vision. The accuracy and lightweight of target detection algorithms are also being studied continuously, and many landing products have been produced.

[0003] Although target detection technology has made great progress, the detection accuracy in more complex environments still needs to be improved, and higher requirements are also placed on the proposed target detection algorithm. Summary of the invention

[0004] Among the many technical fields of computer vision, target detection is also a very basic task. Image segmentation, object tracking, key point detection, etc. usually rely on target detection. However, due to the fact that the target objects are blocked, too small, and affected by environmental factors in real scenes, the accuracy of detection needs to be improved.

[0005] In order to solve the above pain points, the present invention proposes a target detection method based on self-distillation and self-attention, abandons the traditional two-stage algorithm of knowledge distillation, uses a one-stage self-distillation algorithm to extract the complex features of the detection target using mixed images, and introduces the Transformer architecture to combine the local features of the convolutional neural network with the global features of the Transformer. The three-level threshold is directly used on Faster-RCNN to screen out high-level candidate boxes, and finally accurate target detection is performed.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A method for object detection based on self-distillation and self-attention, characterized in that it comprises the following steps:

[0008] Step S1: Introduce the self-distillation mechanism on the general target detection dataset to train the feature extraction network;

[0009] Step S2: extract the network features of each layer obtained by training in step S1, and use the Transformer network to perform secondary feature extraction;

[0010] Step S3: The features extracted from step S1 and step S2 are fused using a feature fusion strategy and an attention mechanism;

[0011] Step S4: Use the Faster-RCNN target detection model to classify and regress the acquired features to obtain the target detection results.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] Step S11: Obtain a public general object detection dataset from the Internet and obtain relevant annotations of the training data;

[0014] Step S12: Randomly sample two images from the training set of the general target detection dataset, and select them as p and 1 and p 2 , in p 1 and p 2 Perform pixel-level image blending between them to construct a mixed image p mixup ; Mixed image p mixup Tags mixup is the true label q encoded with one-hot in the same mixing ratio 1 and q 2 Linear interpolation between 1 and q 2 They represent the original image p 1 and p 2 The true label of

[0015] Step S13: Mix the mixed image p mixup and two original images p 1 and p 2 Put them into ResNet101 model training together, the original image p 1 and p 2 The inferred probability distribution is and And use linear interpolation to infer the interpolation probability distribution of the mixed image: Mixed image p mixup The inferred probability distribution is By inferring the interpolation probability distribution of the mixed image Probability distribution of mixed images predicted by the model Perform supervision; interpolation probability distribution of mixed images The calculation formula is as follows:

[0016]

[0017] Among them, λ represents the original image p 1 and p 2 Mixing ratio, inferred probability distribution of mixed images H, W, C represent the height, width, and number of channels of the feature probability distribution inferred from the image, respectively, and R represents a real number set;

[0018] Step S14: Using square L 2 The norm realizes the mutual alignment between the interpolated probability distribution of the mixed image and the predicted probability distribution of the mixed image. The calculation formula is as follows:

[0019]

[0020] Among them, L feature It represents the loss value used to align the interpolated probability distribution and the predicted probability distribution of the mixed image;

[0021] Step S15: Introduce the discriminator to the feature map of the same stage The discriminator classifies the feature maps generated from the linear interpolation or mixed images. The loss function L of the discriminator is skd The binary cross entropy loss used is calculated as follows:

[0022]

[0023] Among them, K represents the number of pictures input to the model at one time, and k is an enumeration of K from the beginning.

[0024] Further, step S2 includes the following steps:

[0025] Step S21: In step S13, the image is put into the ResNet101 model for feature extraction, and the multi-layer features of the extracted ResNet101 model are introduced into the Transformer architecture to extract more advanced features: the third layer feature output by the ResNet101 model is defined as f res3 , define the fourth layer feature output of the ResNet101 model as f res4 , define the third layer feature output of the Transformer model as s res3 , define the fourth layer feature output of the Transformer model as s res4 , introduce the self-attention mechanism, first res3 and f res4Downsample, and then f res3 and f res4 RelAttention mechanism is used to extract more advanced features, and the output new advanced feature map is f′ res3 and f′ res4 ;

[0026] Step S22: In order to consolidate the original information, the FFN network is introduced to obtain the new high-level feature map f′ res3 and f′ res4 Input it, and introduce the residual mechanism on this basis, and convert f res3 and f res4 Adding the obtained high-level features, we finally get the two-layer features output by the Transformer architecture, which are s res3 and res4 , which is calculated as follows:

[0027] s res3 =f res3 +FFN(RelAttention(downsample(f res3 )))

[0028] s res4 =f res4 +FFN(RelAttention(downsample(f res4 )))

[0029] Among them, FFN(·) represents a feed-forward neural network, and downsample(·) represents a downsampling operation.

[0030] Furthermore, step S3 specifically includes the following steps:

[0031] Step S31: The high-level features s obtained from the Transformer model res3 and res4 Perform dimension matching and directly convert s res3 Perform a 1×1 convolution to obtain features of dimension C;

[0032] Step S32: Perform feature fusion on the features obtained in step S1 and step S2. First, s res3 and f res3 To perform feature fusion and make full use of the captured location information so that the area of ​​interest can be accurately captured, the CoordinateAttention mechanism is introduced to the fused s′ res3 Perform feature enhancement;

[0033] Step S33: res4 、fres4 , s′ res3 To perform feature fusion, the activated neurons are first passed through an LRN module to suppress the surrounding neurons to increase the generalization ability of the features. Finally, the three features are spliced ​​together along the channel number dimension to obtain the final high-level feature s final .

[0034] Furthermore, step S4 specifically includes the following steps:

[0035] Step S41: Using Faster-RCNN, first use the RPN structure to generate candidate boxes, project the candidate boxes generated by the RPN onto the feature map to obtain the corresponding feature matrix; when generating candidate boxes, increase the threshold for screening candidate boxes, and increase the number of selected candidate boxes, and screen the candidate boxes successively with thresholds of 0.5, 0.6, and 0.7, respectively, and finally obtain high-quality candidate boxes;

[0036] Step S42: Each feature matrix is ​​scaled to a feature map of size 7×7 through the ROI Pooling layer, and then the feature map is flattened and passed through a series of fully connected layers to obtain the regression and classification prediction results of target detection.

[0037] Compared with the prior art, the present invention and its preferred embodiment have the following beneficial effects:

[0038] 1. Unlike traditional knowledge distillation, there is no need to introduce two stages. The network itself can directly generate soft labels to supervise itself, which greatly reduces the complexity of the network and avoids two-stage model training.

[0039] 2. The features of mixed images are fully utilized for self-distillation, which greatly improves the generalization ability of the model.

[0040] 3. In view of the fact that traditional convolutional networks can extract local information, while Transformer networks can extract global information, we combine the advantages of the two and fuse the features, which can extract more advanced features and achieve better detection results.

[0041] 4. In order to adapt to objects of various sizes and better locate the target, a three-level progressive candidate box threshold is introduced for screening, which can obtain higher-quality candidate boxes and improve the accuracy of the target detection regression task. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] Figure 1 It is a flow chart of the method implementation of the embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:

[0045] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] like Figure 1 As shown, an embodiment of the present invention provides a target detection method based on self-distillation and self-attention, comprising the following steps:

[0048] Step S1: Introduce the self-distillation mechanism on the general target detection dataset to train the feature extraction network. Specifically, it includes the following steps:

[0049] Step S11: Obtain a public general object detection dataset from the Internet and obtain relevant annotations of the training data;

[0050] Step S12: Randomly sample two images from the training set of the general target detection dataset, and select them as p and 1 and p 2 , in p 1 and p 2 Perform pixel-level image blending between them to construct a mixed image p mixup . Mixed image p mixup Tags mixup is the true label q encoded with one-hot in the same mixing ratio 1 and q 2 Linear interpolation between 1 and q 2 They represent the original image p 1 and p 2 The true label of

[0051] Step S13: Mix the mixed image p mixup and two original images p 1 and p 2 Put them into ResNet101 model training together, the original image p 1and p 2 The inferred probability distribution is and And use linear interpolation to infer the interpolation probability distribution of the mixed image: Mixed image p mixup The inferred probability distribution is The interpolation probability distribution of the inferred mixed image can be To predict the probability distribution of the mixed image Interpolation probability distribution of mixed images The calculation formula is as follows:

[0052]

[0053] Among them, λ represents the original image p 1 and p 2 Mixing ratio, inferred probability distribution of mixed images H, W, C represent the height, width, and number of channels of the feature probability distribution inferred from the image, respectively, and R represents the set of real numbers;

[0054] Step S14: Using square L 2 The norm is used to realize the mutual alignment between the interpolation probability distribution of the mixed image and the predicted probability distribution of the mixed image. The calculation formula is as follows:

[0055]

[0056] Among them, L feature It represents the loss value used to align the interpolated probability distribution and the predicted probability distribution of the mixed image;

[0057] Step S15: Inspired by adversarial feature distillation, a discriminator is further introduced into the feature map of the same stage The discriminator classifies the feature maps generated from the linear interpolation or mixed images. The loss function L of the discriminator is skd The binary cross entropy loss used is calculated as follows:

[0058]

[0059] Among them, K represents the number of pictures input to the model at one time, and k is an enumeration of K from the beginning.

[0060] Step S2: Extract the network features of each layer trained in step S1, and use the Transformer network to perform secondary feature extraction. Specifically, it includes the following steps:

[0061] Step S21: In step S13, the image is placed in the ResNet101 model for feature extraction. The multi-layer features of the extracted ResNet101 model are introduced into the Transformer architecture to extract more advanced features. The third layer feature of the ResNet101 model is defined as f res3 , define the fourth layer feature of the ResNet101 model as f res4 , define the third layer feature of the Transformer model as s res3 , define the fourth layer feature of the Transformer model as s res4 , introduce the self-attention mechanism, first res3 and f res4 Downsample, and then f res3 and f res4 RelAttention mechanism is used to extract more advanced features, and the output new advanced feature map is f′ res3 and f′ res4 ;

[0062] Step S22: In order to consolidate the original information, the FFN network is introduced to obtain the new high-level feature map f′ res3 and f′ res4 Input it, and introduce the residual mechanism on this basis, and convert the previous f res3 and f res4 Adding the obtained high-level features, we finally get the two-layer features of the Transformer architecture, which are s res3 and res4 , which is calculated as follows:

[0063] S res3 =f res3 +FFN(RelAttention(downsample(f res3 )))

[0064] s res4 =f res4 +FFN(RelAttention(downsample(f res4 )))

[0065] Among them, FFN(·) represents a feed-forward neural network, and downsample(·) represents a downsampling operation.

[0066] Step S3: The features extracted from step S1 and step S2 are fused using feature fusion strategy and attention mechanism. Specifically, the following steps are included:

[0067] Step S31: The high-level features s obtained from the Transformer model res3 and res4 Perform dimension matching and directly convert s res3 Perform a 1×1 convolution to obtain features of dimension C;

[0068] Step S32: Perform feature fusion on the features obtained in step S1 and step S2. First, s res3 and f res3 To perform feature fusion and make full use of the captured location information so that the area of ​​interest can be accurately captured, the CoordinateAttention mechanism is introduced to the fused s′ res3 Perform feature enhancement;

[0069] Step S33: res4 、f res4 , s′ res3 To perform feature fusion, we first pass through an LRN (Local Response Normalization) module. The LRN module draws on the concept of lateral inhibition in neurobiology, that is, activated neurons inhibit surrounding neurons, which increases the generalization ability of features. Finally, the three features are spliced ​​together along the channel number dimension to obtain the final high-level feature s final .

[0070] Step S4: Use the Faster-RCNN target detection model to classify and regress the acquired features to obtain good target detection results. Specifically, the following steps are included:

[0071] Step S41: The target detection model used in the method of this embodiment is Faster-RCNN, which has high detection accuracy. First, the RPN structure is used to generate candidate boxes, and the candidate boxes generated by the RPN are projected onto the feature map to obtain the corresponding feature matrix. Different from the traditional method, as a preferred embodiment, in generating candidate boxes, the threshold for screening candidate boxes is increased, and the number of selected candidate boxes is increased. In addition, the candidate boxes are screened successively with thresholds of 0.5, 0.6, and 0.7 in a layer-by-layer progressive manner, and finally high-quality candidate boxes are obtained;

[0072] Step S42: Each feature matrix is ​​scaled to a feature map of size 7×7 through the ROI Pooling layer, and then the feature map is flattened and passed through a series of fully connected layers to obtain the regression and classification prediction results of target detection.

[0073] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0074] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0078] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of target detection methods based on self-distillation and self-attention under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. An object detection method based on self-distillation and self-attention, characterized in that: The following steps are involved: Step S1: Introduce the self-distillation mechanism on the general target detection dataset to train the feature extraction network; Step S2: extract the network features of each layer obtained by training in step S1, and use the Transformer network to perform secondary feature extraction; Step S3: The features extracted from step S1 and step S2 are fused using a feature fusion strategy and an attention mechanism; Step S4: Use the Faster-RCNN target detection model to classify and regress the acquired features to obtain target detection results; Step S1 specifically includes the following steps: Step S11: Obtain a public general object detection dataset from the Internet and obtain relevant annotations of the training data; Step S12: Randomly sample the training set of the general target detection dataset, select two images p1 and p2, perform pixel-level image blending between p1 and p2, and construct a blended image p mixup ; Mixed image p mixup Tags mixup It is a linear interpolation between the true labels q1 and q2 encoded with one-hot in the same mixing ratio, where q1 and q2 represent the true labels of the original images p1 and p2 respectively; Step S13: Mix the mixed image p mixup Put it into ResNet101 model training together with the two original images p1 and p2. The probability distribution inferred from the original images p1 and p2 is and And use linear interpolation to infer the interpolation probability distribution of the mixed image: Mixed image p mixup The inferred probability distribution is By inferring the interpolation probability distribution of the mixed image Probability distribution of mixed images predicted by the model Perform supervision; interpolation probability distribution of mixed images The calculation formula is as follows: Among them, λ represents the mixing ratio of the original images p1 and p2, and the inferred probability distribution of the mixed image H, W, C represent the height, width, and number of channels of the feature probability distribution inferred from the image, respectively, and R represents a real number set; Step S14: using the squared L2 norm to achieve mutual alignment between the interpolated probability distribution of the mixed image and the predicted probability distribution of the mixed image, the calculation formula is as follows: Among them, L feature It represents the loss value used to align the interpolated probability distribution and the predicted probability distribution of the mixed image; Step S15: Introduce the discriminator to the feature map of the same stage The discriminator classifies the feature maps generated from the linear interpolation or mixed images. The loss function L of the discriminator is skd The binary cross entropy loss used is calculated as follows: Among them, K represents the number of pictures input to the model at one time, and k is the enumeration of K from the beginning; Step S2 includes the following steps: Step S21: In step S13, the image is put into the ResNet101 model for feature extraction, and the multi-layer features of the extracted ResNet101 model are introduced into the Transformer architecture to extract more advanced features: the third layer feature output by the ResNet101 model is defined as f res3 , define the fourth layer feature output of the ResNet101 model as f res4 , define the third layer feature output of the Transformer model as s res3 , define the fourth layer feature output of the Transformer model as s res4 , introduce the self-attention mechanism, first res3 and f res4 Downsample, and then f res3 and f res4 RelAttention mechanism is used to extract more advanced features, and the output new advanced feature map is f' res3 and f' res4 ; Step S22: In order to consolidate the original information, the FFN network is introduced to obtain the new high-level feature map f' res3 and f' res4 Input it, and introduce the residual mechanism on this basis, and convert f res3 and f res4 Adding the obtained high-level features, we finally get the two-layer features output by the Transformer architecture, which are s res3 and res4 , which is calculated as follows: s res3 =f res3 +FFN(RelAttention(downsample(f res3 ))) s res4 =f res4 +FFN(RelAttention(downsample(f res4 ))) Among them, FFN(·) represents a feedforward neural network, downsample(·) represents a downsampling operation; Step S3 specifically includes the following steps: Step S31: The high-level features s obtained from the Transformer model res3 and res4 Perform dimension matching and directly convert s res3 Perform a 1×1 convolution to obtain features of dimension C; Step S32: Perform feature fusion on the features obtained in step S1 and step S2. First, s res3 and f res3 To perform feature fusion and make full use of the captured location information so that the area of ​​interest can be accurately captured, the Coordinate Attention mechanism is introduced to the fused s' res3 Perform feature enhancement; Step S33: res4 、f res4 、s' res3 To perform feature fusion, the activated neurons are first passed through an LRN module to suppress the surrounding neurons to increase the generalization ability of the features. Finally, the three features are spliced ​​together along the channel number dimension to obtain the final high-level feature s final .

2. The object detection method based on self-distillation and self-attention according to claim 1, characterized in that Step S4 specifically includes the following steps: Step S41: Using Faster-RCNN, first use the RPN structure to generate candidate boxes, project the candidate boxes generated by the RPN onto the feature map to obtain the corresponding feature matrix; when generating candidate boxes, increase the threshold for screening candidate boxes, and increase the number of selected candidate boxes, and screen the candidate boxes successively with thresholds of 0.5, 0.6, and 0.7, respectively, and finally obtain high-quality candidate boxes; Step S42: Scale each feature matrix to a feature map of size 7×7 through the ROIPooling layer, and then flatten the feature map through a series of fully connected layers to obtain the regression and classification prediction results of target detection.

Images