Anti-shielding target detection method and device and storage medium thereof

Abstract

The invention relates to a target detection method under a shielding condition. The method comprises the following steps: creating a shielding data set, randomly sampling a background part, and then obtaining shielding samples according to different filling modes; according to the method, in order to better extract information of a space dimension and a channel dimension, an attention mechanism fusing the space and channel information is added, a multi-scale attention mechanism using a sliding window is added, and the accuracy of the attention mechanism is improved. The relation between global and local features is enhanced; a knowledge distillation strategy is used to guide learning of features of the visible part of the target; calculating classification loss, regression loss and distillation loss; calculating an overall loss value, and training the model by using the overall loss value; and performing target detection on the to-be-detected image by using the target detection model. The method effectively reduces the influence of shielding on the target detection performance.

Description

Technical Field

[0001] The invention relates to an image detection method, in particular to an anti-occlusion target detection method, a device and a storage medium thereof, belonging to the technical field of image processing. Background Art

[0002] In recent years, with the development of deep learning, especially the introduction of convolutional neural networks (CNN) and attention mechanisms, target detection algorithms have been significantly improved in accuracy and speed. Convolutional neural networks (CNN) are one of the most commonly used algorithm frameworks in the current field of target detection. Convolutional neural networks extract spatial features of images through convolutional layers. They are the core technology for processing image data and perform well in processing visual tasks. The R-CNN series is a classic algorithm in target detection. It proposes a detection framework based on region proposals, which greatly promotes the application of deep learning in target detection. The YOLO series is a single-stage target detection method. Its core idea is to transform target detection into a regression problem. The YOLO algorithm predicts target categories and bounding boxes simultaneously through a single convolutional neural network, which is suitable for real-time detection tasks.

[0003] At present, the detection algorithm based on the attention mechanism can significantly improve the feature expression ability of the target. The attention mechanism originated from the field of natural language processing (NLP), but in recent years it has been widely used in computer vision tasks, especially in target detection, which can effectively improve the performance of the model in complex scenes. The attention mechanism can enhance the model's attention to important features and suppress the influence of irrelevant features, thereby improving the accuracy of target detection. For example, Swin Transformer uses a hierarchical window self-attention mechanism (Shifted WindowAttention) to model global and local information in the image. It significantly improves the computational efficiency by calculating attention within the local window and shifting the window at different levels. Now some new studies try to combine CNN and attention mechanism to take advantage of CNN's ability to extract local features, while using the attention mechanism to capture global information and improve the accuracy and robustness of target detection.

[0004] Although various algorithms have shown good detection results in various data sets in the field of target detection, in some cases, the target information will be occluded for various reasons, resulting in significant changes in the appearance of the object under different occlusion states. The detection accuracy of these algorithms will decrease with the increase of the degree of occlusion. In today's robot inspection tasks, detecting the status of the machine is a crucial task. This task can help people manage and maintain the machine efficiently, while reducing the labor intensity and cost associated with inspection. However, in practical applications, it is a common phenomenon that some parts of the machine are occluded. When the object to be detected is occluded, the detection accuracy will be very sensitive to the threshold. The detection threshold of the occluded object is low, resulting in missed detection. If the threshold is lowered, this will greatly increase the risk of false detection in the target detection task. In this case, the occluded part only contains limited target information, resulting in significant changes in the appearance of the object under different occlusion states. The traditional target detection framework is difficult to adapt to the needs of severe occlusion and cannot effectively utilize the features of the unoccluded part. This is mainly because the model cannot effectively learn the significant feature changes caused by occlusion. Therefore, a more sophisticated algorithm is needed to solve the occlusion problem. . Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an anti-occlusion target detection method. A new attention module is added on the basis of the YOLOV8 model. This module can simultaneously perceive global coarse-grained information and local fine-grained information, thereby enhancing the learning of occluded target features; an occlusion data set is designed, and random occlusion of the image is added to improve the detection capability when there is occlusion; a knowledge distillation strategy is used, in which the teacher network is trained using a data set without occlusion, and the student network is trained using an occlusion data set, thereby guiding the student network to learn from the teacher network on a local feature alignment scale, thereby reducing the distance between the features extracted in the occluded and unoccluded cases, thereby enhancing the robustness of the model to occluded target detection.

[0006] In order to solve the above technical problems, the present invention provides an anti-occlusion target detection method, comprising the following steps:

[0007] ⑴ Construct training samples, add random occlusions on the basis of the original samples, randomly sample the background outside the target, and fill it into the target object, using different filling methods to construct an occlusion data set;

[0008] (2) Input the occluded data samples into the model, extract features from the samples through the feature extraction network, and obtain fine features through the attention network;

[0009] ⑶ Use the unoccluded data set to train a teacher network. The teacher network is trained based on the classification loss and regression loss. The teacher network can effectively extract the feature information of the unoccluded image, and the student network is trained using the knowledge distillation method.

[0010] (4) Calculate the overall loss value based on the classification loss, regression loss and distillation loss, and use the overall loss value to train the student network to obtain a student network that can detect occlusion;

[0011] (5) Using the student network to perform target detection on the image to be detected to detect the occluded target information in the image to be detected;

[0012] (6) Use the post-processing module to filter the occluded target information and output the optimal prediction result for each target.

[0013] The further limiting technical feature of the present invention is that the aforementioned anti-occlusion target detection method builds the occlusion data set, samples the background area of ​​the sample using a random sampling method, and fills it into the target area using different filling methods. The specific method for constructing the occlusion data set is: directly sampling the image background area, randomly sampling the background outside the target, and filling it into the target object; the filling adopts different filling method combinations such as top, bottom, left, right and grid filling methods when training the student network.

[0014] In the aforementioned anti-occlusion target detection method, the size of the target detection image is C×H×W, where H×W is the image size and C is the number of image channels; when the target detection image is subjected to feature extraction based on the YOLOV8 backbone network, an attention enhancement network is added after the feature extraction module, and the attention enhancement network is composed of a convolutional attention module and a multi-scale attention module;

[0015] The convolutional attention module consists of two sub-modules: a spatial attention module and a channel attention module;

[0016] The multi-scale attention module consists of a local attention module and a global attention module;

[0017] The feature extraction module obtains the feature map F, which is used as the input of the attention enhancement network. The convolutional attention module obtains the channel attention and spatial attention enhanced feature F caug ; The feature map F is passed through the multi-scale attention module to obtain the multi-scale attention feature F maug , combined with F caug and F maug Get the final attention-enhanced features.

[0018] The aforementioned anti-occlusion target detection method, the knowledge distillation training method uses the information of the unoccluded part to infer the information of the entire target, and the specific training method is:

[0019] First, a teacher network is trained using an unoccluded dataset, and the teacher network extracts feature information of the unoccluded image. Then, a student network is trained using the occluded data and the features extracted by the teacher network. Through training, the features extracted by the student network are guided to align with the teacher network, reducing the distance between the features extracted in the occluded and unoccluded conditions, thereby inferring the information of the entire target by utilizing the information of the unoccluded part.

[0020] In the aforementioned anti-occlusion target detection method, the classification loss adopts the improved cross entropy loss, and the calculation method is:

[0021]

[0022] Among them, q is the intersection-over-union ratio of the predicted box and the true box. The intersection-over-union ratio is calculated by dividing the intersection of the predicted box and the true box by the union of the two boxes. p is the predicted score, that is, the probability. When the two boxes intersect, q>0, which is a positive sample, otherwise it is a negative sample. α and γ are hyperparameters used to adjust the proportion of positive and negative samples in the loss.

[0023] The regression loss is calculated as:

[0024]

[0025] Among them, IoU is the intersection over union ratio, b and b gt Represent the predicted box and the true box respectively, ρ represents the Euclidean distance between the two boxes, c represents the diagonal distance of the closed area of ​​the two boxes, v represents the consistency of the relative proportion between the two boxes, and α is a hyperparameter that adjusts the weight of the loss ratio;

[0026] The calculation method of the distillation loss is:

[0027]

[0028] Among them, F teacher and F student Represent the features extracted from the teacher network and the student network respectively.

[0029] The overall loss value is calculated as:

[0030] L sum =αL cls +βL reg +γL Dist

[0031] Among them, α, β, and γ are all hyperparameters used to adjust the weights of classification loss, regression loss, and distillation loss.

[0032] The convolutional attention enhancement network of the present invention sequentially calls the channel attention network based on the feature map F to calculate the channel attention F c And the spatial attention network calculates the spatial attention F s , the feature map F after convolutional attention enhancement caug The calculation method is:

[0033] The input feature map F∈R H×W×C After global maximum pooling and global average pooling based on width and height, two 1×1×C feature maps are obtained. Then, they are sent to a two-layer neural network (MLP). Then, the features output by MLP are added element by element, and then activated by sigmoid to generate the final channel attention weight. Finally, the attention weight and the input feature map F are element-wise multiplied to generate the input feature F required by the spatial attention module. c .

[0034] The feature map F output by the channel attention module c As the input feature map of the spatial attention module. First, perform a global maximum pooling and global average pooling based on the channel to obtain two H×W×1 feature maps, and then concatenate these two feature maps based on the channel. Then, after a 7×7 convolution operation, the dimension is reduced to 1 channel. Then, the spatial attention feature F is generated through sigmoid. s Finally, the feature is multiplied by the input feature map of the module to obtain the final convolutional attention enhanced feature F caug .

[0035] According to the method of the first aspect, the feature map F extracted by the convolution block is used as the input of the multi-scale attention enhancement network, and the multi-scale attention enhancement network sequentially calls the global attention network based on the feature map F to calculate the global attention F g And the local attention network calculates the local attention F l , the feature map F after multi-scale attention enhancement maug The calculation method is:

[0036] The input feature map F∈R H×W×C The width and height dimensions are divided into 16 windows of size Each block is projected and pooled, and then attention is calculated to obtain a 16×16×C attention score, which is finally multiplied to obtain the global attention feature F. g , perform the same calculation in each block, and concatenate them to get the local attention feature F l, then the two feature maps are concatenated based on the channel, and then a 7×7 convolution operation is performed to reduce the dimension to C channels to obtain the feature map F after multi-scale attention enhancement maug .

[0037] An electronic device comprises a memory and a processor, wherein the memory stores a computer-readable program executable by the processor, and when the processor executes the computer program, the steps in the aforementioned anti-occlusion target detection method are implemented.

[0038] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the aforementioned anti-occlusion target detection method are implemented.

[0039] The beneficial effects of the present invention are as follows: the method of the present invention can flexibly produce data sets to cope with different occlusion modes, add a variety of attention mechanisms on the basis of the existing target detection network, the proposed attention strategy takes into account both performance improvement and faster detection speed, and performs model training through the knowledge distillation method, which is robust to occluded target detection; by testing on the occlusion data set, the present invention has achieved ideal results. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall process of target detection in the prior art.

[0041] Figure 2 A schematic diagram of the principle of constructing a data set according to the present invention.

[0042] Figure 3 Schematic diagram of the principle of channel attention according to an embodiment of the present invention.

[0043] Figure 4 Schematic diagram of the principle of a spatial attention network according to an embodiment of the present invention.

[0044] Figure 5 Schematic diagram of the principle of a multi-scale attention network according to an embodiment of the present invention.

[0045] Figure 6 FIG. 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] This embodiment provides an anti-occlusion target detection method, comprising the following steps:

[0047] S1. Construct training samples. Add random occlusions to the original samples. Randomly sample the background outside the target and fill it into the target object. Use different filling methods to construct an occlusion dataset.

[0048] In this embodiment, the occlusion data set is constructed, such as Figure 2 As shown, unlike some methods that simulate occlusion by adding random noise or filling color, this embodiment adds random occlusion on the basis of the original image, directly samples the background area of ​​the sample image, randomly samples the background outside the target multiple times, and fills it into the target object. Since the positions and shapes of occluders in practical applications are various, various filling methods are used when filling, such as top, bottom, left, right, grid, etc., and different combinations are used during training to ensure the diversity of occlusion samples. The method of constructing an occlusion data set can simulate scenes where training samples do not appear, and can learn different occlusion models to enhance the generalization ability of the model.

[0049] S2. Input the occluded data samples into the model, extract features from the samples through the feature extraction network, and obtain more refined features through the attention module.

[0050] In this example, the attention module attention enhancement network consists of a convolutional attention module and a multi-scale attention module. The convolutional attention module consists of two independent sub-modules: the spatial attention module and the channel attention module. The multi-scale attention module consists of a local attention module and a global attention module.

[0051] The feature extraction module obtains the feature map F, which is used as the input of the attention enhancement network. After the convolutional attention module, the channel attention and spatial attention enhanced feature F is obtained. caug ; The feature map F is passed through the multi-scale attention module to obtain the multi-scale attention feature F maug , combined with F caug and F maug The final attention enhancement feature is obtained. The feature map F extracted by the feature extraction module is used as the input of the convolutional attention enhancement network. The convolutional attention enhancement network sequentially calls the channel attention network based on the feature map F to calculate the channel attention feature F. c And the spatial attention network calculates the spatial attention F s ,like Figure 3 As shown, the feature map F after convolutional attention enhancement caug The calculation method is:

[0052] F c =σ(MLP(F aυg )+MLP(F max ))

[0053] F s =σ(f 7×7 ([F aυg , F max ]))

[0054] F caug=F s ☉(F c ☉F).

[0055] Specifically, Figure 3 As shown, the input feature map F∈R H×W×C After global maximum pooling and global average pooling based on width and height, two 1×1×C feature maps are obtained, and then they are sent to a two-layer neural network (MLP). Then, the features output by the MLP are added element by element, and then activated by sigmoid to generate the final channel attention weight. Finally, the attention weight and the input feature map F are element-wise multiplied to generate the input feature F required by the spatial attention module. c .

[0056] like Figure 4 As shown, the feature map F output by the channel attention module c As the input feature map of the spatial attention module. First, perform a global maximum pooling and global average pooling based on the channel to obtain two H×W×1 feature maps, and then concatenate these two feature maps based on the channel. Then, after a 7×7 convolution operation, the dimension is reduced to 1 channel. Then, the spatial attention feature F is generated through sigmoid. s Finally, the feature is element-wise multiplied with the input feature map of the module to obtain the final convolutional attention enhanced feature F caug .

[0057] In this example, the feature map F extracted by the convolution block is used as the input of the multi-scale attention enhancement network. The multi-scale attention enhancement network sequentially calls the global attention network to calculate the global attention F based on the feature map F. g And the local attention network calculates the local attention F l , the feature map F after multi-scale attention enhancement maug The calculation method is:

[0058]

[0059] F maug =σ(f 7×7 ([F g , F l ])).

[0060] like Figure 5 As shown, the input feature map F∈R H×W×C The width and height dimensions are divided into 16 windows of size Each block is projected and pooled to obtain the block features, and then the attention score F of 16×16×C is obtained by attention calculation. Att , and finally multiply to get the global attention feature Fg , perform the same calculation in each block, and concatenate them to get the local attention feature F l Then the two feature maps are concatenated based on the channels, and after a 7×7 convolution operation, the dimension is reduced to C channels to obtain the feature map F after multi-scale attention enhancement. maug .

[0061] S3. Use the unoccluded dataset to train a teacher network. Train the teacher network based on classification loss and regression loss. The teacher network can effectively extract the feature information of the unoccluded image, and use the knowledge distillation method to train the student network.

[0062] In this example, the classification loss uses improved cross entropy loss, which is calculated as follows:

[0063]

[0064] Among them, q is the intersection-over-union ratio of the predicted box and the true box. The intersection-over-union ratio is calculated by dividing the intersection of the predicted box and the true box by the union of the two boxes. p is the predicted score, that is, the probability. When the two boxes intersect, q>0, which is a positive sample, otherwise it is a negative sample. α and γ are hyperparameters used to adjust the proportion of positive and negative samples in the loss.

[0065] In this example, the regression loss is calculated as:

[0066]

[0067] Among them, IoU is the intersection over union ratio, b and b gt They represent the predicted box and the true box respectively, ρ represents the Euclidean distance between the two boxes, c represents the diagonal distance of the closed area of ​​the two boxes, v represents the consistency of the relative proportion between the two boxes, and α is a hyperparameter that adjusts the weight of the loss ratio.

[0068] S4. Calculate the overall loss value based on the classification loss, regression loss and distillation loss, and use the overall loss value to train the student network to obtain a student network that can detect occlusion.

[0069] In this example, the calculation method for distillation loss is:

[0070]

[0071] Among them, F teacher and F student Represent the features extracted by the teacher network and the student network respectively. In this example, the overall loss value of training the student network is calculated as:

[0072] L sum =αL cls +βL reg +γLDist

[0073] Among them, α, β, and γ are all hyperparameters used to adjust the weights of classification loss, regression loss, and distillation loss.

[0074] In this example, the knowledge distillation strategy first uses the unoccluded data set to train a teacher network, which can effectively extract the feature information of the unoccluded image. Then, the student network is trained using the occluded data and the features extracted by the teacher network. Through training, the features extracted by the student network are guided to align with the teacher network, reducing the distance between the features extracted in the occluded and unoccluded conditions. This strategy can use the information of the unoccluded part to infer the information of the entire target, more accurately identify and use the key features of the unoccluded part, and achieve more effective knowledge transfer and model training.

[0075] S5. Use the student network to perform target detection on the image to be detected to detect the occluded target information in the image to be detected.

[0076] In this example, after obtaining the student network detection model, the target detection model can be used to detect the image to be detected. It can better identify and utilize the key features of the unobstructed part, use the information of the unobstructed part to infer the information of the entire target, and improve the algorithm's anti-occluded target detection capability.

[0077] S6. Use the post-processing module to filter the blocked target information and output the optimal prediction result for each target.

[0078] In this example, the student network determines the confidence of the target in the prediction box based on the learned parameters, and outputs the target with high confidence as the prediction information. The prediction information is input into the post-processing module, so that only one prediction result is retained for each target, and other redundant results are removed; the prediction results after post-processing are input into the output module, and the output module outputs the final detection results after annotation.

[0079] In the second aspect, the present invention also provides an anti-occlusion target detection device based on a multi-head attention mechanism, comprising: a processor and a memory; a computer-readable program executable by the processor is stored on the memory; when the processor executes the computer-readable program, the steps in the anti-occlusion target detection method based on the multi-head attention mechanism are implemented as above.

[0080] An electronic device structure in an embodiment of the present invention, such as Figure 6As shown, the electronic device includes multiple components connected by a system bus: a processor, a memory, a communication interface, a display screen, and an input device. Specifically, the processor is responsible for performing calculation and control tasks. The memory is composed of a non-volatile storage medium and a memory, wherein the non-volatile storage medium is used to store an operating system and a computing program, and the memory provides support for the operation of the operating system and the program. The communication interface is used to achieve wired or wireless communication with an external terminal, and supports multiple communication methods such as WIFI, mobile network, and near field communication (NFC). The display screen of the electronic device may be a liquid crystal display screen or an electronic ink screen, and the input device may be a touch screen covering the display screen, or may be an input device such as a physical button, a trackball, or a touchpad on the housing of the electronic device.

[0081] In a third aspect, the present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the anti-occlusion target detection method based on the multi-head attention mechanism as described above.

[0082] In summary, the present invention proposes an anti-occlusion target detection method, device and storage medium, which can be easily integrated into the existing target detection algorithm framework. The method can flexibly produce data sets to cope with different occlusion modes, add multiple attention mechanisms on the basis of the existing target detection network, and the proposed attention strategy takes into account both performance improvement and faster detection speed. The model is trained by the knowledge distillation method, and the detection of occluded targets is robust. By testing on the occlusion data set, the present invention has achieved ideal results.

[0083] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A method for detecting an object with anti-occlusion, characterized in that: The following steps are involved: ⑴ Construct training samples, add random occlusions on the basis of the original samples, randomly sample the background outside the target, and fill it into the target object, using different filling methods to construct an occlusion data set; (2) Input the occluded data samples into the model, extract features from the samples through the feature extraction network, and obtain fine features through the attention network; ⑶ Use the unoccluded data set to train a teacher network. The teacher network is trained based on the classification loss and regression loss. The teacher network can effectively extract the feature information of the unoccluded image, and the student network is trained using the knowledge distillation method. (4) Calculate the overall loss value based on the classification loss, regression loss and distillation loss, and use the overall loss value to train the student network to obtain a student network that can detect occlusion; (5) Using the student network to perform target detection on the image to be detected to detect the occluded target information in the image to be detected; (6) Use the post-processing module to filter the occluded target information and output the optimal prediction result for each target.

2. The anti-occlusion target detection method according to claim 1, characterized in that: The occlusion dataset is constructed, the background area of ​​the sample is sampled by random sampling, and different filling methods are used to fill it into the target area. The specific method for constructing the occlusion dataset is: directly sampling the background area of ​​the image, randomly sampling the background outside the target, and filling it into the target object; the filling adopts the top, bottom, left, right and grid filling methods, and different filling method combinations are used when training the student network.

3. The anti-occlusion target detection method according to claim 1, characterized in that: When performing feature extraction on the target detection image based on the YOLOV8 backbone network, an attention enhancement network is added after the feature extraction module, and the attention enhancement network consists of a convolutional attention module and a multi-scale attention module; The convolutional attention module consists of two sub-modules: a spatial attention module and a channel attention module; The multi-scale attention module consists of a local attention module and a global attention module; The feature extraction module obtains a feature map F , feature map F As the input of the attention enhancement network, the channel attention and spatial attention enhancement features are obtained through the convolutional attention module F caug ; Feature map F The multi-scale attention features are obtained through the multi-scale attention module F maug , combined with F caug and F maug Get the final attention-enhanced features.

4. The anti-occlusion target detection method according to claim 1, characterized in that: The training method of the knowledge distillation uses the information of the unoccluded part to infer the information of the entire target. The specific training method is as follows: First, a teacher network is trained using an unoccluded dataset, and the teacher network extracts feature information of the unoccluded image. Then, a student network is trained using the occluded data and the features extracted by the teacher network. Through training, the features extracted by the student network are guided to align with the teacher network, reducing the distance between the features extracted in the occluded and unoccluded conditions, thereby inferring the information of the entire target by utilizing the information of the unoccluded part.

5. The anti-occlusion target detection method according to claim 1, characterized in that: The classification loss adopts improved cross entropy loss, and the calculation method is: ， in, q is the intersection-over-union ratio of the predicted box and the true box. The intersection-over-union ratio is calculated by dividing the intersection of the predicted box and the true box by the union of the two boxes. p is the predicted score, i.e. probability; when two boxes intersect, q >0, it is a positive sample, otherwise it is a negative sample. α, γ It is a hyperparameter used to adjust the proportion of positive and negative samples in the loss.

6. The anti-occlusion target detection method according to claim 5, characterized in that: The regression loss is calculated as: ， in, IoU is the intersection-and-union ratio, b and b gt Represent the predicted box and the true box respectively, ρ represents the Euclidean distance between two boxes, c Represents the diagonal distance between the closed areas of the two boxes. v Indicates the consistency of the relative proportions between the two boxes, α It is a hyperparameter that adjusts the weight of the loss ratio.

7. The anti-occlusion target detection method according to claim 6, characterized in that: The calculation method of the distillation loss is: ， in, F teacher and F student Represent the features extracted from the teacher network and the student network respectively.

8. The anti-occlusion target detection method according to claim 7, characterized in that: The overall loss value is calculated as follows: ， in, α,β,γ They are all hyperparameters used to adjust the weights of classification loss, regression loss, and distillation loss.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer-readable program that can be executed by the processor, and when the processor executes the computer program, the steps in the anti-occlusion target detection method described in any one of claims 1-8 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the anti-occlusion target detection method according to any one of claims 1 to 8 are implemented.

Images