Small Object Target Detection Method and System Based on Lightweight Multi-Scale Attention Mechanism

Abstract

The present invention provides a small object target detection method and system with a lightweight multi-scale attention mechanism. The method includes the following steps: Step 1, using GhostNet as the backbone feature extraction network of the YOLOv4 target detection architecture to extract features; Step 2, using a multi-scale attention module for the features extracted in Step 1 to capture discriminative features in the small target image from both spatial and channel dimensions; Step 3, adopting the Soft-NMS algorithm for the feature map output in Step 2 to reduce the confidence of the detection boxes overlapping with the current best detection box. The network structure of the present invention is small in size, fast in detection speed, and good in the detection effect of small targets, fully meeting the requirements of real-time scenarios and having very high practical value.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and particularly relates to a small object target detection method and system with a lightweight multi-scale attention mechanism. Background Art

[0002] In recent years, with the rapid development of computer vision based on deep learning, target detection has gradually become a hot research direction in computer vision and is widely applied to many fields such as video surveillance, industrial inspection, and medical treatment. It has important practical significance to reduce the consumption of human and material resources by using computer vision.

[0003] Target detection is a very basic and important task. Image segmentation, object tracking, key point detection, etc. usually rely on target detection. When performing target detection, since the number, size, and pose of objects in each image are different, that is, the output is unstructured, which is very different from image classification.

[0004] However, in actual scenarios, deep learning-based target detection is very sensitive to the scale and changes of targets, especially for the detection of small targets. The main reasons for this phenomenon are as follows:

[0005] First, if the scale of the target to be detected is small, then as the training network deepens, the target to be detected is likely to lose features such as edge information and gray-scale information, and less high-level semantic information will be obtained. In addition, there may be some noise information in the image that misleads the training network to learn wrong features;

[0006] Second, the size of the receptive field mapped to the original image also plays an important role in the success of target detection. When the receptive field is small, more spatial structure features are retained, but less abstract semantic information may be available. On the contrary, when the receptive field is large, the retained semantic information is relatively richer, but the spatial structure information of the target may be lost;

[0007] Third, the extraction of features by the convolutional neural network is discretely implemented and it is difficult to achieve sub-pixel accuracy. When it comes to small targets, the neural network may differ by one pixel in the deep layer of the network and by 8 pixels or 16 pixels or even more in the shallow layer. This has little impact on large targets, but has a great impact on small targets. Therefore, it is very important to improve the detection effect of small targets and reduce the size of the model without reducing the accuracy.

[0008] Currently, the main directions of target detection methods for small targets are as follows:

[0009] First, using the idea of an image pyramid, perform scale transformation on the input image to be detected, that is, magnify or reduce it. On this basis, an image pyramid can be constructed with the image scale increasing or decreasing from top to bottom. Then, use a fixed-size window to slide on each layer of the image to detect the target of interest. However, since images of different resolutions need to pass through the convolutional neural network, the computational complexity is relatively large, resulting in a very slow detection speed;

[0010] Second, fuse the image features, which can enhance the semantic information of the shallow features and the spatial structure information of the deep features. However, since feature-level fusion extracts image features as the fusion information, a lot of detailed features will be lost;

[0011] Third, adjust the scale and distribution of the anchor boxes. However, in actual use, a large number of anchor boxes are usually required to ensure sufficient overlap with the ground truth boxes, resulting in only a small part of the anchor boxes overlapping with the ground truth boxes, which will cause a huge imbalance between positive and negative anchor boxes and slow down the training speed.

[0012] Existing research can only handle the detection problem of small targets. However, improving the robustness of the algorithm to target scale changes and achieving lightweight small target detection are still relatively difficult tasks in object detection. Summary of the Invention

[0013] The purpose of the present invention is to provide a small object target detection method and system with a lightweight multi-scale attention mechanism, which improves the detection accuracy of small targets while reducing the size of the model, and solves the problem that the existing research methods cannot coexist the detection accuracy and the lightweight network.

[0014] The present invention solves the above problems through the following technical means:

[0015] The first aspect of the present invention provides a small object target detection method based on YOLOv4 with a lightweight multi-scale attention mechanism, including the following steps:

[0016] Step 1, use GhostNet as the backbone feature extraction network of the YOLOv4 object detection architecture to extract features;

[0017] Step 2, use a multi-scale attention module to capture discriminative features in the small target image from both spatial and channel dimensions for the features extracted in Step 1;

[0018] Step 3, use the Soft-NMS algorithm to reduce the confidence of the detection boxes that overlap with the current best detection box for the feature map with discriminative features for small targets obtained in Step 2.

[0019] The second aspect of the present invention provides a small object target detection system based on a lightweight multi-scale attention mechanism of YOLOv4, including:

[0020] A first feature extraction module that uses GhostNet as the backbone feature extraction network of the YOLOv4 target detection architecture for feature extraction;

[0021] A second feature extraction module, connected to the first feature extraction module, uses a multi-scale attention module to capture discriminative features in the small target image in both spatial and channel dimensions for the features extracted by the first feature extraction module;

[0022] A detection output module, connected to the second feature extraction module, uses the Soft-NMS algorithm to reduce the confidence of the detection boxes that overlap with the current best detection box in the feature map output by the second feature extraction module.

[0023] The third aspect of the present invention provides a small object target detection device, including:

[0024] A memory; and

[0025] A processor coupled to the memory, the processor being configured to execute the small object target detection method based on the lightweight multi-scale attention mechanism of YOLOv4 based on instructions stored in the memory.

[0026] The fourth aspect of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the small object target detection method based on the lightweight multi-scale attention mechanism of YOLOv4.

[0027] Compared with the existing technology, the beneficial effects of the present invention are as follows:

[0028] The present invention first uses GhostNet as the backbone feature extraction network of the YOLOv4 target detection architecture to extract features, reducing the weight of the network for the first time while ensuring the accuracy; secondly, a multi-scale attention module is provided to reduce the weight of the network for the second time, and capture important discriminative features in the small target image in both spatial and channel dimensions, and reduce the confidence of the detection boxes that overlap with the current best detection box through the Soft-NMS algorithm. Only by modifying very few parameters can the small object categories in the picture be obtained in real time, efficiently and accurately, and for images obtained by different image acquisition devices and different scenarios, the method of the present invention can be used to obtain the small object categories in the image, which has strong robustness. Description of the Drawings

[0029] To more clearly introduce the technical solutions in the embodiments of the present invention, the following will briefly introduce the attached drawings required in the embodiments. Obviously, the attached drawings shown below are only individual cases of the present invention. For researchers in the field, this method can be applied to real-time object detection in different scenarios through simple reproduction of the present invention;

[0030] Figure 1 is the flowchart of the method of the present invention.

[0031] Figure 2 is the multi-scale attention module diagram mentioned in the present invention.

[0032] Figure 3 is the effect diagram of the present invention applied to detecting small objects in different images. Detailed implementation manners

[0033] In order to better understand the above objects, features and advantages of the present invention, the present invention will be further described in detail below in conjunction with the attached drawings and specific implementation manners. It should be noted that this embodiment is only a part of the examples of the present invention, rather than all examples. All other examples obtained without making innovative work belong to the protection scope of the present invention.

[0034] Example 1

[0035] This embodiment proposes a small object target detection method based on a lightweight multi-scale attention mechanism of YOLOv4, including the following steps:

[0036] Step 1, use GhostNet as the backbone feature extraction network of the YOLOv4 object detection architecture to extract features;

[0037] As Figure 1 shown, the YOLOv4 object detection architecture includes:

[0038] Step 1.1, the original image is extracted to obtain preliminary features through the YOLOv4 object detection architecture with GhostNet as the backbone network;

[0039] Step 1.2, convey strong semantic features from top to bottom through the FPN layer for the extracted preliminary features, and then convey strong localization features from bottom to top through the PAN structure to perform feature aggregation on different detection layers from different backbone layers.

[0040] Step 2, use the multi-scale attention module for the features extracted in Step 1 to capture discriminative features in the small target image from both spatial and channel dimensions;

[0041] As Figure 2As shown, the specific steps of using the multi-scale attention module to capture discriminative features in small target images from both spatial and channel dimensions are as follows:

[0042] Construct a spatial attention mechanism module and a channel attention mechanism module; among them, the spatial attention mechanism module can enable the convolutional neural network to efficiently learn the regions that need to be focused on, so as to map the spatial information in the original image to another space to retain the important features in the image. Combining max pooling and average pooling can adaptively learn discriminative global features and local features. The channel attention mechanism of the channel attention mechanism module represents the correlation between the feature maps of n channels and important features by adding a weight to each feature map of the channels. The larger the weight, the more important features the feature map of the channel contains; in this embodiment, common spatial attention mechanism modules and channel attention mechanism modules can be constructed, such as SENet, ECA, CBAM;

[0043] Combine the constructed spatial attention mechanism module and channel attention mechanism module to construct a multi-scale attention mechanism; among them, the multi-scale attention mechanism uses 4 branches to perform multi-scale feature extraction on the input feature map; the first branch uses a 1×1 convolutional operation to adjust the number of channels to the same size as the feature maps output by the other three branches; the second branch uses two cascaded 1×3 convolutional operations and 3×1 convolutional operations, and the third branch uses two cascaded 1×5 convolutional operations and 5×1 convolutional operations. The two cascaded asymmetric convolutional operations effectively reduce the number of network parameters, and at the same time can introduce more non-linear activation layers to enhance the non-linear learning ability; the fourth branch first uses a 3×3 max pooling operation to extract feature textures, and then performs a 1×1 convolutional operation to adjust the number of channels to the same size as the feature maps output by the other three branches;

[0044] First, input the feature map tensor into the spatial attention mechanism module for calculation to add spatial attention, and obtain the feature map tensor , where w, h, and c are the width, height, and number of channels of the feature map respectively;

[0045] Then, use 1×1 convolutional kernels to perform convolutional operations on the feature map tensor S to obtain the feature map tensor ;

[0046] Next, use the 4 branches of the multi-scale attention mechanism to perform multi-scale feature extraction on the feature map tensor D respectively to obtain multi-scale feature map tensors , , , ; The Concat operation is used to perform feature fusion on the feature map tensors P1, P2, P3, and P4 to obtain the feature map tensor ; Then, the feature map tensor Q is input into the channel attention mechanism module for calculation to add channel attention, obtaining the feature map tensor ;

[0047] Finally, the Add operation is used to perform feature fusion on the feature map tensors S and C to obtain the feature map tensor , which is used as the output of the multi-scale attention mechanism.

[0048] Step 3, the Soft-NMS algorithm is used to reduce the confidence of the detection boxes that overlap with the current best detection box for the feature map with discriminative features for small objects obtained in Step 2;

[0049] The attenuation formula of the Soft-NMS algorithm for reducing the confidence of the detection boxes that overlap with the current best detection box is:

[0050]

[0051] where S i is the confidence, b i is the detection box, which is used to adjust the attenuation degree.

[0052] Effect comparison

[0053] Using the small object detection method based on the lightweight multi-scale attention mechanism of YOLOv4 provided by this embodiment, accurate categories of small objects can be obtained when detecting images. Figure 3 The performance comparison of the method of this embodiment with different algorithms in different scenarios is given. Among them, the first column is the labeled image, the second column represents the detection results after adding GhostNet on the basis of YOLOv4, the third column represents the detection results after adding Soft-NMS on the basis of YOLOv4, and the fourth column represents the detection results of this method. In the second and third columns, there are many small targets such as people, cars, animals, etc. that are missed, while the method of this embodiment can accurately detect all small targets without missing detections. For blurred small targets or cases with front and rear occlusions and dense distributions, the method of this embodiment can accurately detect the object categories, while other algorithms cannot. The results show that the method of this embodiment is superior to the YOLOv4 method.

[0054] Embodiment 2

[0055] This embodiment provides a small object target detection system based on the lightweight multi-scale attention mechanism of YOLOv4, including:

[0056] The first feature extraction module uses GhostNet as the backbone feature extraction network of the YOLOv4 object detection architecture for feature extraction;

[0057] The second feature extraction module, connected to the first feature extraction module, uses a multi-scale attention module to capture discriminative features in the small target image from both spatial and channel dimensions for the features extracted by the first feature extraction module;

[0058] The detection output module, connected to the second feature extraction module, uses the Soft-NMS algorithm to reduce the confidence of the detection boxes that overlap with the current best detection box in the feature map output by the second feature extraction module.

[0059] For the specific implementation method of the system in this embodiment, refer to the method described in Embodiment 1, which will not be elaborated here.

[0060] Embodiment 3

[0061] This embodiment provides a small object target detection device, including:

[0062] A memory; and

[0063] A processor coupled to the memory, the processor being configured to execute the small object target detection method based on the YOLOv4-based lightweight multi-scale attention mechanism described in Embodiment 1 based on instructions stored in the memory.

[0064] Among them, the memory can include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory stores, for example, an operating system, application programs, a boot loader, and other programs.

[0065] The small object target detection device may further include an input / output interface, a network interface, a storage interface, etc. These interfaces and the memory and the processor can be connected through a bus, for example. Among them, the input / output interface provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface provides a connection interface for various networking devices. The storage interface provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0066] Embodiment 4

[0067] This embodiment provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the small object target detection method based on the YOLOv4-based lightweight multi-scale attention mechanism described in Embodiment 1

[0068] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer non-transitory readable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer program code.

[0069] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0070] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0072] As mentioned above, only the specific embodiments of the present invention are described, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, and all should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A small object target detection method based on a lightweight multi-scale attention mechanism of YOLOv4, characterized in that, It includes the following steps: Step 1: Use GhostNet as the backbone feature extraction network of the YOLOv4 object detection architecture to extract features; Step 2: Use the multi-scale attention module for the features extracted in Step 1 to capture discriminative features in small target images in terms of both spatial and channel dimensions; The specific steps of using the multi-scale attention module to capture discriminative features in small target images in terms of both spatial and channel dimensions are as follows: Construct a spatial attention mechanism module and a channel attention mechanism module; among them, the channel attention mechanism of the channel attention mechanism module represents the correlation between the feature map of each of the n channels and important features by adding a weight to the feature map of each channel. The larger the weight, the more important features the feature map of this channel contains; Combine the constructed spatial attention mechanism module and channel attention mechanism module to construct a multi-scale attention mechanism; among them, the multi-scale attention mechanism uses 4 branches to perform multi-scale feature extraction on the input feature map. The first branch uses a 1×1 convolution operation, the second branch uses two cascaded 1×3 convolution operations and 3×1 convolution operations, the third branch uses two cascaded 1×5 convolution operations and 5×1 convolution operations, and the fourth branch uses two cascaded 3×3 max pooling operations and 1×1 convolution operations; First, input the feature map tensor into the spatial attention mechanism module for calculation to add spatial attention, and obtain the feature map tensor , where w, h, and c are the width, height, and number of channels of the feature map respectively; Then, use a 1×1 convolutional kernel to perform a convolution operation on the feature map tensor S, obtaining the feature map tensor ; Next, four branches using the multi-scale attention mechanism respectively perform multi-scale feature extraction on the feature map tensor D to obtain multi-scale feature map tensors , , , ; The Concat operation is used to perform feature fusion on the feature map tensors P1, P2, P3, and P4 to obtain the feature map tensor ; Then, the feature map tensor Q is input into the channel attention mechanism module for calculation to add channel attention, obtaining the feature map tensor ; Finally, the Add operation is used to perform feature fusion on the feature map tensors S and C to obtain the feature map tensor , which serves as the output of the multi-scale attention mechanism; Step 3: Use the Soft-NMS algorithm for the feature map with discriminative features for small targets obtained in Step 2 to reduce the confidence of the detection boxes overlapping with the current best detection box.

2. The small object target detection method based on the lightweight multi-scale attention mechanism of YOLOv4 according to claim 1, characterized in that, The specific steps of using GhostNet as the backbone feature extraction network of the YOLOv4 object detection architecture to extract features are as follows: Step 1.1: The original image is passed through the YOLOv4 object detection architecture with GhostNet as the backbone network to extract preliminary features; Step 1.2: Convey strong semantic features from top to bottom through the FPN layer for the extracted preliminary features, and then convey strong localization features from bottom to top through the PAN structure to perform feature aggregation on different detection layers from different backbone layers.

3. The small object target detection method based on the lightweight multi-scale attention mechanism of YOLOv4 according to claim 1, characterized in that, The attenuation formula of the Soft-NMS algorithm for using the Soft-NMS algorithm to reduce the confidence of the detection boxes overlapping with the current best detection box is as follows: Among them, S i is the confidence level, and b i is the detection box, which is used to adjust the attenuation degree.

4. A small object target detection system based on a lightweight multi-scale attention mechanism of YOLOv4, characterized in that, It includes: The first feature extraction module uses GhostNet as the backbone feature extraction network of the YOLOv4 object detection architecture to extract features; The second feature extraction module is connected to the first feature extraction module and uses the multi-scale attention in the multi-scale attention module for the features extracted by the first feature extraction module to capture discriminative features in small target images in terms of both spatial and channel dimensions; The specific steps of using the multi-scale attention module to capture discriminative features in small target images in terms of both spatial and channel dimensions are as follows: Construct a spatial attention mechanism module and a channel attention mechanism module; among them, the channel attention mechanism of the channel attention mechanism module represents the correlation between the feature map of each of the n channels and important features by adding a weight to the feature map of each channel. The larger the weight, the more important features the feature map of this channel contains; Combine the constructed spatial attention mechanism module and channel attention mechanism module to construct a multi-scale attention mechanism. Among them, the multi-scale attention mechanism uses four branches to perform multi-scale feature extraction on the input feature map. The first branch uses a 1×1 convolution operation, the second branch uses two cascaded 1×3 convolution operations and 3×1 convolution operations, the third branch uses two cascaded 1×5 convolution operations and 5×1 convolution operations, and the fourth branch uses two cascaded 3×3 max pooling operations and 1×1 convolution operations. First, input the feature map tensor into the spatial attention mechanism module for calculation to add spatial attention, obtaining the feature map tensor , where w, h, and c are the width, height, and number of channels of the feature map, respectively; Then, use a 1×1 convolutional kernel to perform a convolution operation on the feature map tensor S, obtaining the feature map tensor ; Next, four branches using the multi-scale attention mechanism respectively perform multi-scale feature extraction on the feature map tensor to obtain multi-scale feature map tensors , , , ; The Concat operation is used to perform feature fusion on the feature map tensors P1, P2, P3, and P4 to obtain the feature map tensor ; Then, the feature map tensor Q is input into the channel attention mechanism module for calculation to add channel attention, obtaining the feature map tensor ; Finally, the Add operation is used to perform feature fusion on the feature map tensors S and C to obtain the feature map tensor , which is used as the output of the multi-scale attention mechanism; The detection output module, connected to the second feature extraction module, uses the Soft-NMS algorithm to reduce the confidence of the detection boxes that overlap with the current best detection box in the feature map output by the second feature extraction module.

5. The small object target detection system based on the lightweight multi-scale attention mechanism of YOLOv4 according to claim 4, characterized in that, The specific steps of using GhostNet as the backbone feature extraction network of the YOLOv4 object detection architecture for feature extraction are as follows: Step 1.1, the original image is extracted to obtain preliminary features through the YOLOv4 object detection architecture with GhostNet as the backbone network. Step 1.2, the strong semantic features are conveyed top-down through the FPN layer for the extracted preliminary features, and then the strong localization features are conveyed bottom-up through the PAN structure, and feature aggregation is performed on different detection layers from different backbone layers.

6. The small object target detection system based on the lightweight multi-scale attention mechanism of YOLOv4 according to claim 4, characterized in that, The attenuation formula of the Soft-NMS algorithm for using the Soft-NMS algorithm to reduce the confidence of the detection boxes that overlap with the current best detection box is: Among them, S i is the confidence level, and b i is the detection box, which is used to adjust the attenuation degree.

7. A small object target detection device, comprising: A memory; And A processor coupled to the memory, the processor being configured to execute the small object target detection method based on the lightweight multi-scale attention mechanism of YOLOv4 according to any one of claims 1-3 based on the instructions stored in the memory.

8. A non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the small object target detection method based on the lightweight multi-scale attention mechanism of YOLOv4 according to any one of claims 1-3.

Images