Small target detection method and small target detection system

Abstract

The application belongs to the technical field of computers and specifically relates to a small target detection method and a small target detection system. The small target detection method comprises the following steps: adjusting the fusion proportion of an upper feature map in adjacent two layers of feature maps by using an attention mechanism to obtain a fused feature map; and performing small target detection based on the fused feature map to obtain a small target detection result. The detection method of the application adjusts the fusion proportion of the upper feature map in the process of feature fusion by using the attention, filters out semantic features suitable for the position of the small target, and greatly improves the precision of small target detection.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a small target detection method and a small target detection system. Background Technology

[0002] In recent years, object detection, as an important branch of computer vision, has played a significant role in multiple fields. In real-world scenarios, due to the abundance of small targets, small target detection has broad application prospects in fields such as healthcare, intelligent transportation, smart retail, security and criminal investigation, and national defense. In object detection tasks, the accuracy of object detection plays a crucial role in event triggering; however, the detection accuracy of small targets remains unsatisfactory, becoming a major challenge for the industry.

[0003] Small targets have a low pixel count, resulting in a small coverage area and limited information content, which is the fundamental reason why they are difficult to detect in images. Existing small target detection methods often employ multi-scale detection in single-stage algorithms. This involves first adjusting the number of channels in the feature map from the previous layer provided by the backbone network using a 1×1 convolution, then performing bilinear interpolation to make the H×W size of the feature map the same as the next layer's feature map. Finally, the interpolated feature map and the lower-layer feature map are simply added together. However, since deep semantic features contain semantic information not only for small targets but also for medium / large targets, simple addition and fusion will incorporate semantic information from both types of targets. This introduces two types of semantic information, which are useless for small target detection, into the feature map at the small target detection level, adding interference noise and resulting in low accuracy in small target detection.

[0004] Therefore, how to effectively detect small targets and improve their detection accuracy has become an urgent technical problem to be solved. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, this application provides a small target detection method and a small target detection system.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, this application adopts the following technical solution:

[0009] In a first aspect, embodiments of this application provide a small target detection method based on attention-adaptive fusion features, the method comprising:

[0010] The fusion ratio of the upper feature map in two adjacent feature maps is adjusted using an attention mechanism to obtain a fused feature map.

[0011] Small target detection is performed based on the fused feature map to obtain small target detection results.

[0012] Optionally, the method includes the following steps:

[0013] S1. Obtain the image to be detected;

[0014] S2. Input the image to be detected into a pre-trained small target detection model to obtain the corresponding small target detection result; wherein, the small target detection model includes a backbone network module for extracting multi-scale feature maps, a feature fusion module for using an attention mechanism to adjust the fusion ratio of the upper feature maps in adjacent feature maps to perform feature fusion, and a detection head module for performing small target detection on the fused feature map.

[0015] Optionally, the step of the backbone network module extracting multi-scale feature maps includes:

[0016] The image to be detected is first passed through a 7×7 convolution with a stride of 2, a 2×2 max pooling layer with a stride of 2, and then through residual blocks of varying numbers of repeated stacks, resulting in images with sizes corresponding to the original values. Figure 1 The residual blocks are composed of 1×1 convolutions and 3×3 convolutions, and the C2 feature maps, C3 feature maps, C4 feature maps, and C5 feature maps are 4, 1 / 8, 1 / 16, and 1 / 32 respectively.

[0017] Optionally, the feature fusion module uses an attention mechanism to adjust the fusion ratio of the upper-level feature map in two adjacent feature maps to perform feature fusion, including:

[0018] After adjusting the fusion ratio of the upper-layer feature maps using an attention mechanism according to the following formula, the feature maps of adjacent layers are aggregated:

[0019]

[0020] Among them, P i It is the fused feature map after attention is added. This indicates that convolution is performed using a 3×3 convolution kernel. This indicates that convolution processing using a 1×1 kernel is used for channel number matching, f upsample Indicates upsampling, C i T represents the feature map of this layer. i+1 f represents the feature map of the upper layer. att This indicates that attention is added to the input feature map.

[0021] Optionally, attention can be added to the input feature map according to the following formula:

[0022] f att (xin )=x in *(sigmoid(conv 1×1 (conv 1×1 (x in )))

[0023] Where, x in The input feature map, conv 1×1 This indicates that a convolution operation is performed using a 1×1 convolution kernel, and sigmoid represents the activation function.

[0024] Optionally, the method by which the detection head module detects the fused feature map includes:

[0025] The fused feature map is classified and detected according to the following formula to obtain the class probability of each predicted bounding box at each spatial location. During the detection process, the receptive field is expanded from a square receptive field to a rectangular receptive field through an adaptive receptive field selection module:

[0026] f output1 =conv 3×3 (Rconv 3×3 (RFASM(RFASM(Rconv 3×3 (x)))))

[0027] The fused feature map is subjected to regression detection according to the following formula to obtain the offset of each reference anchor box at each spatial location to determine the predicted bounding box position:

[0028] f output2 =conv 3×3 (Rconv 3×3 (Rconv 3×3 (Rconv 3×3 (Rconv 3×3 (x)))))

[0029] Among them, f output1 f represents the prediction result of the classification detection. output2 This represents the prediction result of regression detection, where x represents the input feature map, and conv 3×3 Rconv indicates that a 3×3 convolution kernel is used for convolution processing. 3×3 This indicates that a 3×3 convolution is followed by a ReLU activation function, and RFASM represents the receptive field adaptive selection module.

[0030] Optionally, the adaptive receptive field selection module expands the square receptive field to a square receptive field and a rectangular receptive field according to the following formula:

[0031] f next =conv 1×1 (m)+m1

[0032] Among them, f next This represents the output of the receptive field adaptive selection module, where m is obtained by normalizing m2, m3, m4, and m5.

[0033]

[0034] Wherein, m1, m2, m3, m4, and m5 are calculated according to the following formula:

[0035] m1 = conv 3×3 (x)

[0036] m2 = conv 1×3 (conv 1×1 (x))

[0037] m3 = conv 3×1 (conv 1×1 (x))

[0038] m4 = conv 1×3 (conv 1×3 (conv 1×1 (x)))

[0039] m5 = conv 3×1 (conv 3×1 (conv 1×1 (x)))

[0040] w2, w3, w4, and w5 are obtained by splitting w according to channels, based on the formula below:

[0041] w = conv 1×1 (m2+m3+m4+m5)

[0042] Where x represents the input feature map, conv 1×3 conv represents convolution processing with a 1×3 kernel. 3×1 This indicates convolution processing with a 3×1 kernel, where ε is 0.0001.

[0043] Optionally, after S1 and before S2, the following may also be included:

[0044] The image to be detected is preprocessed to obtain a preprocessed image to be detected. The preprocessing includes one or more of padding, changing the image size, and data augmentation.

[0045] Optionally, S2 may be followed by:

[0046] S3. After post-processing, overlapping detection boxes are deleted to obtain the final small target detection result.

[0047] Secondly, embodiments of this application provide a small target detection system, which includes a video acquisition subsystem and a small target detection subsystem;

[0048] The video acquisition subsystem is connected to the small target detection subsystem and is used to acquire video images of the target area through the video acquisition device and send the video images of the target area to the small target detection subsystem.

[0049] The small target detection subsystem is used to receive the video image and perform real-time small target detection on the video image using the attention-adaptive fusion feature-based small target detection method as described in any of the first aspects above.

[0050] Thirdly, embodiments of this application provide a computer-readable storage medium storing a small target detection program based on attention-adaptive fusion features. When the small target detection program is executed by a processor, the processor performs the steps of the small target detection method based on attention-adaptive fusion features as described in any of the first aspects above.

[0051] Fourthly, embodiments of this application provide a computer device including a memory and a processor. The memory stores a small target detection program based on attention-adaptive fusion features. When the small target detection program is executed by the processor, the processor performs the steps of the small target detection method based on attention-adaptive fusion features as described in any of the first aspects above.

[0052] (III) Beneficial Effects

[0053] The beneficial effects of this application are as follows: This application proposes a small target detection method and a small target detection system, wherein the small target detection method includes: using an attention mechanism to adjust the fusion ratio of the upper feature map in two adjacent feature maps to obtain a fused feature map; and performing small target detection based on the fused feature map to obtain a small target detection result.

[0054] The small target detection method provided in this application selects the semantic features of the location of the small target during the feature fusion process of upper and lower layers by focusing on attention. This allows the feature map of the small target detection layer to only fuse the semantic features of the small target, thereby greatly improving the accuracy of small target detection. Moreover, it is simple and convenient to deploy, and can be used plug and play.

[0055] Furthermore, during feature detection, the Receptive Field Adaptive Selection (RFASM) module expands the rectangular receptive field using attention weighting, enabling the network to adaptively select which appropriate receptive field to use to capture the target at the current location. This enhances the ability to capture small targets, whether extremely tall or wide, in datasets and real-life scenarios, thereby significantly improving the accuracy of small target detection.

[0056] Furthermore, the small object detection method provided in this application adopts the form of attention, and the weights of the attention module change during training depending on the dataset, thus expanding the scope of application. Attached Figure Description

[0057] This application is described with reference to the following figures:

[0058] Figure 1 This is a schematic diagram of the small target detection method based on attention adaptive fusion features in one embodiment of this application;

[0059] Figure 2 This is a schematic diagram of a small target detection method based on attention adaptive fusion features in another embodiment of this application;

[0060] Figure 3 This is a schematic diagram of the training process of a small target detection model in another embodiment of this application;

[0061] Figure 4 This is a schematic diagram of a small target detection model structure in another embodiment of this application;

[0062] Figure 5 This is a flowchart illustrating the feature fusion process in another embodiment of this application;

[0063] Figure 6 This is a schematic diagram of the attention module's processing flow of the input feature map in another embodiment of this application;

[0064] Figure 7 This is a structural diagram of the receptive field adaptive selection module in another embodiment of this application;

[0065] Figure 8 A visualization of the attention mechanism of the existing RetinaNet model after feature fusion in the first layer of FPN;

[0066] Figure 9 This is a visualization of the attention of the small object detection model after feature fusion in the first layer of FPN in another embodiment of this application;

[0067] Figure 10 This is a schematic diagram of a small target detection system in one embodiment of this application;

[0068] Figure 11 This is a schematic diagram of the architecture of a computer device in one embodiment of this application. Detailed Implementation

[0069] To better explain and facilitate understanding of the present invention, it is described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described below are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other; for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0070] Although the definition of small targets varies across different scenarios, and a unified standard has not yet been established, existing definitions of small targets mainly fall into two categories: those based on relative scale and those based on absolute scale. In the relative scale definition, objects whose median ratio of bounding box area to image area is between 0.08% and 0.58% are considered small targets. In the absolute scale definition, targets with a resolution less than 32×32 pixels are considered small targets. In the small target detection method of this application, small targets refer to those defined using the absolute scale definition.

[0071] Example 1

[0072] Figure 1 This is a schematic diagram of a small target detection method based on attention-adaptive fusion features in one embodiment of this application, as shown below. Figure 1 As shown, the small target detection method based on attention adaptive fusion features in this embodiment includes the following steps:

[0073] S1. Use the attention mechanism to adjust the fusion ratio of the upper feature map in two adjacent feature maps to obtain the fused feature map;

[0074] S2. Small target detection is performed based on the fused feature map to obtain the small target detection results.

[0075] The small target detection method in this application selects the semantic features of the location of the small target by using attention during the feature fusion process. This allows the feature map of the small target detection layer to fuse only the semantic features of the small target, thereby significantly improving the accuracy of small target detection.

[0076] The small target detection method of this embodiment can be applied to computer devices. The computer device can execute the small target detection method of this embodiment by running a software system. The type of computer device can be a laptop, a computer, a server, etc. This application does not specifically limit the type of computer device.

[0077] It is understood that the small target detection method in this embodiment can be executed either by the client device or the server device alone, or by the client device and the server device working together. The server can be a single server or a cloud-based system built from server machines.

[0078] For example, small target detection methods can be integrated into the client. After receiving a small target detection request, the client device can perform small target detection using its own hardware environment.

[0079] For example, small target detection methods can be integrated into server-side devices. After receiving a small target detection request, the server-side device can execute the small target detection method through its own hardware environment.

[0080] To better understand this invention, the implementation process of the method in this embodiment will be described below with the server as the execution subject.

[0081] In this embodiment, small object detection is performed using a pre-trained small object detection model. The implementation process of the small object detection method includes:

[0082] Acquire the image to be detected;

[0083] The image to be detected is input into a pre-trained small object detection model to obtain the corresponding small object detection results. The small object detection model includes a backbone network module for extracting multi-scale feature maps, a feature fusion module for performing attention-based feature fusion on the multi-scale feature maps, and a detection head module for performing small object detection on the fused feature maps.

[0084] In this embodiment, the image to be detected can be obtained by segmenting frames from a video image, and then small target detection is performed on each frame of the video. The video image can be a video image captured in real time by a video acquisition device located in the environment where the small target object is located, or it can be a pre-recorded video image read from a video storage system.

[0085] For example, a communication connection can be established between the video capture device and the server executing the method of this embodiment via a wireless network or a wired network. The server receives video images sent by the video capture device through the established communication connection.

[0086] When the video image is a real-time captured video stream, acquiring the image to be detected may include: acquiring keyframe images. Specifically, keyframe images can be extracted through the following steps:

[0087] The acquired real-time video stream is decoded to obtain multiple original frame images corresponding to the real-time video stream;

[0088] Based on preset frame extraction rules, keyframe images are extracted from the original frame images;

[0089] The extracted keyframe images are used as the images to be detected.

[0090] For example, the frame rate of a video is typically 30 frames per second, and can be reduced to a minimum of 25 frames per second. Therefore, decoding can be performed based on the frame rate of the real-time video stream. Specifically, the monitoring video can be decoded simultaneously upon real-time reception to obtain multiple original frame images corresponding to the real-time video stream. After obtaining multiple original frame images, due to limitations in processing resources and the requirements of the target business, it is generally not necessary to process each original frame image. Therefore, frame images can be extracted from the multiple original frame images based on certain frame extraction rules, such as extracting one frame every 10 frames or one frame every 5 frames, as keyframe images.

[0091] It should be noted that the frame skipping interval can be set according to the actual situation, and this application does not impose any restrictions on it.

[0092] In this embodiment, the backbone network module in the small target detection model can be ResNet50 or ResNet101.

[0093] It should be noted that other neural networks can also be used for the backbone network. This embodiment does not specifically limit the neural network structure of the backbone network module.

[0094] The steps for the backbone network module to extract multi-scale feature maps include:

[0095] The image to be detected is first passed through a 7×7 convolution with a stride of 2, a 2×2 max pooling layer with a stride of 2, and then through residual blocks of varying numbers of repeated stacks, resulting in images with sizes corresponding to the original image size. Figure 1 The residual blocks are composed of 1×1 convolutions and 3×3 convolutions, and the C2 feature maps, C3 feature maps, C4 feature maps, and C5 feature maps are 4, 1 / 8, 1 / 16, and 1 / 32 respectively.

[0096] In this embodiment, the feature fusion module uses an attention mechanism to adjust the fusion ratio of the upper-level feature map in two adjacent feature maps to perform feature fusion, including:

[0097] According to formula (1), the feature maps of adjacent layers are aggregated after adjusting the fusion ratio of the upper layer feature maps using the attention mechanism:

[0098]

[0099] Among them, P i It is the fused feature map after attention is added. This indicates that convolution is performed using a 3×3 convolution kernel. This indicates that convolution processing using a 1×1 kernel is used for channel number matching, f upsample Indicates upsampling, C i T represents the feature map of this layer.i+1 f represents the feature map of the upper layer. att This indicates that attention is added to the input feature map.

[0100] Specifically, attention is added to the input feature map according to formula (2):

[0101] f att (x in )=x in *(sigmoid(conv 1×1 (conv 1×1 (x in (2)

[0102] Where, x in The input feature map, conv 1×1 This indicates that a convolution operation is performed using a 1×1 convolution kernel, and sigmoid represents the activation function.

[0103] By adding attention to the input feature map using formula (2), irrelevant semantic information is avoided from being added to the feature map at the level specifically for detecting small targets, thus improving the accuracy of small target detection.

[0104] Example 2

[0105] Based on Embodiment 1, this embodiment proposes a feature detection head module that expands the rectangular receptive field through an adaptive receptive field selection module.

[0106] In this embodiment, the method by which the feature detection head module detects the fused feature map may include:

[0107] The classification detection head performs classification detection on the fused feature map according to formula (3), and obtains the class probability of each predicted bounding box at each spatial location. During the detection process, the receptive field is expanded from a square receptive field to a rectangular receptive field through the receptive field adaptive selection module:

[0108] f output1 =conv 3×3 (Rconv 3×3 (RFASM(RFASM(Rconv 3×3 (x))))) (3)

[0109] The regression detection head performs regression detection on the fused feature map according to formula (4) to obtain the offset of each reference anchor box at each spatial location to determine the predicted bounding box position:

[0110] f output2 =conv 3×3 (Rconv 3×3 (Rconv 3×3 (Rconv3×3 (Rconv 3×3 (x)))))(4)

[0111] Among them, f output1 f represents the prediction result of the classification detection. output2 This represents the prediction result of regression detection, where x represents the input feature map, and conv 3×3 Rconv indicates that a 3×3 convolution kernel is used for convolution processing. 3×3 This indicates that a 3×3 convolution is followed by a ReLU activation function, and RFASM represents the receptive field adaptive selection module.

[0112] Specifically, the detection head module includes a receptive field adaptive selection module RFASM, which expands the square receptive field to a square receptive field and a rectangular receptive field according to formula (5):

[0113] f next =conv 1×1 (m)+m1(5)

[0114] Among them, f next This represents the output of the receptive field adaptive selection module, where m is obtained by normalizing m2, m3, m4, and m5.

[0115]

[0116] Wherein, m1, m2, m3, m4, and m5 are calculated according to formulas (7)-(11):

[0117] m1 = conv 3×3 (x)(7)

[0118] m2 = conv 1×3 (conv 1×1 (x))(8)

[0119] m3 = conv 3×1 (conv 1×1 (x))(9)

[0120] m4 = conv 1×3 (conv 1×3 (conv 1×1 (x)))(10)

[0121] m5 = conv 3×1 (conv 3×1 (conv 1×1 (x)))(11)

[0122] w2, w3, w4, and w5 are obtained by splitting w according to channels. Since w is a weight of h×w×4, where 4 is the number of channels, the weights obtained by splitting according to channels are 4 weights of h×w×1, namely w2, w3, w4, and w5. And w is calculated according to formula (12):

[0123] w = conv 1×1 (m2+m3+m4+m5)(12)

[0124] Where x represents the input feature map, conv 1×3 conv represents convolution processing with a 1×3 kernel. 3×1 This indicates convolution processing with a 3×1 kernel, and ε is set to 0.0001 to prevent the denominator from being 0.

[0125] Formula (5) expands the rectangular receptive field of the input feature map using 1×3 and 3×1 convolutions, and replaces 1×5 and 5×1 convolutions by stacking two 1×3 and 3×1 convolutions. All 1×3 and 3×1 convolutions are depthwise separable convolutions, which not only obtain a larger receptive field, accommodating small targets that are extremely tall or narrow in the scene, but also save computational and storage resources. Furthermore, the Receptive Field Adaptive Selection (RFASM) module uses attention weighting, allowing the network to adaptively select the appropriate receptive field to capture the target at the current location, thereby enhancing the network model's detection accuracy for small targets.

[0126] The small target detection method in this embodiment treats the feature fusion process as an encoder and feature detection as a decoder. By refining the encoder's encoding capabilities, more information about small targets is added to the feature map, while simultaneously enhancing the decoder's decoding capabilities, thereby improving the algorithm model's ability to detect small targets. Specifically, attention is used during the feature fusion process to filter out the semantic features of the small target's location, ensuring that the feature map for the small target detection layer only incorporates the semantic features of small targets. Furthermore, during feature detection, an "adaptive receptive field selection module" expands the "rectangular receptive field" to capture extremely tall or wide small targets in the dataset and real-life scenarios, significantly improving the accuracy of small target detection and enhancing the model's detection performance.

[0127] Example 3

[0128] Figure 2 This is a flowchart illustrating a small target detection method based on attention-adaptive fusion features in another embodiment of this application, as shown below. Figure 2 As shown, the method includes the following steps:

[0129] Step S10: Establish and train a small target detection model.

[0130] Figure 3 This is a schematic diagram of the training process of a small object detection model in another embodiment of this application, as shown below. Figure 3 As shown, the training process of the small target detection model in this embodiment includes:

[0131] Step S11: Create a dataset. Collect images and manually annotate the corresponding labels for each image, ensuring a one-to-one correspondence between images and labels. In this embodiment, the COCO dataset is selected. The COCO dataset is currently the largest and most authoritative dataset in the field of object detection, capable of evaluating the model's average accuracy, small target accuracy, medium target accuracy, and large target accuracy.

[0132] Step S12: Divide the dataset into training set, validation set, and test set.

[0133] If the dataset is small, it can be divided according to a 6:2:2 ratio. If the dataset is large (e.g., it contains hundreds of thousands of images), it can be divided according to an 8:1:1 ratio.

[0134] Step S13: Data loading.

[0135] Data is loaded using the Dataloader function provided in the deep learning framework PyTorch. During the data loading process, the number of images loaded each time, whether random sampling is used, and the number of CPUs used can be determined.

[0136] Step S14: Data preprocessing.

[0137] In computer vision tasks, neural networks require that the height and width of the input image data be consistent. Therefore, image preprocessing is necessary, including padding and resizing. For object detection tasks, since bounding boxes need to be drawn and labeled in the original image, remapping back to the original image is also required. Furthermore, data preprocessing also includes operations such as data augmentation.

[0138] Step S15: Forward Inference of the Model. The loaded image data is fed into the initialized model, and the weights of the data and the model are calculated layer by layer to finally obtain the forward inference result of the model.

[0139] Step S16: Obtain the forward inference results of the model and calculate the loss value.

[0140] Using a predefined loss function, the loss value is calculated based on the model's forward inference results and the corresponding image labels.

[0141] Step S17: Calculate the gradient of each layer of the network based on the loss value, backpropagate, and update the network weights.

[0142] Calculate the gradient of each layer of weights in the network based on the loss value, and use the current weight value to subtract the learning rate × gradient value to obtain the updated weight value, and save the model.

[0143] Step S18: Determine whether the preset termination condition is met:

[0144] If the model has not reached the specified number of rounds or the loss value is still greater than the threshold, return to step S13;

[0145] If the model reaches the specified number of rounds or the loss value is less than the threshold, the model weight file is saved, training is terminated, and the process ends.

[0146] During training, the initial small object detection model is trained based on the data in the training set, and the model network parameters are determined. The hyperparameters of the model are adjusted using the validation set to obtain the trained small object detection model. Then, the accuracy of the trained model is tested on the test set.

[0147] The specific testing methods are existing technologies and will not be elaborated here.

[0148] Step S20: Obtain the image to be detected and preprocess it.

[0149] The image to be detected is resized to a preset size by adjusting the image size or padding. This size is usually acceptable to the network model, such as 1333×800 pixels.

[0150] Step S30: Input the preprocessed image into the small target detection model.

[0151] Step S40: The small object detection model obtains the initial result of small object detection through forward inference. The image data is fed into the small object detection model network for calculation, and the result of the model's forward inference is obtained, which is the initial result of small object detection.

[0152] Step S50: After post-processing, overlapping detection boxes are deleted to obtain the final small target detection result.

[0153] Because the model produces a large number of results after forward inference, rendering all predictions as bounding boxes would result in many overlapping and mutually inclusive bounding boxes, and a significant number of these boxes would be meaningless. Therefore, after detection, a post-processing step is needed to filter out these numerous overlapping and meaningless bounding boxes. Nonmaximum suppression (NMS) and Soft-NMS can be used for this post-processing.

[0154] Step S60: Remap the small target detection results and present them as frames in the original image.

[0155] The post-processed small target detection results are remapped back to the original image, and the target location and category are marked with bounding boxes in the original image. Finally, the detection results are saved or displayed.

[0156] This embodiment provides an attention-based adaptive fusion method during feature fusion. By using an attention module to filter out the semantic features of the location of small targets in the feature map after linear interpolation, irrelevant semantic features of medium / large targets are eliminated. This allows the feature map for detecting small targets to only fuse the semantic features of small targets, greatly improving the accuracy of the model in detecting small targets.

[0157] Because of the use of an attention module, the model can perform adaptive fusion during feature fusion. That is, the model automatically selects semantic features of small targets based on the current dataset images to complete the fusion, rather than manually designing hyperparameters or other methods. Furthermore, an adaptive receptive field selection module that expands the rectangular receptive field is added to the detection head, enabling the network to adaptively capture small targets of different shapes, greatly increasing the network's target detection capability. Therefore, the method in this embodiment is simple and convenient to deploy, plug and play, and further improves the accuracy of small target detection.

[0158] To verify the technical effectiveness of the method in this embodiment, object detection was performed using images from the test set in the COCO dataset.

[0159] Table 1

[0160]

[0161] Table 1 compares the object detection results on the COCO dataset using several existing basic models and the models after adding the module of this invention. The data in the table represents the object detection accuracy.

[0162] As can be seen from the table, compared with the original model and the same model after being replaced with the module designed in this invention, the model using the module of this invention achieves better results in small object accuracy. After adding this invention, the small object accuracy of RetinaNet increased by 2%, and the small object detection accuracy of the FCOS model increased by 0.5%, verifying that this invention can achieve better feature fusion effect and feature detection accuracy than the original model.

[0163] In summary, the feature fusion module and receptive field adaptive selection module proposed in this invention can effectively improve the adaptability of the model and enhance the performance after feature fusion and detection. In the feature fusion stage, more information about small targets can be encoded into the feature map, and in the feature detection stage, more information can be decoded, i.e., more results related to small targets can be predicted, thereby improving the accuracy of the detection model in detecting small targets and achieving better small target detection results.

[0164] The structure of the small target detection model in this embodiment will be described below. Figure 4 This is a schematic diagram of the small target detection model structure in another embodiment of this application, as shown below. Figure 4 As shown, the small target detection model includes a backbone module, a feature fusion module, and a detection head module. The loss functions are Focal Loss and L1 Loss.

[0165] The backbone network is used to extract features at different scales, resulting in feature maps at multiple scales.

[0166] Feature pyramid network is used to achieve feature fusion based on multi-scale learning to obtain fused features;

[0167] The detection head is used to detect the category of a target and its corresponding location based on its features, and to predict the type of target.

[0168] The backbone network uses ResNet50, which has four stages. The feature maps of each stage are reduced to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively.

[0169] Specifically, in this embodiment, the image to be detected is input into the backbone network, and four feature maps are obtained as the feature maps to be detected of the image to be detected. The four feature maps are denoted as C2, C3, C4 and C5 respectively.

[0170] The feature fusion module is implemented based on the Feature Pyramid Network (FPN).

[0171] Feature fusion methods include:

[0172] According to formula (1), adjacent feature layers are aggregated. The upper feature map is upsampled by linear interpolation. Attention is added to the upsampled upper feature map according to formula (2) to obtain the upper feature map with added attention. The upper feature map with added attention is added and fused with the current feature map after channel matching. Attention is added to the result of the addition and fusion to obtain the fused feature map with added attention.

[0173] It should be noted that the upsampling here can be bilinear interpolation or nearest neighbor interpolation. This embodiment does not specifically limit the upsampling method.

[0174] Specifically, the features to be detected in the image output by the backbone network. Figure 5 This is a flowchart of feature fusion in another embodiment of this application, where the triangle portion represents the attention module.

[0175] like Figure 5As shown, the feature fusion process includes:

[0176] A1. Use 1×1 convolution to adjust the number of channels of each feature map to be consistent for the C3, C4 and C5 feature maps extracted from the backbone network, and obtain new features with C channels. The new feature maps obtained by convolution are T3, T4 and T5.

[0177] A2. Perform bilinear interpolation on the T5 feature map, and then filter out useless semantic information by the attention module. At this time, add the filtered feature map and T4 together to obtain a new feature map NT4.

[0178] Figure 6 This is a schematic diagram of the attention module's processing flow of the input feature map in another embodiment of this application, as shown below. Figure 6 As shown, the processing flow of the attention module includes:

[0179] The number of channels in the feature map to be filtered for irrelevant information is reduced to 64 by using 1×1 convolution, which means that 64 1×1 convolution kernels are used to convolve the feature map. At this time, the shape of the feature map is H×W×64.

[0180] Next, the result after convolution is normalized;

[0181] The normalized result is reduced to 1 dimension by 1×1 convolution to obtain the output. At this time, the shape of the output feature map is H×W×1.

[0182] The result is transformed to a range of 0 to 1 using the Sigmoid function, forming a probability graph of the location.

[0183] The obtained location probability map is multiplied by the input feature map to obtain the result feature map after filtering out useless information, and then output.

[0184] A3. Similarly, after performing the same operation as step 2 on NT4, we obtain a new feature map NT3 after fusing it with T3.

[0185] A4. The obtained T5, NT4, and NT3 feature maps are first smoothed and fused by 3×3 convolution. Then, the convolutional feature maps are filtered out again by the attention module to remove irrelevant noise information that is not suitable for the target size task of this layer, and P3, P4, and P5 are obtained.

[0186] A5. The P5 feature map is obtained by performing a 3×3 convolution with a stride of 2.

[0187] A6. The P6 feature map is first activated by the ReLU non-linear function, and then passed through a 3×3 convolution with a stride of 2 to obtain the P7 feature map.

[0188] A7. Use the feature maps P3, P4, P5, P6, and P7 as the fused feature maps after adding attention.

[0189] In this embodiment, the five feature maps P3, P4, P5, P6, and P7 are input into the detection head module. The probability of the corresponding category of the reference box on the feature map can be obtained through the classification unit, and the corresponding offset can be obtained through the detection box regression unit.

[0190] The detection head module includes a classification unit and a bounding box regression unit. The classification unit predicts the class probability of each reference box (anchor) (number A) at each location (the dataset has K classes). This step is equivalent to predicting the probability of each anchor in each class and then taking the highest probability. The bounding box regression unit predicts the offset between each reference box and the ground truth box at each location (there are 4 offsets, so the final result is 4A). The classification unit includes two 3×3 convolutions (containing ReLU activation function, 256 channels) and two receptive field adaptive selection modules (RFASM), followed by a 3×3 convolution (without ReLU function), outputting KA channels. Finally, sigmoid activation yields the predicted probability of each anchor in each class. Each location is equivalent to KA binary classification problems. Similar to the classification unit, the bounding box regression unit contains four 3×3 convolutions (with ReLU activation function, and 256 channels), and finally a 3×3 convolution (without ReLU function). However, the final output has 4A channels, which indicates that the bounding box regression is class-independent.

[0191] Specifically, Figure 7 This is a structural diagram of the receptive field adaptive selection module in another embodiment of this application, as shown below. Figure 7 As shown, the receptive field adaptive fusion process is as follows:

[0192] B1. Perform a 3×3 convolution on the input feature map x to obtain the feature map m1;

[0193] B2. Perform a 1×1 convolution on the input feature map x to reduce its dimension to 64 channels, and then use a 1×3 convolution to expand the receptive field of the vertical rectangle to obtain the output feature map m2.

[0194] B3. Perform a 1×1 convolution on the input feature map x to reduce its dimension to 64 channels, and then use a 3×1 convolution to expand the receptive field of the horizontal rectangle to obtain the output feature map m3.

[0195] B4. Perform a 1×1 convolution on the input feature map x to reduce the dimension to 64 channels, and then stack two 1×3 convolutions to expand the receptive field to a larger vertical rectangle, to obtain the output feature map m4.

[0196] B5. Perform a 1×1 convolution on the input feature map x to reduce the dimension to 64 channels, and then stack two 3×1 convolutions to expand the receptive field to a larger horizontal rectangle, to obtain the output feature map m5.

[0197] B6. Add and fuse feature maps m2, m3, m4, and m5 together, and then reduce the dimension of the fused feature map to 4 channels through 1×1 convolution to obtain 4×h×w weights, that is, one channel corresponds to one feature map, and each channel is w2, w3, w4, and w5 respectively.

[0198] B7. Next, the weights of 4×h×w are quickly normalized. The [i,j] positions of the four branches (m2, m3, m4, m5) are normalized and summed according to formula (6) (ε is 0.0001 to prevent the denominator from being 0).

[0199] B8. The normalized and summed feature map m is increased in dimension by 1×1 convolution and then added to m1 to obtain the output feature map output.

[0200] Next, overlapping detection boxes are removed through post-processing. In this embodiment, non-maximum suppression (NMS) algorithm is used for post-processing. For the prediction of each feature map layer, the top 1K prediction results (filtered according to the largest class probability of each reference box) are first taken (that is, the Anchor corresponding probability and corresponding offset mentioned above). Then, an unqualified result is filtered out with a threshold of 0.05. At this point, the number of prediction results is greatly reduced. Decoding the prediction boxes of the prediction results at this time, instead of decoding all prediction results predicted by the model, can improve the inference speed.

[0201] Finally, the prediction results of all layer feature maps are merged together, and overlapping boxes are filtered using the NMS algorithm with IoU=0.5 to obtain the final inference result.

[0202] While lower-level features have less semantic information but accurate target location, higher-level features have richer semantic information but coarser target location. This invention uses an attention module to filter semantic information irrelevant to the current layer's detection task during the FPN feature fusion process. This makes the P3, P4, P5, P6, and P7 feature maps more focused on the target size that this layer should detect. Since the feature fusion process mainly occurs in the P3 and P4 feature maps, which are specifically designed for small target detection, filtering out irrelevant semantic information significantly improves the algorithm model's accuracy in detecting small targets.

[0203] To verify the technical effectiveness of the method in this embodiment, the visualization effect of feature fusion on images in the test set of the COCO dataset was analyzed, and the trained model was used to perform object detection on the images. Figure 8 This is a visualization of the attention mechanism after feature fusion in the first layer of FPN for the existing RetinaNet model. Figure 9 This is a visualization of the attention process after feature fusion in the first layer of the FPN for a small object detection model in another embodiment of this application. Figure 8 As shown in the figure, the gray-white areas indicate that the corresponding position values ​​on the feature map are large, indicating high attention; conversely, the gray-black areas indicate that the corresponding position values ​​on the feature map are small, indicating low attention. It can be observed that... Figure 8 In the initial analysis, almost all areas appear grayish-white, with some areas appearing grayish-black, resulting in low distinguishability and an inability to effectively differentiate between the background and target areas. This means that semantic information irrelevant to the current detection task has been added. However, after the attention model filters out the useless information, the result of feature fusion is as follows: Figure 9 As shown. It can be clearly seen that, Figure 9 The background region in the middle part of the image is visualized as gray-black, while the region with the target is visualized as gray and white. This successfully filters out most of the irrelevant semantic information, reducing the difficulty of subsequent detection by the detection head and significantly improving the detection accuracy of small targets.

[0204] In summary, this invention uses an attention module to filter out the semantic features of the location of small targets in the feature map after linear interpolation, thereby eliminating irrelevant semantic features of medium / large targets. This allows the feature map for detecting small targets to only incorporate the semantic features of small targets, significantly improving the accuracy of the detection model for small targets. The method in this embodiment can achieve excellent small target detection results.

[0205] Example 4

[0206] A second aspect of this application provides a small target detection system. Figure 10 This is a schematic diagram of the small target detection system structure in one embodiment of this application. Please refer to [link / reference]. Figure 10 The system includes a video acquisition subsystem 10 and a small target detection subsystem 20;

[0207] The video acquisition subsystem 10 is connected to the small target detection subsystem 20 and is used to acquire video images of the target area through the video acquisition device and send the video images of the target area to the small target detection subsystem 20.

[0208] The small target detection subsystem 20 is used to receive video images and perform real-time small target detection on the video images using the attention-adaptive fusion feature-based small target detection method as described in Embodiment 1 above.

[0209] In this embodiment, the video acquisition device can be a surveillance camera installed in the target monitoring area. The surveillance camera captures video of the monitored area in real time. It should be noted that, in this step, the surveillance camera can be, but is not limited to, a network surveillance camera. First, the network surveillance camera is placed in a location that can capture the target area, and then the video images are accessed and read from the camera via network, local transmission, or other means.

[0210] By employing the small target detection method based on attention-adaptive fusion features from Example 1, the accuracy of small target detection is improved.

[0211] Example 5

[0212] A third aspect of this application provides a computer device, including a memory and a processor. The memory stores a small target detection program based on attention-adaptive fusion features. When the small target detection program is executed by the processor, the processor performs the steps of the small target detection method based on attention-adaptive fusion features as described in any of the above embodiments.

[0213] Figure 11 This is a schematic diagram of the architecture of a computer device in one embodiment of this application.

[0214] Figure 11 The computer device shown may include at least one processor 101, at least one memory 102, at least one network interface 104, and other user interfaces 103. The various components of the computer device are coupled together via a bus system 105. It is understood that the bus system 105 is used to implement communication between these components. In addition to a data bus, the bus system 105 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 11 The general labeled all buses as Bus System 105.

[0215] The user interface 103 may include a display, keyboard, or clicking device (e.g., mouse, trackball, or touchpad).

[0216] It is understood that the memory 102 in this embodiment can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 102 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0217] In some implementations, memory 102 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 1021 and application program 1022.

[0218] The operating system 1021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 1022 includes various applications used to implement various application functions. Programs implementing the methods of this embodiment of the invention can be included in the application program 1022.

[0219] In this embodiment of the invention, the processor 101 executes the method steps provided in the first aspect by calling the program or instructions stored in the memory 102, specifically the program or instructions stored in the application program 1022.

[0220] The methods disclosed in the above embodiments of the present invention can be applied to processor 101, or implemented by processor 101. Processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 101 or by instructions in the form of software. The processor 101 may be a general-purpose processor, digital signal processor, application-specific integrated circuit, off-the-shelf programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in the art. The storage medium is located in memory 102, and processor 101 reads the information in memory 102 and completes the steps of the above method in combination with its hardware.

[0221] In addition, in conjunction with the small target detection method based on attention adaptive fusion features in the above embodiments, the present invention can provide a computer-readable storage medium storing a small target detection program based on attention adaptive fusion features. When the small target detection program is executed by a processor, the processor performs the steps of any of the small target detection methods based on attention adaptive fusion features in the above method embodiments.

[0222] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. The use of the terms first, second, third, etc., is for convenience only and does not indicate any order. These terms can be understood as part of the component names.

[0223] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0224] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0225] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A small target detection method based on attention-adaptive fusion features, characterized in that, Small targets are those with a resolution of less than 32×32 pixels. The method includes the following steps: S1. Obtain the image to be detected; S2. Input the image to be detected into a pre-trained small target detection model to obtain the corresponding small target detection result; wherein, the small target detection model includes a backbone network module for extracting multi-scale feature maps, a feature fusion module for using an attention mechanism to adjust the fusion ratio of the upper feature maps in adjacent feature maps for feature fusion, and a detection head module for detecting small targets in the fused feature maps; The feature fusion module uses an attention mechanism to adjust the fusion ratio of the upper-level feature map in two adjacent feature maps for feature fusion of multi-scale feature maps. This includes: upsampling the upper-level feature map through linear interpolation, adding attention to the upsampled upper-level feature map to obtain an attention-added upper-level feature map, adding and fusing the attention-added upper-level feature map with the channel-matched current-level feature map, and adding attention to the result of the addition and fusion to obtain an attention-added fused feature map. The method by which the detection head module detects the fused feature map includes: The fused feature map is classified and detected according to the following formula to obtain the class probability of each predicted bounding box at each spatial location. During the detection process, the receptive field is expanded from a square receptive field to a rectangular receptive field through an adaptive receptive field selection module: ； The fused feature map is subjected to regression detection according to the following formula to obtain the offset of each reference anchor box at each spatial location to determine the predicted bounding box position: ； in, This represents the prediction result of the classification detection. This indicates the prediction results of the regression test. Represents the input feature map, This indicates that a 3×3 convolution kernel is used for convolution processing. This indicates that a 3×3 convolution is followed by a ReLU activation function, and RFASM indicates the receptive field adaptive selection module.

2. The small target detection method based on attention-adaptive fusion features according to claim 1, characterized in that, The steps for the backbone network module to extract multi-scale feature maps include: The image to be detected is first passed through a 7×7 convolution with a stride of 2, a 2×2 max pooling layer with a stride of 2, and then through residual blocks with different numbers of repeated stacks, to obtain C2 feature maps, C3 feature maps, C4 feature maps, and C5 feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. The residual blocks are composed of 1×1 convolutions and 3×3 convolutions.

3. The small target detection method based on attention-adaptive fusion features according to claim 1, characterized in that, The feature fusion module uses an attention mechanism to adjust the fusion ratio of the upper-level feature map in two adjacent feature maps to perform feature fusion. The method includes: After adjusting the fusion ratio of the upper-layer feature maps using an attention mechanism according to the following formula, the feature maps of adjacent layers are aggregated: ； in, It is the fused feature map after attention is added. This indicates that convolution is performed using a 3×3 convolution kernel. This indicates that convolution processing using a 1×1 convolution kernel is used for channel number matching. This indicates that upsampling is being performed. This represents the feature map of this layer. f represents the feature map of the upper layer. att This indicates that attention is added to the input feature map.

4. The small target detection method based on attention-adaptive fusion features according to claim 3, characterized in that, Add attention to the input feature map according to the following formula: ； in, The input feature map, conv 1×1 This indicates that a convolution operation is performed using a 1×1 convolution kernel, and sigmoid represents the activation function.

5. The small target detection method based on attention-adaptive fusion features according to claim 1, characterized in that, The adaptive receptive field selection module expands the square receptive field to both a square and a rectangular receptive field according to the following formula: ； in, This represents the output of the adaptive receptive field selection module. Depend on , , , Normalization yields: ； in, , , , , Calculated using the following formula: ； ； ； ； ； , , , Calculated according to the following formula Decomposed by channel: ； in, Indicates the input feature map, This indicates convolution processing with a 1×3 kernel. This indicates convolution processing with a 3×1 kernel. The value is 0.0001.

6. The small target detection method based on attention-adaptive fusion features according to claim 1, characterized in that, After S1 and before S2, it also includes: The image to be detected is preprocessed to obtain a preprocessed image to be detected. The preprocessing includes one or more of padding, changing the image size, and data augmentation.

7. The small target detection method based on attention-adaptive fusion features according to claim 1, characterized in that, S2 is followed by: S3. After post-processing, overlapping detection boxes are deleted to obtain the final small target detection result.

8. A small target detection system, characterized in that, The system includes a video acquisition subsystem and a small target detection subsystem; The video acquisition subsystem is connected to the small target detection subsystem and is used to acquire video images of the target area through the video acquisition device and send the video images of the target area to the small target detection subsystem. The small target detection subsystem is used to receive the video image and perform real-time small target detection on the video image using the small target detection method based on attention adaptive fusion features as described in any one of claims 1 to 7.

Images