Adaptive tiny target detection method based on attention mechanism

Abstract

The invention discloses a self-adaptive tiny target detection method based on an attention mechanism, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a data set; constructing a feature enhancement module; constructing an adaptive sparse convolution detection head; constructing a self-adaptive tiny target detection model based on an attention mechanism, adding a feature enhancement module into a backbone network, and replacing an original detection head with a self-adaptive sparse convolution detection head; and determining a final tiny target detection model. According to the method, an adaptive sparse convolution head is constructed, the use of sparse convolution corresponding to detection heads in different FPN layers is dynamically adjusted through a channel attention mechanism, the model complexity and the detection precision are effectively balanced, and the calculation efficiency is kept while the detection precision of a tiny object is improved; a feature enhancement module is constructed, global context information is fused by using global average pooling, the feature representation capability of the model for tiny objects is enhanced, and the detection performance of the model for objects of different sizes in a complex environment is improved.

Description

Technical Field

[0001] The invention relates to an adaptive small target detection method based on an attention mechanism, and belongs to the technical field of computer vision. Background Art

[0002] With the rapid development of computer vision technology, small target detection has become increasingly important in many fields, such as autonomous driving, aerial surveillance, remote sensing image analysis, etc. In these applications, it is crucial to accurately and quickly detect small-sized targets in images or videos. However, since small targets occupy a limited number of pixels in an image, they are often easily affected by background noise, lighting changes, occlusions, etc., making it difficult for traditional computer vision algorithms to accurately locate and identify these targets.

[0003] Traditional multi-scale feature-based methods are represented by feature pyramid networks (FPNs), which integrate deep and shallow features to improve the detection capabilities of objects of different sizes. For example, in some advanced target detection algorithms, the FPN structure is used to extract information from feature maps at different levels, so that the model can focus on the overall semantic information and local detail features of the object at the same time. However, with the increase in the number of network layers and the need for multi-scale feature processing, the complexity of the model also increases, and the demand for computing resources increases significantly. This makes it difficult for some devices with limited computing resources, such as embedded devices and mobile devices, to meet the requirements of real-time detection.

[0004] In order to meet the challenge of limited computing resources, many studies have tried to adopt sparse convolution technology. Sparse convolution focuses on calculating important features by performing convolution operations only in key areas of the image, thereby reducing computational load and memory usage. He S, Jiang C, Dong D, et al. proposed a sparse dynamic convolution in their paper "Sd-conv: Towards the parameter-efficiency of dynamic convolution" (Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2023.). The authors first designed a binary mask that prunes static convolution kernels based on a learnable threshold, removes redundancy from a large network to obtain a lightweight network, thereby significantly reducing parameters and computational costs. Du B, Huang Y, Chen J, Huang D proposed a global context enhanced adaptive sparse convolutional network (CEASC) in the paper "Adaptive Sparse Convolutional Networks with Global Context Enhancement for Faster Object Detection on Drone Images." (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023.), which saves computing resources by using sparse convolution in the detection head. The CESC module in the network uses point-wise convolution to generate global feature information. In addition, the authors also use the mean and variance of the global feature information to group normalize the feature map, thereby compensating for the missing context information to improve detection accuracy. However, when sparse convolution is applied to the detection heads corresponding to different layers of FPN, it brings new problems. At the lower layers of FPN, images have higher spatial resolution and rich detail information, which are suitable for capturing tiny objects. At this time, using sparse convolution can effectively reduce the amount of computation and retain key details. However, at the higher layers of FPN, tiny objects occupy limited space in the image after multiple downsamplings, and detailed information has been greatly reduced. Applying sparse convolution may further ignore this limited information, resulting in excessive sparseness of high-level feature maps, thereby losing contextual information, and ultimately affecting the detection accuracy of tiny objects.Therefore, how to dynamically adjust the use of sparse convolution according to the feature characteristics of different levels of FPN and supplement effective contextual feature information is a key issue in optimizing small target detection performance.

[0005] In summary, while improving detection performance, existing tiny object detection technologies face many challenges such as limited computing resources, insufficient feature representation, and difficulty in balancing detection efficiency. A new technical solution is urgently needed to solve these problems in order to meet the growing needs of practical applications. Summary of the invention

[0006] The present invention aims to provide an adaptive small target detection method based on attention mechanism, and the present invention aims to solve the technical problems existing in the prior art.

[0007] To achieve the above object, the present invention provides the following technical solution, an adaptive small target detection method based on attention mechanism, comprising the following steps: S1, obtaining a data set: collecting small target images through an image acquisition device, annotating the images, and dividing the data set into a training set and a validation set; S2, construct feature enhancement module; S3, build adaptive sparse convolution detection head; S4, build an adaptive small target detection model based on attention mechanism: including backbone network, neck network, detection head, add feature enhancement module to the backbone network, and use adaptive sparse convolution detection head instead of original detection head; S5, determine the final tiny target detection model: use the training set to train the adaptive tiny target detection model based on the attention mechanism, and use the validation set to evaluate the tiny target detection model.

[0008] Furthermore, in S1, various types of drones are used to photograph pedestrians, vehicles, and bicycles of different densities in different environments as target data sets.

[0009] Furthermore, based on the RetinaNet framework, the characteristics of each layer of FPN are optimized to improve the performance of tiny object detection.

[0010] Furthermore, in S2, the feature enhancement module The global feature vector is obtained by performing global average pooling on the layer feature map. Convolution adjusts the number of channels of the vector, and after bilinear interpolation upsampling, Layer-wise lateral connection feature map addition and fusion to improve FPN The feature map representation of the layer includes the following steps: S21, global average pooling, this operation is performed on the highest layer feature map Perform global average pooling, that is, average the feature values ​​of each channel to obtain a channel with the same number of channels and spatial dimension The global feature vector represents the global context information of the entire image. S22, Convolution adjustment: through The convolutional layer adjusts the number of channels of the global feature vector obtained by global average pooling to match the number of channels of the original highest-level feature map to facilitate subsequent fusion operations. S23, bilinear interpolation upsampling, will be passed through The global feature vector adjusted by convolution is upsampled by bilinear interpolation to make its spatial size consistent with The layer feature map is matched. S24, feature fusion, adds and fuses the global feature map after bilinear interpolation upsampling with the horizontally connected feature map of the bilinear interpolation layer.

[0011] Furthermore, in S3, the adaptive sparse convolution detection head utilizes a channel attention mechanism to quantify the importance of feature channels of each FPN layer to obtain weights, and dynamically determines whether to use a sparse convolution detection head or a normal detection head based on these weights.

[0012] The common detection head is a detection head that does not use sparse convolution. It adopts a standard convolutional neural network structure, including a series of convolutional layers, for extracting features and classifying and locating targets. They perform convolution operations at every position in the feature map without considering the importance of spatial position.

[0013] The sparse convolution detection head uses sparse convolution to focus on key areas in the image and utilizes global context information, effectively enhancing the recognition ability of small targets while significantly reducing the computational resource consumption on extensive invalid background areas. Sparse convolution By applying a binary mask, the sparse convolution detection head can determine which feature units participate in the convolution process, thereby reducing the amount of computation in invalid background areas.

[0014] The adaptive process of the adaptive sparse convolution detection head includes the following steps: S31, use the channel attention mechanism to quantify the importance of the feature channels of each FPN layer, obtain the weight, and define the output of FPN as a list of feature maps , where each value in the list Corresponding to the feature map output of a specific level of FPN, Represents the FPN Layer, where , It is expressed as: ; Using Channel Attention Network For feature map list Process and get a list of attention scores ,in Corresponding FPN Layer attention score output, It is expressed as: , S32, setting a threshold, comparing the attention score in step S31 with the threshold to determine whether to use a detection head with sparse convolution at a specific FPN level, the threshold is expressed as ,if , go to step S33, otherwise, go to step S34; S33, FPN Layer uses a detection head with sparse convolution Perform detection and output the test results , It is expressed as: ; S34, using a common detection head Conduct testing, test results for: .

[0015] Furthermore, in S31, the channel attention network is composed of a global average pooling layer and two fully connected layers connected in series in sequence, the first fully connected network uses a ReLU activation function, and the second fully connected layer uses a Sigmoid activation function. The specific steps include: Step 1: Given a feature map, its size is , perform global average pooling on the feature map, and use the following formula to compress the size of the feature map to , in Indicates The global average of the channels, which represents the Compressed to The feature map of Indicates the number of channels. represents the height of the feature map, represents the width of the feature map, Indicates that the input feature map is in the channel and spatial location Elements of Step 2: The input is processed into two fully connected layers. The first fully connected network is used for feature dimensionality reduction, and the second fully connected layer further converts the feature vector into a single scalar output. For the first fully connected layer, use the following formula: in, It is the result after the first fully connected layer and ReLU activation function. and are the weights and biases of the first fully connected layer, ReLU is the activation function, For the second fully connected layer, use the following formula: in, For the final output, and are the weights and biases of the second fully connected layer, is the Sigmoid activation function, where It is a value between 0 and 1, representing the attention weight. An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the adaptive small target detection method based on the attention mechanism when executing the program.

[0016] A computer-readable storage medium stores computer instructions on which, when executed by a processor, an adaptive small target detection method based on an attention mechanism is implemented.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses an adaptive sparse convolution head to dynamically adjust the use of sparse convolution in the detection heads corresponding to different FPN layers through a channel attention mechanism, that is, a channel attention module is inserted after the output of each level of FPN. The module quantifies the importance of feature channels at each level to obtain weights, and makes decisions based on the set thresholds to determine whether the detection heads corresponding to each level of FPN use sparse convolution. This method effectively reduces unnecessary computational overhead, while retaining the ability to accurately detect small targets at the key feature level; 2. The present invention adopts a channel attention mechanism to calculate weights. The design of the channel attention mechanism is relatively simple and computationally efficient. It is mainly composed of global average pooling and a two-layer fully connected network, which avoids complex spatial operations and a large number of parameter increases. This enables the channel attention mechanism to be effectively integrated into various network architectures without significantly increasing the computational burden. 3. The present invention adopts a feature enhancement module to fuse global context information using global average pooling. Global average pooling is an operation with very few or even no parameters, and will not significantly increase the number of parameters and the amount of calculation of the model. Compared with complex context modeling methods, GAP provides a global perspective through simple operations and avoids over-reliance on context information. This not only improves the overall detection accuracy while introducing global information, but also keeps the resource consumption of the model within a controllable range. By adding a feature enhancement module, the model's feature representation capability for tiny objects is enhanced, thereby improving the model's detection performance for objects of different sizes in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a basic flow chart of the adaptive small target detection method in the present invention; Figure 2 It is a basic flow chart of the feature enhancement module in the present invention; Figure 3 It is a basic flow chart of channel attention in the present invention; Figure 4 This is an adaptive small target detection model diagram based on the attention mechanism in the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0020] Example: Figure 1 As shown, the present invention discloses an adaptive small target detection method based on attention mechanism, comprising the following steps: S1, obtaining a data set: collecting small target images through an image acquisition device, annotating the images, and dividing the data set into a training set and a validation set; In S1, various types of drones are used to photograph pedestrians, vehicles, and bicycles of different densities in different environments as target data sets.

[0021] S2, construct feature enhancement module; In S2, the feature enhancement module The global feature vector is obtained by performing global average pooling on the layer feature map. Convolution adjusts the number of channels of the vector, and after bilinear interpolation upsampling, Layer-wise lateral connection feature map addition and fusion to improve FPN The feature map representation of the layer.

[0022] like Figure 2 As shown, the operation process of the feature enhancement module includes the following steps: S21, global average pooling, this operation is performed on the highest layer feature map Perform global average pooling, that is, average the feature values ​​of each channel to obtain a channel with the same number of channels and spatial dimension The global feature vector represents the global context information of the entire image.

[0023] S22, Convolution adjustment: through The convolutional layer adjusts the number of channels of the global feature vector obtained by global average pooling. The purpose of this step is to match the number of channels of the original highest-level feature map to facilitate subsequent fusion operations.

[0024] S23, bilinear interpolation upsampling, will be passed through The global feature vector adjusted by convolution is upsampled by bilinear interpolation to make its spatial size consistent with The layer feature maps are matched.

[0025] S24, feature fusion, adds and fuses the global feature map after bilinear interpolation upsampling with the horizontally connected feature map of the bilinear interpolation layer.

[0026] The introduction of the feature enhancement module can improve the detection ability of the target detection network for targets of different scales, thereby improving the overall detection performance.

[0027] S3, build adaptive sparse convolution detection head; In S3, an adaptive sparse convolution detection head is used. This structure adopts different detection heads for each level of FPN, and uses a channel attention mechanism to quantify the importance of the feature channels of each FPN layer to obtain weights. Based on these weights, it is dynamically decided whether to use a sparse convolution detection head or a normal detection head.

[0028] The common detection head is a detection head that does not use sparse convolution. It adopts a standard convolutional neural network structure, including a series of convolutional layers, for extracting features and classifying and locating targets. They perform convolution operations at every position in the feature map without considering the importance of spatial position.

[0029] The sparse convolution detection head uses sparse convolution to focus on key areas in the image and utilizes global context information, effectively enhancing the recognition ability of small targets while significantly reducing the computational resource consumption on extensive invalid background areas. Sparse convolution By applying a binary mask, the sparse convolution detection head can determine which feature units participate in the convolution process, thereby reducing the amount of computation in invalid background areas.

[0030] The adaptive process of the adaptive sparse convolution detection head includes the following steps: S31, use the channel attention mechanism to quantify the importance of the feature channels of each FPN layer, obtain the weight, and define the output of FPN as a list of feature maps , where each value in the list Corresponding to the feature map output of a specific level of FPN, Represents the FPN Layer, where , It is expressed as: ; Using Channel Attention Network For feature map list Process and get a list of attention scores ,in Corresponding FPN Layer attention score output, It is expressed as: , In the S31, Figure 3 As shown in Figure 1, the channel attention network consists of a global average pooling layer and two fully connected layers connected in sequence. The first fully connected layer uses the ReLU activation function, and the second fully connected layer uses the Sigmoid activation function.

[0031] The specific steps include: Step 1: Given a feature map, its size is , perform global average pooling on the feature map, and use the following formula to compress the size of the feature map to , in Indicates The global average of the channels, which represents the Compressed to The feature map of Indicates the number of channels. represents the height of the feature map, represents the width of the feature map, Indicates that the input feature map is in the channel and spatial location Elements of Step 2: The input is processed into two fully connected layers. The first fully connected network is used for feature dimensionality reduction, and the second fully connected layer further converts the feature vector into a single scalar output. For the first fully connected layer, use the following formula: in, It is the result after the first fully connected layer and ReLU activation function. and are the weights and biases of the first fully connected layer, ReLU is the activation function, For the second fully connected layer, use the following formula: in, For the final output, and are the weights and biases of the second fully connected layer, is the Sigmoid activation function, where is a value between 0 and 1, representing the attention weight. S32, setting a threshold, comparing the attention score in step S31 with the threshold to determine whether to use a detection head with sparse convolution at a specific FPN level, the threshold is expressed as ,if , go to step S33, otherwise, go to step S34; S33, FPN Layer uses a detection head with sparse convolution Perform detection and output the test results , It is expressed as: ; S34, using a common detection head Conduct testing, test results for: .

[0032] The channel attention network processes the feature maps passed in by each FPN layer separately and outputs a weight value between 0 and 1. We set the threshold to 0.5 and use the threshold to determine whether to apply the sparse convolution detection head at this level. If the weight value exceeds the threshold, the sparse convolution detection head is selected; otherwise, the normal detection head is selected. Through this method, the model can adaptively select the most suitable detection head based on the specific context information of each level, thereby optimizing the overall performance and computational efficiency.

[0033] S4, build an adaptive small target detection model based on attention mechanism: including backbone network, neck network, detection head, such as Figure 4 As shown in the figure, the feature enhancement module is added to the backbone network, and the adaptive sparse convolution detection head is used instead of the original detection head; In S4, the model is built based on the RetinaNet framework, and the characteristics of each layer of FPN are optimized to improve the small object detection performance.

[0034] S5, determine the final tiny target detection model: use the training set to train the adaptive tiny target detection model based on the attention mechanism, and use the validation set to evaluate the tiny target detection model.

[0035] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. An adaptive small target detection method based on attention mechanism, characterized by: The method comprises the following steps: S1, obtain the data set: collect small target images through the image acquisition device, annotate the images, and divide the data set into a training set and a validation set. S2, build feature enhancement module, S3, build an adaptive sparse convolutional detection head, S4, build an adaptive small target detection model based on attention mechanism: including backbone network, neck network, detection head, add feature enhancement module to the backbone network, and use adaptive sparse convolution detection head instead of original detection head. S5, determine the final tiny target detection model: use the training set to train the adaptive tiny target detection model based on the attention mechanism, and use the validation set to evaluate the tiny target detection model.

2. The method for adaptive small target detection based on attention mechanism according to claim 1, characterized in that: In S1, various types of drones are used to photograph pedestrians, vehicles, and bicycles of different densities in different environments as target data sets.

3. The method for adaptive small target detection based on attention mechanism according to claim 1, characterized in that: In step S4, based on the RetinaNet framework, the characteristics of each layer of FPN are optimized to improve the performance of small object detection.

4. The method for adaptive small target detection based on attention mechanism according to claim 1, characterized in that: In S2, the feature enhancement module The global feature vector is obtained by performing global average pooling on the layer feature map. Convolution adjusts the number of channels of the vector, and after bilinear interpolation upsampling, Layer-wise lateral connection feature map addition and fusion to improve FPN The feature map representation of the layer includes the following steps: S21, global average pooling, this operation is performed on the highest layer feature map Perform global average pooling, that is, average the feature values ​​of each channel to obtain a channel with the same number of channels and spatial dimension The global feature vector represents the global context information of the entire image. S22, Convolution adjustment: through The convolutional layer adjusts the number of channels of the global feature vector obtained by global average pooling to match the number of channels of the original highest-level feature map to facilitate subsequent fusion operations. S23, bilinear interpolation upsampling, will be passed through The global feature vector adjusted by convolution is upsampled by bilinear interpolation to make its spatial size consistent with The layer feature map is matched. S24, feature fusion, adds and fuses the global feature map after bilinear interpolation upsampling with the horizontally connected feature map of the bilinear interpolation layer.

5. The method for adaptive small target detection based on attention mechanism according to claim 1, characterized in that: In S3, the adaptive sparse convolution detection head uses the channel attention mechanism to quantify the importance of the feature channels of each FPN layer to obtain weights, and dynamically decides whether to use the sparse convolution detection head or the ordinary detection head according to these weights. The adaptive process of the adaptive sparse convolution detection head includes the following steps: S3, build an adaptive sparse convolutional detection head, S31, use the channel attention mechanism to quantify the importance of the feature channels of each FPN layer, obtain the weight, and define the output of FPN as a list of feature maps , where each value in the list Corresponding to the feature map output of a specific level of FPN, Represents the FPN Layer, where , It is expressed as: ； Using Channel Attention Network For feature map list Process and get a list of attention scores ,in Corresponding FPN Layer attention score output, It is expressed as: ， S32, setting a threshold, comparing the attention score in step S31 with the threshold to determine whether to use a detection head with sparse convolution at a specific FPN level, the threshold is expressed as ,if , go to step S33, otherwise, go to step S34; S33, FPN Layer uses a detection head with sparse convolution Perform detection and output the test results , It is expressed as: ； S34, using a common detection head Conduct testing, test results for: 。 6. The method for adaptive small target detection based on attention mechanism according to claim 5, characterized in that: In S31, the channel attention network is composed of a global average pooling layer and two fully connected layers connected in series in sequence. The first fully connected network uses the ReLU activation function, and the second fully connected layer uses the Sigmoid activation function. The specific steps include: Step 1: Given a feature map, its size is , perform global average pooling on the feature map, and use the following formula to compress the size of the feature map to , in Indicates The global average of the channels, which represents the Compressed to The feature map of Indicates the number of channels. represents the height of the feature map, represents the width of the feature map, Indicates that the input feature map is in the channel and spatial location Elements of Step 2: The input is processed into two fully connected layers. The first fully connected network is used for feature dimensionality reduction, and the second fully connected layer further converts the feature vector into a single scalar output. For the first fully connected layer, use the following formula: in, It is the result after the first fully connected layer and ReLU activation function. and are the weights and biases of the first fully connected layer, ReLU is the activation function, For the second fully connected layer, use the following formula: in, For the final output, and are the weights and biases of the second fully connected layer, is the Sigmoid activation function, where It is a value between 0 and 1, representing the attention weight.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, an adaptive small target detection method based on the attention mechanism as described in any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instruction is executed by the processor, an adaptive small target detection method based on the attention mechanism as described in any one of claims 1-6 is implemented.

Images