A small target detection method based on multi-scale feature fusion and feature enhancement

Abstract

The application discloses a small target detection method based on multi-scale feature fusion and feature enhancement, and belongs to the technical field of target detection.The method comprises the following steps: introducing a feature fusion module, a feature enhancement module and a feature reconstruction module, constructing a small target detection model based on multi-scale feature fusion and feature enhancement, training the constructed small target detection model, and using the trained small target detection model to realize small target detection.The small target detection method based on multi-scale feature fusion and feature enhancement provided by the application uses the feature fusion and feature enhancement method, and solves the problems of insufficient feature information of the existing small target detection algorithm, easy occlusion of small targets and difficulty in distinguishing small targets from complex backgrounds by means of feature reconstruction, thereby improving the small target detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly to a small target detection method based on multi-scale feature fusion and feature enhancement. BACKGROUND

[0002] Small target detection is an important research direction in the field of computer vision, aiming to classify and locate instances with limited areas from images with complex backgrounds. In the COCO dataset, targets with an area less than 32x32 pixels are generally classified as small targets. In recent years, object detection in remote sensing images has received extensive attention. Small targets are ubiquitous in remote sensing images, and detecting small objects in remote sensing images is of great significance for various application scenarios such as military reconnaissance, maritime rescue, and traffic management. In recent years, with the development of deep learning, many detection models based on deep learning have been proposed and have made important progress in object detection. Currently, small target detection algorithms mainly include two-stage and one-stage methods:

[0003] Two-stage detection algorithms divide the detection process into two parts: the first stage generates candidate regions, and the second stage classifies and locates these regions. By training two parts separately, such detectors can adapt to various target detection tasks, providing higher detection accuracy and precision, especially in large-scale target detection tasks. Common two-stage object detection algorithms include the R-CNN series, Faster R-CNN series, and Mask R-CNN series. However, two-stage detection algorithms usually include two main stages, each of which requires independent computation, resulting in longer overall inference time. Moreover, when the candidate regions are large, they consume a lot of memory. Furthermore, due to the long inference time, two-stage detection algorithms are generally not suitable for real-time application scenarios such as autonomous driving and unmanned aerial vehicle monitoring.

[0004] One-stage detection algorithms have a relatively simple structure, are easy to implement and deploy, and directly predict the position and class of objects from images without first generating candidate regions. With an end-to-end training method and high real-time detection performance, one-stage algorithms are suitable for most target detection tasks, especially those that require fast detection, and have a great advantage. Common one-stage detection algorithms include the YOLO series, SSD, and FCOS network. However, one-stage algorithms are slightly inferior to two-stage algorithms in terms of accuracy, especially when dealing with small targets. They are prone to misdetecting noise in the background as targets, which makes one-stage detectors unable to accurately locate and classify small target objects.

[0005] However, whether it is a single-stage small target detection algorithm or a two-stage small target detection algorithm, due to the small pixel area of small targets in the image, the feature information of small targets is insufficient, and small targets are located in a complex environment background, such as urban streets or natural environments such as forests, so that the interference in the background may be mistaken for a target or the real target is difficult to identify due to being blocked by the background. These factors often lead to a decrease in detection accuracy when positioning and classifying small targets, especially in remote sensing images. Therefore, enhancing the feature representation of small targets and enhancing the distinction between small targets and the background are still challenging tasks.

[0006] At present, existing small target detection algorithms can be summarized as follows:

[0007] (1) Small target detection algorithm based on multi-scale representation learning: Multi-scale feature fusion has made significant progress in computer vision, especially in the field of target detection. However, compared with medium and large targets, small targets have fewer pixels, making feature extraction more difficult. Moreover, as the number of network layers increases, the feature and position information is gradually lost, making it difficult to be effectively detected. Although multi-scale feature fusion methods, such as FPN, can improve target positioning and classification to some extent by combining shallow and deep features, these methods still have shortcomings. First, the features of small and large targets in existing multi-scale fusion methods may be confused at certain scales, even at fine-grained scales, and may still be disturbed by background noise or large targets. Second, effectively fusing features of different scales is still a challenge. If the fusion strategy is not appropriate, it may lead to information loss or redundancy. And existing multi-scale methods can capture targets of different sizes, but in some cases, smaller targets may still not be effectively distinguished from the background, especially when the target edge is blurred or the target itself is too small and lacks information, the algorithm may have difficulty in accurately positioning the target. Although FPN and its improved versions (such as CE-FPN, PANet, NAS-FPN, BiFPN, and AugFPN) have improved multi-scale fusion to some extent, they have not solved the problem of noise or large target interference and the difficulty of distinguishing small targets from complex backgrounds in small target detection, affecting the accuracy and robustness of detection.

[0008] (2) Feature enhancement-based small target detection algorithm: Feature enhancement in small target detection aims to strengthen the feature representation of small targets in the image through specific algorithms or techniques, making it easier for the detector to capture and identify these targets. The core idea of this method is to process the original input data or intermediate layer feature maps to improve the saliency of small targets relative to the background or other interference factors, and to emphasize the use of contextual information to help classify and locate small targets, thereby improving detection accuracy. However, some feature enhancement methods still have certain limitations. Some researchers have proposed feature enhancement modules specifically for small target detection in remote sensing images, which assign weights to feature maps from spatial and channel directions, making the model not only effectively suppress the impact of noise but also highlight useful features, improving target classification accuracy and positioning precision. However, current feature enhancement methods often generate a large number of enhanced features, but not all of these features are helpful for small target detection. In some cases, redundant enhanced features may cause information overload, making it more difficult for the network to extract useful signals, thereby affecting detection accuracy. Secondly, some feature enhancement methods may strengthen background noise or irrelevant areas, especially when the enhancement process does not fully consider the characteristics of complex backgrounds, which may introduce noise interference to small target detection, leading to false positives or false negatives.

[0009] (3) Attention mechanism-based small target detection algorithm: The application of attention mechanism in small target detection aims to simulate the characteristics of the human visual system, which can automatically focus on the most relevant or most important part of the image. This mechanism allows the model to dynamically allocate weights to different regions when processing input data, thereby improving the focus on key information, especially in the task of detecting small targets in complex backgrounds. The core of this method is to assign different weights to different parts of the input feature map, so that the network focuses more on those regions that are most important for the current task. In recent years, in order to solve the problem that different scale feature maps have different importance, some researchers have proposed to dynamically adjust the importance of different scale feature maps through attention mechanism, thereby reducing the redundant information in the feature layer.

[0010] In summary, most of the existing small target detection algorithms rely on a cascaded method, which first roughly extracts the important region of the feature, and then uses this as a basis to guide it to highlight the small target area and the surrounding context information conducive to the small target. However, this method requires multiple stages to gradually refine the detection results, and the network of each stage usually needs to be trained separately or optimized jointly. Due to the dependency between stages, it is difficult to achieve complete parallel processing. In addition, some current research methods start from the perspective of data imbalance of small targets, and through label optimization strategy, the proportion of small targets in the image is equivalent to that of large targets. However, due to the small size of the small target, even with the label optimization strategy, it may not be able to completely distinguish the small target from the background noise. Therefore, the current small target detection is faced with the problems of insufficient feature information and easy to be partially or completely blocked by other objects or background, especially in dense scenes, making it difficult for small targets to separate from the background. Although the existing methods have made efforts from various aspects to solve the above problems, most methods ignore the advantages of high-resolution feature maps and the characteristics of small targets are not fully represented, making it difficult to achieve accurate detection results. SUMMARY

[0011] The present application provides a small target detection method based on multi-scale feature fusion and feature enhancement to solve the technical problem that the existing small target detection method is difficult to achieve accurate detection results.

[0012] To solve the above technical problems, the present application provides the following technical solutions:

[0013] On the one hand, the present application provides a small target detection method based on multi-scale feature fusion and feature enhancement, which comprises:

[0014] Constructing a small target detection model based on multi-scale feature fusion and feature enhancement;

[0015] Training the constructed small target detection model;

[0016] Using the trained small target detection model to realize small target detection.

[0017] Further, the small target detection model comprises an input module, a backbone network, a feature fusion module, a feature enhancement module, a feature reconstruction module, and a detector.

[0018] The feature reconstruction module is only used in the model training stage and is discarded in the inference stage.

[0019] The process of the small target detection model to realize small target detection comprises:

[0020] The to-be-detected image is input into the backbone network through the input module, processed by the backbone network, and multi-scale feature maps are obtained. Then, the multi-scale feature maps are sent to the feature fusion module. In the feature fusion module, the multi-scale feature maps are first optimized, and then the optimized multi-scale feature maps are fused from top to bottom to obtain feature-fused feature maps. Subsequently, the feature-fused feature maps are sent to the feature enhancement module for processing. The feature representation of small targets is enhanced by capturing local detail information of small targets and global information captured by interaction between pixels, to obtain feature-enhanced feature maps. Finally, the feature-enhanced feature maps are sent to the detector to realize classification and positioning of targets.

[0021] Further, the feature fusion module comprises a channel attention module and a spatial attention module.

[0022] The feature fusion module first optimizes the multi-scale feature maps output by the backbone network through the channel attention module to highlight important regions in different scale feature maps, and then fuses the optimized multi-scale feature maps through the spatial attention module to obtain feature-fused feature maps.

[0023] Further, the optimization of the multi-scale feature maps output by the backbone network through the channel attention module to highlight important regions in different scale feature maps comprises:

[0024] The channel attention module first performs channel processing on the multi-scale feature maps output by the backbone network through adaptive max pooling. Then, the pooled feature vectors are fused and passed through a Sigmoid function to generate final channel attention weights. Finally, the original feature maps are multiplied by the channel attention weights to enhance features useful for small target detection and suppress irrelevant features. The features obtained after multiplying the original feature maps by the channel attention weights are point-convolved so that the number of feature mappings of each scale is 256, to obtain the optimized multi-scale feature maps to highlight important regions in different scale feature maps.

[0025] Further, the fusion of the optimized multi-scale feature maps through the spatial attention module to obtain feature-fused feature maps comprises:

[0026] The low-level feature map and the high-level feature map in the feature map optimized by the channel attention module are respectively unfolded into sequences, the sequence corresponding to the high-level feature map is taken as the query, the sequence corresponding to the low-level feature map is taken as the key and the value, the Self-Attention is performed between the high-level feature map and the low-level feature map, the Self-Attention result is changed into a two-dimensional image, then the dimension of the two-dimensional image is matched with the low-level feature map through upsampling, finally the low-level feature map is fused with the two-dimensional image after upsampling, and a feature map after feature fusion is obtained.

[0027] Further, the feature enhancement module comprises a global information extraction module, a local information extraction module and an inverted residual structure.

[0028] The global information extraction module captures global information by performing perception and interaction between pixels; the local information extraction module completes local information extraction through parallel calculation of three ordinary convolutions and three expansion convolutions; wherein the expansion rates of the three expansion convolutions are 2, 4 and 2 respectively; the features obtained by fusing the local information and the global information are sent into the inverted residual structure to obtain a feature map after feature enhancement.

[0029] Further, the global information extraction module first unfolds a two-dimensional image in units of pixels to change it into a spatially continuous sequence, then performs an average operation on the channels, then performs linear calculation through an FFN and obtains the probability distribution of the features in their spatial dimensions by using a Softmax activation function, obtains the weight between pixels on the feature map, and finally reshapes it into a two-dimensional image.

[0030] Further, the feature reconstruction module realizes feature reconstruction based on a mask autoencoder.

[0031] In still another aspect, the present application also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0032] In still another aspect, the present application also provides a computer readable storage medium, which stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0033] The technical solution provided by the present application has at least the following beneficial effects:

[0034] 1. Compared with other small target detection methods based on multi-scale representation learning: the technical solution provided by the present application fully considers the importance of different scale feature maps, and proposes a gradual and refined feature fusion strategy. The strategy first uses a channel attention module to filter the feature map, which can automatically learn the important features for small target detection. Then, the feature fusion is performed through a top-down feature fusion module, which helps the model to understand the complex scene and the surrounding environment. Not only does it reduce the redundant information in the multi-scale feature fusion process, but it also fully utilizes the advantages of high-level and low-level feature maps, which helps to solve the problem of distinguishing small targets from complex backgrounds.

[0035] 2. Compared with small target detection methods based on feature enhancement: small objects often need to use surrounding environment information for more accurate classification and positioning. For this, the feature enhancement method proposed in the technical solution of the present application focuses on both local detail information and global context information of small targets. It not only highlights the importance of surrounding environment information, but also fully utilizes the advantages of local detail information, greatly improving the accuracy of small target detection. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0037] Figure 1 is the execution flow diagram of the small target detection method based on multi-scale feature fusion and feature enhancement provided by the embodiment of the present application;

[0038] Figure 2 is the architecture diagram of the small target detection model provided by the embodiment of the present application;

[0039] Figure 3 is the SAFPN module architecture diagram provided by the embodiment of the present application;

[0040] Figure 4 is the system block diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0042] First, it should be pointed out that in the embodiments of the present application, the words "exemplarily", "for example" and the like are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of the word "exemplarily" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be either one of the two.

[0043] First embodiment

[0044] The present embodiment provides a small target detection method based on multi-scale feature fusion and feature enhancement, aiming to establish an effective and fine single-stage small target detection technology. The method focuses on the important area of small targets through a feature fusion module SAFPN, solving the problem that small targets are difficult to distinguish from the background. High-resolution feature maps can often retain more detailed information of small targets, such as the boundaries, textures and shapes of small targets, which are particularly important for small target detection. For example, the texture features of the target can effectively help the model to distinguish adjacent complex backgrounds. Therefore, we introduce a new feature enhancement module LGFE to enhance the feature representation of small targets. This module captures local information through convolution and global information between pixels, thereby obtaining the context information and detailed information of small targets. We also additionally introduce a feature reconstruction module FR, which can serve as auxiliary information to force the model to learn more rich and detailed feature representations, alleviating the problem of information loss caused by down-sampling of small targets in deep neural networks and the problem of small targets being difficult to distinguish from complex backgrounds.

[0045] The method can be implemented by an electronic device, and the execution flow of the method is as shown in Figure 1 , including the following steps:

[0046] S1, constructing a small target detection model based on multi-scale feature fusion and feature enhancement;

[0047] S2, training the constructed small target detection model;

[0048] S3, using the trained small target detection model to realize small target detection.

[0049] Specifically, as shown in Figure 2 , the small target detection model of the present embodiment is improved on the basis of the FCOS network model, which includes an input module, a backbone network, a feature fusion SAFPN module, a feature enhancement LGFE module, a feature reconstruction FR module and a detector. Based on this, the working process of the small target detection model can be described as follows:

[0050] Firstly, input data is processed by backbone network (ResNet50) to obtain multi-scale feature maps, and then the multi-scale feature maps are sent to the SAFPN module. In the SAFPN module, feature optimization is first performed, and then feature fusion from top to bottom is performed, and in turn, high-resolution feature maps to low-resolution feature maps are obtained, respectively P2, P3, P4, P5 and P6. Subsequently, the high-resolution feature map P2 passes through the feature enhancement LGFE module, and the feature representation of the small target is enhanced by capturing the local detail information of the small target and the global information captured by the interaction between pixels. Finally, the multi-scale feature maps are sent to the detector (FCOS network detector) for classification and positioning.

[0051] Next, the core modules (SAFPN module, LGFE module, FR module) in the model are introduced.

[0052] 1. Multi-scale feature fusion module-SAFP

[0053] This module is an effective and fine method of multi-scale feature fusion for small target features. By capturing important features of small targets in multi-scale feature maps, the discrimination between small targets and complex backgrounds is improved, and the model focuses more on important areas related to small targets, thereby improving the detection performance of the model. First, it optimizes the multi-scale features passing through the channel attention module, thereby highlighting important areas in different scale feature maps, and then performs feature fusion through the spatial attention module. The module includes two parts: feature optimization and feature fusion. As shown in Figure 3 .

[0054] Feature optimization: from the perspective of feature channels, first, the channel attention module first processes the input feature map through adaptive max pooling. Then, the feature vectors after pooling are fused, and the final channel attention weight is generated through the Sigmoid function. Finally, the original feature map is multiplied by the channel attention weight to enhance the features useful for small target detection and suppress irrelevant features. The important features obtained are passed through PWConv, so that the number of feature mappings of each scale is 256, so that feature mapping matching can be performed on different scales. Adaptive max pooling selects the most significant or most important response value in each channel, emphasizing local, prominent feature points, which can better capture key but small targets or detailed information in the image, and enhance the feature selection ability of the model in the channel dimension, which helps to extract more accurate feature information from each channel.

[0055] Feature fusion: In the multi-scale feature map, the low-level feature map is derived from the shallow network and rich in spatial information. The spatial information has a high feature resolution, which can better maintain the spatial structure information of the image and help better capture the local details in the image. However, due to the lack of high-level semantic content expression and the weak understanding of objects and context in complex scenes, the high-level feature map is derived from the deep network and has rich high-level semantic information. The semantic information has a low feature resolution, such as scene context and semantic relationship, which helps to aggregate global information, but loses the detail information in the image, especially when dealing with small targets. The positioning accuracy of small targets is not very accurate. The SAFF module in this embodiment fully utilizes the advantages of low-level feature maps with accurate detail information and high-level feature maps with rich semantic information. It not only promotes the exchange between features of different levels, enabling the model to dynamically adjust the importance of features according to the input content, but also helps the model better understand the relationship between the target and its surrounding environment by calculating the interaction between different positions of the feature map. For example Figure 3 As shown in FIG. 7, the low-level feature map and the high-level feature map are respectively unfolded into a sequence. The sequence of the high-level feature map is Q, and the sequence of the low-level feature map is K and V. The Self-Attention is performed between the low-level feature map and the high-level feature map, and then it is changed into a two-dimensional image. The two-dimensional image is matched with the dimension of the low-level feature map through upsampling, and finally the two-dimensional image is fused.

[0056] In summary, the embodiment constructs a gradual and refined feature pyramid feature fusion module SAFPN according to the characteristics of multi-scale features. First, the features of different scales are selected in the channel dimension, and then the low-level feature map is guided to perform feature fusion based on the high-level feature map, as shown in FIG. 8. In the figure, CA represents the channel attention module, PWconv represents the point convolution, Conv represents the 3x3 convolution, Q, K and V represent query, key and value respectively. The feature fusion SAFPN module enhances the discrimination of small target features and large target features and complex background through the feature optimization module and the feature fusion module, so that the model can more easily distinguish small targets from complex backgrounds, and the problem of insufficient small target feature information is also solved. Figure 3

[0057] 2, Feature enhancement module LGFE:

[0058] The embodiment designs a feature enhancement module LGFE, as shown in FIG. 9. In the figure, CA represents the channel attention module, PWconv represents the point convolution, Conv represents the 3x3 convolution, and Q, K and V represent query, key and value respectively. Figure 2 ​As shown, the module fuses local information and global information to improve the accuracy of small target detection. First, we introduce the Patch Aware Attention (PAA) module, which is responsible for pixel-to-pixel perception and interaction to capture global information. The PAA module first expands the two-dimensional image in pixel units to become a spatially continuous sequence, then performs an average operation in the channel, and then performs linear calculation through a Feed-Forward Neural Network (FFN) and uses a Softmax activation function to obtain the probability distribution of the feature in its spatial dimension, obtain the weight between pixels on the feature map, and finally reshape it into a two-dimensional image. The capture of local information is completed by three ordinary convolutions and three extended convolutions calculated in parallel, where the expansion rates of the extended convolutions are 2, 4, and 2, respectively. The receptive field of the ordinary convolution is directly related to the size of the convolution kernel, which can accurately capture local information. The extended convolution expands the receptive field by inserting holes between the convolution kernel elements, which can effectively increase the receptive field without changing the size of the convolution kernel, which is very beneficial to maintaining image detail information, but a larger expansion rate may cause the captured long-distance information to be irrelevant. The fusion of ordinary convolution and extended convolution can compensate for the shortcomings of each other. The inverted residual structure in MobileNetv2 has been proven to not only help the back propagation of gradients and reduce computational complexity, but also enable the model to better maintain the integrity of the input feature information, ensuring effective information transmission. Therefore, finally, we send the feature representation obtained by fusing local information and global information into the inverted residual structure to ensure that the enhanced small target feature representation can be effectively transmitted. The fusion of local information and global information in high-resolution feature maps makes up for the loss of detailed information in global features and the lack of rich semantic information in local features, reduces the information loss problem of small targets in the down-sampling process, and enables the model to obtain more rich and accurate feature representation, improving the model's understanding of the environment around small targets.

[0059] 3. Feature reconstruction module FR:

[0060] Masked Autoencoders (MAE) is a sub-supervised learning method that masks part of the input image and reconstructs the entire image based on the unmasked part, not only enhancing the model's understanding of the image, but also generating powerful feature representation. As Figure 2As shown, the feature reconstruction module proposed in this paper first generates a binary image object map with the same size as the original image according to the gt bboxes (the real detection boxes labeled in the picture), where 1 represents the region where the small target is located, and 0 represents the background region. Then, the target map is obtained by maximum pooling, so that the binary image object map matches the size of the high-resolution feature map P2. Then, we upsample the P4 feature map and the P6 feature map to obtain f4 and f6, so that they can match the size of the high-resolution feature map P2. MAE will mask a large part of the random image patch subset, and MAE has proved that the mask ratio of 75% has the best effect of identifying the effect. In the feature reconstruction module in this paper, the mask is generated according to the binary image object map. First, the binary image is unfolded into a group of continuous patches in space according to the patch size of 8, and the pixel sum in each patch is calculated. If the pixel sum is greater than or equal to 1, it means that there is a small target in the patch, and the mask is performed. Then, the proportion of the masked patches is counted. If it does not reach 75%, the mask is randomly performed from the unmasked patches to ensure that the mask ratio of the patches is 75%. Then, according to the generated mask and the binary image target map, the high-resolution feature map P2, and f4 and f6 are reconstructed into the original image by the decoder of MAE, respectively. The reconstructed original image and the binary image target map are made into mean square error loss, and finally the loss of feature reconstruction is added to obtain the final auxiliary loss. It is worth noting that the feature reconstruction module proposed in this paper is only used in the training stage, and the module is discarded in the inference stage.

[0061] In summary, the feature reconstruction module is constructed according to the advantages of the high-level feature map in this embodiment. Then, the P2, P4 and P6 feature maps are respectively sent into the feature reconstruction FR module for feature reconstruction, so as to force the model to learn more rich and detailed feature representations, such as Figure 2 As shown, wherein, “DW Conv” represents a depth convolution, which applies a convolution kernel to each input channel separately, while reducing the number of parameters and still retaining the basic characteristics of traditional convolution. “DConv” represents an inflation convolution. Among them, the high-resolution feature map contains more detailed information of the small target. Based on the high-resolution feature map, the local detailed information and the global context information of the high-resolution feature map are captured at the same time, which can improve the understanding ability of the model to the complex background and the surrounding environment.

[0062] Next, the effectiveness of the scheme proposed in the present application is verified.

[0063] Table 1 provides the performance comparison results of this method with other state-of-the-art methods on the AI-TOD-v2 dataset. As shown in Table 1, compared with other methods, the feature fusion and feature enhancement as well as the design of the feature reconstruction module of this method make the model accuracy reach 26.9%, which exceeds other methods and has a high AP 50 , AP 75 , AP vt , AP t and AP s Among the indicators, this method also achieved the highest and outperformed other methods. The experimental results verified the effectiveness of the method proposed in this paper.

[0064] Table 1 Comparison of the performance of this method with other state-of-the-art methods on the AI-TOD-v2 dataset

[0065]

[0066] The AI-TOD-v2 dataset contains eight common small object categories, including airplanes (AI), bridges (BR), storage tanks (ST), ships (SH), swimming pools (SP), vehicles (SE), people (PE), and windmills (WM). As shown in Table 2, we experimentally evaluate the AP for each category in the AI-TOD-v2 dataset. The experimental data demonstrates that our method outperforms previous methods on all categories in the AI-TOD-v2 dataset. These results highlight the superior performance of our method on this challenging small object dataset.

[0067] Table 2 AP evaluation results of various categories of AI-TOD-v2 dataset

[0068]

[0069]

[0070] As shown in Table 3, on the SODA-A dataset, compared to other rotated-box-based methods, our method achieves an AP of 33.7%, outperforming other methods. Furthermore, across categories, our method outperforms other methods in AP metrics across multiple categories. Overall, the proposed method demonstrates comparable results to existing methods on this challenging small-object dataset, demonstrating its effectiveness on the SODA-A dataset.

[0071] Table 3 Comparison results of this method with other rotation box based methods on the SODA-A dataset

[0072]

[0073]

[0074] From the above, it can be seen that through extensive experiments on the AI-TOD-v2 and SODA-A data sets, it is shown that the method proposed in the application is superior to the performance of other methods. The method proposed in the application provides a feasible solution to solve the problems of insufficient feature information of small targets and difficulty in distinguishing small targets from complex backgrounds, and provides a new direction and idea for further research on small target detection algorithms.

[0075] In summary, the embodiment provides a small target detection method based on multi-scale feature fusion and feature enhancement, designs a gradual and refined feature fusion method, which first selects features for multi-scale features, and then performs fine feature fusion based on the advantages of high-level feature maps and low-level feature maps. A feature enhancement LGFE module is introduced, which is based on high-resolution feature maps, and simultaneously captures local detail information and global context information of small targets, so as to enhance the feature representation of small targets and reduce the influence of redundant noise. An additional feature reconstruction FR module is introduced, which forces the model to learn more rich and detailed feature representation through feature reconstruction, solves the problems of information loss of small targets in the down-sampling process and difficulty in distinguishing small targets from complex backgrounds. Thus, the feature fusion and feature enhancement methods are used, and the feature reconstruction method is used to solve the problems of insufficient feature information of existing small target detection algorithms, small targets being easily occluded, and difficulty in distinguishing small targets from complex backgrounds, thereby effectively improving the small target detection performance.

[0076] The method of the embodiment can be applied to multiple fields, such as in the monitoring and security field, small target detection in monitoring cameras is one of the key tasks, in monitoring videos, small targets such as intruders and suspicious objects need to be detected to ensure public safety, and accurately detecting the category and position is crucial for the normal operation of urban safety and traffic management systems; in the field of aerospace and military applications, small target detection is a key problem in devices such as unmanned aerial vehicles (UAVs) and aircrafts, for example, in military reconnaissance, it is necessary to detect small targets such as soldiers, vehicles, and weapons, and accurately detecting these targets plays an important role in military decision-making and flight safety; in the medical field, small target detection can be used to detect small lesion areas such as tumor cells and blood vessel abnormalities, such as detecting small nodules in early lung cancer screening; in ophthalmic disease diagnosis, retinal lesions are detected to accurately identify small abnormal structures in medical images, which is crucial for early disease diagnosis and treatment; in the field of autonomous driving, small target detection can be used to detect small targets such as traffic signs, pedestrians, and bicycles in the distance to ensure driving safety and compliance with traffic rules, and the detection accuracy of small targets directly affects the reaction time and safety of the system; in the industrial detection field, small target detection can be used to detect defects or small parts in products to ensure product quality.

[0077] Second embodiment

[0078] The embodiment provides an electronic device, such as Figure 4 As shown in the figure, the electronic device comprises a processor and a memory; wherein the processor and the memory can be connected through a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment. In addition, the electronic device can further comprise a transceiver, the processor and the transceiver can be connected through a communication bus, and the transceiver is used for communication with other devices.

[0079] Next, the method will be described in detail in combination with Figure 4 The various components of the electronic device will be described in detail:

[0080] The processor is the control center of the electronic device, and the electronic device can comprise a plurality of processors, each of which can be a single-CPU or a multi-CPU. The processor herein can be a processor or a general term of a plurality of processing elements. For example, the processor is one or more central processing units (CPU), which can also be other general-purpose processors, application specific integrated circuits (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor can perform various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0081] In a specific implementation, as an embodiment, the processor can comprise one or more CPUs, such as CPU0 and CPU1 shown in Figure 4 of course, this is only an exemplary description.

[0082] The memory is used to store software programs for implementing the scheme of the present application, and is controlled by the processor to execute, and the specific implementation manner can refer to the above-mentioned method embodiments, which will not be described here.

[0083] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Figure 4 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0084] The transceiver may include a receiver and a transmitter ( Figure 4 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Figure 4 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0085] In addition, it should be noted that Figure 4 The structure of the electronic device shown in the figure does not constitute a limitation on the device. The actual device may include more or fewer components than shown, or may combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment can refer to the technical effects described in the first embodiment above, and therefore will not be repeated here.

[0086] Third embodiment

[0087] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device. The instructions stored therein can be loaded by a processor in a terminal to execute the method described above.

[0088] Moreover, it should be noted that the present application can be provided as a method, an apparatus, or a computer program product. Therefore, the embodiments of the present application can take the form of an entirely or partially hardware embodiment, an entirely or partially software embodiment, or an embodiment combining software and hardware aspects. Furthermore, when implemented in software, the embodiments of the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, a computer diskette, an optical storage medium, a magnetic storage medium, and a semiconductor memory device). The computer program product includes one or more computer instructions that when loaded and executed by a computer, cause the computer to carry out the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, such as from a website, a computer, a server, or a data center to another website, computer, server, or data center through a wired (for example, infrared, wireless, microwave, or the like) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, data center, or the like, including one or more collections of available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0089] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, an embedded processor, or a processor of another programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device that implements the flow Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0090] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the flow Figure 1 one or more flows and / or blocks Figure 1the functions specified in the individual block or blocks. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, thus the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flowchart block(s). Figure 1 the functions specified in the individual block or blocks. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, thus the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flowchart block(s). Figure 1 the functions specified in the individual block or blocks. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, thus the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flowchart block(s).

[0091] It should also be noted that, in the present document, the terms such as first and second, etc. are merely used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or terminal device. Without more limitations, the element defined by the statement "including a…", does not exclude the presence of other identical elements in the process, method, article or terminal device including the element. In addition, the term "and / or" is merely a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. In addition, the character " / " in the present document generally represents an "or" relationship between the front and rear associated objects, but it can also represent an "and / or" relationship, which can be understood in the context before and after. "One or more" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0092] In addition, it can be understood that in various embodiments of the present application, the size of the sequence number of the above processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0093] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0094] In the several embodiments provided herein, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of functional modules / units is merely a logical functional division. In actual implementation, other division methods may be used, such as multiple units or components being combined or integrated into another device, or some features being ignored or not implemented. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interface, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs. In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0095] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0096] Finally, it should be noted that the above description is only the preferred embodiment of the application, it should be pointed out that although the preferred embodiment of the application has been described, for those skilled in the art, once the basic creative concept of the application is known, several improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements should also be considered as the protection scope of the application. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the application.

Claims

1. A small target detection method based on multi-scale feature fusion and feature enhancement, characterized in that: include: Construct a small target detection model based on multi-scale feature fusion and feature enhancement; Train the constructed small target detection model; Use the trained small target detection model to realize small target detection; The small target detection model includes: an input module, a backbone network, a feature fusion module, a feature enhancement module, a feature reconstruction module and a detector; The feature reconstruction module is only used in the model training phase and is discarded in the inference phase. The process of implementing small target detection by the small target detection model includes: The image to be detected is input into the backbone network through the input module, and is processed by the backbone network to obtain a multi-scale feature map. Then, the multi-scale feature map is sent to the feature fusion module. In the feature fusion module, the multi-scale feature map is first feature optimized, and then the optimized multi-scale feature map is subjected to top-down feature fusion to obtain a feature map after feature fusion. Subsequently, the feature map after feature fusion is sent to the feature enhancement module for processing. The feature representation of the small target is enhanced by capturing the local detail information of the small target and the global information captured by the interaction between pixels, to obtain a feature map after feature enhancement. Finally, the feature map after feature enhancement is sent to the detector to achieve target classification and positioning; The feature fusion module includes a channel attention module and a spatial attention module; The feature fusion module first optimizes the multi-scale feature map output by the backbone network through the channel attention module to highlight the important areas in the feature maps of different scales, and then performs feature fusion on the optimized multi-scale feature map through the spatial attention module to obtain a feature map after feature fusion; The feature enhancement module includes a global information extraction module, a local information extraction module, and an inverted residual structure. The global information extraction module captures global information by perceiving and interacting between pixels. The local information extraction module extracts local information through three ordinary convolutions and three dilated convolutions in parallel. The dilation rates of the three dilated convolutions are 2, 4, and 2, respectively. The features obtained by fusing the local information and the global information are fed into the inverted residual structure to obtain a feature map after feature enhancement. The feature reconstruction module realizes feature reconstruction based on a mask autoencoder.

2. The small target detection method based on multi-scale feature fusion and feature enhancement according to claim 1, characterized in that: The multi-scale feature map output by the backbone network is optimized by the channel attention module to highlight the important areas in the feature maps of different scales, including: The channel attention module first performs channel processing on the multi-scale feature map output by the backbone network through adaptive maximum pooling, then fuses the pooled feature vectors and generates the final channel attention weight through the Sigmoid function, and finally multiplies the original feature map with the channel attention weight to enhance the features useful for small target detection and suppress irrelevant features. Finally, the features obtained by multiplying the original feature map with the channel attention weight are point convolved so that the number of feature maps at each scale is 256, and the optimized multi-scale feature map is obtained to highlight the important areas in the feature maps of different scales.

3. The small target detection method based on multi-scale feature fusion and feature enhancement according to claim 1, characterized in that: The feature fusion of the optimized multi-scale feature map by the spatial attention module to obtain the feature fusion feature map includes: The low-level feature maps and high-level feature maps in the feature map optimized by the channel attention module are expanded into sequences respectively, and the sequence corresponding to the high-level feature map is used as the query, and the sequence corresponding to the low-level feature map is used as the key and value. Self-Attention is performed between the high-level feature map and the low-level feature map, and the self-attention result is converted into a two-dimensional image. Then, the dimension of the two-dimensional image is matched with the low-level feature map through upsampling. Finally, the low-level feature map is fused with the upsampled two-dimensional image to obtain the feature map after feature fusion.

4. The small target detection method based on multi-scale feature fusion and feature enhancement according to claim 1, characterized in that: The global information extraction module first expands the two-dimensional image in pixels to make it a spatially continuous sequence. Then, it performs an averaging operation on the channel, and then uses FFN to perform linear calculations and use the Softmax activation function to obtain the probability distribution of features in their spatial dimensions, obtain the weights between pixels on the feature map, and finally reshape it into a two-dimensional image.

Images