Small target detection method based on progressive multi-scale feature fusion network

Abstract

The invention belongs to the technical field of small target detection, particularly relates to a small target detection method based on a progressive multi-scale feature fusion network, and solves the problem that semantic information in deep features of a feature pyramid network in an existing small target detection method is lost. The method comprises the following steps: uniformly adjusting the size of a small target image to be detected, and inputting the adjusted image into a backbone network part of a small target detection network; the backbone part performs feature extraction on an input to-be-detected image to obtain feature maps of four scales; the neck part is used for carrying out feature augmentation on four-scale feature maps output by the backbone network; performing scale alignment on the feature maps, and performing up-sampling or down-sampling on features of different levels to realize scale adjustment on each feature map; and inputting the processed feature map into a detection head to predict confidence, classification and a bounding box.

Description

Technical Field

[0001] The present invention relates to the technical field of small target detection, and in particular to a small target detection method based on a progressive multi-scale feature fusion network. Background Art

[0002] Object detection is a basic task in the field of computer vision. It is a key task to classify and locate objects in images or videos. With the help of large amounts of data and powerful learning capabilities of deep convolutional neural networks, it has made significant progress in recent years. As a subfield of general object detection, small object detection focuses on detecting objects with smaller sizes. It has important theoretical and practical significance in various scenarios such as surveillance, drone scene analysis, pedestrian detection, and traffic sign detection. In fields such as autonomous driving, small object detection has long been a difficult problem. Its goal is to accurately detect small objects with very few visual features in images.

[0003] Small target detection is crucial for real-world visual applications. For example, in aerospace images or video surveillance taken at a long distance, with the help of computers, the captured high-quality image data can be effectively analyzed and processed to identify different categories of targets and mark their locations. This technology is widely used in various mission scenarios, helping to accurately locate and identify targets, saving a lot of manpower and time costs, so it has important research significance and practical application value.

[0004] The main difference between small target detection and general target detection is the size of the objects being processed. Small targets usually have low resolution and low-quality appearance, and there are varying degrees of occlusion, blur, and incompleteness, so it is difficult to discern and learn representations from their structures. At the same time, the regional features of all objects are more easily contaminated by the background and other instances, introducing noise into further learning representations. In summary, the feature representations of small objects are easily affected by noise, which hinders subsequent detection. Improving the ability to extract small target features and the quality of feature fusion is the key to solving the problem.

[0005] Existing small target detection methods are mainly divided into the following types: The first type is multi-scale feature extraction. Traditional target detection algorithms are designed for ordinary targets. When extracting features, since the target has enough features, it is not sensitive to the partial loss of target features. However, for small targets, features are lost during the interaction process, which causes the features of small targets to disappear completely when they are input into the detection module. The second type of method is multi-scale feature fusion. When performing feature fusion, each spatial position may encounter conflicts between multiple object information. The feature pyramid network effectively enhances the detection ability of targets of different scales by constructing a multi-scale feature hierarchy, especially by fusing high-level semantic features and detail features to improve the detection performance of small targets, but multiple downsampling leads to the degradation of fragile small target features. Based on this, it is necessary to invent a new small target detection method to solve the above problems existing in the existing small target detection methods. Summary of the invention

[0006] In order to overcome the technical defects of the existing small target detection methods, that is, the semantic information is lost in the deep features of the feature pyramid network, and the small target features are lost when the shallow information is downsampled and fused by the path using a bottom-up path, the present invention provides a small target detection method based on a progressive multi-scale feature fusion network.

[0007] The present invention provides a small target detection method based on a progressive multi-scale feature fusion network, the steps are:

[0008] S1. Adjust the size of the small target image to be detected to 640×640, and input the adjusted image into the backbone network part of the small target detection network;

[0009] S2. In the backbone of the small target detection network, feature extraction is performed on the input image to be detected to obtain feature maps of four scales. , , and , the extracted feature map is input into the neck network of the small target detection network, and its sub-steps are:

[0010] S21, the input image to be detected is subjected to two convolutions with a kernel size of 3×3 and a step size of 1 to extract features;

[0011] S22: Process the features obtained in step S21 through a spatial depth convolution module to obtain a feature map ;

[0012] S23. Feature map The feature map is obtained through a basic convolution module with a convolution kernel size of 3×3 and a step size of 1 and a spatial depth convolution module. ;

[0013] S24. Feature map Repeat step S23 once to obtain the feature map ;

[0014] S25. Feature map Features are extracted through a basic convolution module with a convolution kernel size of 3×3 and a step size of 1 and a spatial depth convolution module;

[0015] S26: Process the features obtained in step S25 through the global attention module to obtain features ;

[0016] S3. Feature maps of four scales output by the backbone network in the neck part of the small target detection network , , and Perform feature augmentation, the sub-steps are:

[0017] S31. Feature map After convolution, the kernel size is , the feature is obtained by dilated convolution with a dilation rate of 6 ; For feature map The convolution kernel size is , the feature is obtained by dilated convolution with a dilation rate of 3 ; For feature map The convolution kernel size is , the feature is obtained by dilated convolution with a dilation rate of 1 ;

[0018] S32, dilated convolution results for all sizes , and Aggregate to form an enhanced feature map , the calculation formula for the aggregation operation is:

[0019] (1),

[0020] In formula (1) Indicates that for The feature map is at scale The weight under is the i-th feature map;

[0021] S33, use residual connection to convert the original feature map With the enhanced feature map Add together to get the final enhanced feature map ;

[0022] S34. Feature map Repeat steps S31, S32 and S33 to obtain the final enhanced feature map ;

[0023] S4. Enhanced feature map obtained in S3 and Scale alignment is performed by upsampling or downsampling the features at different levels to adjust the scale of each feature map and obtain the adjusted features. The calculation formula is:

[0024] (2),

[0025] In formula (2), Represents the feature map of the current scale i, represents the feature map after scale adjustment, represents the upsampling operation, represents the downsampling operation, Indicates the current scale;

[0026] S5. After the scale adjustment in step S4, the attention mechanism is used to integrate the horizontal and vertical feature information to generate a comprehensive attention map for the key area to focus on. The calculation formula for generating the attention map is:

[0027] (3),

[0028] (4),

[0029] In formulas (3) and (4), and Represent horizontal and vertical convolutions respectively, Represents the Sigmoid function; It is a feature map after applying convolution operations in the horizontal and vertical directions to capture the contextual relationship of the target. is the generated attention map;

[0030] S6. In the feature fusion stage, residual connections are added to ensure that global information is not lost in the deep layers of the network, while also maintaining attention to details to obtain feature maps , where the calculation formula for feature fusion is:

[0031] (5),

[0032] In formula (5), represents element-wise multiplication, is the attention map corresponding to the feature map of the current scale i, Represents the result of feature fusion;

[0033] S7. Feature map of backbone network output Repeat steps S31, S32 and S33 to obtain the final enhanced feature map ;

[0034] S8. Enhanced feature map and feature map Repeat steps S4, S5 and S6 to obtain the fused feature map ;

[0035] S9. Feature map of backbone network output Repeat steps S31, S32 and S33 to obtain the enhanced feature map ;

[0036] S10, enhancing feature map and feature map Repeat steps S4, S5 and S6 to obtain the fused feature map ;

[0037] S11. Output feature map of backbone network And fusion feature map Input to the detection head to predict confidence, classification and bounding box.

[0038] Preferably, in step S1, the size of the small target images to be detected is uniformly adjusted to 640×640.

[0039] Compared with the prior art, the technical solution provided by the present invention has the following technical effects: first, compared with the first type of small target detection method, the method described in the present invention adopts a global attention mechanism, and by modeling the relationship between different positions of the feature map, the model can capture global dependencies, thereby enhancing the representation ability of small targets in complex backgrounds; second, compared with the second type of small target detection method, the method described in the present invention adopts an improved multi-scale feature fusion strategy, which fuses and enhances shallow small target features in the early stage, and then performs in-depth feature interaction. This strategy avoids the degradation or loss of small target features during the downsampling process, and effectively improves the feature expression ability of small targets. Test experiments show that the small target detection method based on the progressive multi-scale feature fusion network described in the present invention is superior to the three detection indicators of YOLOv5 that does not use this method for small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 It is a flowchart of a small target detection method based on a progressive multi-scale feature fusion network described in an embodiment of the present invention; Figure 2 A feature augmentation model diagram in a certain embodiment of the present invention; Figure 3 A context attention fusion model diagram in an embodiment of the present invention; Figure 4 This is a diagram showing the results of detection using the method in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0045] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0046] In one embodiment, Figure 1 As shown in the figure, the small target detection method based on the progressive multi-scale feature fusion network has the following steps:

[0047] S1. The size of the small target image to be detected is uniformly adjusted, and the adjusted image is input into the backbone network part of the small target detection network;

[0048] S2. In the backbone of the small target detection network, feature extraction is performed on the input image to be detected to obtain feature maps of four scales. , , and , the extracted feature map is input into the neck network of the small target detection network, and its sub-steps are:

[0049] S21, the input image to be detected is subjected to two convolutions with a kernel size of 3×3 and a step size of 1 to extract features;

[0050] S22: Process the features obtained in step S21 through a spatial depth convolution module to obtain a feature map ;

[0051] S23. Feature map The feature map is obtained through a basic convolution module with a convolution kernel size of 3×3 and a step size of 1 and a spatial depth convolution module. ;

[0052] S24. Feature map Repeat step S23 once to obtain the feature map ;

[0053] S25. Feature map Features are extracted through a basic convolution module with a convolution kernel size of 3×3 and a step size of 1 and a spatial depth convolution module;

[0054] S26: Process the features obtained in step S25 through the global attention module to obtain features ;

[0055] S3. Feature maps of four scales output by the backbone network in the neck part of the small target detection network , , and Perform feature augmentation, the sub-steps are:

[0056] S31. Feature map After convolution, the kernel size is , the feature is obtained by dilated convolution with a dilation rate of 6 ; For feature map The convolution kernel size is , the feature is obtained by dilated convolution with a dilation rate of 3 ; For feature map The convolution kernel size is , the feature is obtained by dilated convolution with a dilation rate of 1 ;

[0057] S32, dilated convolution results for all sizes , and Aggregate to form an enhanced feature map , the calculation formula for the aggregation operation is:

[0058] (1),

[0059] In formula (1) Indicates that for The feature map is at scale The weight under It is feature maps;

[0060] S33, use residual connection to convert the original feature map With the enhanced feature map Add together to get the final enhanced feature map ;

[0061] S34. Feature map Repeat steps S31, S32 and S33 to obtain the final enhanced feature map ;

[0062] S4. Enhanced feature map obtained in S3 and Scale alignment is performed by upsampling or downsampling the features at different levels to adjust the scale of each feature map and obtain the adjusted features. The calculation formula is:

[0063] (2),

[0064] In formula (2), Represents the feature map of the current scale i, represents the feature map after scale adjustment, represents the upsampling operation, represents the downsampling operation, Indicates the current scale;

[0065] S5. After the scale adjustment in step S4, the attention mechanism is used to integrate the horizontal and vertical feature information to generate a comprehensive attention map for the key area to focus on. The calculation formula for generating the attention map is:

[0066] (3),

[0067] (4),

[0068] In formulas (3) and (4), and Represent horizontal and vertical convolutions respectively, Represents the Sigmoid function; It represents the feature map after applying convolution operations in the horizontal and vertical directions, which is used to capture the contextual relationship of the target. is the generated attention map;

[0069] S6. In the feature fusion stage, residual connections are added to ensure that global information is not lost in the deep layers of the network, while also maintaining attention to details to obtain feature maps , where the calculation formula for feature fusion is:

[0070] (5),

[0071] In formula (5), represents element-wise multiplication, is the attention map corresponding to the feature map of the current scale i, Represents the result of feature fusion;

[0072] S7. Feature map of backbone network output Repeat steps S31, S32 and S33 to obtain the final enhanced feature map ;

[0073] S8. Enhanced feature map and feature map Repeat steps S4, S5 and S6 to obtain the fused feature map ;

[0074] S9. Feature map of backbone network output Repeat steps S31, S32 and S33 to obtain the enhanced feature map ;

[0075] S10, enhancing feature map and feature map Repeat steps S4, S5 and S6 to obtain the fused feature map ;

[0076] S11. Output feature map of backbone network And fusion feature map Input to the detection head to predict confidence, classification and bounding box.

[0077] The technical effect of the method of the present invention can be confirmed by the following simulation experiments.

[0078] All experimental results of this invention are obtained under the following operating environment: Ubuntu 20.04.1, Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz CPU, 90GB memory, RTX 3090(24GB)*1.

[0079] The simulation experiment in this specific embodiment is used to detect the performance of the method described in the present invention for small target detection. The data set used in the simulation experiment is the VisDrone2019 data set, which is provided by the AISKYEYE team of the Machine Learning and Data Mining Laboratory of Tianjin University. The data set contains a large amount of image data, which are taken using drones in different environments, times and places. The diversity of the data is very high, including different urban landscapes, rural landscapes, different weather conditions and a variety of scenes. The data set contains 8599 photos, including 6471 training sets, 548 validation sets, and 1580 test sets, with a ratio of approximately 25:2:6.

[0080] In order to verify the effectiveness of the experimental results, multiple sets of fusion evaluation indicators were used, including mean Average Precision (mAP), precision, and recall.

[0081] The average accuracy means the average accuracy of each category detection. The optimal value of the average accuracy is 1. The closer the value is to 1, the better the detection performance of the model. The average accuracy is obtained by the following formula:

[0082] ,

[0083] Where K is the number of categories, AP i is the average precision of the ith category. i Obtained by the following formula:

[0084] ,

[0085] mAP@0.5 is the average mAP when the IoU threshold is greater than 0.5. mAP@0.5:0.95 is the average mAP at different IoU thresholds (from 0.5 to 0.95, with a step size of 0.05). The mAP of multiple IoU thresholds provides a more comprehensive performance evaluation, ensuring that the model can achieve good performance under different accuracy requirements. IoU (Intersection over Union) measures the degree of overlap of two regions by calculating the ratio of their intersection area to their union area, and is used to express the accuracy of the detection box. The formula for calculating IoU is as follows:

[0086] .

[0087] The precision rate represents the ratio of positive samples in the recognized images. The precision rate is obtained by the following formula:

[0088] ,

[0089] Among them, TP (True Positives) is a positive sample that is correctly identified as a positive sample, and FP (False Positives) is a positive sample that is correctly identified as a positive sample. The higher the precision value, the better the detection accuracy.

[0090] The recall rate indicates the ratio of samples correctly identified as positive by the model to the total number of samples in the actual positive class. The recall rate is obtained by the following formula:

[0091] ,

[0092] Among them, FN (False Negatives) is the number of false negative classes, that is, the number of positive samples mistakenly identified as negative classes.

[0093] Therefore, the criterion for evaluating the performance of small target detection is: for the same set of comparative experiments, if the average accuracy, precision and recall rate of the image obtained by a certain small target detection method is larger, it means that the detection result obtained by the small target detection method is relatively good.

[0094] Table 1

[0095] method Precision Recall mAP@0.5 mAP@0.5:0.95 YOLOv5 44% 33.7% 32.5% 17.6% The present invention 54% 43.5% 44.3% 25.5%

[0096] Compared with YOLOv5, the small target detection results of the present invention have improved the average accuracy, precision and recall of the detection. The improvement of mAP indicates that the small target detection effect of this method in different IoU threshold ranges has been improved. At the same time, higher precision and recall rates indicate that this method is more suitable for the detection of various small targets, and can detect targets of different sizes, shapes and positions more accurately. Table 1 shows the evaluation indicators of the results of several fusion methods, and the bold numbers are the relative optimal values.

[0097] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. A small target detection method based on a progressive multi-scale feature fusion network, characterized in that: The steps are: S1. The size of the small target image to be detected is uniformly adjusted, and the adjusted image is input into the backbone network part of the small target detection network; S2. In the backbone of the small target detection network, feature extraction is performed on the input image to be detected to obtain feature maps of four scales. , , and , the extracted feature map is input into the neck network of the small target detection network, and its sub-steps are: S21, the input image to be detected is subjected to two convolutions with a kernel size of 3×3 and a step size of 1 to extract features; S22: Process the features obtained in step S21 through a spatial depth convolution module to obtain a feature map ; S23. Feature map The feature map is obtained through a basic convolution module with a convolution kernel size of 3×3 and a step size of 1 and a spatial depth convolution module. ; S24. Feature map Repeat step S23 once to obtain the feature map ; S25. Feature map Features are extracted through a basic convolution module with a convolution kernel size of 3×3 and a step size of 1 and a spatial depth convolution module; S26: The features obtained in step S25 are obtained through the global attention module ; S3. Feature maps of four scales output by the backbone network in the neck part of the small target detection network , , and Perform feature augmentation, the sub-steps are: S31. Feature map After convolution kernel size is , the feature is obtained by dilated convolution with a dilation rate of 6 ; For feature map The convolution kernel size is , the feature is obtained by dilated convolution with a dilation rate of 3 ; For feature map The convolution kernel size is , the feature is obtained by dilated convolution with a dilation rate of 1 ; S32, dilated convolution results for all sizes , and Aggregate to form an enhanced feature map , the calculation formula for the aggregation operation is: （1）， In formula (1) Indicates that for The feature map is at scale The weight under represents the jth feature map; S33, use residual connection to convert the original feature map With the enhanced feature map Add together to get the final enhanced feature map ; S34. Feature map Repeat steps S31, S32 and S33 to obtain the final enhanced feature map ; S4. Enhanced feature map obtained in S3 and Scale alignment is performed by upsampling or downsampling the features at different levels to adjust the scale of each feature map and obtain the adjusted features. The calculation formula is: （2）， In formula (2), Represents the current scale The feature map of represents the feature map after scale adjustment, represents the upsampling operation, represents the downsampling operation, Indicates the current scale; S5. After the scale adjustment in step S4, the attention mechanism is used to integrate the horizontal and vertical feature information to generate a comprehensive attention map for the key area to focus on. The calculation formula for generating the attention map is: （3）， （4）， In formulas (3) and (4), and Represent horizontal and vertical convolutions, respectively. Represents the Sigmoid function; Represents the feature map after applying convolution operations in the horizontal and vertical directions to capture the contextual relationship of the target. represents the generated attention map; S6. In the feature fusion stage, residual connections are added to ensure that global information is not lost in the deep layers of the network, while also maintaining attention to details to obtain feature maps , where the calculation formula for feature fusion is: （5）， In formula (5), represents element-wise multiplication, is the attention map corresponding to the feature map of the current scale i, Represents the result of feature fusion; S7. Feature map of backbone network output Repeat steps S31, S32 and S33 to obtain the final enhanced feature map ; S8. Enhanced feature map and feature map Repeat steps S4, S5 and S6 to obtain the fused feature map ; S9. Feature map of backbone network output Repeat steps S31, S32 and S33 to obtain the enhanced feature map ; S10, enhancing feature map and feature map Repeat steps S4, S5 and S6 to obtain the fused feature map ; S11. Output feature map of backbone network And fusion feature map Input to the detection head to predict confidence, classification and bounding box.

2. The small target detection method based on the progressive multi-scale feature fusion network according to claim 1 is characterized in that: In step S1, the size of the small target images to be detected is uniformly adjusted to 640×640.

Images