A robust small target detection method under resource-constrained conditions

Abstract

This invention provides a robust small target detection method under resource-constrained conditions, belonging to the field of computer vision technology. The method includes: simulating feature maps F4 and F5 using dilated convolution with an attention mechanism, and fusing features with learnable weights to obtain feature map #imgabs0#; constructing a fine-grained multi-scale target detection framework, including adding a small target detection head corresponding to feature map F2 and removing the large detection head corresponding to feature map F5; processing feature map F5 using deconvolution and dilated convolution to obtain feature map F4 as a residual, and fusing features with learnable weights to obtain feature map #imgabs1#. This invention effectively solves the problems of limited small target sample quantity, poor model robustness, limited available features easily overwhelmed by environmental noise, and difficulty in optimizing the model for small targets under resource-constrained conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a robust small target detection method under resource-constrained conditions. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, with the rapid development of the Internet of Things, intelligent transportation, intelligent security and other fields, more and more application scenarios require accurate detection and identification of small targets, especially the demand for resource-constrained small target detection is becoming more and more urgent. UAV small target detection has broad application prospects in resource-constrained fields such as UAV monitoring, weather forecasting, and military reconnaissance. It can not only replace manpower to complete the work that is difficult to complete with manual monitoring, but also reduce manpower and material resources and improve work efficiency. At present, there have been a lot of research works at home and abroad, among which deep learning methods have achieved excellent results. The methods based on deep learning are roughly divided into data enhancement, multi-scale feature fusion, context information learning, super-resolution, loss function, attention mechanism and other methods, and combined with the current mainstream detectors to achieve good results, promoting the rapid development of small target detection under resource-constrained conditions.

[0004] Although the rapid development of deep learning in recent years has enabled target detection algorithms to achieve good results on large and medium-sized targets, the performance of small target detection is still not ideal under resource-constrained conditions due to the small number of samples and poor model robustness, few available features and easy to be drowned by environmental noise, and difficulty in optimizing small targets. Previous studies have found that small target detection usually has key and difficult problems such as insufficient feature information, small target aggregation, scattered and complex target scenes, and imbalanced positive and negative samples. Using existing methods to detect small targets under such resource-constrained conditions, it is difficult to obtain sufficient supporting features and positive samples to help the model optimize small targets, resulting in the accuracy of small target detection being much lower than that of medium and large targets.

[0005] Based on the above analysis, although the feature pyramid method has achieved good results, by fusing shallow and deep feature maps, the feature map can have both deep semantic information and shallow detail information, which has a good effect on small target detection. However, this method will lose a lot of small target detail information due to the use of convolution and pooling when obtaining the semantic information of the deep feature map, making the fine-grained information after feature fusion less relevant to the semantic information, making it difficult to detect small targets. Moreover, most of the existing detections are based on anchor frames, and the manual design of anchor frames is difficult to be accurate, which makes it difficult for the detector to focus on the optimization of small targets. Therefore, how to design a more effective method to deal with resource-constrained small target detection is becoming more and more important and gradually becoming one of the most cutting-edge research directions in the field of robust small target detection under resource-constrained conditions. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a robust small target detection method under resource-constrained conditions, which effectively solves the problems of small target samples being small in number and poor model robustness, few available features being easily submerged by environmental noise, and difficulty in optimizing the model for small targets under resource-constrained conditions.

[0007] In a first aspect, the present invention provides a robust small target detection method under resource-constrained conditions, wherein feature maps F2, F3, F4, and F5 are obtained by performing 1, 2, 3, and 4 downsampling operations on a feature pyramid in a feature extraction stage, respectively, for a target detection model, the method comprising:

[0008] The feature map F2 is simulated by the hole convolution to simulate the feature maps F4 and F5, learn the semantic information related to the fine-grained information, and use the attention mechanism to process it, and obtain the feature map by fusion of features through learnable weights ;

[0009] Construct a fine-grained target multi-scale detection framework, including adding a small target detection head corresponding to feature map F2 and removing the large detection head corresponding to feature map F5 to achieve scale-level alignment of anchor boxes and feature maps;

[0010] The feature map F5 is processed by deconvolution and dilated convolution to obtain the feature map F4 in a residual manner. The features are fused by learnable weights to obtain the feature map .

[0011] Furthermore, it also includes: data set preprocessing, including:

[0012] Perform unified formatting on the data in the dataset;

[0013] The dataset is enhanced by copying, randomly transforming, and pasting small objects for images that are mainly small objects; randomly cropping and scaling multiple images and stitching them into one image for images that are mainly large objects; and adding noise, changing brightness, imitating lighting changes, and blurring the images.

[0014] Furthermore, the feature map F2 is simulated by the dilated convolution and processed by the attention mechanism, and the feature map F4 and F5 are obtained by fusing the features with the learnable weights. ,include:

[0015] The feature map F2 that has not been processed by the feature extraction module is input with dilated convolutions with dilated rates of 1, 2, and 3, respectively, to obtain three feature maps F with different receptive fields. 21 、F 22 、F 23 , used to retain the existing feature map information, simulate the P4 level feature map features, and simulate the P5 feature map features. The formula is: , DilatedConv is a dilated convolution;

[0016] The three feature maps F 21 、F 22 、F 23 Use a learnable feature fusion method to fuse and combine it with the attention mechanism to obtain features , the formula is: , EMA refers to the attention mechanism, and Fusion refers to the learnable weight fusion;

[0017] And the feature map after feature extraction And the feature map F3 is fused to get , the formula is: , Cat refers to learnable weight fusion.

[0018] Furthermore, the feature map F5 is processed by deconvolution and dilated convolution to obtain the feature map F4 in a residual manner, and the feature is fused by learnable weights to obtain the feature map ,include:

[0019] The feature map F5 is upsampled using transposed convolution , the formula is: , TransposeConv refers to transposed convolution;

[0020] The feature map F5 is fused with dilated convolutions with dilation rates of 1, 3, and 5 using learnable weights to obtain a feature map with a larger receptive field. , the formula is: , Fusion is learnable weight fusion; , DilatedConv is a dilated convolution;

[0021] Combined with the residual structure, feature map F4 is obtained by fusion with feature map F3 through a learnable fusion method. , the formula is: , Cat is the learnable weight fusion.

[0022] Furthermore, it also includes: inputting the data set,into the target detection model for training;

[0023] Combined with contrast learning, the predicted box is brought closer to the real box, and the generated predicted box is optimized;

[0024] The classification and regression losses are calculated by combining the true box and the predicted box. Finally, after multiple iterations, the model parameters are continuously optimized until the loss no longer decreases and fluctuates or the number of training times is reached; the model weight at the minimum loss value is saved.

[0025] Furthermore, it also includes:

[0026] After the target detection model is trained, the optimal model weights are loaded, and the image or video data to be tested is input for prediction to obtain a picture or video file with a prediction box, category, and confidence level, or the camera is directly connected for real-time detection.

[0027] The beneficial effects of the present invention are as follows: the robust small target detection method under resource-constrained conditions proposed by the present invention can achieve scale-level matching of anchor frames and feature maps on all existing multi-scale prediction models with feature pyramid structures, add rich semantic features related to fine-grained information to the P2-level feature map and greatly reduce the loss of small target detail information, and can also enable the detection model to pay more attention to the optimization of small targets without losing coarse-grained semantic features. The accuracy of small target detection can be greatly improved under the premise of increasing the amount of calculation very little, and combined with the data enhancement method, the model has good robustness and generalization performance. The present invention has good versatility and can be simply applied to all models with multi-scale detection. It has achieved good results on multiple remote sensing and drone aerial photography data sets, and has promoted the further development of small target detection under resource-constrained conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0030] Figure 2 A schematic diagram of an improved feature extraction module provided in an embodiment of the present invention.

[0031] Figure 3 A schematic diagram of the image classification system model framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] Most small target detection methods are improved on existing mainstream models. Currently, the multi-scale target detection model based on feature pyramid network has achieved good results by fusing shallow and deep feature maps.

[0034] Therefore, the target detection model based on the present invention selects a suitable mainstream target detector model as the basic detector in the feature extraction stage. If high precision is required, a two-stage method such as FasterRCNN is selected. If high speed is required, a single-stage method such as the YOLO series is selected. Feature extraction is performed on the image to obtain a shallow feature map with rich detail information and a deep feature map with rich semantic features.

[0035] Specifically, most of the small target detection methods are improved on the basis of existing general target detection. Common methods are divided into single-stage and two-stage methods. Both methods have their own advantages and can be selected according to needs. The single-stage method first performs downsampling feature extraction through convolution or pooling operations to obtain feature maps with different receptive fields. Secondly, the feature maps with different receptive fields are fused in combination with the feature pyramid structure, and finally the fused feature maps are used for multi-scale detection. The two-stage method first uses a region proposal network to generate a large number of candidate boxes, and performs convolution operations on the candidate boxes to extract information such as color, texture, and shape. Secondly, the extracted features are fused with the position information of the candidate boxes, and the candidate boxes are classified. Finally, the classified candidate boxes are regressed to determine the exact position of the target. The method proposed in the present invention can be simply used in the feature pyramid structure and feature fusion stage.

[0036] It should be noted that the P2, P3, P4, and P5 level feature maps F2, F3, F4, and F5 mentioned in the present invention respectively refer to feature maps obtained by performing 1, 2, 3, and 4 downsampling operations in the feature extraction stage of the target detection model.

[0037] The purpose of the present invention is to retain more detailed information of P2 and P4 level feature maps, increase the model's attention to small target optimization, and achieve better classification results in the field of small target detection under resource-constrained conditions. Based on this purpose, Figure 1 As shown, an embodiment of the present invention provides a robust small target detection method under resource-constrained conditions. The method improves the feature maps F2, F3, F4, and F5 obtained by performing 1, 2, 3, and 4 downsampling operations on the feature pyramid in the feature extraction stage of the target detection model, and specifically includes:

[0038] Step 110: For feature map F2, simulate feature maps F4 and F5 through dilated convolution, learn semantic information related to fine-grained information, and use attention mechanism to process and obtain feature map by fusion feature through learnable weights ;

[0039] Step 120: construct a fine-grained target multi-scale detection framework, including adding a small target detection head corresponding to the feature map F2 and removing a large detection head corresponding to the feature map F5 to achieve scale-level alignment between the anchor frame and the feature map;

[0040] Step 130: Deconvolution and dilated convolution are performed on the feature map F5 to obtain the feature map F4 in a residual manner. The features are fused through learnable weights to obtain the feature map .

[0041] The existing target detection model will lose a lot of small target information when acquiring large receptive semantic information. Therefore, in order to solve the problem that small targets have few available features and are easily submerged by environmental noise, the embodiment of the present invention proposes a semantic imitation and fine-grained information supplementation module (SFSM for short). The module is implemented based on dilated convolution, attention mechanism and learnable weight fusion. By simulating P4 and P5 level feature maps and fusing with P2 and P3 level feature maps, it can obtain large receptive field semantic information related to fine-grained information while rarely losing small target detail information, retaining more small target detail information and enhancing the correlation between fine-grained information and semantic information. And further combined with the attention mechanism, the model pays more attention to the foreground target under the premise of having more detailed information, improving the problem that small targets are easily submerged by environmental noise.

[0042] Most existing models are designed based on anchor frames, but it is difficult to set a fixed value for such manually designed components so that the model can adapt to the detection of small targets and regular targets at the same time. Therefore, the present invention further proposes a fine-grained target multi-scale detection framework to address the problem that the model is difficult to optimize for small targets. This method introduces a tiny detection head and removes the large detection head to achieve scale-level alignment between the anchor frame and the feature map, so that the model can better focus on the optimization of small targets.

[0043] In order to reduce the impact of removing the large detection head in the previous operation, the present invention finally proposes a coarse-grained feature enhancement module (CFEM), which is implemented based on transposed convolution, dilated convolution and learnable weight fusion. It can obtain semantic information with a larger receptive field, enrich the input features of the medium detection head, and alleviate the impact of removing the large detection head on large target detection.

[0044] The method proposed in this invention can be easily integrated into all models with feature pyramid structures. The improved feature extraction module is as follows: Figure 2 As shown in the figure, it has been tested on multiple data sets and can stably improve the detection effect of small targets, with good robustness. It can significantly improve the detection accuracy of small targets under resource-constrained conditions with a small increase in computational complexity. Combining it with the existing single-stage method can improve the detection effect while ensuring its detection speed, laying a solid foundation for promoting the development of small target detection under resource-constrained conditions and its industrialization.

[0045] Optionally, as an embodiment of the present invention, it further includes: data set preprocessing, including:

[0046] Perform unified formatting on the data in the dataset;

[0047] The dataset is enhanced by copying, randomly transforming, and pasting small objects for images that are mainly small objects; randomly cropping and scaling multiple images and stitching them into one image for images that are mainly large objects; and adding noise, changing brightness, imitating lighting changes, and blurring the images.

[0048] The data enhancement method provided by the embodiment of the present invention increases the number of small targets in the training samples on the one hand, and increases the number of small targets in the positive samples on the other hand. In addition, by adding noise to the data set to change the brightness to simulate the problem of illumination change and shooting blur, the model has better robustness and generalization, solving the problem of the complex and changeable environment of small targets, the small number of small target samples and the poor robustness of the model.

[0049] Optionally, as an embodiment of the present invention, for the feature map F2, the feature maps F4 and F5 are simulated by dilated convolution to learn semantic information related to fine-grained information, and the attention mechanism is used to obtain the feature map by fusion of features with learnable weights. ,include:

[0050] The feature map F2 that has not been processed by the feature extraction module is input with dilated convolutions with dilated rates of 1, 2, and 3, respectively, to obtain three feature maps F with different receptive fields. 21 、F 22 、F 23 , used to retain the existing feature map information, simulate the P4 level feature map features, and simulate the P5 feature map features. The formula is: , DilatedConv is a dilated convolution;

[0051] The three feature maps F 21 、F 22 、F 23 Use a learnable feature fusion method to fuse and combine it with the attention mechanism to obtain features Since the small targets are sparsely distributed on the feature map F2, the feature , enhancing the focus on the foreground target in the feature map, which is more helpful for target detection. The formula is: , EMA refers to the attention mechanism, and Fusion refers to the learnable weight fusion;

[0052] And the feature map after feature extraction And the feature map F3 is fused to get , the formula is: , Cat refers to learnable weight fusion. The three feature maps are fused using a learnable feature fusion method, and the contributions of the three feature maps can be continuously adjusted during the training process.

[0053] The method provided in this embodiment can be used as a semantic imitation and fine-grained information supplement module, incorporated into the target detection model, to solve the problem that in the feature extraction stage of the existing model, due to multiple downsampling operations, small target information is lost and small targets have few available features and are easily submerged by environmental noise. Through this embodiment, a P2-level feature map containing both a large amount of detailed information and rich semantic information related to fine-grained information is obtained.

[0054] In order to be more suitable for small target detection, existing methods often add tiny detection heads. However, if four detection heads are used, the features learned by the detection heads will be similar, resulting in overfitting and difficulty in optimization. Therefore, the present invention proposes a fine-grained target multi-scale detection framework, which achieves scale-level alignment of anchor frames and feature maps by adding tiny target detection heads and removing large detection heads that have little effect on small target detection.

[0055] It should be noted that the void rate here is calculated. The receptive field of the F2 feature map is 5×5, and the receptive fields of F4 and F5 are 9×9 and 11×11 respectively. ,in represents the target receptive field, represents the current receptive field, S i represents the step length, k Indicates the size of the convolution kernel. When the convolution kernel size is 3×3, the step size d It is 1. It can be calculated that the hole rates used to obtain 9×9 and 11×11 receptive fields using a 5×5 receptive field are 2 and 3 respectively.

[0056] Optionally, as an embodiment of the present invention, the feature map F5 is processed by deconvolution and dilated convolution to obtain the feature map F4 in a residual manner, and the feature is fused by learnable weights to obtain the feature map ,include:

[0057] The feature map F5 is upsampled using transposed convolution , the formula is: , TransposeConv refers to transposed convolution;

[0058] The feature map F5 is fused with dilated convolutions with dilation rates of 1, 3, and 5 using learnable weights to obtain a feature map with a larger receptive field. , the formula is: , Fusion is learnable weight fusion; , DilatedConv is a dilated convolution;

[0059] Combined with the residual structure, feature map F4 is obtained, which is fused with feature map F3 through a learnable fusion method. , the formula is: , Cat is the learnable weight fusion.

[0060] The method provided in this embodiment can be incorporated into the target detection model as a coarse-grained feature enhancement module. The above steps enrich the input feature map of the medium detection head, further make full use of the limited features, make full use of the large receptive field semantic information and reduce the impact of removing the large detection head, thereby improving the large target detection performance and small target classification accuracy under limited resource conditions.

[0061] Optionally, as an embodiment of the present invention, the method further includes: inputting the data set into a target detection model for training;

[0062] Combined with contrast learning, the predicted box is brought closer to the real box, and the generated predicted box is optimized;

[0063] The classification and regression losses are calculated by combining the true box and the predicted box. Finally, after multiple iterations, the model parameters are continuously optimized until the loss no longer decreases and fluctuates or the number of training times is reached; the model weight at the minimum loss value is saved.

[0064] Optionally, as an embodiment of the present invention, it also includes:

[0065] After the target detection model is trained, the optimal model weights are loaded, and the image or video data to be tested is input for prediction to obtain a picture or video file with a prediction box, category, and confidence level, or the camera is directly connected for real-time detection.

[0066] An embodiment of the present invention also provides an image classification system constructed based on the target detection model of the aforementioned embodiment. Figure 3 The system in the dotted box corresponding to the figure is the system module that mainly performs the classification function. The network model has saved parameters after training. The user only needs to input the image data to be tested into the network, and the model will automatically load the model parameters to perform feature extraction, shallow semantic feature supplementation, deep feature fusion and multi-scale detection on the input image or video, and finally output the prediction results to interact with the user.

[0067] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention or any person skilled in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and shall be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A robust small target detection method under resource-constrained conditions, characterized in that: For the target detection model, in the feature extraction stage, feature graphs F2, F3, F4, and F5 are obtained by performing 1, 2, 3, and 4 downsampling operations on the feature pyramid, respectively. The method includes: The feature map F2 is simulated by the dilated convolution and processed by the attention mechanism, and the feature map is obtained by fusing the features with the learnable weights. Construct a fine-grained target multi-scale detection framework, including adding the small target detection head corresponding to the feature map F2 and removing the large detection head corresponding to the feature map F5; The feature map F5 is processed by deconvolution and dilated convolution to obtain the feature map F4 in a residual manner. The features are fused by learnable weights to obtain the feature map The feature map F2 is simulated by the hole convolution and the feature maps F4 and F5 are processed by the attention mechanism, and the feature map is obtained by the learnable weight fusion feature. include: The feature map F2 that has not been processed by the feature extraction module is input with dilated convolutions with dilated rates of 1, 2, and 3, respectively, to obtain three feature maps F with different receptive fields. 21 、F 22 、F 23 , used to retain the existing feature map information, simulate the P4 level feature map features, and simulate the P5 feature map features. The formula is: F 21 , F 22 , F 23 =DilatedConv(F2), DilatedConv is a dilated convolution; The three feature maps F 21 、F 22 、F 23 Use a learnable feature fusion method to fuse and combine it with the attention mechanism to obtain features The formula is: EMA refers to the attention mechanism, and Fusion refers to the learnable weight fusion; It is fused with the feature map F′2 and feature map F3 after feature extraction to obtain The formula is: Cat refers to learnable weight fusion; The feature map F5 is processed by deconvolution and dilated convolution to obtain the feature map F4 in a residual manner, and the features are fused by learnable weights to obtain the feature map include: The feature map F5 is upsampled using transposed convolution The formula is: TransposeConv refers to transposed convolution; The feature map F5 is fused with dilated convolutions with dilation rates of 1, 3, and 5 using learnable weights to obtain a feature map with a larger receptive field. The formula is: Fusion is a learnable weight fusion; F 51 , F 53 , F 55 =DilatedConv(F5), DilatedConv is a dilated convolution; Combined with the residual structure, feature map F4 is obtained, which is fused with feature map F3 through a learnable fusion method. The formula is: Cat is a learnable weight fusion.

2. The method according to claim 1, characterized in that Also includes: Dataset preprocessing, including: Perform unified formatting on the data in the dataset; The dataset is enhanced by copying, randomly transforming, and pasting small objects for images that are mainly small objects; randomly cropping and scaling multiple images and stitching them into one image for images that are mainly large objects; and adding noise, changing brightness, imitating lighting changes, and blurring the images.

3. The method according to claim 1, characterized in that Also includes: Input the data set into the target detection model for training; Combined with contrast learning, the predicted box is brought closer to the real box, and the generated predicted box is optimized; Combine the real box and the predicted box to calculate the classification and regression loss. Finally, after multiple iterations, continuously optimize the model parameters until the loss no longer decreases and fluctuates or reaches the training number. Save the model weights at the minimum loss value.

4. The method according to claim 3, characterized in that Also includes: After the target detection model is trained, the optimal model weights are loaded, and the image or video data to be tested is input for prediction to obtain a picture or video file with a prediction box, category, and confidence level, or the camera is directly connected for real-time detection.

Images