A method for detecting small targets of UAV based on multi-scale feature fusion
By adopting the multi-scale feature fusion method with pixel shuffling upsampling technology in the RetinaNet detection framework, the problems of feature offset distortion and insufficient semantic information transmission in the detection of small targets of drones are solved, and better detection results are achieved.
Patent Information
- Application Number
- CN202210155269.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-18
- Publication Date
- 2025-02-18
- Estimated Expiration
- 2042-02-18
AI Technical Summary
The prior art performs poorly in UAV small target detection, especially due to the problem of feature offset distortion and insufficient deep semantic information transmission in the original FPN method.
The multi-scale feature fusion method based on the RetinaNet detection framework is adopted, and the channel dimension information of deep features is converted into spatial information through pixel shuffling upsampling technology to realize the effective transmission of deep semantic information to the shallow layer.
It improves the detection performance of small targets of drone, enhances the feature representation ability of shallow small targets, and improves the detection effect of small targets of drone.
Smart Images
Figure CN114581799B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicle small target detection, and in particular relates to a unmanned aerial vehicle small target detection method based on multi-scale feature fusion. Background Art
[0002] In recent years, as the manufacturing cost and use difficulty of drones continue to decrease, drones have been rapidly applied to all walks of life, and ordinary people can easily control drones, which has brought great convenience to people's production and life. However, at the same time, the "illegal flying" of drones has also brought huge challenges to the supervision of air traffic, public safety and other fields. Therefore, as an important part of drone supervision, the detection of drone targets has become a key issue that needs to be solved urgently.
[0003] There are many solutions for drone detection, based on radar, sound, radio, optoelectronic equipment, etc. These technologies that use the physical properties of drones to locate drones are very common, but these detection solutions often require very expensive equipment and strict configuration. Vision-based methods are lower in cost, relatively simple to configure, and easy to deploy.
[0004] Now, with the development of deep learning, it has made a lot of breakthroughs in the field of computer vision, and many excellent works have emerged. Early works such as Fast R-CNN, Faster R-CNN, YOLO, etc. have achieved good results on various general data sets. Later, He Kaiming et al. proposed the Feature Pyramid Network (FPN), which consists of two bottom-up and top-down pathways and a horizontal connection. First, the bottom-up pathway extracts features, then the horizontal connection aligns the number of channels, and then the top-down feature fusion is performed, and finally the feature map with multi-scale features is obtained for prediction. This method realizes multi-scale feature fusion by propagating semantic information from deep layers to shallow layers, so that the rich semantic information of deep layers can be transmitted to shallow layers, solving the problem of missing shallow semantic information and improving the network's detection performance for targets of different scales, especially small targets. Therefore, FPN is widely used in various general target detection methods, such as RetinaNet, FCOS and other works.
[0005] However, drone targets are different from general-scale targets. They are extremely small, usually accounting for less than 0.1% of the pixels in the entire image. In addition, the hollow appearance of small drones makes them blend into the background, and their features are not obvious, making detection very difficult. The FPN method performs poorly in drone target detection tasks mainly for the following reasons:
[0006] 1) The nearest neighbor interpolation method used in the original FPN upsampling will cause feature offset distortion, which will have a great impact on the detection of small targets in drones;
[0007] 2) Small targets are usually detected in the shallow layer of the network, but the lack of semantic information of the shallow features of the network leads to poor performance of the network in detecting small targets. Therefore, the original FPN method transfers the semantic information of the deep layer to the shallow layer to alleviate this problem. However, in the actual operation of FPN, the channel dimension alignment method still produces a lot of channel information attenuation, resulting in insufficient transmission of deep semantic information to the shallow layer. For extremely small targets such as drones that need more deep semantic information supplementation, insufficient semantic information will have a greater impact. Summary of the invention
[0008] The present invention provides a method for detecting small targets in unmanned aerial vehicles based on multi-scale feature fusion, which solves the problems existing in the unmanned aerial vehicle target detection task in the prior art and provides a new multi-scale feature fusion method. Based on the RetinaNet detection framework, the original FPN method used by RetinaNet can be replaced. The RetinaNet of the present invention is used in the unmanned aerial vehicle small target detection task to obtain better detection effect.
[0009] To achieve the above objectives, the present invention adopts the following technical solutions:
[0010] A method for detecting small targets of unmanned aerial vehicles based on multi-scale feature fusion includes the following steps:
[0011] Step 1: UAV dataset creation: collect UAV images, clean the collected images, and then mark the small UAV targets in them;
[0012] Step 2: Build a drone detection model: Based on the drone dataset, train a drone detection model;
[0013] Step 3: Test image detection process: Use the trained target detection network and network weight parameters to detect the drone in the test image and output the detection results.
[0014] In the above steps, in step 1, the collected images are cleaned to delete blurry images and images that do not contain the target;
[0015] Step 2 specifically includes the following steps:
[0016] Step (2.1) obtains the feature layer from the backbone network:
[0017] In the {C2, C3, C4, C5} feature layers of the backbone feature extraction network ResNet50, that is, the features of the 2nd, 3rd, 4th and 5th layers of ResNet50, {C3, C4, C5} are selected as several fusion feature layers for multi-scale feature fusion, and the number of channels corresponding to the feature layers is {512, 1024, 2048};
[0018] Step (2.2) pre-fusion processing:
[0019] Feature fusion requires the alignment of feature dimensions of different layers, which include both channel dimensions and spatial dimensions, i.e., width and height. Therefore, the method of this step is to upsample the deep features and align the deep feature dimensions with the shallow feature dimensions, so that the deep features can be fused with the shallow features. Specifically, the present invention uses a pixel shuffling upsampling method, which has the characteristics of high efficiency, fast speed, and no parameters. More importantly, it can make full use of a large amount of channel information in the deepest layer of the backbone network, and use it as the pixel information required for pixel shuffling to perform pixel shuffling upsampling. In short, the channel dimension information can be converted into spatial information and retained, which not only avoids the defect of simple nearest neighbor difference feature distortion, but also allows the rich semantic information of the deep layer to be transmitted to the shallow layer to the greatest extent, so that the shallow layer can obtain richer deep semantic information, improve the shallow layer UAV small target feature representation, and thus improve the network's detection performance for UAV small targets.
[0020] In order to transfer the deep features to the shallow layers, the present invention adopts the pixel shuffling upsampling method. First, the input features are amplified by 2 times through convolution with a convolution kernel of 1, and then upsampled by 2 times through pixel shuffling. At this time, the resolution is expanded by 2 times, and the channel dimension is reduced to 0.5 times of the original. Finally, the upsampling result is obtained, which is recorded as {P3, P4}, and the number of channels is {512, 1024} respectively;
[0021] The input features in step (2.2) are the two features C4 and C5 in the output features {C3, C4, C5} of step (2.1), and the corresponding number of channels is {1024, 2048}.
[0022] The upsampling steps are summarized as follows:
[0023] y′=PS(x′)
[0024] PS stands for PixelShuffle, which is the abbreviation of pixel shuffle. The default expansion factor (Scale) is 2, which means that the pixel shuffle method is used for 2 times upsampling, where the input is x′∈R w×h×4c Its number of channels is 4c, and the output is y′∈R 2w×2h×cThe number of its channels is c. In this process, the number of channels is reduced from 4c to c, and the image size is expanded from w×h to 2w×2h.
[0025] Step (2.3) fusion operation:
[0026] Add the output features {P3, P4} of step (2.2) and the features C3 and C4 in the output features {C3, C4, C5} of step (2.1) to obtain the features {T3, T4}, whose corresponding channel numbers are {512, 1024}. At the same time, directly obtain C5 in the output features {C3, C4, C5} of step (2.1) as T5. Finally, the output features of this step are {T3, T4, T5}, and the corresponding channel numbers are {512, 1024, 2048} respectively.
[0027] Step (2.4) post-fusion processing:
[0028] The input features of this step are {T3, T4, T5}, and the corresponding channel numbers are {512, 1024, 2048}. This step first uses convolution with a convolution kernel of 3 to remove aliasing on the three input features, and then uses convolution with a convolution kernel of 1 to perform channel compression, and uniformly compresses the number of channels of the three features to 256. Finally, the output features of this step are {F3, F4, F5}, and the corresponding channel numbers are {256, 256, 256}.
[0029] The formula is written as:
[0030] y = Conv-1×1(Conv-3×3(x))
[0031] The input is denoted by x∈R h×w×c , the output feature map is recorded as y∈R h×w×256 , the size of convolution Conv-1×1 is 1×1×256, and the size of convolution Conv-3×3 is 3×3×c;
[0032] Step (2.5) Target detection:
[0033] For {F3, F4, F5} that integrates multi-scale features in step (2.4), target category classification and target bounding box regression are performed respectively to obtain the final detection result.
[0034] Furthermore, in step (2.5):
[0035] The target category classification uses the Focal Loss loss with a balancing factor:
[0036] FL(p t )=-α t (1-p t ) γ log(pt )
[0037] Where FL stands for Focal Loss, α t is the weighted balance parameter of positive and negative samples. The larger the value, the greater the weight of the positive sample. γ is the focusing coefficient, which controls the update rate of the sample weight and is a hyperparameter. t Indicates the probability that the sample belongs to a positive sample, (1-p t ) γ is the weight expression;
[0038] The bounding box regression is the standard Smooth L1 loss;
[0039] Step (2.6) train the detection model:
[0040] The pre-trained model used is a model pre-trained on Imagenet provided by Pytorch. All images were resized to 1333×800 before being sent to the model. The number of training iterations was set to 12, the momentum was 0.9, the batch size was 4, the initial learning rate was 0.0005, and the weight decayed to 0.3 times the original weight every 4 epochs. The network model parameters were obtained based on the above training configuration;
[0041] Step 3 specifically includes the following steps:
[0042] Step (3.1) sends the test image into the network model;
[0043] Step (3.2) extracts features through the Resent50 backbone network to obtain a feature map;
[0044] Step (3.3) uses the improved multi-scale feature fusion network in step 2 to perform multi-scale feature fusion;
[0045] Step (3.4) performs target category prediction and bounding box regression on the fused feature map;
[0046] Step (3.5) sets the threshold and uses non-maximum suppression to filter out the final detection results.
[0047] Beneficial effect: The present invention provides a method for detecting small targets of UAVs based on multi-scale feature fusion. The method is based on RetinaNet and is targetedly improved according to the characteristics of small targets of UAVs. A new feature fusion method is adopted. The feature fusion method designs a pixel shuffle feature fusion network PSFF based on a pixel shuffle upsampling module PSU, which improves the problem of feature distortion caused by the upsampling method, so that the network can more effectively transfer deep semantic information to the shallow layer, thereby enhancing the feature representation of shallow small targets. Experiments are carried out on a self-built swarm UAV data set, which verifies that the method of the present invention can improve the detection effect of small targets of UAVs. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;
[0049] Figure 2 This is a schematic diagram of the process of building a drone monitoring model in an embodiment of the present invention;
[0050] Figure 3 This is a schematic diagram of the overall framework of the method in the embodiment of the present invention;
[0051] Figure 4 Schematic diagram of a pixel shuffling feature fusion network in an embodiment of the present invention;
[0052] Figure 5 This is a schematic diagram of a pixel shuffling upsampling module in an embodiment of the present invention;
[0053] Figure 6 This is a schematic diagram of the pixel shuffling upsampling principle in an embodiment of the present invention;
[0054] Figure 7 Schematic diagram of visual detection in an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:
[0056] In order to solve the technical problems mentioned in the background technology, the present invention adopts a new multi-scale feature fusion method, called Pixel Shuffle Feature Fusion (PSFF), which is based on the RetinaNet detection framework and can replace the original FPN method used by RetinaNet. The RetinaNet of the present invention can achieve better detection results in the small target detection task of drones.
[0057] like Figure 1 As shown, a method for detecting small targets of unmanned aerial vehicles based on multi-scale feature fusion includes the following steps:
[0058] Step (1) UAV dataset preparation: collect UAV images, clean the collected images, delete blurry images and images that do not contain targets, and then annotate the images that meet the requirements. The VOC annotation format is used here, and 70% of the annotated data is used as a training set and 30% as a test set;
[0059] Step (2) Building a drone detection model: Based on the drone dataset, train the drone detection model;
[0060] Step (3) Test image detection process: Use the trained target detection network and network weight parameters to detect the drone in the test image and output the detection results.
[0061] In the above steps, if Figure 2 As shown, the technical solution of the present invention in step (2) is a method for detecting small targets of unmanned aerial vehicles based on multi-scale feature fusion, which includes the following steps:
[0062] Step (2.1) obtains the feature layer from the backbone network:
[0063] In the {C2, C3, C4, C5} feature layers of the backbone feature extraction network ResNet50, that is, the features of the 2nd, 3rd, 4th and 5th layers of ResNet50, {C3, C4, C5} are selected as several fusion feature layers for multi-scale feature fusion, and the number of channels corresponding to the feature layers is {512, 1024, 2048};
[0064] Step (2.2) pre-fusion processing:
[0065] Feature fusion requires the alignment of feature dimensions of different layers, which includes both channel dimension and spatial dimension. Spatial dimension is width and height. Therefore, the approach of this step is to upsample the deep features and align the deep feature dimensions with the shallow feature dimensions, so that the deep features can be fused with the shallow features.
[0066] Specifically, the present invention uses a pixel shuffling upsampling method, which is efficient, fast, and parameter-free. More importantly, it can make full use of a large amount of channel information in the deepest layer of the backbone network, and use it as the pixel information required for pixel shuffling to perform pixel shuffling upsampling. In short, the channel dimension information can be converted into spatial information and preserved. In this way, not only the defects of simple nearest neighbor difference feature distortion are avoided, but also the rich semantic information of the deep layer can be transmitted to the shallow layer to the greatest extent, so that the shallow layer can obtain richer deep semantic information, improve the shallow layer UAV small target feature representation, and thus improve the network's detection performance for UAV small targets;
[0067] In order to transfer the deep features to the shallow layers, the present invention adopts the pixel shuffling upsampling method. First, the input features are amplified by 2 times through convolution with a convolution kernel of 1, and then upsampled by 2 times through pixel shuffling. At this time, the resolution is expanded by 2 times, and the channel dimension is reduced to 0.5 times of the original. Finally, the upsampling result is obtained, denoted as {P3, P4}, and the number of channels is {512, 1024} respectively;
[0068] The input features in step (2.2) are the two features C4 and C5 in the output features {C3, C4, C5} of step (2.1), and the corresponding number of channels is {1024, 2048};
[0069] The upsampling steps are summarized as follows
[0070] This approach can be organized into a formula:
[0071] y′=PS(x′)
[0072] PS stands for PixelShuffle, which is the abbreviation of pixel shuffle. The default expansion factor (Scale) is 2, which means that the pixel shuffle method is used for 2 times upsampling, where the input is x′∈R w×h×4c Its number of channels is 4c, and the output is y′∈R 2w×2h×c The number of its channels is c. In this process, the number of channels is reduced from 4c to c, and the image size is expanded from w×h to 2w×2h.
[0073] Step (2.3) Fusion operation: (pixel addition)
[0074] Add the output features {P3, P4} of step (2.2) and the features C3 and C4 in the output features {C3, C4, C5} of step (2.1) to obtain the features {T3, T4}, whose corresponding channel numbers are {512, 1024}. At the same time, directly obtain C5 in the output features {C3, C4, C5} of step (2.1) as T5. Finally, the output features of this step are {T3, T4, T5}, and the corresponding channel numbers are {512, 1024, 2048} respectively.
[0075] Step (2.4) Post-fusion processing: (remove aliasing and align the number of channels)
[0076] The input features of this step are {T3, T4, T5}, and the corresponding channel numbers are {512, 1024, 2048}. This step first uses convolution with a convolution kernel of 3 to remove aliasing on the three input features, and then uses convolution with a convolution kernel of 1 to perform channel compression, and uniformly compresses the number of channels of the three features to 256. Finally, the output features of this step are {F3, F4, F5}, and the corresponding channel numbers are {256, 256, 256};
[0077] The formula is written as:
[0078] y = Conv-1×1(Conv-3×3(x))
[0079] The input is denoted by x∈R h×w×c , the output feature map is recorded as y∈R h×w×256 , the size of convolution Conv-1×1 is 1×1×256, and the size of convolution Conv-3×3 is 3×3×c.
[0080] Step (2.5) Target detection: For {F3, F4, F5} which integrates multi-scale features in step (2.4), target category classification and target bounding box regression are performed respectively to obtain the final detection result.
[0081] Furthermore, in step (2.5):
[0082] The target category classification uses the Focal Loss loss with a balancing factor:
[0083] FL(p t )=-α t (1-p t ) γ log(p t )
[0084] Where FL stands for Focal Loss, α t is the weighted balance parameter of positive and negative samples. The larger the value, the greater the weight of the positive sample. It is set to 0.25 here. γ is the focusing coefficient, which controls the update rate of the sample weight. It is a hyperparameter greater than 0 and is set to 2 in this paper. t Indicates the probability that the sample belongs to a positive sample, (1-p t ) γ is the weight expression;
[0085] The bounding box regression is the standard Smooth L1 loss;
[0086] Step (2.6) Training the detection model: The pre-trained model used is a model pre-trained on Imagenet provided by Pytorch. All images were uniformly resized to 1333×800 before being fed into the model. The number of training iterations was set to 12, the momentum was 0.9, the batch size was 4, the initial learning rate was 0.0005, and the weight decayed to 0.3 times the original after every 4 epochs of training. The network model parameters were obtained based on the above training configuration.
[0087] Step (3) specifically includes the following steps:
[0088] Step (3.1) sends the test image into the network model;
[0089] Step (3.2) extracts features through the Resent50 backbone network to obtain a feature map;
[0090] Step (3.3) uses the improved multi-scale feature fusion network of the present invention to perform multi-scale feature fusion;
[0091] Step (3.4) performs target category prediction and bounding box regression on the fused feature map;
[0092] In step (3.5), the threshold is set to 0.5, and the final detection result is screened out through non-maximum suppression.
[0093] like Figure 7 The visual detection results shown in Figure 1 and Table 1 are as follows. The following are our numerical statistical results. The results show that the above method improves the detection effect of small drone targets.
[0094] Table 1. Statistical results of visualization detection
[0095]
[0096] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.
Claims
1. A method for detecting small targets of unmanned aerial vehicles based on multi-scale feature fusion, characterized in that: The following steps are involved: Step 1: UAV dataset creation: collect UAV images, clean the collected images, and then mark the small UAV targets in them; Step 2: Build a drone detection model: Based on the drone dataset, train the drone detection model; specifically, the following steps are included: Step (2.1) obtains the feature layer from the backbone network: In the {C2, C3, C4, C5} feature layers of the backbone feature extraction network ResNet50, {C3, C4, C5} are selected as several fusion feature layers for multi-scale feature fusion, and the number of channels corresponding to the feature layers is {512, 1024, 2048}; Step (2.2) pre-fusion processing: Using the pixel shuffle upsampling method, firstly, the output features {C3, C4, C5} of step (2.1) are subjected to 2-fold channel amplification by convolution with a convolution kernel of 1, and then 2-fold upsampling is performed by pixel shuffling. At this time, the resolution is expanded by 2 times, and the channel dimension is reduced to 0.5 times of the original. Finally, the upsampling result is obtained, denoted as {P3, P4}, and the number of channels is {512, 1024} respectively; the sampling steps are summarized as the following formula: y′=PS(x′), PS stands for PixelShuffle, which is the abbreviation of pixel shuffle. The default expansion factor is 2, which means that the pixel shuffle method is used for 2x upsampling, where the input is x′∈R w×h×4c Its number of channels is 4c, and the output is y′∈R 2w×2h×c The number of its channels is c. In this process, the number of channels is reduced from 4c to c, and the image size is expanded from w×h to 2w×2h. Step (2.3) fusion operation: Add the output features {P3, P4} of step (2.2) and the features C3 and C4 in the output features {C3, C4, C5} of step (2.1) to obtain the features {T3, T4}, whose corresponding channel numbers are {512, 1024}. At the same time, directly obtain C5 in the output features {C3, C4, C5} of step (2.1) as T5. Finally, the output features of this step are {T3, T4, T5}, and the corresponding channel numbers are {512, 1024, 2048} respectively. Step (2.4) post-fusion processing: The formula is written as: y=Conv-1×1(Conv-3×3(x)); The input is denoted by x∈R h×w×c , the output feature map is recorded as y∈R h×w×256 , the size of convolution Conv-1×1 is 1×1×256, and the size of convolution Conv-3×3 is 3×3×c; The input features are {T3, T4, T5}, and the corresponding channel numbers are {512, 1024, 2048}. First, the three input features are de-aliased using convolution with a convolution kernel of 3, and then the convolution with a convolution kernel of 1 is used for channel compression to uniformly compress the three feature channels to 256. Finally, the output features of this step are {F3, F4, F5}, and the corresponding channel numbers are {256, 256, 256}. Step (2.5) Target detection: For {F3, F4, F5} that integrates multi-scale features in step (2.4), target category classification and target bounding box regression are performed respectively to obtain the final detection result; Step (2.6) train the detection model: The pre-trained model used is a model pre-trained on Imagenet provided by Pytorch. All images were resized to 1333×800 before being sent to the model. The number of training iterations was set to 12, the momentum was 0.9, the batch size was 4, the initial learning rate was 0.0005, and the weight decayed to 0.3 times the original weight every 4 epochs. The network model parameters were obtained based on the above training configuration; Step 3: Test image detection process: Use the trained target detection network and network weight parameters to detect the drone in the test image and output the detection results.
2. The method for detecting small drone targets based on multi-scale feature fusion according to claim 1 is characterized in that: In step 1, the collected images are cleaned to delete blurry images and images that do not contain the target.
3. The method for detecting small drone targets based on multi-scale feature fusion according to claim 1, characterized in that: The target category classification described in step (2.5) uses FocalLoss loss with a balancing factor: FL(p t )=-a t (1-p t ) γ log(p t ); Where FL stands for Focal Loss, α t is the weighted balance parameter of positive and negative samples. The larger the value, the greater the weight of the positive sample. γ is the focusing coefficient, which controls the update rate of the sample weight and is a hyperparameter. t Indicates the probability that the sample belongs to a positive sample, (1-p t ) γ is a weight expression.
4. The method for detecting small drone targets based on multi-scale feature fusion according to claim 1 or 3, characterized in that: The target bounding box regression described in step (2.5) is the standard Smooth L1 loss.
5. The method for detecting small targets of unmanned aerial vehicles based on multi-scale feature fusion according to claim 1, characterized in that: Step 3 specifically includes the following steps: Step (3.1) sends the test image into the network model; Step (3.2) extracts features through the Resent50 backbone network to obtain a feature map; Step (3.3) uses the improved multi-scale feature fusion network in step 2 to perform multi-scale feature fusion; Step (3.4) performs target category prediction and bounding box regression on the fused feature map; Step (3.5) sets the threshold and uses non-maximum suppression to filter out the final detection results.