Infrared small target detection method and device for aerial photography of unmanned aerial vehicle

Abstract

The invention discloses an infrared small target detection method and device for aerial photography of an unmanned aerial vehicle. The method comprises the following steps: making a real scene data set; constructing a deep learning model for infrared small target detection, wherein the deep learning model comprises an infrared small target feature extraction backbone network, a feature focusing pyramid neck network and a detection head; performing iterative training on the deep learning model by using the real scene data set to obtain a trained deep learning model, and deploying the trained deep learning model to an embedded platform to obtain a target detection model; and detecting a to-be-identified infrared small target image through the target detection model to obtain a detection result. According to the method, the infrared small target detection performance of the model is effectively improved, and the parameter quantity of the model is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an infrared small target detection method and device for high-altitude aerial photography by unmanned aerial vehicles. Background Art

[0002] With its flexibility and stealth, drones are playing an increasingly important role in disaster relief, traffic monitoring and other fields. Infrared imaging has the advantages of long detection distance and strong all-weather capability, which can further enhance the effectiveness of drones, making infrared target detection a current research hotspot. However, infrared images also have the characteristics of low contrast and low signal-to-noise ratio, especially when dealing with small objects such as people or vehicles at a distance. Due to the small size of the target and the weak infrared radiation signal, there is a lack of obvious feature information in the infrared image, which greatly increases the difficulty of detection.

[0003] With the rapid development of artificial intelligence technology, traditional target detection algorithms have transitioned to the current mainstream deep learning-based methods, such as YOLO, SSD, RCNN, transformer-based algorithms, etc. However, there are still many challenges in applying deep learning-based target detection algorithms to infrared small targets, especially how to adapt to the unique characteristics of infrared images and improve the detection accuracy of small targets. Many researchers have improved the performance of infrared target detection by improving the algorithm framework, integrating attention modules, and improving loss functions. For example, a study proposed an attention-based improved YOLO algorithm FA-YOLO, which uses GAN to generate infrared images, and proposed a negative sample focusing mechanism and an extended convolution attention module to improve the detection accuracy of infrared-occluded vehicles. There are also studies that propose a hybrid feature extraction network HB-YOLOv5, which combines convolution and self-attention mechanisms, uses a two-dimensional Gaussian distribution to represent the bounding box, and significantly improves the accuracy and speed of infrared small target detection. In terms of datasets, although there are some public infrared datasets, they are usually smaller in scale, have fewer categories, and have lower resolution than visible datasets, which to some extent limits the research on infrared target detection based on deep learning.

[0004] However, most of the existing algorithms have large number of parameters and computational complexity, and their small target detection performance is poor. The current drone high-altitude aerial infrared data set is small in scale, low in resolution, with few labeled categories and single scenes, which to a certain extent leads to poor model generalization ability and makes it difficult to meet the application needs of actual scenarios. Summary of the invention

[0005] (I) Purpose of the invention

[0006] The purpose of the present invention is to provide an infrared small target detection method and device for unmanned aerial vehicle high-altitude aerial photography. The detection method effectively improves the infrared small target detection performance of the model and reduces the number of parameters of the model by proposing a small target feature extraction backbone network and a feature focusing pyramid neck network.

[0007] (II) Technical solution

[0008] In order to solve the above problems, the first aspect of the present invention provides an infrared small target detection method for UAV high-altitude aerial photography, comprising:

[0009] Create a real-world dataset;

[0010] Construct a deep learning model for infrared small target detection, the deep learning model includes an infrared small target feature extraction backbone network, a feature focusing pyramid neck network and a detection head, the infrared small target feature extraction backbone network uses the first extraction layer, the second extraction layer and the third extraction layer to extract features from the input image, outputs scale feature maps of different extraction layers and passes them to the feature focusing pyramid neck network, and the feature focusing pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head;

[0011] Iteratively training the deep learning model using the real scene data set to obtain a trained deep learning model, and deploying the trained deep learning model on an embedding platform to obtain a target detection model;

[0012] The target detection model is used to detect the infrared small target image to be identified to obtain a detection result.

[0013] Furthermore, the infrared small target feature extraction backbone network includes multiple convolutions and multiple feature refinement modules, the multiple convolutions and the multiple feature refinement modules are alternately connected, and the multiple feature refinement modules include a first feature refinement module, a second feature refinement module and a third feature refinement module.

[0014] Furthermore, the infrared small target feature extraction backbone network also includes a squeeze excitation feature selection module, and the squeeze excitation feature selection module is located in the third extraction feature layer of the infrared small target feature extraction backbone network.

[0015] Furthermore, the infrared small target feature extraction backbone network uses the first extraction layer, the second extraction layer and the third extraction layer to extract features from the input image, and outputs scale feature maps of different extraction layers, including:

[0016] Receive a feature map of size 640×640×3;

[0017] The 640×640×3 feature map is convolutionally downsampled and then subjected to feature extraction by a first feature refinement module, and a first extraction layer feature map is outputted, with a size of 160×160×64;

[0018] After convolution downsampling the feature map of the first extraction layer, feature extraction is performed through the second feature refinement module, and the feature map of the second extraction layer is output with a size of 80×80×128;

[0019] The feature map of the second extraction layer is convolutionally downsampled and then extracted through the third feature refinement module, and then passed through the fast spatial pyramid pooling module to obtain an adaptive fusion feature map;

[0020] The adaptive fusion feature map is input into the squeeze excitation feature selection module, and a third extraction layer feature map with a size of 40×40×256 is output; wherein, the infrared small target feature extraction backbone network also includes a fast spatial pyramid pooling module, and the fast spatial pyramid pooling module is connected between the third feature refinement module and the squeeze excitation feature selection module.

[0021] Furthermore, the feature-focused pyramid neck network includes a feature-focused module, and the feature-focused pyramid neck network fuses the multiple scale feature maps based on the feature-focused module.

[0022] Furthermore, the feature-focused pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head, including:

[0023] Receiving a second extraction layer feature map;

[0024] After convolution downsampling, the feature map of the second extraction layer is concatenated with the feature map of the third extraction layer, and then enters the fourth feature refinement module for feature extraction to obtain the target feature map of the third extraction layer and output it to the detection head with a size of 40×40×256;

[0025] After upsampling, the feature map of the second extraction layer is concatenated with the feature map of the first extraction layer, and then enters the fifth feature refinement module for feature extraction to obtain the input map of the first extraction layer focusing module, which has a size of 160×160×64;

[0026] Input the third extraction layer target feature map, the first extraction layer focusing module input map and the second extraction layer feature map into the feature focusing module to obtain a focusing module fusion feature map with a size of 80×80×192;

[0027] After upsampling, the target feature map of the third extraction layer is spliced ​​with the fusion feature map of the focusing module, and then passes through the sixth feature refinement module to obtain the target feature map of the second extraction layer and output it to the detection head, with a size of 80×80×128;

[0028] After upsampling the second extraction layer target feature map, it is spliced ​​with the fifth feature refinement module, and then passes through the seventh feature refinement module to output the first extraction layer target feature map with a size of 160×160×64.

[0029] Furthermore, the third extraction layer target feature map, the first extraction layer focus module input map and the second extraction layer feature map are all input into the feature focus module to obtain a focus module fusion feature map with a size of 80×80×192, including:

[0030] The target feature map of the third extraction layer is upsampled to obtain a feature map of size 80×80×256, and then the number of channels is changed by convolution, and the output feature map size is 80×80×hid_c;

[0031] The second extraction layer feature map is convolved to change the number of channels, and the output feature map size is 80×80×hid_c;

[0032] The input image of the first extraction layer focusing module is convolutionally downsampled and the number of channels is changed, and the output feature map size is 80×80×hid_c;

[0033] The three 80×80×hid_c feature maps are spliced ​​for the first time in the channel dimension to obtain a feature map after the first splicing, with a size of 80×80×192, and then the number of channels is adjusted to 64 through convolution, and the size of the output feature map is 80×80×64. The 80×80×64 feature map is subjected to two maximum pooling operations successively to obtain a first maximum pooling feature map and a second maximum pooling feature map, respectively, and then the 80×80×64 feature map, the first maximum pooling feature map and the second maximum pooling feature map are spliced ​​to obtain a feature map after the second splicing, with a size of 80×80×192. After the 80×80×192 feature map is convolved, it is added to the feature map after the first splicing to obtain a focusing module fusion feature map with a size of 80×80×192.

[0034] Furthermore, it is characterized in that the adaptive fusion feature map is input into the squeeze excitation feature selection module, and the third extraction layer feature map is output, including:

[0035] Perform global average pooling on the input adaptive fusion feature map to obtain a first output dimension feature map;

[0036] Input the first output dimension feature map into the fully connected layer to obtain the second output dimension feature map;

[0037] The second output dimension feature map and the input adaptive fusion feature map are subjected to channel weighting processing to obtain a third extraction layer feature map.

[0038] Furthermore, it is characterized in that the step of inputting the first output dimensional feature map into the fully connected layer to obtain the second output dimensional feature map comprises:

[0039] The feature channel dimension of the first output dimensional feature map is reduced to 1 / r of the initial feature channel dimension using a dimensionality reduction layer to obtain a first dimensionality reduction output feature map;

[0040] Processing the first dimensionality reduction output feature map through a dimensionality increase layer to obtain a second output dimensionality feature map;

[0041] The fully connected layer includes a dimensionality reduction layer and a dimensionality increase layer connected in sequence, and r is a compression ratio.

[0042] Another aspect of the present invention provides an infrared small target detection device for high-altitude aerial photography of unmanned aerial vehicles, comprising:

[0043] Production module, used to produce real scene data sets;

[0044] A construction module is used to construct a deep learning model for infrared small target detection, wherein the deep learning model includes an infrared small target feature extraction backbone network, a feature focusing pyramid neck network and a detection head, wherein the infrared small target feature extraction backbone network uses a first extraction layer, a second extraction layer and a third extraction layer to extract features from an input image, outputs scale feature maps of different extraction layers and passes them to the feature focusing pyramid neck network, and the feature focusing pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head;

[0045] A training module, used to iteratively train the deep learning model using the real scene data set to obtain a trained deep learning model, and deploy the trained deep learning model on an embedding platform to obtain a target detection model;

[0046] The detection module is used to detect the infrared small target image to be identified through the target detection model to obtain the detection result.

[0047] The above technical scheme of the present invention has the following beneficial technical effects: the present invention aims at the problem of infrared small target detection in the high-altitude aerial photography scene of unmanned aerial vehicles, proposes an infrared small target feature extraction backbone network, proposes a feature focusing module that can fully integrate features of various scales, and proposes a feature focusing pyramid neck network based on the feature focusing module. The algorithm proposed by the present invention has a small number of parameters and good infrared small target detection performance; the present invention combines public data sets and collected data sets to construct a large-scale real-scene infrared database, which can effectively provide data support for target detection algorithms based on deep learning; BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of an infrared small target detection method for UAV high-altitude aerial photography based on the present invention;

[0049] Figure 2 This is a schematic diagram of a backbone network for extracting infrared small target features based on an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a squeeze excitation feature selection module based on an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of a feature-focused pyramid neck network based on an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of a feature focusing module based on an embodiment of the present invention;

[0053] Figure 6 is a model conversion flow chart based on an embodiment of the present invention;

[0054] Figure 7 This is a flow chart of an infrared small target detection device for high-altitude aerial photography of unmanned aerial vehicles based on the present invention;

[0055] Figure 8 This is a picture of the detection effect of scenario 1 of the embodiment of the present invention;

[0056] Fig. 9 This is a picture of the detection effect of scenario 2 of the embodiment of the present invention.

[0057] Figure Number:

[0058] 1. The detection object is a person; 2. The detection object is a vehicle; 3. The detection object is a two-wheeled vehicle. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0060] The first aspect of the present invention provides an infrared small target detection method for UAV high-altitude aerial photography, such as Figure 1 As shown, the following steps are included:

[0061] S101, create a real scene dataset;

[0062] S102, constructing a deep learning model for infrared small target detection, the deep learning model comprising an infrared small target feature extraction backbone network, a feature focusing pyramid neck network and a detection head, the infrared small target feature extraction backbone network respectively uses a first extraction layer, a second extraction layer and a third extraction layer to extract features from an input image, outputs scale feature maps of different extraction layers and passes them to the feature focusing pyramid neck network, the feature focusing pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head;

[0063] S103, iteratively training the deep learning model using the real scene data set to obtain a trained deep learning model, and deploying the trained deep learning model on an embedding platform to obtain a target detection model;

[0064] S104, detecting the infrared small target image to be identified by using the target detection model to obtain a detection result.

[0065] Furthermore, the infrared small target feature extraction backbone network includes multiple convolutions and multiple feature refinement modules, the multiple convolutions and the multiple feature refinement modules are alternately connected, and the multiple feature refinement modules include a first feature refinement module, a second feature refinement module and a third feature refinement module.

[0066] Furthermore, the infrared small target feature extraction backbone network also includes a squeeze excitation feature selection module, and the squeeze excitation feature selection module is located in the third extraction feature layer of the infrared small target feature extraction backbone network.

[0067] Furthermore, the infrared small target feature extraction backbone network uses the first extraction layer, the second extraction layer and the third extraction layer to extract features from the input image, and outputs scale feature maps of different extraction layers, including:

[0068] Receive a feature map of size 640×640×3;

[0069] The 640×640×3 feature map is convolutionally downsampled and then subjected to feature extraction by a first feature refinement module, and a first extraction layer feature map is outputted, with a size of 160×160×64;

[0070] After convolution downsampling the feature map of the first extraction layer, feature extraction is performed through the second feature refinement module, and the feature map of the second extraction layer is output with a size of 80×80×128;

[0071] The feature map of the second extraction layer is convolutionally downsampled and then extracted through the third feature refinement module, and then passed through the fast spatial pyramid pooling module to obtain an adaptive fusion feature map;

[0072] The adaptive fusion feature map is input into the squeeze excitation feature selection module, and a third extraction layer feature map with a size of 40×40×256 is output; wherein, the infrared small target feature extraction backbone network also includes a fast spatial pyramid pooling module, and the fast spatial pyramid pooling module is connected between the third feature refinement module and the squeeze excitation feature selection module.

[0073] Furthermore, the feature-focused pyramid neck network includes a feature-focused module, and the feature-focused pyramid neck network fuses the multiple scale feature maps based on the feature-focused module.

[0074] Furthermore, the feature-focused pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head, including:

[0075] Receiving a second extraction layer feature map;

[0076] After convolution downsampling, the feature map of the second extraction layer is concatenated with the feature map of the third extraction layer, and then enters the fourth feature refinement module for feature extraction to obtain the target feature map of the third extraction layer and output it to the detection head with a size of 40×40×256;

[0077] After upsampling, the feature map of the second extraction layer is concatenated with the feature map of the first extraction layer, and then enters the fifth feature refinement module for feature extraction to obtain the input map of the first extraction layer focusing module, which has a size of 160×160×64;

[0078] Input the third extraction layer target feature map, the first extraction layer focusing module input map and the second extraction layer feature map into the feature focusing module to obtain a focusing module fusion feature map with a size of 80×80×192;

[0079] After upsampling, the target feature map of the third extraction layer is spliced ​​with the fusion feature map of the focusing module, and then passes through the sixth feature refinement module to obtain the target feature map of the second extraction layer and output it to the detection head, with a size of 80×80×128;

[0080] After upsampling the second extraction layer target feature map, it is spliced ​​with the fifth feature refinement module, and then passes through the seventh feature refinement module to output the first extraction layer target feature map with a size of 160×160×64.

[0081] Furthermore, the third extraction layer target feature map, the first extraction layer focus module input map and the second extraction layer feature map are all input into the feature focus module to obtain a focus module fusion feature map with a size of 80×80×192, including:

[0082] The target feature map of the third extraction layer is upsampled to obtain a feature map of size 80×80×256, and then the number of channels is changed by convolution, and the size of the output feature map is 80×80×hid_c (hid_c is the number of hidden channels set, which is set to 64 here);

[0083] The second extraction layer feature map is convolved to change the number of channels, and the output feature map size is 80×80×hid_c;

[0084] The input image of the first extraction layer focusing module is convolutionally downsampled and the number of channels is changed, and the output feature map size is 80×80×hid_c;

[0085] The three 80×80×hid_c feature maps are spliced ​​for the first time in the channel dimension to obtain a feature map after the first splicing, with a size of 80×80×192, and then the number of channels is adjusted to 64 through convolution, and the size of the output feature map is 80×80×64. The 80×80×64 feature map is subjected to two maximum pooling operations successively to obtain a first maximum pooling feature map and a second maximum pooling feature map, respectively, and then the 80×80×64 feature map, the first maximum pooling feature map and the second maximum pooling feature map are spliced ​​to obtain a feature map after the second splicing, with a size of 80×80×192. After the 80×80×192 feature map is convolved, it is added to the feature map after the first splicing to obtain a focusing module fusion feature map with a size of 80×80×192.

[0086] Furthermore, it is characterized in that the adaptive fusion feature map is input into the squeeze excitation feature selection module, and the third extraction layer feature map is output, including:

[0087] Perform global average pooling on the input adaptive fusion feature map to obtain a first output dimension feature map;

[0088] Input the first output dimension feature map into the fully connected layer to obtain the second output dimension feature map;

[0089] The second output dimension feature map and the input adaptive fusion feature map are subjected to channel weighting processing to obtain a third extraction layer feature map.

[0090] Furthermore, inputting the first output dimensional feature map into the fully connected layer to obtain the second output dimensional feature map includes:

[0091] The feature channel dimension of the first output dimensional feature map is reduced to 1 / r of the initial feature channel dimension using a dimensionality reduction layer to obtain a first dimensionality reduction output feature map;

[0092] Processing the first dimensionality reduction output feature map through a dimensionality increase layer to obtain a second output dimensionality feature map;

[0093] The fully connected layer includes a dimensionality reduction layer and a dimensionality increase layer connected in sequence, and r is a compression ratio.

[0094] The present detection method is described in detail below in conjunction with specific embodiments.

[0095] The first extraction layer, the second extraction layer and the third extraction layer of the backbone network for extracting features of small infrared targets in the present invention are respectively the P2 layer, the P3 layer and the P4 layer in the general target detection algorithm, and the width and height of the corresponding feature maps are 160×160, 80×80 and 40×40 respectively;

[0096] The specific details of step S101 are as follows:

[0097] The present invention produces a large-scale real-scene dataset, which integrates 17,910 images from datasets from different sources, including 16,624 training set images and 1,286 validation set images, and the images come from public datasets and collected images.

[0098] In step S102, the process of building a deep learning model for infrared small target detection is as follows:

[0099] The extraction process of the infrared small target feature extraction backbone network is as follows: Figure 2 As shown, the specific process includes: the feature size of the input image of the present invention is 640×640×3 (for any size of input image, it is generally adjusted to 640×640×3 first), a 3×3 convolution layer is used to perform a convolution operation on the feature, extract the features in the image, and downsample the image; the feature refinement module is used to further extract more complex features, and gradually extract and refine the important features in the infrared image.

[0100] The infrared small target feature extraction backbone network proposed in the present invention downsamples the input features through four 3×3 convolutions to extract features, and further extracts complex features using multiple feature refinement modules in the middle (the multiple feature refinement modules include a first feature refinement module, a second feature refinement module, and a third feature refinement module). Specifically, after downsampling the 640×640×3 feature map by two 3×3 convolutions, feature extraction is performed through the first feature refinement module, and the first extraction layer feature map (P2 layer feature map) is output, with a size of 160×160×64.

[0101] The first extraction layer feature map is subjected to a 3×3 convolution downsampling and then convolution downsampling. After that, feature extraction is performed through the second feature refinement module, and the second extraction layer feature map (P3 layer feature map) is output, with a size of 80×80×128;

[0102] The feature map of the second extraction layer is downsampled by a 3×3 convolution and then extracted by the third feature refinement module, and then passed through the fast spatial pyramid pooling module to obtain an adaptive fusion feature map;

[0103] The adaptive fusion feature map is input into the squeeze excitation feature selection module, and a third extraction layer feature map (P4 layer feature map) with a size of 40×40×256 is output; wherein the infrared small target feature extraction backbone network also includes a fast spatial pyramid pooling module, and the fast spatial pyramid pooling module is connected between the third feature refinement module and the squeeze excitation feature selection module.

[0104] The P2 layer feature map, P3 layer feature map, and P4 layer feature map (the corresponding feature map sizes are 160×160×64, 80×80×128, and 40×40×256, respectively) are output to the feature-focused pyramid neck network.

[0105] Combine the following Figure 3 As shown, the squeeze excitation feature selection module is described as follows:

[0106] In order to further improve the performance of the target detection model, especially when dealing with complex backgrounds and multi-scale targets, the present invention introduces a squeeze-excitation feature selection module. The core purpose of the squeeze-excitation feature selection module is to enhance the representation ability of the network and improve the representation power of features by explicitly modeling the dependencies between channels.

[0107] The squeeze-excitation feature selection module is implemented in two main steps: squeeze and excitation. First, in the squeeze step, the module performs global average pooling on the input adaptive fusion feature map (h×w×c) to obtain the first output dimension feature map (size 1×1×c). This step effectively compresses the spatial information into a channel-level global information. Subsequently, in the excitation step, the first output dimension feature map is fed into a network consisting of two layers of fully connected layers. The fully connected layer consists of two layers. The first layer is a dimensionality reduction layer, which usually reduces the number of feature channels to the original 1 / r (r is the compression ratio, usually 16) to obtain the first reduced dimension output feature map, which helps to reduce model complexity and computational requirements. The second layer is a dimensionality increase layer, which is the first reduced dimension output feature map processed by the dimensionality increase layer to obtain the second output dimension feature map, the purpose of which is to restore the features to the original channel dimension. Then, through a sigmoid activation function, the module outputs a weight coefficient vector with a value between 0 and 1. Finally, the second output dimension feature map and the input adaptive fusion feature map are channel-weighted to obtain the third extraction layer feature map (P4 layer feature map). The weight vector is applied to each channel on the original feature map (referring to the input adaptive fusion feature map) to weight the features of different channels. The calculation formula of the squeeze excitation feature selection module is:

[0108]

[0109] In the formula, The output feature map (i.e., the third extraction layer feature map or the P4 layer feature map) has a size of H×W×C (40×40×256);

[0110] X: Input feature map (adaptively fused feature map: output map after fast spatial pyramid pooling module), size is H×W×C (40×40×256).

[0111] GP(X) performs a global average pooling operation on X, which compresses the H×W spatial dimensions of each channel into a single real number with an output dimension of 1×1×C.

[0112] σ: sigmoid activation function.

[0113] δ: ReLU activation function.

[0114] W1 is the first weight matrix of the fully connected layer. W1 and ReLU activation function are the dimensionality reduction operations of the corresponding fully connected layer.

[0115] W2 is the second weight matrix of the fully connected layer. W2 and the sigmoid activation function correspond to the dimensionality increase operation of the fully connected layer.

[0116] ⊙: Element-wise multiplication, used to apply the calculated scaling factor for each channel back to the original channel features.

[0117] The present invention adds a squeeze-excitation feature selection module to the last P4 layer (the third extraction feature layer) of the infrared small target feature extraction backbone network, so that the model features are optimized and selected in the last step before subsequent feature processing, which helps the model to capture and strengthen key features more accurately. In addition, at the P4 layer at the end of the backbone network, the feature resolution is low and contains rich high-level language information. Adding the squeeze-excitation feature selection module to this position can optimize the model performance without significantly increasing the computational burden.

[0118] Figure 4As shown, the present invention also proposes a feature-focused pyramid neck network, which fuses features from three scales based on a feature-focused module, so that the model can fully learn the feature information of each scale; and then uses the feature-focused pyramid neck network to further fuse features of each scale to obtain a neck output. Specifically, the inputs of the three feature layers P2, P3, and P4 from the infrared small target feature extraction backbone network are obtained; for the P3 layer features (the second extraction layer feature map or the P3 layer feature map), a 3×3 size convolution is performed to downsample the features and splice them with the P4 layer features (the third extraction layer feature map or the P4 layer feature map), and then the fourth feature refinement module is used to adjust the number of output channels to obtain a feature scale of 40×40×256 (the third extraction layer target feature map or the P4 layer target feature map); at the same time, the P3 layer features are upsampled by the nearest neighbor upsampling (nearest neighbor interpolation upsampling, which is a type of upsampling) module, and spliced ​​with the P2 layer features (the first extraction layer feature map or the P2 layer feature map), and then the fifth feature refinement module is used to adjust the number of output channels to obtain a feature scale of 160×160×64 for the first extraction layer focusing module input map (i.e., the P2 layer focusing module input map); then the features obtained after the above two operations and the original P3 layer features are used as input, and the output The P4 layer target feature map is input into the feature focusing module (i.e., the P4 layer target feature map is used as the P4 layer focus module input map, the original P3 layer feature is used as the P3 layer focus module input map, and the P4 layer focus module input map, the P3 layer focus module input map and the P2 layer focus module input map are respectively input into the feature focusing module), and the output feature is obtained, and the focus module fusion feature map is output, and the feature scale size is 80×80×192; the feature processed by the fourth feature refinement module is upsampled, and spliced ​​with the feature focus module output feature (focus module fusion feature map), and then the sixth feature refinement module is used to adjust the feature size to 80×80×128 (P3 layer target feature map or the second extraction layer target feature map); this feature is upsampled again, spliced ​​with the fifth feature refinement module output feature, and then the seventh feature refinement module is used to adjust the feature size to 160×160×64 (P2 layer target feature map or the first extraction layer target feature map), and the output of three feature scales is obtained and passed to the detection head.

[0119] Combine the following Figure 5 As shown, the present invention also proposes a feature focusing module, which fuses features from three scales and performs feature fusion. Specifically,

[0120] For the input image of the P4 layer focusing module, the size is 40×40×256. First, the feature scale is changed to 80×80×256 by using the nearest neighbor upsampling, and then the number of channels is changed by using the 1×1 convolution to change the feature size to 80×80×hid_c.

[0121] For the input image of the focusing module of the P3 layer, the size is 80×80×128. The number of channels is changed by using a 1×1 convolution, and the feature size is changed to 80×80×hid_c.

[0122] For the input image of the focusing module of the P2 layer, the size is 160×160×64. The features are downsampled using a 3×3 convolution and the number of channels is changed to 80×80×hid_c. hid_c is set to 1 / 2 of the number of feature channels of the P3 layer, which is 64 here.

[0123] After the above operations, the features of the three scales are unified to the same scale size (80×80×hid_c), and then the three features of the same scale are spliced ​​together (operation C) to obtain the feature map after the first splicing. At this time, the feature size is 80×80×192. The number of channels of the spliced ​​features is adjusted to 64 using a 1×1 convolution. At this time, the feature size is 80×80×64; then, after two maximum pooling operations, the maximum pooling uses a pooling window of size 5×5 to perform the maximum pooling operation on the input feature map. The window slides 1 pixel each time and adds padding values ​​to the edges of the input feature map to ensure that the size of the output feature map is the same as the input feature map. Then the features of the first three operations (including 1×1 convolution and two maximum pooling) are spliced ​​together (that is, the 80×80×64 feature map after 1×1 convolution, the first maximum pooling feature map and the second maximum pooling feature map are spliced), and the feature scale size is 80×80×192 (the two pooling operations obtain the first maximum pooling feature map and the second maximum pooling feature map respectively); then the features are optimized by 1×1 convolution without changing the feature scale, and finally this feature is added to the feature after the first splicing operation (the feature map after the first splicing) (operation A) and output to obtain the focusing module fusion feature map, and the output feature size is 80×80×192.

[0124] In addition, the characteristic size of the input image of the present invention is 640×640×3, but the input image of any size can be adjusted to other sizes, but the corresponding output image size will also change accordingly.

[0125] Further, in S103, the deep learning model is iteratively trained using the real scene data set to obtain a trained deep learning model, and the trained deep learning model is deployed on the embedded platform to obtain a target detection model, specifically:

[0126] Real scene model training, the image size is set to 416×416, the batch size is set to 32, and 100 rounds of iterative training are performed on the improved model.

[0127] In the present invention, model deployment involves converting the PyTorch format model to ONNX (Open Neural Network Exchange) format, and then to OM (a model suitable for Huawei Ascend series NPU) format. See the model conversion process for details. Figure 6 .

[0128] The PyTorch model is converted to ONNX format. The present invention uses the export function of the YOLO class of the Ultralytics library to achieve this. The conversion of the ONNX model to the OM format is completed by Huawei's ATC (Ascend Tensor Compiler) tool. The ATC tool is designed for the optimization of Huawei's Ascend series AI processors. During the model conversion process, ATC will perform operations such as operator scheduling optimization, weight data rearrangement, and memory usage optimization, and further tune the network model of the open source framework so that it can be efficiently executed on the Ascend AI processor; based on the OM model, the deployment reasoning of the proposed model can be realized with the adapted reasoning code.

[0129] The present invention verifies the deployed model based on real scene images, and verifies the real-time performance of the deployed model based on 1286 verification images in the real scene data set. The present invention measures the speed at which the model processes images, i.e., the frame rate. The test results show that the model achieves an average real-time detection effect of 31.4 frames per second when running on an embedded platform. The results show that the model can process video streams at a fast enough speed and is suitable for application scenarios that require fast response.

[0130] Further, in S104, the infrared small target image to be identified is detected by the target detection model to obtain a detection and identification result. The present invention inputs the acquired real scene image into the model for detection and outputs the detection result. Figure 8 and Fig. 9 They are respectively the detection effect diagrams under different scenarios of the present invention, Figure 8 and Fig. 9 In the figure, 1-blue box indicates that it is detected as a person, 2-green box indicates that it is detected as a vehicle, and 3-red box indicates that it is detected as a two-wheeled vehicle. It can be seen that the detection performance of the model of the present invention is generally good, and it can correctly detect most of the people and vehicles in the scene.

[0131] The second aspect of the present invention provides an infrared small target detection device for high-altitude aerial photography of unmanned aerial vehicles, such as Figure 7 As shown, including:

[0132] A production module 201, used for producing a real scene data set;

[0133] A construction module 202 is used to construct a deep learning model for infrared small target detection, wherein the deep learning model includes an infrared small target feature extraction backbone network, a feature focusing pyramid neck network and a detection head, wherein the infrared small target feature extraction backbone network uses a first extraction layer, a second extraction layer and a third extraction layer to extract features from an input image, outputs scale feature maps of different extraction layers and passes them to the feature focusing pyramid neck network, and the feature focusing pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head;

[0134] A training module 203 is used to iteratively train the deep learning model using the real scene data set to obtain a trained deep learning model, and deploy the trained deep learning model on an embedded platform to obtain a target detection model;

[0135] The detection module 204 is used to detect the infrared small target image to be identified through the target detection model to obtain a detection result.

[0136] Aiming at the problem of infrared small target detection in the high-altitude aerial photography scene of unmanned aerial vehicles, the present invention proposes a small target feature extraction backbone network to fully extract small target features, integrates a squeeze excitation feature selection module to further optimize and select features from the small target feature extraction backbone network, proposes a feature focusing module, and based on the feature focusing module, proposes a feature focusing pyramid neck network, which can fully integrate the three-scale features from the backbone, effectively improve the infrared small target detection performance of the model, reduce the number of parameters of the model, and make the model easy to deploy on an embedded platform; and constructs a data set of real scenes, which can effectively improve the detection effect of the model in various scenes. Compared with the existing algorithm, the improved algorithm constructed by the present invention has a small number of parameters and a good small target detection effect.

[0137] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A method for detecting small infrared targets for UAV aerial photography, characterized in that: include: Create a real-world dataset; Construct a deep learning model for infrared small target detection, the deep learning model includes an infrared small target feature extraction backbone network, a feature focusing pyramid neck network and a detection head, the infrared small target feature extraction backbone network uses the first extraction layer, the second extraction layer and the third extraction layer to extract features from the input image, outputs scale feature maps of different extraction layers and passes them to the feature focusing pyramid neck network, and the feature focusing pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head; Iteratively training the deep learning model using the real scene data set to obtain a trained deep learning model, and deploying the trained deep learning model on an embedding platform to obtain a target detection model; The target detection model is used to detect the infrared small target image to be identified to obtain a detection result.

2. The infrared small target detection method for UAV high-altitude aerial photography according to claim 1 is characterized in that: The infrared small target feature extraction backbone network includes multiple convolutions and multiple feature refinement modules, the multiple convolutions and the multiple feature refinement modules are alternately connected, and the multiple feature refinement modules include a first feature refinement module, a second feature refinement module and a third feature refinement module.

3. The infrared small target detection method for UAV high-altitude aerial photography according to claim 1 is characterized in that: The infrared small target feature extraction backbone network also includes a squeeze excitation feature selection module, and the squeeze excitation feature selection module is located in the third extraction feature layer of the infrared small target feature extraction backbone network.

4. The infrared small target detection method for UAV high-altitude aerial photography according to claim 3 is characterized in that: The infrared small target feature extraction backbone network uses the first extraction layer, the second extraction layer and the third extraction layer to extract features from the input image, and outputs scale feature maps of different extraction layers, including: Receive a feature map of size 640×640×3; The 640×640×3 feature map is convolutionally downsampled and then subjected to feature extraction by a first feature refinement module, and a first extraction layer feature map is outputted, with a size of 160×160×64; After convolution downsampling the feature map of the first extraction layer, feature extraction is performed through the second feature refinement module, and the feature map of the second extraction layer is output with a size of 80×80×128; The feature map of the second extraction layer is convolutionally downsampled and then extracted through the third feature refinement module, and then passed through the fast spatial pyramid pooling module to obtain an adaptive fusion feature map; Input the adaptive fusion feature map into the squeeze excitation feature selection module, and output a third extraction layer feature map with a size of 40×40×256; Among them, the infrared small target feature extraction backbone network also includes a fast spatial pyramid pooling module, and the fast spatial pyramid pooling module is connected between the third feature refinement module and the squeeze excitation feature selection module.

5. The infrared small target detection method for UAV high-altitude aerial photography according to claim 1 is characterized in that: The feature focusing pyramid neck network includes a feature focusing module, and the feature focusing pyramid neck network fuses the multiple scale feature maps based on the feature focusing module.

6. The infrared small target detection method for UAV high-altitude aerial photography according to claim 5 is characterized in that: The feature-focused pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head, including: Receiving a second extraction layer feature map; After convolution downsampling, the feature map of the second extraction layer is concatenated with the feature map of the third extraction layer, and then enters the fourth feature refinement module for feature extraction to obtain the target feature map of the third extraction layer and output it to the detection head with a size of 40×40×256; After upsampling, the feature map of the second extraction layer is concatenated with the feature map of the first extraction layer, and then enters the fifth feature refinement module for feature extraction to obtain the input map of the first extraction layer focusing module, which has a size of 160×160×64; Input the third extraction layer target feature map, the first extraction layer focusing module input map and the second extraction layer feature map into the feature focusing module to obtain a focusing module fusion feature map with a size of 80×80×192; After upsampling, the target feature map of the third extraction layer is spliced ​​with the fusion feature map of the focusing module, and then passes through the sixth feature refinement module to obtain the target feature map of the second extraction layer and output it to the detection head, with a size of 80×80×128; After upsampling the second extraction layer target feature map, it is spliced ​​with the fifth feature refinement module, and then passes through the seventh feature refinement module to output the first extraction layer target feature map with a size of 160×160×64.

7. The infrared small target detection method for UAV high-altitude aerial photography according to claim 6 is characterized in that: The third extraction layer target feature map, the first extraction layer focus module input map and the second extraction layer feature map are all input into the feature focus module to obtain a focus module fusion feature map with a size of 80×80×192, including: The target feature map of the third extraction layer is upsampled to obtain a feature map of size 80×80×256, and then the number of channels is changed by convolution, and the output feature map size is 80×80×hid_c; The second extraction layer feature map is convolved to change the number of channels, and the output feature map size is 80×80×hid_c; The input image of the first extraction layer focusing module is convolutionally downsampled and the number of channels is changed, and the output feature map size is 80×80×hid_c; The three 80×80×hid_c feature maps are spliced ​​for the first time in the channel dimension to obtain a feature map after the first splicing, with a size of 80×80×192, and then the number of channels is adjusted to 64 through convolution, and the size of the output feature map is 80×80×64. The 80×80×64 feature map is subjected to two maximum pooling operations successively to obtain a first maximum pooling feature map and a second maximum pooling feature map, respectively, and then the 80×80×64 feature map, the first maximum pooling feature map and the second maximum pooling feature map are spliced ​​to obtain a feature map after the second splicing, with a size of 80×80×192. After the 80×80×192 feature map is convolved, it is added to the feature map after the first splicing to obtain a focusing module fusion feature map with a size of 80×80×192.

8. The infrared small target detection method for UAV high-altitude aerial photography according to claim 4 is characterized in that: Inputting the adaptive fusion feature map into the squeeze excitation feature selection module and outputting the third extraction layer feature map comprises: Perform global average pooling on the input adaptive fusion feature map to obtain a first output dimension feature map; Input the first output dimension feature map into the fully connected layer to obtain the second output dimension feature map; The second output dimension feature map and the input adaptive fusion feature map are subjected to channel weighting processing to obtain a third extraction layer feature map.

9. The infrared small target deep learning detection method for UAV high-altitude aerial photography according to claim 8 is characterized in that: Inputting the first output dimensional feature map into the fully connected layer to obtain the second output dimensional feature map includes: The feature channel dimension of the first output dimensional feature map is reduced to 1 / r of the initial feature channel dimension using a dimensionality reduction layer to obtain a first dimensionality reduction output feature map; Processing the first dimensionality reduction output feature map through a dimensionality increase layer to obtain a second output dimensionality feature map; The fully connected layer includes a dimensionality reduction layer and a dimensionality increase layer connected in sequence, and r is a compression ratio.

10. An infrared small target detection device for high-altitude aerial photography of unmanned aerial vehicles, characterized in that: include: Production module, used to produce real scene data sets; A construction module is used to construct a deep learning model for infrared small target detection, wherein the deep learning model includes an infrared small target feature extraction backbone network, a feature focusing pyramid neck network and a detection head, wherein the infrared small target feature extraction backbone network uses a first extraction layer, a second extraction layer and a third extraction layer to extract features from an input image, outputs scale feature maps of different extraction layers and passes them to the feature focusing pyramid neck network, and the feature focusing pyramid neck network is used to fuse multiple scale feature maps and output them to the detection head; A training module, used to iteratively train the deep learning model using the real scene data set to obtain a trained deep learning model, and deploy the trained deep learning model on an embedding platform to obtain a target detection model; The detection module is used to detect the infrared small target image to be identified through the target detection model to obtain the detection result.

Images