Dual-light image target detection method under view angle of unmanned aerial vehicle

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a dual-light image target detection method under the view angle of an unmanned aerial vehicle, and the method comprises the steps: constructing a dual-light image target detection model which comprises a dual-light feature extraction and fusion network, a feature pyramid network and a rotating target detection head network; preprocessing the visible light and infrared light image data under the view angle of the unmanned aerial vehicle; inputting the preprocessed image pair into a dual-light feature extraction and fusion network to obtain fusion feature maps of different levels; inputting the fusion feature maps of different levels into a feature pyramid network, and outputting a multi-scale fusion feature map; the multi-scale fusion feature map is input into a rotating target detection head network, regression is carried out to obtain a rotating surrounding frame, a category and a confidence coefficient of the target under the view angle of the unmanned aerial vehicle, and the target detection capability in a complex environment is enhanced by a dual-light feature extraction and fusion network; the rotating bounding box regression strategy solves the problem that an existing horizontal bounding box is inaccurate in positioning when the target rotates.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for detecting a rotating target by combining visible light and infrared light images under the viewing angle of an unmanned aerial vehicle. Background Art

[0002] In recent years, with the maturity of drone technology, drones equipped with cameras have been widely used for target detection tasks in the fields of traffic monitoring, security patrols, disaster relief, etc. Existing technologies mainly rely on processing image information from a single light source (such as visible light or infrared light only), which is obviously insufficient in some cases.

[0003] Visible light cameras can provide clear images when there is sufficient light, but they are not very effective at night or in low light conditions; while infrared thermal imaging sensors can work at night or in bad weather, but their image resolution is low and is easily affected by ambient temperature. The dual-light image target detection method can fuse visible light and infrared light image information in complex environments, which can increase the adaptability of the detection algorithm. However, the existing dual-light image target detection methods are mostly based on vehicle-mounted perspectives or other ground perspectives, and use horizontal bounding boxes to represent the position of the target. In the perspective of drones, the target usually rotates, and the horizontal bounding box cannot accurately describe the target position. Summary of the invention

[0004] In order to solve the technical problem that the existing dual-light image target detection method cannot be applied to the perspective of a drone, the present invention proposes a "dual-light image target detection method under the perspective of a drone", comprising the following steps:

[0005] Step 1: Build a dual-light image target detection model, including a dual-light feature extraction and fusion network, a feature pyramid network, and a rotating target detection head network, to pre-process the visible light and infrared light image data pairs from the perspective of the drone;

[0006] Step 2: Input the preprocessed image pair into a dual-light feature extraction and fusion network, which includes a visible light image branch and an infrared light image branch. The two branches have the same network structure, both consisting of a convolutional layer and a series of residual blocks. A cross-branch spatial attention and feature fusion module is added between the residual blocks to adaptively fuse the information in the visible light image and the infrared light image at different levels to obtain fused feature maps at different levels.

[0007] The spatial attention and feature fusion module includes three branches, two of which are visible light feature enhancement branch and infrared light feature enhancement branch, which have the same network structure and can enhance local features and then output enhanced feature maps; the other branch is a feature interaction branch, which realizes fine-grained interaction between the input visible light and infrared light feature maps and outputs an interactive feature map; the visible light enhanced feature map, infrared light enhanced feature map and interactive feature map output by the three branches are spliced ​​and integrated along the channel dimension to output the final fused feature map;

[0008] Step 3: Input the fused feature maps of different levels into the feature pyramid network. The feature pyramid network further fuses the multi-scale information in the feature maps of multiple levels and outputs the multi-scale fused feature map.

[0009] Step 4: Input the multi-scale fusion feature map into the rotating target detection head network, and regress the rotating bounding box, category and confidence of the target from the perspective of the drone. The rotating target detection head network includes an anchor frame adjustment network, an anchor frame alignment convolution network and a regression prediction network. The anchor frame adjustment network is used to adjust the initial horizontal anchor frame. The adjusted rotating anchor frame is used to guide the anchor frame alignment convolution network to extract the anchor frame alignment fusion feature; the anchor frame alignment convolution network includes multiple anchor frame alignment convolution layers, which adjust the sampling position of the convolution operation under the guidance of the rotating anchor frame; the regression prediction network can use the extracted alignment features to regress and obtain the rotating target detection result.

[0010] Compared with the prior art, the technical effects of the present invention are as follows:

[0011] The present invention discloses a dual-light image target detection method from the perspective of an unmanned aerial vehicle. By constructing a dual-light feature extraction and fusion network, the target detection capability in a complex environment is enhanced. A spatial attention and feature fusion module is added to the dual-light feature extraction and fusion network, and the features of the two branches are adaptively enhanced and fine-grainedly interacted to generate fused features. The module can dynamically and adaptively fuse visible light and infrared light image features to generate a more adaptive multimodal feature representation, thereby improving the adaptability of target detection.

[0012] The rotating bounding box regression strategy is introduced through the rotating target detection head network to solve the problem of inaccurate positioning of the existing horizontal bounding box when the target is rotated. The preset horizontal anchor box is initially adjusted through the anchor frame adjustment network to generate a rotated anchor frame. This narrows the gap between the anchor frame and the rotated target and helps to regress the rotated bounding box more accurately. At the same time, the rotated anchor frame can guide the subsequent feature extraction process and improve the spatial consistency between the feature and the target. The anchor frame adjustment network and the anchor frame alignment convolutional network reduce the difficulty of rotating bounding box regression and improve the accuracy and efficiency of rotating target detection from the perspective of the drone.

[0013] The present invention is suitable for visible light-infrared light target detection from the perspective of a UAV, and provides an effective solution for target detection of UAVs in the fields of traffic monitoring, security patrol, disaster relief, etc., and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the overall process of the present invention and serves as an abstract drawing.

[0015] Figure 2 Schematic diagram of the dual-light feature extraction and fusion network structure in step 2.

[0016] Figure 3 Schematic diagram of the structure of the spatial attention and feature fusion module 1 in the dual-light feature extraction and fusion network.

[0017] Figure 4 Schematic diagram of the head network structure for rotating target detection in step 4. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in the field without creative work by adopting the embodiments of the present invention are within the scope of protection of the present invention.

[0019] The present invention takes the dual-light image vehicle detection task under the perspective of a drone as an example to specifically illustrate the workflow of the present invention. The examples fully implemented according to the content of the present invention are as follows:

[0020] (1) Using the visible light camera and infrared thermal imaging sensor carried by the UAV, under the precise positioning of the GPS and inertial navigation system, the visible light and infrared dual-light image data pairs containing the vehicle target are synchronously collected in the target area.

[0021] (2) Figure 1 As shown in the figure, a dual-light image vehicle detection network model from the perspective of a drone is constructed, including dual-light data preprocessing, dual-light feature extraction and fusion network, feature pyramid network and rotating target detection head network;

[0022] like Figure 2As shown, a dual-light feature extraction and fusion network is constructed, including a visible light image branch and an infrared light image branch. The two branches have the same network structure, consisting of 1 convolution layer and 5 residual blocks, which are recorded as residual block 1, residual block 2, residual block 3, residual block 4, and residual block 5. Each residual block consists of 1 convolution layer with a step size of 2 and multiple residual structures stacked. The convolution layer with a step size of 2 can achieve effective downsampling while maintaining high feature information integrity, which not only improves the learning ability and accuracy of the model, but also makes the network structure simpler and more unified, achieving a better balance between detection accuracy and computational efficiency. Each of the residual structures contains 2 convolution layers, and the number of stacked residual structures in residual block 1, residual block 2, residual block 3, residual block 4, and residual block 5 is 1, 2, 4, 8, and 2, respectively.

[0023] A cross-branch spatial attention and feature fusion module is added between residual blocks 3, 4, and 5 of the two branches. Specifically, the visible light image and the infrared light image are respectively output through the convolution layer and residual blocks 1, 2, and 3 to output the visible light feature map and the infrared light feature map. The visible light feature map and the infrared light feature map are input into the spatial attention and feature fusion module 1 to obtain the fused feature. Figure 1 ; The visible light feature map and the infrared light feature map continue to pass through residual block 4 and residual block 5 respectively, and are input into the corresponding spatial attention and feature fusion modules to obtain the other two levels of fusion features Figure 2 and fusion features Figure 3 , Figure 2 The fusion features in Figure 1 ~ Fusion features Figure 3 That is the dual-light fusion feature after enhanced fusion at each level.

[0024] The feature enhancement and fusion of the spatial attention and feature fusion module are independent of the feature extraction process of the two branches. The feature extraction is not affected by the fusion, and the independence of the feature extraction of the visible light image branch and the infrared image branch is maintained.

[0025] like Figure 3 As shown, the visible light image branch and the infrared light image branch are respectively outputted after passing through their respective residual blocks 3, and then inputted into the spatial attention and feature fusion module 1 to obtain the fusion feature Figure 1 For example, the spatial attention and feature fusion module 1 includes three branches, two of which are the visible light feature enhancement branch and the infrared light feature enhancement branch. They have the same network structure and apply the spatial attention mechanism to process the input visible light feature map and the infrared light feature map respectively, which can enhance the local features and then output the enhanced feature map.

[0026] The other branch is the feature interaction branch, which realizes the fine-grained interaction between the input visible light and infrared light feature maps and outputs the interaction feature map. The visible light enhanced feature map, infrared light enhanced feature map and interaction feature map output by the three branches are spliced ​​along the channel dimension, and then passed through the residual block 6 to further integrate and enhance the feature expression, and the final fusion feature map is output. Figure 1 The residual block 6 here does not perform downsampling and only contains several stacked residual structures.

[0027] Furthermore, the visible light feature enhancement branch and the infrared light feature enhancement branch adopt a spatial attention module (CBAM) to enhance local details and suppress irrelevant areas, so that the features are more focused on the detail information of the target area. The input visible light feature map and infrared light feature map are first subjected to maximum pooling and average pooling, respectively, and the feature maps are compressed along the channel dimension to obtain two feature maps. The two feature maps are stacked along the channel dimension to obtain a two-channel stacked feature map. The stacked feature map is convolved to obtain a spatial attention map. Afterwards, the obtained spatial attention map is used to weight the feature map, and the spatial attention map is element-wise multiplied (Hadamard product) with the input feature map to obtain a visible light enhanced feature map and an infrared light enhanced feature map.

[0028] Furthermore, the feature interaction branch first performs element-by-element product (Hadamard product) on the input visible light and infrared light feature maps to achieve fine-grained interaction between the two, and then applies convolution operation to the interacted features to further integrate the feature information to obtain fine-grained interaction features and output the interaction feature map.

[0029] After the dual-light feature extraction and fusion network, the present invention adds a feature pyramid network to combine the fusion features at different levels. Figure 1 , fusion features Figure 2 and fusion features Figure 3 The input is sent to the feature pyramid network, and multi-scale fusion is performed through the feature pyramid network to output a multi-scale fusion feature map, which is used to predict vehicle targets of different sizes at different scales.

[0030] In images from the ground perspective, the relative position between the acquisition device and the target is constrained by the ground plane, so the target usually remains horizontal. However, in the top-down perspective of a drone, changes in the target's orientation appear as rotations of any angle in the image. When existing ground-perspective horizontal target detection methods are applied to drone-perspective images, two major problems are encountered. First, the horizontal bounding box cannot accurately locate rotated targets, especially when detecting targets with large aspect ratios such as vehicles. In addition, when the target rotates, the sampling position of the convolution operation used in existing target detection may deviate from the target and fail to effectively extract target features, thus affecting the detection performance. Therefore, it is necessary to perform targeted detection of rotated targets.

[0031] like Figure 4 As shown, the specific processing process of the rotating target detection head network is as follows: inputting the multi-scale fusion feature map into the anchor frame adjustment network of the corresponding scale for parameter regression; decoding the output of the anchor frame adjustment network: according to the network output, calculating the anchor frame adjustment parameters, adjusting the initial horizontal anchor frame according to the anchor frame adjustment parameters, and obtaining the corresponding rotated anchor frame; according to the rotated anchor frame, calculating the anchor frame alignment displacement matrix; synchronously inputting the fusion feature and the alignment displacement matrix into the anchor frame alignment convolution network, extracting the anchor frame alignment fusion feature; inputting the anchor frame alignment fusion feature into the regression prediction network, and on the basis of the rotated anchor frame, regressing to obtain the final target rotation bounding box and its category prediction result.

[0032] Each multi-scale fusion feature map output by the feature pyramid network has a corresponding anchor frame adjustment network. Each position on the feature map corresponds to an anchor frame. The anchor frame adjustment network outputs a set of 6 parameters for each position, including 2 displacement parameters (x, y), 2 scaling parameters (w, h), and 1 rotation angle parameter. And a confidence parameter p, which is used to indicate the possibility of the existence of an object at that location. According to these anchor frame adjustment parameters, the initial horizontal anchor frame is adjusted to generate the corresponding rotation anchor frame. Anchor frames with confidence values ​​lower than the preset threshold are not adjusted to reduce the amount of calculation.

[0033] The guiding role of the rotation anchor box is achieved through the anchor box alignment displacement matrix. The anchor box alignment displacement matrix is ​​calculated according to the rotation anchor box to adjust the sampling position of the conventional two-dimensional convolution to better capture the features of the rotated target. This matrix has the same size as the input fusion feature map. Each element corresponds to a position on the feature map, and records the displacement of each sampling position relative to the sampling position of the conventional convolution operation when performing anchor box alignment convolution. For a conventional 3×3 convolution kernel, each matrix element contains 9 displacement vectors. Similar to the anchor box adjustment, the anchor box alignment displacement matrix is ​​only calculated for the position where the target may exist. The fusion feature and the alignment displacement matrix are simultaneously input into the anchor box alignment convolution network, the anchor box alignment fusion feature is extracted, and these features are input into the regression prediction network. Based on the rotation anchor box, the final target rotation bounding box and its category prediction result are regressed.

[0034] (3) Preprocess the visible light and infrared light image pairs from the input drone's perspective, including registration, standardization, and enhancement. First, the visible light and infrared light images are registered to align the two modalities in space to ensure the consistency of feature extraction and fusion. Then, the registered images are standardized, including adjusting the size, brightness, and contrast to adapt to different lighting conditions. Finally, data enhancement processing is applied, including rotation, flipping, and scaling, to increase the diversity of the data and improve the robustness of the dual-light image vehicle detection network model under various environmental conditions. These preprocessing steps lay the foundation for subsequent feature extraction and fusion, and enhance the adaptability and detection performance of the dual-light image vehicle detection network model.

[0035] (4) The preprocessed visible light and infrared light image pairs are input into the dual-light feature extraction and fusion network. Specifically, the visible light image is input into the visible light image branch, and the infrared light image is input into the infrared light image branch. The dual-light feature extraction and fusion network generates a fusion feature map of visible light and infrared light at three levels.

[0036] (5) The fused feature maps of three different levels are input into the feature pyramid network to integrate multi-scale information and generate a multi-scale fused feature map.

[0037] (6) The multi-scale fusion feature map is input into the corresponding scale rotation target detection head network, and the anchor frame adjustment network generates a rotation anchor frame based on the input features.

[0038] (7) According to the rotated anchor frame, the anchor frame alignment displacement matrix is ​​calculated to guide the anchor frame alignment convolutional network to extract the anchor frame alignment fusion features.

[0039] (8) The anchor frame alignment fusion features are input into the regression prediction network. Based on the rotated anchor frame, the rotated bounding box of the vehicle target and the vehicle category prediction result are obtained by regression.

[0040] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

Claims

1. A dual-light image target detection method from the perspective of a drone, characterized in that: The following steps are involved: Step 1: Build a dual-light image target detection model, including a dual-light feature extraction and fusion network, a feature pyramid network, and a rotating target detection head network, to pre-process the visible light and infrared light image data pairs from the perspective of the drone; Step 2: Input the preprocessed image pair into a dual-light feature extraction and fusion network, which includes a visible light image branch and an infrared light image branch. The two branches have the same network structure, both consisting of a convolutional layer and a series of residual blocks. A cross-branch spatial attention and feature fusion module is added between the residual blocks to adaptively fuse the information in the visible light image and the infrared light image at different levels to obtain fused feature maps at different levels. The spatial attention and feature fusion module includes three branches, two of which are visible light feature enhancement branch and infrared light feature enhancement branch, which have the same network structure and can enhance local features and then output enhanced feature maps; the other branch is a feature interaction branch, which realizes fine-grained interaction between the input visible light and infrared light feature maps and outputs an interactive feature map; the visible light enhanced feature map, infrared light enhanced feature map and interactive feature map output by the three branches are spliced ​​and integrated along the channel dimension to output the final fused feature map; Step 3: Input the fused feature maps of different levels into the feature pyramid network. The feature pyramid network further fuses the multi-scale information in the feature maps of multiple levels and outputs the multi-scale fused feature map. Step 4: Input the multi-scale fusion feature map into the rotating target detection head network, and regress the rotating bounding box, category and confidence of the target from the perspective of the drone. The rotating target detection head network includes an anchor frame adjustment network, an anchor frame alignment convolution network and a regression prediction network. The anchor frame adjustment network is used to adjust the initial horizontal anchor frame. The adjusted rotating anchor frame is used to guide the anchor frame alignment convolution network to extract the anchor frame alignment fusion feature. The anchor frame alignment convolution network includes multiple anchor frame alignment convolution layers, which adjust the sampling position of the convolution operation under the guidance of the rotation anchor frame; the regression prediction network can use the extracted alignment features to regress and obtain the rotation target detection result.

2. The method for detecting targets using dual-light images from the perspective of an unmanned aerial vehicle according to claim 1, characterized in that: The preprocessing described in step 1 is specifically as follows: first, the visible light and infrared light images are registered to align the two modalities in space to ensure the consistency of feature extraction and fusion; then the registered images are standardized, including adjusting the size, brightness and contrast; finally, data enhancement processing is applied.

3. The method for detecting targets using dual-light images from the perspective of an unmanned aerial vehicle according to claim 1, characterized in that: The visible light image branch and the infrared light image branch of the dual-light feature extraction and fusion network described in step 2 have the same network structure, consisting of 1 convolutional layer and 5 residual blocks, which are respectively recorded as residual block 1, residual block 2, residual block 3, residual block 4, and residual block 5. A cross-branch spatial attention and feature fusion module is added between residual block 3, residual block 4, and residual block 5 of the two branches; The visible light enhanced feature map, infrared light enhanced feature map and interactive feature map output by the visible light feature enhancement branch, infrared light feature enhancement branch and feature interaction branch of the spatial attention and feature fusion module are spliced ​​along the channel dimension, and then pass through the residual block 6 to further integrate and enhance the feature expression, and output the final fused feature map.

4. The method for detecting targets in dual-light images from the perspective of a drone according to claim 3, characterized in that: Residual block 1, residual block 2, residual block 3, residual block 4, and residual block 5 are composed of a convolution layer with a step size of 2 and multiple stacked residual structures. The convolution layer can achieve effective downsampling while maintaining high feature information integrity. The number of stacked residual structures in residual block 1, residual block 2, residual block 3, residual block 4, and residual block 5 are 1, 2, 4, 8, and 2 respectively. Residual block 6 contains several stacked residual structures.

5. The method for detecting targets using dual-light images from the perspective of an unmanned aerial vehicle according to claim 3, characterized in that: The visible light feature enhancement branch and the infrared light feature enhancement branch adopt a spatial attention module, perform maximum pooling and average pooling on the input visible light feature map and infrared light feature map respectively, compress the feature map along the channel dimension to obtain two feature maps; stack the two feature maps along the channel dimension to obtain a two-channel stacked feature map, and perform a convolution operation on the stacked feature map to obtain a spatial attention map; use the obtained spatial attention map to weight the feature map, perform element-by-element product of the spatial attention map and the input feature map to obtain a visible light enhancement feature map and an infrared light enhancement feature map; the specific processing process of the feature interaction branch is: perform element-by-element product of the input visible light feature map and the infrared light feature map, then apply a convolution operation to obtain fine-grained interaction features, and output an interaction feature map.

6. The method for detecting targets using dual-light images from the perspective of a drone according to claim 1, characterized in that: The specific processing process of the rotation target detection head network is as follows: inputting the multi-scale fusion feature map into the anchor frame adjustment network of the corresponding scale for parameter regression; decoding the output of the anchor frame adjustment network: calculating the anchor frame adjustment parameters according to the network output, and adjusting the initial horizontal anchor frame according to the anchor frame adjustment parameters to obtain the corresponding rotation anchor frame; According to the rotation anchor frame, the anchor frame alignment displacement matrix is ​​calculated; the fusion feature and the alignment displacement matrix are simultaneously input into the anchor frame alignment convolution network to extract the anchor frame alignment fusion feature; The anchor box alignment fusion features are input into the regression prediction network. Based on the rotation of the anchor box, the final target rotation bounding box and its category prediction results are obtained by regression.

7. The method for detecting targets using dual-light images from the perspective of a drone according to claim 6, wherein: Each position on the multi-scale fusion feature map corresponds to an anchor box. The anchor box adjustment network outputs a set of 6 parameters for each position, including 2 displacement parameters (x, y), 2 scaling parameters (w, h), and 1 rotation angle parameter. And a confidence parameter p, which is used to indicate the possibility that a target exists at that location.

Images