Dual-light image target detection method under unmanned aerial vehicle perspective

Abstract

The application belongs to the technical field of image processing, and specifically relates to a dual-light image target detection method under the view angle of a UAV, which constructs a dual-light image target detection model, including a dual-light feature extraction and fusion network, a feature pyramid network and a rotating target detection head network, and performs preprocessing on visible light and infrared light image data pairs under the view angle of the UAV; the preprocessed image pairs are input into the dual-light feature extraction and fusion network to obtain fusion feature maps at different levels; the fusion feature maps at different levels are input into the feature pyramid network to output multi-scale fusion feature maps; the multi-scale fusion feature maps are input into the rotating target detection head network to regress a rotating bounding box, a category and a confidence of a target under the view angle of the UAV, the dual-light feature extraction and fusion network enhances the target detection capability in a complex environment, and the rotating bounding box regression strategy solves the problem that an existing horizontal bounding box is not accurately positioned when a target rotates.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a method for rotating target detection under the visual angle of a UAV by combining visible light and infrared light images. BACKGROUND

[0002] In recent years, with the maturity of UAV technology, UAVs equipped with camera equipment are widely used in target detection tasks in the fields of traffic monitoring, security patrol, disaster rescue, etc. The existing technology mainly relies on processing image information of a single light source (such as visible light or infrared light only), which has obvious shortcomings in some cases.

[0003] A visible light camera can provide clear images when the light is sufficient, but the effect is poor at night or in low light conditions; while an infrared thermal imaging sensor can work at night or in bad weather, but its image resolution is low and is easily affected by the environment temperature. The dual-light image target detection method can fuse the visible light and infrared light image information in complex environments, which can increase the adaptability of the detection algorithm, but the existing dual-light image target detection method is mostly for vehicle-mounted visual angle or other ground visual angle, and uses a horizontal bounding box to represent the position of the target, while in the visual angle of a UAV, the target usually rotates, and the horizontal bounding box cannot accurately describe the target position. SUMMARY

[0004] In order to solve the technical problem that the existing dual-light image target detection method cannot be applied to the visual angle of a UAV, the present application proposes a "dual-light image target detection method under the visual angle of a UAV", which includes the following steps:

[0005] Step 1: constructing a dual-light image target detection model, including a dual-light feature extraction and fusion network, a feature pyramid network and a rotating target detection head network, pre-processing the visible light and infrared light image data pairs under the visual angle of a UAV;

[0006] Step 2: inputting the pre-processed image pair into the dual-light feature extraction and fusion network, the dual-light feature extraction and fusion network including a visible light image branch and an infrared light image branch, the two branch network structures being the same, both being composed of convolution layers and a series of residual blocks, and adding a cross-branch spatial attention and feature fusion module between the residual blocks, for adaptively fusing the information in the visible light image and the infrared light image at different levels to obtain fusion feature maps at different levels;

[0007] The spatial attention and feature fusion module comprises three branches, two of which are a visible light feature enhancement branch and an infrared light feature enhancement branch, have the same network structure, and can enhance local features to output enhanced feature maps; the other branch is a feature interaction branch, which realizes fine-grained interaction between the input visible light and infrared light feature maps, and outputs an interaction feature map; the visible light enhanced feature map, the infrared light enhanced feature map and the interaction feature map output by the three branches are spliced and integrated along the channel dimension to output a final fusion feature map;

[0008] Step 3: input the fusion feature maps at different levels into a feature pyramid network, the feature pyramid network further fuses multi-scale information in the multiple level feature maps, and outputs a multi-scale fusion feature map;

[0009] Step 4: input the multi-scale fusion feature map into a rotating target detection head network, and regress to obtain the rotating bounding box, class and confidence of the target under the perspective of the unmanned aerial vehicle, the rotating target detection head network comprises an anchor box adjustment network, an anchor box alignment convolution network and a regression prediction network, the anchor box adjustment network is used for adjusting the initial horizontal anchor box, and the adjusted rotating anchor box is used for guiding the anchor box alignment convolution network to extract anchor box alignment fusion features; the anchor box alignment convolution network comprises a plurality of anchor box alignment convolution layers, and the sampling positions of the convolution operations are adjusted under the guidance of the rotating anchor box; the regression prediction network can regress the rotating target detection result by using the extracted alignment features.

[0010] Compared with the prior art, the technical effects of the present application are as follows:

[0011] The application discloses a dual-light image target detection method under the perspective of an unmanned aerial vehicle, which enhances the target detection capability in a complex environment by constructing a dual-light feature extraction and fusion network, and adds a spatial attention and feature fusion module in the dual-light feature extraction and fusion network to adaptively enhance and finely interact the features of two branches, thereby generating fusion features.

[0012] The rotation bounding box regression strategy is introduced through the rotating target detection head network, the problem that the existing horizontal bounding box is not accurate in positioning when the target rotates is solved, the preset horizontal anchor box is preliminarily adjusted through the anchor box adjustment network to generate a rotating anchor box, which reduces the gap between the anchor box and the rotating target, and helps to more accurately regress the rotating bounding box. Meanwhile, the rotating anchor box can guide the subsequent feature extraction process, and improve the spatial consistency between the features and the target. The anchor box adjustment network and the anchor box alignment convolution network reduce the regression difficulty of the rotating bounding box, and improve the precision and efficiency of the rotating target detection under the perspective of the unmanned aerial vehicle.

[0013] The application is suitable for visible light-infrared light target detection under the perspective of a UAV, and provides an effective solution for target detection of the UAV in the fields of traffic monitoring, security patrol, disaster rescue and the like, and has a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 It is a whole flowchart of the application and serves as an abstract drawing.

[0015] Figure 2 It is a network structure diagram of double-light feature extraction and fusion in step 2.

[0016] Figure 3 It is a structure diagram of a spatial attention and feature fusion module 1 in the double-light feature extraction and fusion network.

[0017] Figure 4 It is a network structure diagram of a rotating target detection head in step 4. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. All other embodiments obtained by those skilled in the art without creative labor on the basis of the described embodiments belong to the protection scope of the application.

[0019] The application takes a double-light image vehicle detection task under the perspective of a UAV as an embodiment, and specifically describes the working process of the application. The embodiment implemented according to the content of the application is as follows:

[0020] (1) A visible light camera and an infrared thermal imaging sensor carried by a UAV are used to synchronously collect a visible light-infrared light double-light image data pair containing a vehicle target under the accurate positioning of a GPS and an inertial navigation system.

[0021] (2) As shown in Figure 1 , a double-light image vehicle detection network model under the perspective of a UAV is constructed, including double-light data preprocessing, a double-light feature extraction and fusion network, a feature pyramid network and a rotating target detection head network.

[0022] As shown in Figure 2 , the double-light image vehicle detection network model under the perspective of a UAV is constructed, including double-light data preprocessing, a double-light feature extraction and fusion network, a feature pyramid network and a rotating target detection head network.As shown, a dual-light feature extraction and fusion network is constructed, including a visible light image branch and an infrared light image branch, the two branch network structures are the same, which is composed of 1 convolutional layer and 5 residual blocks, which are sequentially denoted as residual block 1, residual block 2, residual block 3, residual block 4, and residual block 5. Each residual block is composed of 1 convolutional layer with a step of 2 and multiple residual structures stacked. The convolutional layer with a step of 2 can effectively down-sample while maintaining high feature information integrity, not only improving the learning ability and accuracy of the model, but also making the network structure more simple and unified, achieving a better balance between detection accuracy and computational efficiency. Each residual structure contains 2 convolutional layers, and the number of residual structures stacked in residual block 1, residual block 2, residual block 3, residual block 4, and residual block 5 is 1, 2, 4, 8, and 2, respectively.

[0023] A cross-branch spatial attention and feature fusion module is added between the residual blocks 3, 4, and 5 of the two branches. Specifically, the visible light image and the infrared light image pass through the convolutional layer and the residual blocks 1, 2, 3, respectively, to output the visible light feature map and the infrared light feature map. The visible light feature map and the infrared light feature map are input into the spatial attention and feature fusion module 1 to obtain the fusion feature Figure 1 ; the visible light feature map and the infrared light feature map continue to pass through the residual blocks 4 and 5, respectively, and are input into the corresponding spatial attention and feature fusion module to obtain another two levels of fusion features Figure 2 and fusion feature Figure 3 , Figure 2 fusion feature Figure 1 ~ fusion feature Figure 3 , which are the dual-light fusion features after each level of enhancement and fusion.

[0024] The spatial attention and feature fusion module enhances and fuses the features independently of the feature extraction process of the two branches, and the feature extraction is not affected by the fusion, maintaining the independence of the feature extraction of the visible light image branch and the infrared light image branch.

[0025] As shown in Figure 3 , the visible light feature map and the infrared light feature map are output after the visible light image branch and the infrared light image branch pass through the respective residual blocks 3, and are input into the spatial attention and feature fusion module 1 to obtain the fusion feature Figure 1 . As an example, the spatial attention and feature fusion module 1 includes three branches, two of which are the visible light feature enhancement branch and the infrared light feature enhancement branch, with the same network structure. The spatial attention mechanism is applied to process the input visible light feature map and infrared light feature map, respectively, to enhance the local features and output the enhanced feature map.

[0026] Another branch is a feature interaction branch, which realizes fine-grained interaction between the input visible light feature map and infrared light feature map, and outputs an interaction feature map. The visible light enhanced feature map, the infrared light enhanced feature map and the interaction feature map output by the three branches are spliced along the channel dimension, and then pass through a residual block 6 to further integrate and enhance the feature expression, and finally output the fused feature Figure 1 Here, the residual block 6 does not perform down-sampling and only contains a plurality of stacked residual structures.

[0027] Further, the visible light feature enhancement branch and the infrared light feature enhancement branch use a spatial attention module (CBAM) to enhance local details and suppress irrelevant areas, so that the features are more focused on the details of the target area. First, the input visible light feature map and infrared light feature map are respectively subjected to maximum pooling and average pooling, and the feature maps are compressed along the channel dimension to obtain two feature maps. The two feature maps are stacked along the channel dimension to obtain a two-channel stacked feature map. Convolution operation is performed on the stacked feature map to obtain a spatial attention map. Then, the spatial attention map is used to weight the feature map, and the spatial attention map and the input feature map are multiplied element by element (Hadamard product) to obtain the visible light enhanced feature map and the infrared light enhanced feature map.

[0028] Further, the feature interaction branch first performs element-by-element multiplication (Hadamard product) on the input visible light and infrared light feature maps to realize fine-grained interaction between them, and then applies convolution operation to the interacted features to further integrate feature information and obtain fine-grained interaction features, and outputs an interaction feature map.

[0029] After the dual-light feature extraction and fusion network, the present application adds a feature pyramid network, which inputs the fusion features Figure 3 of different levels Figure 1 , fusion features Figure 2 and fusion features Figure 3 into the feature pyramid network for multi-scale fusion, and outputs multi-scale fusion feature maps for predicting vehicle targets of different sizes at different scales.

[0030] In the image of the ground view, the relative position between the collection device and the target is constrained by the ground plane, so the target usually remains in a horizontal state. However, in the overhead view of the unmanned aerial vehicle, the orientation change of the target will appear as a rotation at an arbitrary angle in the image. When the existing ground view horizontal target detection method is applied to the image of the unmanned aerial vehicle view, two main problems will be encountered. First, the horizontal bounding box cannot accurately locate the rotated target, especially when detecting long and wide targets such as vehicles. In addition, when the target rotates, the sampling position of the convolution operation used in the existing target detection may deviate from the target, which cannot effectively extract the target features, thereby affecting the detection performance. Therefore, it is necessary to carry out targeted rotation target detection.

[0031] As shown in Figure 4 The specific processing procedure of the rotating target detection head network is as follows: the multi-scale fusion feature map is input into an anchor box adjustment network of a corresponding scale for parameter regression; the output of the anchor box adjustment network is decoded; anchor box adjustment parameters are calculated according to the network output, the initial horizontal anchor box is adjusted according to the anchor box adjustment parameters, and the corresponding rotating anchor box is obtained; the anchor box alignment displacement matrix is calculated according to the rotating anchor box; the fusion feature and the alignment displacement matrix are synchronously input into an anchor box alignment convolution network, and the anchor box alignment fusion feature is extracted; and the anchor box alignment fusion feature is input into a regression prediction network, and the final target rotating bounding box and the category prediction result are regressed on the basis of the rotating anchor box.

[0032] The multi-scale fusion feature map output by the feature pyramid network has a corresponding anchor box adjustment network, and each position on the feature map corresponds to an anchor box. The anchor box adjustment network outputs a set of six parameters for each position, including two displacement parameters (x, y), two scaling parameters (w, h), one rotation angle parameter and one confidence parameter p, which represents the possibility of the existence of a target at the position. According to these anchor box adjustment parameters, the initial horizontal anchor box is adjusted to generate a corresponding rotating anchor box. Anchor boxes with a confidence lower than a preset threshold are not adjusted to reduce the amount of calculation.

[0033] The guiding effect of the rotating anchor box is realized through the anchor box alignment displacement matrix, which is calculated according to the rotating anchor box and used to adjust the sampling position of the conventional two-dimensional convolution, so as to better capture the features of the rotating target. The matrix has the same size as the input fusion feature map, and each element corresponds to a position on the feature map, recording the displacement amount of each sampling position relative to the sampling position of the conventional convolution operation when performing anchor box alignment convolution. For a conventional 3x3 convolution kernel, each matrix element contains nine displacement vectors. Similar to anchor box adjustment, the anchor box alignment displacement matrix is calculated only for positions where a target may exist. The fusion feature and the alignment displacement matrix are synchronously input into the anchor box alignment convolution network to extract the anchor box alignment fusion feature, and these features are input into the regression prediction network to regress the final target rotating bounding box and the category prediction result on the basis of the rotating anchor box.

[0034] (3) Preprocess the input visible light and infrared light image pair under the unmanned aerial vehicle perspective, including registration, standardization and enhancement processing. First, the visible light and infrared light images are registered to align the two modalities in space, ensuring consistency in feature extraction and fusion. Then, the registered images are standardized, including adjusting size, brightness and contrast, to adapt to different lighting conditions. Finally, data augmentation processing is applied, including rotation, flipping and scaling, to increase the diversity of the data and improve the robustness of the dual light image vehicle detection network model under various environmental conditions. These preprocessing steps lay the foundation for subsequent feature extraction and fusion, enhancing the adaptability and detection performance of the dual light image vehicle detection network model.

[0035] (4) The preprocessed visible light and infrared light image pair is input into the dual light feature extraction and fusion network. Specifically, the visible light image is input into the visible light image branch, and the infrared light image is input into the infrared light image branch. The dual light feature extraction and fusion network generates fusion feature maps of visible light and infrared light at three levels.

[0036] (5) The three different levels of fusion feature maps are input into the feature pyramid network to integrate multi-scale information and generate multi-scale fusion feature maps.

[0037] (6) The multi-scale fusion feature maps are input into the corresponding scale of the rotated object detection head network, and the anchor box adjustment network generates rotated anchor boxes according to the input features.

[0038] (7) According to the rotated anchor boxes, the anchor box alignment displacement matrix is calculated to guide the anchor box alignment convolution network to extract anchor box alignment fusion features.

[0039] (8) The anchor box alignment fusion features are input into the regression prediction network, and the rotated bounding box of the vehicle target and the vehicle class prediction results are obtained based on the rotated anchor boxes.

[0040] The preferred embodiments of the present application are described in detail in combination with the drawings, but the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, various simple modifications can be made to the technical solutions of the present application, and these simple modifications all belong to the protection scope of the present application.

Claims

1. A method for dual-light image target detection under the perspective of a UAV, characterized in that, The method comprises the following steps: Step 1: constructing a dual-optical image target detection model, including a dual-optical feature extraction and fusion network, a feature pyramid network and a rotating target detection head network, and pre-processing visible light and infrared light image data pairs under the visual angle of a UAV; Step 2: inputting the pre-processed image pairs into the dual-optical feature extraction and fusion network, wherein the dual-optical feature extraction and fusion network comprises a visible light image branch and an infrared light image branch, the network structures of the two branches are the same, and each branch is composed of a convolution layer and a series of residual blocks, a cross-branch spatial attention and feature fusion module is added between the residual blocks, which is used for adaptively fusing the information in the visible light image and the infrared light image at different levels to obtain fused feature maps at different levels; the spatial attention and feature fusion module comprises three branches, two of which are a visible light feature enhancement branch and an infrared light feature enhancement branch, have the same network structure, and can enhance local features to output enhanced feature maps; the other branch is a feature interaction branch, which realizes fine-grained interaction between the input visible light and infrared light feature maps, and outputs an interaction feature map; the visible light enhanced feature map, the infrared light enhanced feature map and the interaction feature map output by the three branches are spliced and integrated along the channel dimension to output a final fused feature map; Step 3: inputting the fused feature maps at different levels into the feature pyramid network, and the feature pyramid network further fuses multi-scale information in the multiple level feature maps to output multi-scale fused feature maps; Step 4: inputting the multi-scale fused feature maps into the rotating target detection head network to regress the rotating bounding box, the class and the confidence of the target under the visual angle of the UAV, wherein the rotating target detection head network comprises an anchor box adjustment network, an anchor box alignment convolution network and a regression prediction network, the anchor box adjustment network is used for adjusting the initial horizontal anchor box, and the adjusted rotating anchor box is used for guiding the anchor box alignment convolution network to extract anchor box alignment fused features; the anchor box alignment convolution network comprises a plurality of anchor box alignment convolution layers, and the sampling positions of the convolution operations are adjusted under the guidance of the rotating anchor box; the regression prediction network can regress the rotating target detection result by using the extracted alignment features. 2.The method of claim 1, wherein: In step 1, the pre-processing specifically comprises: firstly, performing registration processing on the visible light and infrared light images to align the two modalities in space and ensure the consistency of feature extraction and fusion; then performing standardization processing on the registered images, including adjusting the size, brightness and contrast; finally applying data enhancement processing. 3.The method of claim 1, wherein: In step 2, the network structures of the visible light image branch and the infrared light image branch of the dual-optical feature extraction and fusion network are the same, each branch is composed of 1 convolution layer and 5 residual blocks, and each residual block is sequentially marked as residual block 1, residual block 2, residual block 3, residual block 4 and residual block 5; a cross-branch spatial attention and feature fusion module is added between the residual blocks 3, 4 and 5 of the two branches; The visible light enhanced feature map, the infrared light enhanced feature map and the interaction feature map output by the visible light feature enhancement branch, the infrared light feature enhancement branch and the feature interaction branch of the spatial attention and feature fusion module are spliced along the channel dimension, and then pass through the residual block 6 to further integrate and enhance the feature expression, and output the final fusion feature map.

4. The dual-light image target detection method under the perspective of the UAV according to claim 3, characterized in that: The residual block 1, the residual block 2, the residual block 3, the residual block 4 and the residual block 5 are composed of one convolution layer with a step of 2 and a plurality of residual structures stacked, the convolution layer can realize effective down sampling while maintaining high feature information integrity, the number of residual structures stacked in the residual block 1, the residual block 2, the residual block 3, the residual block 4 and the residual block 5 is 1, 2, 4, 8 and 2 respectively, and the residual block 6 comprises a plurality of stacked residual structures.

5. The method of claim 3, wherein the method further comprises: The visible light feature enhancement branch and the infrared light feature enhancement branch adopt a spatial attention module, respectively perform maximum pooling and average pooling on the input visible light feature map and infrared light feature map, compress the feature maps along the channel dimension to obtain two feature maps, stack the two feature maps along the channel dimension to obtain a two-channel stacked feature map, and perform convolution operation on the stacked feature map to obtain a spatial attention map; the obtained spatial attention map is used for feature weighting, the spatial attention map and the input feature map are multiplied element by element to obtain the visible light enhanced feature map and the infrared light enhanced feature map; the specific processing process of the feature interaction branch is that the input visible light feature map and infrared light feature map are multiplied element by element, and then convolution operation is applied to obtain fine-grained interaction features, and an interaction feature map is output.

6. The dual-light image target detection method under the UAV perspective according to claim 1, characterized in that: The specific processing process of the rotating target detection head network is that the multi-scale fusion feature map is input into the anchor box adjustment network of the corresponding scale for parameter regression; the output of the anchor box adjustment network is decoded: according to the network output, the anchor box adjustment parameters are calculated, the initial horizontal anchor box is adjusted according to the anchor box adjustment parameters to obtain the corresponding rotating anchor box; According to the rotating anchor box, the anchor box alignment displacement matrix is calculated; the fusion feature and the alignment displacement matrix are input into the anchor box alignment convolution network at the same time to extract the anchor box alignment fusion feature; The anchor box alignment fusion feature is input into the regression prediction network, and the final target rotating bounding box and its category prediction result are obtained based on the rotating anchor box.

7. The method of claim 6, wherein the method further comprises: Each position on the multi-scale fusion feature map corresponds to 1 anchor box, and the anchor box adjustment network outputs a set of 6 parameters for each position, including 2 displacement parameters (x, y), 2 scaling parameters (w, h), 1 rotation angle parameter, and 1 confidence parameter p to represent the possibility of the presence of a target at the position. and 1 confidence parameter p to represent the possibility of the presence of a target at the position.

Images