Dual-mode unmanned aerial vehicle recognition method for adaptive fusion of visible light and infrared images

Abstract

The application discloses a dual-mode unmanned aerial vehicle recognition method for self-adaptive fusion of visible light and infrared images, relates to the technical field of computer vision, and solves the technical problem of poor detection effect of an unmanned aerial vehicle under a complex environment by using a visible light camera alone, and the technical solution points are as follows: the complementarity of visible light images and infrared images is utilized; a self-adaptive fusion module is used to distribute weights to dual-mode inputs according to image feature quality, and then a light dual-mode network is used to perform real-time recognition on the unmanned aerial vehicle. The algorithm can meet the real-time requirement, can recognize a micro target under a complex light environment, can cope with the unmanned aerial vehicle detection task under the complex environment, and has a good application prospect in the military field. Moreover, the network structure is simple, the calculation amount is small, the detection effect on a small target is good, and the network has strong robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to multi-modal perception technology, and more particularly to a dual-modal unmanned aerial vehicle recognition method for adaptively fusing visible light and infrared images. BACKGROUND

[0002] With the development of unmanned aerial vehicle technology, unmanned aerial vehicles have become an important new type of combat weapon in military technology wars. The application of unmanned aerial vehicles on the battlefield mainly includes reconnaissance and surveillance, precision strikes and swarm combat. More importantly, unmanned aerial vehicles are replacing some human operations with higher efficiency and performance due to their simple structure and light weight. Unmanned aerial vehicles have been widely used in civilian fields such as exploration, rescue, reconnaissance and express delivery, but the detection task of unmanned aerial vehicles is often affected by poor lighting conditions, occlusion and noise points.

[0003] Currently, unmanned aerial vehicle detection methods mainly include audio signal analysis, radar data analysis, radio frequency signal analysis and computer vision technology. Muhammad et al. proposed a new machine learning framework for detecting and classifying various sounds in a noisy environment, using sound detection unmanned aerial vehicles, which is not suitable in a noisy environment. Cheng et al. proposed a radar detection and tracking method for detecting unmanned aerial vehicles in a cluttered environment, but in snowy weather, the detection range of the radar is very short. Liu et al. developed a small target detection method based on YOLOv3 from the perspective of a special unmanned aerial vehicle, but visual-based detection methods will fail in poor lighting conditions. Sonain et al. detected and located malicious unmanned aerial vehicles through sound and image information, which is computationally intensive and not suitable for onboard computers. Current target detection algorithms are mainly divided into two-stage methods and one-stage methods. Two-stage methods first generate region proposals, and then classify samples through convolutional neural networks. Common two-stage target detection algorithms are R-CNN, SPPNet, Fast R-CNN and R-FCN

[19] , etc. One-stage methods directly extract features through convolutional neural networks to predict target classification and location, including YOLO series networks and SSD. However, most detection algorithms only have visible light pictures as input, which means that the effect of the algorithm is highly dependent on lighting conditions. Therefore, a lightweight and efficient micro unmanned aerial vehicle detection algorithm with strong robustness is an important technology for the development of unmanned aerial vehicle applications.

[0004] How to build a lightweight and efficient dual-modal unmanned aerial vehicle detection network to enable unmanned aerial vehicles to complete tasks stably in complex scenes is a problem that needs to be solved. SUMMARY

[0005] The application provides a dual-mode unmanned aerial vehicle recognition method for adaptively fusing visible light and infrared images, which aims to improve the detection capability of small targets, improve the robustness of the algorithm, and enable unmanned aerial vehicles to complete more complex tasks.

[0006] The above technical purpose of the application is achieved by the following technical solutions:

[0007] A dual-mode unmanned aerial vehicle recognition method for adaptively fusing visible light and infrared images, which is realized by a deep learning network including a first backbone network, a second backbone network, an adaptive fusion module, a channel attention module, and an FPN structure. The method comprises:

[0008] S1: Obtain the visible light image and the infrared image of the unmanned aerial vehicle, input the visible light image into the first backbone network, and input the infrared image into the second backbone network;

[0009] S2: Extract the visible light feature map from the visible light image by the first backbone network, and extract the infrared feature map from the infrared image by the second backbone network, then input the visible light feature map and the infrared feature map into the adaptive fusion module;

[0010] S3: The adaptive fusion module assigns weights to the visible light image and the infrared image according to the feature quality of the visible light feature map and the infrared feature map, multiplies the visible light feature map and the infrared feature map by their respective weights, and splices the weighted visible light feature map and the infrared feature map to obtain a feature map p0;

[0011] S4: The channel attention module assigns weights w c to the feature map p0 in each channel, multiplies w c with the feature map p0 to obtain a feature map p1;

[0012] S5: Perform multi-scale feature fusion operation on the feature map p1 by the FPN structure to obtain a detection layer p5;

[0013] S6: Obtain the unmanned aerial vehicle recognition result from the detection layer p5.

[0014] The application has the beneficial effects that the dual-mode unmanned aerial vehicle recognition method for adaptively fusing visible light and infrared images uses the characteristics of visible light images and infrared images as the input of the network, assigns different weights to the visible light image and the infrared image by the adaptive fusion module, and then outputs the detection layer by fusing multi-scale features through the FPN structure, finally obtains the detection result.

[0015] The unmanned aerial vehicle recognition method can not only detect micro targets in real time, but also stably detect unmanned aerial vehicles in an environment with poor light conditions through adaptive feature fusion of visible light images and infrared images, and has a good application prospect. Experiments prove that the algorithm has simple structure, low calculation amount, is suitable for small target detection, and has strong robustness. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 a flowchart of the method described in the application;

[0017] Figure 2 a schematic diagram of the adaptive fusion module;

[0018] Figure 3 a schematic diagram of the channel attention module;

[0019] Figure 4 a structural composition schematic diagram of the recognition network. DETAILED DESCRIPTION

[0020] The technical solutions of the application will be described in detail below with reference to the drawings.

[0021] The adaptive fusion of visible light and infrared images of the dual-mode unmanned aerial vehicle recognition method described in the application is realized through a deep learning network, and the deep learning network includes a first backbone network, a second backbone network, an adaptive fusion module, a channel attention module and an FPN structure. As shown in the figure, the method includes: Figure 1

[0022] S1: Obtain the visible light image and the infrared image of the unmanned aerial vehicle, input the visible light image into the first backbone network, and input the infrared image into the second backbone network.

[0023] S2: Extract features of the visible light image through the first backbone network to obtain a visible light feature map, extract features of the infrared image through the second backbone network to obtain an infrared feature map, and then input the visible light feature map and the infrared feature map into the adaptive fusion module.

[0024] Specifically, the visible light image is obtained by a visible light camera, and has a width*height*channel number of 640*640*3; the infrared image is obtained by an infrared camera, and has a width*height*channel number of 640*640*1, and the infrared image is converted into a three-channel image. Finally, the visible light feature map and the infrared feature map with a size of 640*640*8 are obtained through the backbone network.

[0025] ​S3: The adaptive fusion module performs weight distribution on the visible light image and the infrared image according to the feature quality of the visible light feature map and the infrared feature map, multiplies the visible light feature map and the infrared feature map by respective weights, and splices the weighted visible light feature map and the weighted infrared feature map to obtain a feature map p0.

[0026] Specifically, as shown in the figure, Figure 2 if the adaptive fusion module inputs a feature map with a width*height*channel number of W*H*3, performs a maximum pooling operation and an average pooling operation on the feature map to obtain a maximum pooling layer with a width*height*channel number of W*H*1 and an average pooling layer with a width*height*channel number of W*H*1, splices the maximum pooling layer with a width*height*channel number of W*H*1 and the average pooling layer with a width*height*channel number of W*H*1 directly to form a feature map with a width*height*channel number of W*H*2, inputs the feature map with a width*height*channel number of W*H*2 to an artificial convolution layer to obtain a spatial feature map with a width*height*channel number of W*H*1, adds the maximum value and the average value of the spatial feature map to obtain a score value, and finally inputs the score value into a Sigmoid activation function to obtain a pre-weight ω.

[0027] The weight w v of the visible light feature map and the weight w i of the infrared feature map are respectively represented as:

[0028]

[0029] f d =Conv(Concat(MaxPool(F),AvgPool(F)));

[0030] ω=Sig(Mean(f d )+Max(f d ));

[0031] wherein w1 represents the pre-weight of the visible light feature map; w2 represents the pre-weight of the infrared feature map; ω represents the pre-weight; f d represents a spatial feature map; F represents an input visible light feature map or an infrared feature map; MaxPool(F) represents a maximum pooling operation on a feature map; AvgPool(F) represents an average pooling operation on a feature map; Concat represents splicing of the feature maps subjected to the maximum pooling operation and the average pooling operation in a certain dimension; Conv represents a convolution layer; Max(f d ) represents a maximum value on a spatial feature map; Mean(f d ) represents a global average value of a spatial feature map; and Sig represents a Sigmoid activation function.

[0032] The feature map p0 is obtained by multiplying the visible light feature map by w v and multiplying the infrared feature map by w iIt is formed by multiplying and splicing, and its width * height * number of channels is W * H * 2C.

[0033] S4: Assign weights w to feature map p0 in each channel using the channel attention module. c , will w c Multiplying it by feature map p0 yields feature map p1.

[0034] like Figure 3 As shown, the channel attention module can assign different weights to each channel of the feature map. Specifically, it performs global max pooling and average pooling operations on the input feature map, then feeds the two new feature maps into a shared perceptron to generate two feature vectors, adds them together, and finally passes them through a channel sigmoid activation function to obtain the channel attention weight w. c The entire process of channel attention can be represented by the following formula:

[0035]

[0036] w c =Sig(MLP(AvgPool(p0))+MLP(MaxPool(p0)));

[0037] Where MLP stands for Multilayer Perceptron; w c That is, the weights of the channel attention module.

[0038] S5: The detection layer p5 is obtained by performing multi-scale feature fusion operation on the feature map p1 through the FPN structure.

[0039] Specifically, the FPN structure has 11 layers, namely Conv layer, C3 layer, Conv layer, C3 layer, Conv layer, C3 layer, SPPF layer, Conv layer, Upsample layer, Concat layer and C3 layer.

[0040] Figure 4In the embodiment, the whole recognition network has 16 layers, in order to avoid losing small target information, the whole network adopts 8 times down-sampling, and a detection layer is adopted for small target detection. The feature pyramid network (FPN) structure can obtain multi-level fusion feature maps, and can effectively reduce the loss of small and medium targets. The main modules of the recognition network include a Conv module, a C3 module, a SPPF module, a Concat module, an Upsample module and an AF module. The Conv module is the most basic module, including a convolution layer, a batch normalization layer and a Silu activation function. The function of the C3 module is to learn the remaining characteristics. The SPPF module (single layer) is a spatial pyramid pool, which can obtain features from different scales. The Concat module is to directly splice two feature maps in a certain dimension. The Upsample module is an up-sampling module, which up-samples the size of the feature map to detect small targets. The AF module includes an adaptive fusion module and a channel attention module.

[0041] S6: obtaining the unmanned aerial vehicle recognition result by the detection layer p5.

[0042] Specifically, the detector detects on the detection layer, and the detection result is obtained by regression of the training loss function of the recognition network. The result mainly includes the detection frame position, the detected category and the confidence.

[0043] Since the recognition method described in the present application is only used for detecting one category of unmanned aerial vehicles, classification errors are not considered here, and the loss function of the recognition network is represented as:

[0044]

[0045]

[0046]

[0047] wherein, L EIOU represents an effective bounding box regression loss; ρ 2 represents the Euclidean distance between the center points of the prediction box and the target box, b represents the center point of the prediction box; b gt represents the center point of the target box; w represents the width of the prediction box; w gt represents the width of the target box; h represents the height of the prediction box; h gt represents the height of the target box; c represents the diagonal distance of the minimum box covering two detection boxes; C w represents the width of the minimum box covering two detection boxes; C h represents the height of the minimum box covering two detection boxes; L boj represents the confidence loss; p o represents the target confidence score in the prediction box; p iouan iou value representing a prediction box and a corresponding target box; represents a binary cross-entropy loss; w obj represents the weight of a positive sample;

[0048] K represents an output feature map, S 2 represents a unit cell of the output feature map, and B represents a prediction anchor box in each unit cell; represents whether the kth output feature map, the ith unit cell, and the jth anchor box are a positive sample; x p represents a prediction vector; x gt represents a true value vector; a box represents the weight of a position error; a obj represents the weight of a confidence error.

[0049] The above is an exemplary embodiment of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A dual-modal UAV recognition method that adaptively fuses visible light and infrared images, characterized in that, This recognition method is implemented using a deep learning network, which includes a first backbone network, a second backbone network, an adaptive fusion module, a channel attention module, and an FPN structure. The method includes: S1: Acquire visible light and infrared images of the UAV, input the visible light image into the first backbone network, and input the infrared image into the second backbone network; S2: The visible light image is extracted using the first backbone network to obtain a visible light feature map, and the infrared image is extracted using the second backbone network to obtain an infrared feature map. The visible light feature map and the infrared feature map are then input into the adaptive fusion module. S3: The adaptive fusion module assigns weights to the visible light image and the infrared image based on the feature quality of the visible light feature map and the infrared feature map, multiplies the visible light feature map and the infrared feature map by their respective weights, and then concatenates the weighted visible light feature map and the infrared feature map to obtain the feature map p0. S4: Assign weights w to feature map p0 in each channel using the channel attention module. c , will w c Multiply by feature map p0 to obtain feature map p1; S5: Perform multi-scale feature fusion operation on feature map p1 through FPN structure to obtain detection layer p5; S6: Obtain the drone identification result based on the detection layer p5; If the adaptive fusion module inputs a feature map with width * height * number of channels of W * H * 3, max pooling and average pooling operations are performed on the feature map to obtain a max pooling layer of W * H * 1 and an average pooling layer of W * H * 1, respectively. The max pooling layer of W * H * 1 and the average pooling layer of W * H * 1 are directly concatenated to form a feature map of W * H * 2. The feature map of W * H * 2 is then input into an artificial convolutional layer to obtain a spatial feature map of W * H * 1. The maximum value and the average value of the spatial feature map are added together to obtain a score value. Finally, the score value is passed into the Sigmoid activation function to obtain the preweight ω. In step S3, the weight w of the visible light feature map v The weight w of the infrared feature map i They are represented as follows: f d =Conv(Concat(MaxPool(F),AvgPool(F)))); ω=Sig(Mean(f d )+Max(f d )); Where w1 represents the pre-weight of the visible light feature map; w2 represents the pre-weight of the infrared feature map; ω represents the pre-weight; f d F represents the spatial feature map; F represents the input visible light feature map or infrared feature map; MaxPool(F) represents the max pooling operation on the feature map; AvgPool(F) represents the average pooling operation on the feature map; Concat represents the concatenation of the feature maps from the max pooling and average pooling operations along a certain dimension; Conv represents the convolutional layer; Max(f) represents the maximum pooling operation on the feature map ... d Mean(f) represents the maximum value on the spatial feature map; d ) represents the global average value of the spatial feature map; Sig represents the Sigmoid activation function.

2. The method as described in claim 1, characterized in that, The visible light image was obtained by a visible light camera and its width*height*channel number is 640*640*3; the infrared image was obtained by an infrared camera and its width*height*channel number is 640*640*1; after preprocessing and backbone network, the size of both the visible light feature map and the infrared feature map is 640*640*8.

3. The method as described in claim 1, characterized in that, In step S4, feature map p1 is represented as: w c =Sig(MLP(AvgPool(p0))+MLP(MaxPool(p0))); Where MLP stands for Multilayer Perceptron; w c That is, the weights of the channel attention module.

4. The method as described in claim 3, characterized in that, The loss function of the recognition network is expressed as: Among them, L EIOU This represents an effective bounding box regression loss; ρ 2 represents the Euclidean distance between points in the predicted bounding box and points in the target bounding box, and b represents the midpoint of the predicted bounding box; b gt w represents the midpoint of the target bounding box; w represents the width of the predicted bounding box; w gt h represents the width of the target bounding box; h represents the height of the predicted bounding box; h gt The height of the target bounding box is represented by 'c'; the diagonal distance between the two smallest bounding boxes covering the target bounding boxes is represented by 'c'. w This represents the width of the smallest bounding box covering the two detection boxes; C h L represents the height of the smallest bounding box covering the two detection boxes. obj p represents the confidence loss; o p represents the target confidence score in the prediction box. iou This represents the IoU value between the predicted bounding box and the corresponding target bounding box; Represents the cross-entropy loss in binary classification; w obj Indicates the weight of positive samples; K represents the output feature map, S 2 The cell represents the output feature map, and B represents the predicted anchor box in each cell; This indicates whether the k-th output feature map is a positive sample when mapping the i-th cell to the j-th anchor box; x p Represents the prediction vector; x gt Represents the truth vector; α box The weights representing the positional error; α obj The weights represent the confidence error.

5. The method as described in claim 1, characterized in that, Both the first backbone network and the second backbone network include Conv modules and C3 modules, and the output size of both the first backbone network and the second backbone network is 640*640*8.

6. The method as described in claim 1, characterized in that, The FPN structure has 11 layers, namely Conv layer, C3 layer, Conv layer, C3 layer, Conv layer, C3 layer, SPPF layer, Conv layer, Upsample layer, Concat layer and C3 layer.

Images