A method and device for detecting a weak and small target in an infrared remote sensing image of a UAV

Abstract

The present application relates to a kind of unmanned aerial vehicle infrared remote sensing image weak small target detection method and device, method includes: obtaining the infrared remote sensing image to be measured;The infrared remote sensing image to be measured is input into the target detection model of preestablished, and weak small target detection result is output, wherein, the target detection model is obtained by input training set, and is trained using the WIoU loss function based on distance attention mechanism, the training set includes infrared remote sensing image and corresponding target label, the target detection model is constructed by introducing scale internal feature efficient interaction mechanism and non-maximum suppression mechanism in the target detector based on yolo.The present application can effectively utilize the feature information in infrared remote sensing image, improve the detection accuracy and robustness of weak small target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of infrared weak small target detection, in particular to a UAV infrared remote sensing image weak small target detection method and device. BACKGROUND

[0002] As a new emerging safe and efficient detection method, UAV infrared remote sensing image target detection technology has the potential to be widely applied in the field of infrared weak small target detection. UAV infrared remote sensing system can operate all-weather and high-precision, making it possible to detect infrared weak small targets in complex backgrounds. UAV infrared remote sensing image target detection technology has wide coverage and strong flexibility, and is suitable for disaster monitoring, military reconnaissance, etc.

[0003] In recent years, the rapid development of deep learning technology has provided powerful tools for image recognition and detection. However, traditional target detectors have certain limitations in UAV infrared remote sensing image target detection. Existing target detectors are not ideal in terms of speed and lightweight, and cannot meet the application requirements of real-time or mobile devices and other edge devices. Moreover, the weak small targets in UAV infrared remote sensing images are usually small in size and low in contrast, and traditional detection methods have challenges in accuracy and robustness. Therefore, developing a faster, lighter and more accurate target detection algorithm is of great significance for UAV infrared remote sensing image target detection. SUMMARY

[0004] The purpose of the present application is to solve the problems of low detection accuracy, high weak small target missing detection rate and high computational complexity of existing algorithms, and to provide a UAV infrared remote sensing image weak small target detection method and device. The scale-in feature efficient interaction and fusion module, the WIoU loss function based on distance attention mechanism and the soft-NMS are introduced, and at the same time, a lightweight network is designed to reduce the complexity of the network, which can effectively utilize the feature information in the infrared remote sensing image, improve the detection accuracy and robustness of the weak small target.

[0005] To achieve the above purpose, on the one hand, the present application provides a UAV infrared remote sensing image weak small target detection method, comprising:

[0006] acquiring an infrared remote sensing image to be tested;

[0007] inputting the infrared remote sensing image to be tested into a preset target detection model, and outputting a weak small target detection result, wherein the target detection model is obtained by inputting a training set and training using a WIoU loss function based on a distance attention mechanism, the training set comprises an infrared remote sensing image and a corresponding target label, and the target detection model is constructed by introducing a scale-in feature efficient interaction mechanism and a non-maximum suppression mechanism in a yolo-based target detector.

[0008] Optionally, the acquiring the training set comprises:

[0009] Acquiring a plurality of original infrared remote sensing images containing dim targets, and converting the original infrared remote sensing images from a single channel to a pseudo three-channel RGB format;

[0010] Labeling the dim target class and the upper left corner and lower right corner coordinates of the dim target in the infrared remote sensing image after format conversion, and acquiring a final infrared remote sensing image.

[0011] Optionally, the target detection model comprises:

[0012] A backbone network for feature extraction, interaction and fusion of an input image, outputting a plurality of first feature maps of different scales;

[0013] A neck network for feature splicing and extraction of the plurality of first feature maps of different scales, outputting a plurality of second feature maps of different scales;

[0014] A head network for dim target detection of the plurality of second feature maps of different scales, outputting a detection frame.

[0015] Optionally, in the backbone network, the input image is subjected to a convolution layer, a feature extraction module, a fast spatial pyramid pooling layer, and a cross-stage efficient attention mechanism, outputting three feature maps I b1 , I b2 and I b3 of different scales, the feature maps I b1 and I b2 are connected to the feature extraction module and the neck network through feature splicing, the feature map I b3 is connected to the cross-stage efficient attention mechanism and the neck network through feature splicing, and the feature map I b3 is subjected to self-attention operation and scale-intra feature interaction through a scale-intra feature efficient interaction and fusion module, and is connected to the feature splicing in the neck network, the neck network outputs three feature maps I n1 , I n2 and I n3 of different scales, which are input into detection heads in the head network, and output dim target detection frames.

[0016] Optionally, in the scale-intra feature efficient interaction and fusion module, two-dimensional sine-cosine position encoding is added to the input feature, and then the feature is subjected to a multi-head attention mechanism, residual connection and layer normalization, and a feedforward fully connected network, outputting the interaction and fusion feature.

[0017] Optionally, after obtaining a plurality of weak small target detection boxes of the head network output, a non-maximum suppression mechanism is combined for splicing and dimension transformation operation to output a final weak small target detection box.

[0018]

[0019] wherein s i is a boundary box score, b i is an initial boundary box, is a boundary box with the maximum score, D is a final result box set, and sigma is a standard deviation of a Gaussian function.

[0020] Optionally, the WIoU loss function based on the distance attention mechanism is:

[0021] L WIoUv1 = R WIoU * L IoU ;

[0022]

[0023] wherein L WIoUv1 is a WIoU v1 function, L IoU is an IOU loss function, R WIoU is a penalty term of WIOU, x and y are coordinates of the upper left corner of an anchor box, x gt and y gt are coordinates of the upper left corner of a target box, W g and H g represent the width and height of the minimum boundary box, and the superscript * indicates that W g and H g are separated from the calculation graph.

[0024] In another aspect, the application also provides a weak small target detection device for an unmanned aerial vehicle infrared remote sensing image, comprising:

[0025] An image acquisition device is configured to acquire an infrared remote sensing image to be measured.

[0026] An image processing device is configured to input the infrared remote sensing image to be measured into a preset target detection model to output a weak small target detection result, wherein the target detection model is obtained by inputting a training set and training using a WIoU loss function based on a distance attention mechanism, the training set comprises an infrared remote sensing image and a corresponding target label, and the target detection model is constructed by introducing a scale-in feature efficient interaction mechanism and a non-maximum suppression mechanism into a yolo-based target detector.

[0027] The application has the following beneficial effects:

[0028] The application effectively captures the detailed features of the weak and small target in the infrared remote sensing image by introducing the scale-in feature efficient interaction and fusion module, realizes the cross-channel and cross-space information fusion between the feature maps. At the same time, the detection accuracy is improved and the missed detection is reduced by using the distance attention mechanism based WIoU loss function and the non-maximum suppression soft-NMS. The application has the advantages of rapidity, lightness and accuracy, is suitable for real-time application and various scenes of infrared image weak and small target detection, and has wide application prospect in the fields of environmental monitoring and military reconnaissance. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0030] Figure 1 A schematic diagram of the composition structure of the unmanned aerial vehicle infrared remote sensing image weak and small target detection device of the embodiment of the present application, wherein 101 is a computer, 102 is an unmanned aerial vehicle infrared remote sensing imaging device, 103 is a transceiver device, and 104 is a detected area.

[0031] Figure 2 A schematic diagram of the target detection model structure of the embodiment of the present application.

[0032] Figure 3 A schematic diagram of the scale-in feature efficient interaction and fusion module structure of the embodiment of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0034] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0035] The embodiment of the present application provides an unmanned aerial vehicle infrared remote sensing image weak and small target detection device, which comprises:

[0036] An image acquisition device is used to acquire an infrared remote sensing image to be measured.

[0037] The image processing device is used for inputting the to-be-detected infrared remote sensing image into a preset target detection model, and outputting a weak small target detection result, wherein the target detection model is obtained by inputting a training set and using a distance attention mechanism-based WIoU loss function for training, the training set comprises infrared remote sensing images and corresponding target labels, and the target detection model is constructed by introducing a scale-in feature efficient interaction mechanism and a non-maximum suppression mechanism in a yolo-based target detector.

[0038] Specifically, as shown in the figure, Figure 1 In the embodiment, the image acquisition device is an unmanned aerial vehicle infrared remote sensing imaging device 102, which collects infrared remote sensing images of a detected area 104, and sends the images to the image processing device, i.e., a computer 101, through a transceiver device 103, so as to realize weak small target detection in the infrared remote sensing images.

[0039] Based on the above device, the embodiment further provides a weak small target detection method for unmanned aerial vehicle infrared remote sensing images, comprising:

[0040] acquiring a to-be-detected infrared remote sensing image;

[0041] inputting the to-be-detected infrared remote sensing image into a preset target detection model, and outputting a weak small target detection result, wherein the target detection model is obtained by inputting a training set and using a distance attention mechanism-based WIoU loss function for training, the training set comprises infrared remote sensing images and corresponding target labels, and the target detection model is constructed by introducing a scale-in feature efficient interaction mechanism and a non-maximum suppression mechanism in a yolo-based target detector.

[0042] Specifically, the embodiment introduces a scale-in feature efficient interaction and fusion module, effectively captures the detailed features of weak small targets in the infrared remote sensing images, realizes cross-channel and cross-space information fusion between feature maps, uses a distance attention mechanism-based WIoU loss function and a non-maximum suppression soft-NMS, and improves the detection accuracy and reduces the missed detection.

[0043] Further, acquiring the training set comprises:

[0044] acquiring a plurality of original infrared remote sensing images containing weak small targets, and converting the original infrared remote sensing images from a single channel to a pseudo three-channel RGB format;

[0045] labeling weak small target categories and weak small target upper left and lower right coordinates in the infrared remote sensing images after format conversion, and acquiring a final infrared remote sensing image.

[0046] Specifically, the embodiment captures a plurality of infrared remote sensing images containing weak and small targets by the unmanned aerial vehicle infrared remote sensing imaging device 102, converts the single-channel images into pseudo three-channel RGB format by inputting the computer 101 through the transceiver device 103, and constructs an infrared remote sensing image dataset I h = [I h1 , I h2 ,...I hK ], wherein the total number of elements in the dataset I h is K, the image size is n ch ×h×w, n ch is the image channel, h is the image height, and w is the image width. The image label uses the open source tool labelImg, and the label content is the infrared weak and small target category and the coordinates of the upper left corner and the lower right corner of the target. The information file after labeling is in the txt format.

[0047] Further, the target detection model comprises:

[0048] a backbone network, configured to perform feature extraction, interaction and fusion on an input image, and output a plurality of first feature maps of different scales;

[0049] a neck network, configured to perform feature splicing and extraction on the plurality of first feature maps of different scales, and output a plurality of second feature maps of different scales;

[0050] a head network, configured to perform weak and small target detection on the plurality of second feature maps of different scales, and output a detection frame.

[0051] The specific structure of the target detection model is shown in Figure 2 , wherein an RGB image with a size of n ch ×h×w is used as input to enter the backbone network of the target detection model, and is output through a convolutional layer, a feature extraction module, a fast spatial pyramid pooling layer, a cross-stage efficient attention mechanism, and three feature maps I b1 , I b2 and I b3 of different scales, wherein the scales are n c1 ×h / 8×w / 8, n c2 ×h / 16×w / 16 and n c3 ×h / 32×w / 32. Among them, the feature maps I b1 and I b2 are spliced to connect the feature extraction module in the backbone network and the feature transmission of the neck network, and the feature map I b3 is spliced to connect the cross-stage efficient attention mechanism in the backbone network and the feature transmission of the neck network. Then, the feature map I b3The scale-in feature efficient interaction and fusion module is connected to the backbone network, and performs self-attention operation and scale-in feature interaction to realize information fusion. The output of the scale-in feature efficient interaction and fusion module is consistent with the input feature scale.

[0052] Three feature maps I n1 , I n2 and I n3 with different scales are output in the neck network, and the scales are n c1 ×h / 8×w / 8, n c2 ×h / 16×w / 16 and n c3 ×h / 32×w / 32 respectively, realizing multi-scale feature fusion and improving the performance of small target detection. n1 , I n2 and I n3 are input into the detection head of the head network, and three images containing small target detection frames are output.

[0053] In the scale-in feature efficient interaction and fusion module, two-dimensional sine-cosine position coding is added to the input feature, and then the multi-head attention mechanism, residual connection and layer normalization, and feedforward fully connected network are used to output the interactive and fused features.

[0054] Specifically, as shown in Figure 3 , in the forward propagation process, the shape of the input feature is [B, C, H, W] (batch size B, channel number C, height H, width W), which is flattened and transposed into the format [B, HxW, C] before processing, then two-dimensional sine-cosine position coding (2D Sin-Cos Positional Embedding) is added and embedded into the feature to provide position information, and finally the output feature is transposed back to the original format [B, C, H, W].

[0055] The multi-head attention mechanism is improved on the basis of the self-attention mechanism. When generating q, k, and v (query, index, and content), q, k, and v are divided into num_heads (attention heads) respectively, and self-attention operation is performed on each part, and finally the results are spliced.

[0056] Residual connection and layer normalization are composed of Add and Norm. The calculation process of Add&Norm layer can be expressed by mathematical formula as follows:

[0057] Add&Norm(X)=LayerNorm(X+MultiHeadAttention(X));

[0058] Wherein, Add indicates X+MultiHeadAttention(X), residual connection is performed; Norm indicates LayerNorm(), that is, LayerNormalization, layer normalization is performed. X indicates a feature map input vector.

[0059] The feedforward full connection network, referred to as FFN, is essentially a two-layer full connection layer, the activation function of the first layer is Relu, and the second layer does not use the activation function. The calculation process can be expressed by a mathematical formula as:

[0060] FFN(X)=max(0,XW1+b1)W2+b2;

[0061] Wherein, W1 and W2 are weight matrices, which respectively map the input dimension to the hidden dimension, and then map back to the output dimension; b1 and b2 are bias terms. max(0,x) can change negative numbers to 0, enhance nonlinearity, and reflect the activation function Relu; X indicates a feature map input vector.

[0062] Further, the target detection model is trained:

[0063] The weights of the feature extraction module in the backbone network are first frozen, and multiple epochs are trained, and then all the weights are unfrozen, and multiple epochs are trained. The network training parameters are set: learning rate lr, batchsize, training set verification set division, optimizer and training period.

[0064] The loss function used is a WIoU loss function based on distance attention mechanism, which effectively eliminates the influence of hindering convergence, and does not introduce new indicators (such as aspect ratio), and the calculation process can be expressed by a mathematical formula as:

[0065] L WIoUv1 =R WIoU ×L IoU ;

[0066] Wherein, L WIoUv1 is a WIoU v1 function, L IoU is an IOU loss function, R WIoU is a penalty term of WIOU, calculated by the following formula:

[0067]

[0068] Wherein, x and y are the coordinates of the upper left corner of the anchor frame, x gt and y gt are the coordinates of the upper left corner of the target frame, W g and H g represent the width and height of the minimum bounding box; at the same time, in order to eliminate the influence of R WIoU hindering convergence, W g and Hg Separate from the computational graph (the superscript * indicates this operation).

[0069] Further, target detection model testing and application:

[0070] Using the trained target detection model for prediction, inputting a test image, and outputting the target frame of the infrared remote sensing image weak target prediction. First, the image to be tested I t is input into the network, and the image size is n ch ×h×w. After network inference, the output of the detection head is obtained. The output feature map has three feature maps with scales of 80×80, 40×40 and 20×20. The classification and regression prediction results are extracted from the feature maps with different scales, and are subjected to splicing and dimension transformation operations. In order to facilitate processing, the original channel dimension is transposed to the end, and the shapes of the class prediction branch and the bbox prediction branch are (b, 8400, 80) and (b, 8400, 4) respectively. Then, soft-NMS is used to replace the traditional NMS for non-maximum suppression, and the confidence of the overlapping frame is reduced through Gaussian weighting, and the formula is as follows:

[0071]

[0072] Where s i is the boundary box score, b i is the initial boundary box, is the boundary box with the highest score, D is used to store the final result frame set, and σ is the standard deviation of the Gaussian function.

[0073] By converting the pixel coordinates of the target detection frame into a normalized scale and visualizing the detection results in the image, it is determined whether a target exists according to whether there is a detection frame. If the input contains the detection frame parameters (x, y, w, h), the normalization operation is performed as (X, Y, W, H), and a rectangular frame is drawn, and it is determined that a target is found; otherwise, it is determined that no target is found.

[0074] The unmanned aerial vehicle infrared remote sensing image weak target detection device and the specific target detection model have the advantages of being fast, lightweight and accurate, and are suitable for real-time application of infrared image weak target detection and various scenes, and have a wide application prospect in the fields of environmental monitoring and military reconnaissance.

[0075] The above-described embodiments only describe the preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A method for detecting a weak and small target in an infrared remote sensing image of a UAV, characterized in that, The method comprises: acquiring an infrared remote sensing image to be tested; inputting the infrared remote sensing image to be tested into a preset target detection model, and outputting a weak small target detection result, wherein the target detection model is obtained by inputting a training set and training by using a distance attention mechanism-based WIoU loss function, the training set comprises infrared remote sensing images and corresponding target labels, and the target detection model is constructed by introducing a scale-in feature efficient interaction mechanism and a non-maximum suppression mechanism into a yolo-based target detector; the target detection model comprises: a backbone network for performing feature extraction, interaction and fusion on an input image, and outputting a plurality of first feature maps of different scales; a neck network for performing feature splicing and extraction on the plurality of first feature maps of different scales, and outputting a plurality of second feature maps of different scales; a head network for performing weak small target detection on the plurality of second feature maps of different scales, and outputting a detection frame; The input image in the backbone network passes through a convolutional layer, a feature extraction module, a fast spatial pyramid pooling layer, and a cross-stage efficient attention mechanism, and outputs three feature maps of different scales , and The feature maps , are connected to the feature transmission of the neck network through feature map splicing, the feature maps are connected to the feature transmission of the neck network through feature map splicing of the cross-stage efficient attention mechanism, and the feature maps are connected to the feature map splicing in the neck network after self-attention operation and intra-scale feature interaction and fusion, and the neck network outputs three feature maps of different scales , and , which are respectively input into the detection head in the head network, and output small target detection boxes; in the scale-in feature efficient interaction and fusion module, two-dimensional sine-cosine position coding is added to input features, and then the input features are outputted through a multi-head attention mechanism, residual connection and layer normalization, and a feedforward fully connected network.

2. The method of claim 1, wherein, acquiring the training set comprises: acquiring a plurality of original infrared remote sensing images containing weak small targets, and converting the original infrared remote sensing images from a single channel to a pseudo three-channel RGB format; labeling weak small target categories and weak small target upper left and lower right coordinates in the infrared remote sensing images after format conversion, and acquiring a final infrared remote sensing image.

3. The method of claim 1, wherein, after acquiring a plurality of weak small target detection frames output by the head network, splicing and dimension transformation operations are performed in combination with the non-maximum suppression mechanism to output a final weak small target detection frame, wherein the non-maximum suppression mechanism is: ； wherein, is the bounding box score, is the initial bounding box, is the bounding box with the highest score, D is the final result box set, is the standard deviation of the Gaussian function.

4. The method of claim 1, wherein, the distance attention mechanism-based WIoU loss function is: ； ； where, is the WloU v1 function, is the IOU loss function, is the penalty term for WIOU, and is the coordinate of the top-left corner of the anchor box, and is the coordinate of the top-left corner of the target box, W g and H g denote the width and height of the minimum bounding box, and the superscript * indicates that W g and H g are separated from the computational graph.

5. An apparatus for detecting a weak and small target in an unmanned aerial vehicle (UAV) infrared remote sensing image, which is used to implement the method for detecting a weak and small target in an unmanned aerial vehicle (UAV) infrared remote sensing image according to claim 1, characterized in that, The method comprises: an image acquisition device for acquiring an infrared remote sensing image to be tested; an image processing device for inputting the infrared remote sensing image to be tested into a preset target detection model, and outputting a weak small target detection result, wherein the target detection model is obtained by inputting a training set and training by using a distance attention mechanism-based WIoU loss function, the training set comprises infrared remote sensing images and corresponding target labels, and the target detection model is constructed by introducing a scale-in feature efficient interaction mechanism and a non-maximum suppression mechanism into a yolo-based target detector.

Images