An unmanned aerial vehicle aerial infrared small target detection method and device based on LMI-DETR

Abstract

The application discloses a kind of based on LMI-DETR's unmanned aerial vehicle aerial infrared small target detection method and device, comprising: obtaining infrared image dataset and pre-processing, obtain pre-processed dataset;LMI-DETR model is constructed, including Stem module is replaced as EdgeFusionStem module to fuse original infrared image, edge enhancement image and gradient information;Replace backbone network with structure based on light weight reparameterization convolution module;AIFI module in neck network is replaced as frequency domain-Mamba interactive module;According to pre-processed dataset, LMI-DETR model is trained, and obtains the LMI-DETR model after training;Target infrared image data are input into the LMI-DETR model after training, and small target in unmanned aerial vehicle aerial infrared image is detected.The application improves the real-time performance and robustness of unmanned aerial vehicle aerial infrared small target detection.

Description

Technical Field

[0001] This invention belongs to the field of infrared small target detection technology in computer vision, and particularly relates to a method and device for detecting infrared small targets in UAV aerial photography based on LMI-DETR. Background Technology

[0002] In the development of low-altitude technologies, using unmanned aerial vehicles (UAVs) as mobile platforms and airborne cameras for ground target detection has wide applications in various fields, such as traffic congestion tracing and monitoring, disaster investigation and emergency search and rescue, agricultural surveys and disease control, and military reconnaissance and target strike. However, due to the small size and limited features of the targets, accurate detection is extremely difficult. Therefore, small ground target detection, as a key technology for UAVs, has always been a hot topic and a challenge in this field of research.

[0003] Due to limitations in detection technology and data acquisition equipment, early UAVs typically relied on visible light images for target identification. However, these images are significantly hampered by lighting conditions, weather, and other complex environmental factors. Visible light images become blurry and unclear at night, in shadow areas, in low light conditions, or during fog or dust storms. Compared to visible light, infrared imaging technology offers numerous advantages, including lighter weight, higher precision, smaller size, better concealment, all-weather operation, and stronger penetration. Therefore, mounting infrared imaging detection devices on UAVs for data acquisition and target detection has become a key research focus when performing target identification tasks in complex environments.

[0004] In recent years, deep learning object detection methods, represented by convolutional neural networks (CNNs), have been continuously developing. Currently, the mainstream object detectors are mainly dominated by the You Only Look Once (YOLO) series of models. Unlike two-stage detectors such as R-CNN, Mask R-CNN, and Faster R-CNN, which first select regions of interest and then accurately predict classification and localization, single-stage detectors like YOLO directly generate object categories and location coordinates through predefined anchor points, significantly reducing the time required for object detection and exhibiting excellent accuracy. Although CNN-based methods achieve fast and lightweight object recognition due to their local receptive field advantage, their effective local receptive field is limited to the central region of the image, lacking long-range correlation and making it difficult to capture long-distance dependencies.

[0005] In 2020, Facebook AI proposed the first end-to-end object detection Transformer model, DEtectionTRansformer (DETR). DETR treats object detection as a direct set prediction problem, directly matching predicted bounding boxes with ground truth boxes, thus eliminating traditional steps such as candidate region generation and NMS. In 2023, Real-Time DEtectionTRansformer (RT-DETR), based on the Transformer architecture, became the first real-time end-to-end object detection framework. By efficiently handling attention-based intra-scale feature interaction (AIFI) and CNN-driven cross-scale feature fusion (CCFF), it surpassed the YOLO detector of similar scale in both efficiency and accuracy. Although the Transformer has a global receptive field and powerful long-range modeling capabilities, its computational complexity increases quadratically, resulting in higher computational overhead compared to CNNs.

[0006] Recently, Mamba, as the first foundational model based on a state-space model, has solved the computational efficiency problem of Transformers when processing long sequences due to its linear computational complexity. It has also attracted widespread attention from researchers for its superior ability to capture long-range dependencies. In the field of vision, numerous studies have applied SSMs (Search Engine Models) to various tasks.

[0007] Currently, using drones equipped with edge computing devices for infrared small target detection remains a highly challenging problem due to the following issues:

[0008] (1) Small targets are widely present in infrared images acquired by UAVs. Not only are they numerous, but due to the long detection distance, they are easily submerged by noise and background clutter, resulting in blurred edges. Conventional target detection methods are unable to extract their effective features.

[0009] (2) Due to the limited computing power of drones, complex models are difficult to deploy on drones to play their due role;

[0010] Therefore, there is an urgent need to propose a method and device for detecting small infrared targets in UAV aerial photography based on LMI-DETR. Summary of the Invention

[0011] To address the aforementioned technical issues, this invention proposes a method and apparatus for detecting small infrared targets in UAV aerial photography based on LMI-DETR. This method can significantly reduce model complexity while ensuring detection accuracy, thereby improving the real-time performance and robustness of UAV aerial infrared target detection.

[0012] To achieve the above objectives, this invention provides a method for detecting small infrared targets in UAV aerial photography based on LMI-DETR, comprising:

[0013] Acquire the infrared image dataset and preprocess it to obtain the preprocessed dataset;

[0014] The LMI-DETR model was constructed by replacing the Stem module with the EdgeFusionStem module to fuse the original infrared image, edge enhancement image, and gradient information; replacing the backbone network with a structure based on lightweight, heavily parameterized convolutional modules; and replacing the AIFI module in the neck network with a frequency domain-Mamba interaction module.

[0015] The LMI-DETR model is trained based on the preprocessed dataset to obtain the trained LMI-DETR model;

[0016] The target infrared image data is acquired and input into the trained LMI-DETR model to detect small targets in the UAV aerial infrared image.

[0017] Optionally, the implementation process of the EdgeFusionStem module includes:

[0018] Edge-enhanced images are obtained using a local boundary lifting algorithm;

[0019] Gradient information is extracted using the Sobel operator;

[0020] The original infrared image, edge-enhanced image, and gradient information are stitched together along the channel dimension;

[0021] The concatenated intermediate feature map is input into two convolutional modules. Each convolutional module includes a 3×3 convolution, a batch normalization layer, and a ReLU activation layer in sequence.

[0022] Optionally, the implementation of the lightweight, heavily parameterized convolution module includes: employing a dual-path structure;

[0023] The first path sequentially passes through a convolutional module, a lightweight reparameterized convolution, a batch normalization layer, and a GELU activation layer to obtain the output result; the second path performs the operation through spatial depth convolution when downsampling is required, and does not perform spatial depth convolution when downsampling is not required; the outputs of the two paths are added and fused.

[0024] Optionally, the frequency domain-Mamba interaction module consists of a Mamba block and a frequency domain-channel enhancement feedforward network connected in series; the Mamba block is used for spatial information modeling; and the frequency domain-channel enhancement feedforward network is used for channel frequency domain information enhancement.

[0025] Optionally, the implementation process of the Mamba block includes: employing a double residual structure;

[0026] The number of channels is increased using 1×1 convolutions; the features are divided into two groups along the channel dimension; they are then processed by 3×3 and 5×5 depthwise separable convolutions respectively; the features are then processed by the GELU activation function, layer normalization, and SS2D module; the output is added to the original input features.

[0027] Optionally, the implementation process of the frequency domain-channel enhancement feedforward network includes: employing a residual structure;

[0028] The process sequentially involves layer normalization, channel enhancement units, and frequency domain enhancement units. The channel enhancement unit acquires overall spatial information through global average pooling and models inter-channel dependencies using one-dimensional convolution with a kernel size of 3. The frequency domain enhancement unit divides the features into non-overlapping image blocks, performs a fast Fourier transform on each image block, multiplies it element-wise with learnable frequency domain weights, and reconstructs the frequency domain enhanced features through inverse fast Fourier transform.

[0029] On the other hand, to achieve the above objectives, the present invention also provides an LMI-DETR-based UAV aerial infrared small target detection device, comprising:

[0030] Infrared data acquisition module, used to acquire infrared video stream data;

[0031] The data processing module is used to receive the infrared video stream data and perform format conversion and frame extraction;

[0032] The embedded edge computing module is used to call the LMI-DETR-based infrared small target detection model to perform real-time detection on the processed data and output the recognition results.

[0033] Optionally, the device further includes:

[0034] The power interface module is located on the outer shell of the infrared detection pod and is used to connect the UAV power supply and provide power to the various functional modules inside the infrared detection pod.

[0035] A communication interface module, located on the outer shell of the infrared detection pod, is used to transmit the identification results to the UAV flight control system or ground station via a wired connection.

[0036] A three-axis mechanical stabilization gimbal is used to mount the infrared detection pod.

[0037] Technical effects of this invention: This invention discloses a method and device for detecting small targets in UAV aerial photography based on LMI-DETR. An EdgeFusionStem module is designed to replace the original Stem module, enhancing edge feature representation by fusing the original infrared image, edge-enhanced image, and gradient information. The backbone network is replaced with a structure based on a lightweight, reparameterized convolutional module, aiming to improve the backbone network's lightweight nature using reparameterization technology. This effectively extracts small target features while reducing the model's computational complexity and parameter size. The attention-based same-scale feature interaction (AIFI) module in the neck network is replaced with a frequency-domain-Mamba interaction module (FMIM). By combining Mamba with Fast Fourier Transform (FFT), the model's global modeling capability is enhanced in both the spatial and frequency domains. Attached Figure Description

[0038] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a flowchart illustrating a method for detecting small infrared targets in UAV aerial photography based on LMI-DETR, according to an embodiment of the present invention.

[0040] Figure 2 This is a structural diagram of the infrared small target detection model according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the EdgeFusionStem module structure according to an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the lightweight, heavily parameterized convolution module structure according to an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the frequency domain-Mamba interaction module structure according to an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the structure of an UAV aerial infrared small target detection device based on LMI-DETR according to an embodiment of the present invention. Detailed Implementation

[0045] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0047] like Figure 1 As shown, this embodiment provides a method for detecting small infrared targets in UAV aerial photography based on LMI-DETR, including:

[0048] Obtain a publicly available infrared small target dataset containing various scenes and targets, preprocess the dataset, unify the images to a fixed size of 640×512, and convert them to a unified YOLO annotation format;

[0049] Based on RT-DETR as the baseline model, an EdgeFusionStem module, a lightweight, heavily parameterized convolution module, and a frequency domain-Mamba interaction module (FMIM) are built on it to form the UAV aerial infrared small target detection model LMI-DETR.

[0050] The LMI-DETR model was trained using a preprocessed dataset, and the trained model was deployed in an infrared small target detection device to detect infrared small targets in drone aerial infrared images.

[0051] Furthermore, the following improvements are made to the RT-DETR baseline model infrastructure: Figure 2 As shown: The Stem module is replaced with the EdgeFusionStem module to fuse the original infrared image, the edge enhancement image, and gradient information, thereby enhancing the edge feature representation of the input image;

[0052] The backbone network is replaced with a structure based on lightweight, reparameterized convolutional modules. Reparameterization techniques are used to reduce the computational complexity and parameter size of the model. Specifically, the backbone network is divided into four stages, with downsampling ratios of 4, 8, 16, and 32 for the output feature maps. The EdgeFusionStem module directly outputs a feature map downsampled by 4 times, so no additional downsampling is performed in the first stage. Two lightweight, reparameterized convolutional modules are set up sequentially in each stage for feature extraction. When downsampling is required, the first module in the stage performs the downsampling operation.

[0053] The attention-based same-scale feature interaction (AIFI) module in the neck network is replaced with the frequency domain-Mamba interaction module (FMIM). This module receives feature information from the fourth stage of the backbone network. The features processed by FMIM, together with the second and third stage feature inputs of the backbone network (CCFM) and the decoder, complete the final detection structure output. By combining the Mamba structure with the fast Fourier transform, the model achieves collaborative global modeling capabilities in the spatial and frequency domains.

[0054] Furthermore, such as Figure 3 The EdgeFusionStem module shown acquires an edge-enhanced image through a local boundary lifting algorithm, extracts gradient information using the Sobel operator, and then concatenates the images along the channel dimension before inputting them into two convolutional modules to reduce the image size. The convolutional modules sequentially consist of a 3×3 convolution, a batch normalization layer, and a ReLU activation layer. For a given input image I... input EdgeFusionStem module outputs feature map I output The calculation process is as follows:

[0055] I LBB =B(I input );

[0056] I sobel =G(I LBB );

[0057] I o =C(I input ,I LBB ,I sobel );

[0058] I output =Conv(Conv(I o ));

[0059] Where B represents the local boundary lifting algorithm, G represents the Sobel operator, and I LBB I represents an edge-enhanced image. sobel Represents gradient information, C represents the concatenation operation along the channel dimension, and I o This represents the concatenated intermediate feature map, and Conv represents the convolutional module.

[0060] Furthermore, the local boundary enhancement algorithm constructs a 3×3 sliding window with a step size of 1 to calculate the maximum, minimum, and average values ​​for local regions of the image, which are used to characterize the brightness distribution of that region. For bright regions (pixel value > local average), the maximum value is used to update the center pixel; for dark regions (pixel value ≤ average), the minimum value is used to update the center pixel, thereby adaptively enhancing the boundary sharpness of infrared targets based on local brightness.

[0061] Furthermore, such as Figure 4 The lightweight reparameterized convolution module shown includes two paths: the first path sequentially passes through the convolution module, lightweight reparameterized convolution, batch normalization layer, and GELU activation layer to obtain the output result; the second path achieves size matching with the output feature map of the first path through spatial depth convolution when downsampling is required, and does not perform spatial depth convolution when downsampling is not required. Finally, the outputs of the two paths are added and fused.

[0062] Lightweight reparameterized convolution only applies reparameterized convolution to one-quarter of the input channels, while keeping the remaining channels unchanged, in order to reduce computational complexity and the number of parameters while maintaining feature extraction capabilities.

[0063] Furthermore, such as Figure 5 The Frequency-Mamba Interaction Module (FMIM) shown consists of two parts connected in series: a Mamba block and a Freq-Channel Enhanced FFN (FCEFFN). The Mamba block is used for spatial information modeling, and the FCEFFN is used for channel frequency domain information enhancement.

[0064] The Mamba block employs a dual residual structure. First, a 1×1 convolution is used to expand the number of channels. Then, the features are divided into two groups along the channel dimension, which are processed by 3×3 and 5×5 depthwise and separable convolutions respectively. After GELU activation, the groups are concatenated and restored to the original number of channels using a 1×1 convolution. This is then added to the input features. Following layer normalization and an SS2D module, the output is finally added back to the original input features. For a given input tensor... Mamba block output feature map X output The calculation process is as follows:

[0065]

[0066] X output =LN(SS2D(X) i ))+X input ;

[0067] in, Represents a 1×1 convolution. These represent depthwise separable convolutions of 3×3 and 5×5 respectively, where Chunk represents dividing the convolution into two equal parts along the channel dimension, and X... 3×3 X 5×5 X represents the features obtained after corresponding convolution processing. i δ represents the intermediate features after the first residual fusion, δ represents the GELU activation function, SS2D represents the SS2D module, and LN represents layer normalization.

[0068] Furthermore, SS2D segments the input image into multiple sub-images and scans the image along four directions: top left, bottom right, top right, and bottom left. It then merges the four output features to construct a complete two-dimensional feature map and extracts the features as global features.

[0069] The frequency domain-channel enhancement feedforward network FCEFFN also uses a residual structure, passing through layer normalization, channel enhancement units and frequency domain enhancement units in sequence;

[0070] The channel enhancement unit obtains overall spatial information through global average pooling and models the inter-channel dependencies using a one-dimensional convolution with a kernel size of 3. The frequency domain enhancement unit divides the features into several non-overlapping image blocks, performs a fast Fourier transform (FFT) on each image block, and multiplies the FFT with the learnable frequency domain weights W element-wise to achieve weighted enhancement of features in different frequency bands. Finally, it reconstructs the frequency domain enhanced feature representation through inverse fast Fourier transform (IFFT) and restores the image blocks to their original feature maps before division.

[0071] like Figure 6 As shown, this embodiment also provides an infrared small target detection device for UAV aerial photography based on LMI-DETR. The device is integrated into an independent infrared detection pod, including: a protective shell, an infrared data acquisition module, a data processing module, and an embedded edge computing module.

[0072] Infrared data acquisition module, used to acquire infrared video stream data;

[0073] The data processing module is used to receive video stream data from the infrared data acquisition module, perform format conversion and frame extraction operations on the infrared video stream data, and package the detection results.

[0074] The embedded edge computing module is used to perform real-time detection on preprocessed data based on the LMI-DETR method and output the recognition results including target category, location coordinates and detection confidence.

[0075] Furthermore, the infrared small target detection device also includes: a power interface module, a communication interface module, and a three-axis mechanical stabilization gimbal;

[0076] The power interface module is located on the outer shell of the pod and is used to connect the drone's power supply and provide power to the various functional modules inside the pod.

[0077] The communication interface module is located on the pod shell and is used to transmit the identification results to the UAV flight control system or ground station via wired means;

[0078] The three-axis mechanical stabilization gimbal is used to mount the infrared detection pod to the bottom of the UAV platform to reduce the impact of attitude disturbances during flight on imaging quality.

[0079] This invention designs an EdgeFusionStem module to replace the original Stem module, which enhances edge feature representation by fusing the original infrared image, edge enhancement image, and gradient information. The backbone network is replaced with a structure based on a lightweight, reparameterized convolutional module, aiming to improve the backbone network by using reparameterization technology, which can effectively extract features of small targets while reducing the computational complexity and parameter scale of the model. The attention-based same-scale feature interaction (AIFI) module in the neck network is replaced with the frequency domain-Mamba interaction module (FMIM), which enhances the global modeling capability of the model in both the spatial and frequency domains by combining Mamba with Fast Fourier Transform (FFT).

[0080] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting small infrared targets in UAV aerial photography based on LMI-DETR, characterized in that, include: Acquire the infrared image dataset and preprocess it to obtain the preprocessed dataset; An LMI-DETR model was constructed, which improved upon the baseline model architecture of RT-DETR. This included replacing the Stem module with the EdgeFusionStem module to fuse the original infrared image, edge enhancement image, and gradient information; and replacing the backbone network with a structure based on a lightweight, heavily parameterized convolutional module. Replace the AIFI module in the neck network with a frequency domain-Mamba interaction module; The LMI-DETR model is trained based on the preprocessed dataset to obtain the trained LMI-DETR model; The target infrared image data is acquired and input into the trained LMI-DETR model to detect small targets in the UAV aerial infrared image; The implementation process of the EdgeFusionStem module includes: Edge-enhanced images are obtained using a local boundary lifting algorithm; Gradient information is extracted using the Sobel operator; The original infrared image, edge-enhanced image, and gradient information are stitched together along the channel dimension; The concatenated intermediate feature map is input into two convolutional modules, each of which includes a 3×3 convolution, a batch normalization layer, and a ReLU activation layer in sequence. The local boundary enhancement algorithm constructs a 3×3 sliding window with a step size of 1, and calculates the maximum, minimum, and average values ​​for local regions of the image to characterize the brightness distribution of the region. For bright regions where the pixel value is greater than the local average value, the maximum value is used to update the center pixel; for dark regions where the pixel value is not greater than the average value, the minimum value is used to update the center pixel, thereby adaptively enhancing the boundary sharpness of infrared targets according to local brightness. The frequency domain-Mamba interaction module consists of a Mamba block and a frequency domain-channel enhancement feedforward network connected in series; the Mamba block is used for spatial information modeling; the frequency domain-channel enhancement feedforward network is used for channel frequency domain information enhancement. The implementation process of the frequency domain-channel enhancement feedforward network includes: adopting a residual structure; The process sequentially involves layer normalization, channel enhancement units, and frequency domain enhancement units. The channel enhancement unit acquires overall spatial information through global average pooling and models inter-channel dependencies using one-dimensional convolution with a kernel size of 3. The frequency domain enhancement unit divides the features into non-overlapping image blocks, performs a fast Fourier transform on each image block, multiplies it element-wise with learnable frequency domain weights, and reconstructs the frequency domain enhanced features through inverse fast Fourier transform.

2. The method for detecting small infrared targets in UAV aerial photography based on LMI-DETR as described in claim 1, characterized in that, The implementation process of the lightweight, heavily parameterized convolution module includes: adopting a dual-path structure; The first path sequentially passes through a convolutional module, a lightweight reparameterized convolution, a batch normalization layer, and a GELU activation layer to obtain the output result; the second path performs the operation through spatial depth convolution when downsampling is required, and does not perform spatial depth convolution when downsampling is not required; the outputs of the two paths are added and fused.

3. The method for detecting small infrared targets in UAV aerial photography based on LMI-DETR as described in claim 1, characterized in that, The implementation process of the Mamba block includes: using a double residual structure; The number of channels is increased using 1×1 convolutions; the features are divided into two groups along the channel dimension; they are then processed by 3×3 and 5×5 depthwise separable convolutions respectively; the features are then processed by the GELU activation function, layer normalization, and SS2D module; the output is added to the original input features.

4. A UAV aerial infrared small target detection device based on LMI-DETR, integrated in a separate infrared detection pod, characterized in that, For implementing the LMI-DETR-based UAV aerial infrared small target detection method as described in any one of claims 1-3, the apparatus comprises: Infrared data acquisition module, used to acquire infrared video stream data; The data processing module is used to receive the infrared video stream data and perform format conversion and frame extraction; The embedded edge computing module is used to call the LMI-DETR-based infrared small target detection model to perform real-time detection on the processed data and output the recognition results.

5. The UAV aerial infrared small target detection device based on LMI-DETR as described in claim 4, characterized in that, The device further includes: The power interface module is located on the outer shell of the infrared detection pod and is used to connect the UAV power supply and provide power to the various functional modules inside the infrared detection pod. A communication interface module, located on the outer shell of the infrared detection pod, is used to transmit the identification results to the UAV flight control system or ground station via a wired connection. A three-axis mechanical stabilization gimbal is used to mount the infrared detection pod.

Images