A lightweight real-time small target recognition device and method for UAVs based on SOD-DETR

Abstract

The application discloses a kind of unmanned aerial vehicle lightweight real-time small target identification device and method based on SOD-DETR, wherein small target identification method includes: using small target detection device, the continuous video stream of the region to be detected is acquired;The continuous video stream is input to small target identification model, and the detection result containing small target position information and category label is acquired, wherein the small target identification model is the lightweight small target identification network of improved RT-DETR, the lightweight small target identification network of improved RT-DETR adopts convolution submodule and multi-scale edge information enhancement submodule to construct feature extraction backbone module;Asymmetric rearrangement convolution submodule and double-path feature perception submodule are used to construct target feature enhancement module.The method combines multi-scale feature extraction, edge information enhancement and shallow feature, while ensuring detection accuracy, reduces model parameter quantity and computational complexity.

Description

Technical Field

[0001] This invention belongs to the field of small target detection technology in computer vision, and particularly relates to a lightweight real-time small target recognition device and method for UAVs based on SOD-DETR. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence, computer vision, and edge computing technologies, target recognition systems based on drone platforms have been widely used in emergency rescue, urban security, border patrol, disaster monitoring, and agricultural pest and disease identification. In particular, the need for rapid and accurate identification of small targets (such as people, vehicles, and foreign objects) in complex environments is increasing.

[0003] Small targets are defined in two categories. The first category consists of targets with pixel values ​​below 32×32 in an image. The second category consists of targets whose pixel values ​​are less than 0.12% of the total image pixel values. These targets have relatively few pixels and limited visual detail in the image, making them susceptible to the influence of complex background information.

[0004] With the rapid development of deep learning, Convolutional Neural Networks (CNNs) have overcome the limitations of manual feature extraction in traditional methods through efficient feature learning. Common two-stage object detection algorithms include R-CNN, Fast R-CNN, and Faster R-CNN. These algorithms have high detection accuracy but low detection speed, making them difficult to use for real-time detection. Subsequent single-stage object detection algorithms include Single Shot MultiBoxDetector (SSD) and the YOLO series. These algorithms are known for their real-time object detection capabilities, but due to information loss during the downsampling process of small objects, they suffer from insufficient effective representation in high-level features, posing a challenge to small object detection.

[0005] In 2020, researchers proposed introducing the Transformer architecture into the field of object detection, calling it DETR (Detection Transformer). DETR, based on the Transformer, transforms object detection into ensemble prediction, eliminating the need for traditional techniques such as anchor boxes and non-maximum suppression, achieving end-to-end detection from the original image to the prediction result. In 2023, a real-time end-to-end object detection Transformer called RT-DETR was introduced. RT-DETR efficiently processes multi-scale features through intra-scale interaction and cross-scale fusion, surpassing contemporary real-time detectors in both accuracy and speed. However, it still faces challenges such as excessive computational cost and insufficient accuracy in detecting small objects.

[0006] Currently, using drones as edge computing devices for small target detection remains a highly challenging problem. This is because small target detection presents various issues:

[0007] (1) Small targets in the image from the perspective of UAVs occupy fewer pixels and are partially occluded and have complex background interference, making feature extraction difficult;

[0008] (2) The target scale varies greatly in different scenarios, and a single-scale feature extraction method cannot effectively take into account various types of targets.

[0009] (3) Due to the limitations of computing resources and power consumption of the UAV platform, complex models are difficult to achieve real-time response and efficient operation at the edge.

[0010] Therefore, there is an urgent need for a lightweight real-time small target recognition device and method for UAVs based on SOD-DETR. Summary of the Invention

[0011] To address the aforementioned technical issues, this invention proposes a lightweight real-time small target recognition device and method for UAVs based on SOD-DETR. This method combines multi-scale feature extraction, edge information enhancement, and shallow features, reducing the number of model parameters and computational complexity while ensuring detection accuracy, and can be deployed on UAV platforms.

[0012] To achieve the above objectives, the present invention provides a lightweight real-time small target recognition device for UAVs based on SOD-DETR. The device is mounted on a UAV platform and includes: a data acquisition module, a data interaction module, a data storage module, and an edge computing module.

[0013] The data acquisition module is used to acquire video stream data;

[0014] The data interaction module is used to receive video stream data transmitted by the data acquisition module, perform preprocessing operations on the video stream data, and encapsulate the detection results in a structured manner.

[0015] The edge computing module is used to perform calculations on the processed video stream data and output detection results containing location and category information.

[0016] Optionally, the small target identification device further includes: a device mounting module, a lighting auxiliary module, a power supply module, a data storage module, and a communication module;

[0017] The device mounting module is located at the bottom of the UAV platform and is connected to the data acquisition module and the lighting assistance module through a three-axis mechanical stabilization gimbal to reduce the impact of attitude disturbances during flight on imaging quality.

[0018] The lighting auxiliary module is located on both sides of the three-axis mechanical stabilization gimbal and is used to provide high-brightness LED supplementary lighting at night or in low-light environments.

[0019] The power module is used to provide power to each functional module to ensure stable operation;

[0020] The data storage module is used to store the algorithm model and model parameters;

[0021] The communication module is used to send identification information to the user terminal in real time.

[0022] This invention also provides a lightweight real-time small target recognition method for UAVs based on SOD-DETR, comprising:

[0023] A small target detection device is used to acquire a continuous video stream of the area to be detected;

[0024] The continuous video stream is input into the small target recognition model to obtain detection results containing small target location information and category labels. The small target recognition model is a lightweight small target recognition network based on an improved RT-DERT. The lightweight small target recognition network based on the improved RT-DERT uses a convolutional submodule and a multi-scale edge information enhancement submodule to construct the feature extraction backbone module; and uses an asymmetric rearranged convolutional submodule and a dual-path feature perception submodule to construct the target feature enhancement module.

[0025] Optionally, the continuous video stream needs to be preprocessed before being input into the small target recognition model. The preprocessing of the continuous video stream includes:

[0026] The continuous video stream is sampled at fixed frames, and the resolution of the sampled continuous video stream is uniformly adjusted to obtain an image sequence.

[0027] Optionally, the small target recognition model includes: a feature extraction backbone module, a target feature enhancement module, and an output module;

[0028] The feature extraction backbone module is used to perform multi-scale feature extraction, convolution operation and edge information enhancement processing on the image sequence to obtain enhanced edge information feature maps;

[0029] The target feature enhancement module is used to extract and integrate the enhanced edge information to obtain an integrated feature map;

[0030] The output module is used to output detection results that include small target location information and category labels.

[0031] Optionally, the multi-scale edge information enhancement submodule includes: an adaptive average pooling unit, a convolution unit, and an edge information enhancement unit;

[0032] The adaptive average pooling unit is used for pooling operations on multi-scale features, extracting local information of different sizes, and capturing multi-level features of the image.

[0033] The convolutional unit is used to perform convolutional learning on the multi-level features to obtain a feature map;

[0034] The edge information enhancement unit is used to process the feature map to obtain an enhanced edge information feature map.

[0035] Optionally, the method by which the multi-scale edge information enhancement submodule obtains the enhanced edge information feature map is as follows:

[0036]

[0037] Where MSEIE_in is the input feature map of the scale edge information enhancement submodule, MSEIE_out is the output feature map of the multi-scale edge information enhancement submodule, and Split represents the segmentation operation along the channel dimension. This represents the output after input feature segmentation. Let represent the i-th sub-feature after adaptive average pooling, convolution, edge information enhancement, and upsampling. Conv represents the convolution operation, AvgPool represents adaptive average pooling, E represents the edge information enhancement unit, Upsample represents the upsampling operation, and Concat represents the concatenation operation along the channel dimension.

[0038] Optionally, the asymmetric rearrangement convolution submodule is used to rearrange the features in the P2 detection layer of the backbone module and perform asymmetric convolution feature extraction operations, and then fuse the processed features with the features in the P3-P5 detection layers to obtain fused features.

[0039] The dual-path feature perception submodule is used to process the fused features using dual-path branches to obtain the integrated feature map.

[0040] Optionally, the method for obtaining the fusion module from the asymmetric rearranged convolutional submodule is as follows:

[0041]

[0042] Wherein, given input X, the asymmetric rearrangement convolutional submodule outputs feature map Y, Shuffle means rearranging the input feature map in a specified spatial dimension to a channel dimension, C represents the number of channels, H represents the height, W represents the width, and Conv 3×1Let represent a 3×1 asymmetric convolution kernel, s = (2,1) represent the stride of the convolution in the height and width directions, p = (1,1,0,0) represent the number of padding pixels in the top, bottom, left, and right directions, x1 and x2 represent the first and second parts of the input feature map obtained by segmentation along the channel dimension, x1' and x2' represent the features after different shuffles, and x1" and x2" represent the features after different asymmetric convolutions. H×W×C This represents the spatial dimensions and number of channels of the feature map.

[0043] Optionally, the dual-path feature sensing submodule includes: a first path unit and a second path unit;

[0044] The first pathway unit is used to extract features through a convolutional layer;

[0045] The second pathway unit is used to capture global contextual information, mesoscale features, and local fine-grained features through multi-scale convolutional layers.

[0046] Compared with the prior art, the present invention has the following advantages and technical effects:

[0047] This invention constructs a backbone network by introducing the MSEIE module. MSEIE adopts a CSP structure to extract and enhance multi-scale edge information at different stages, effectively improving the feature extraction capability of small targets under complex backgrounds and occlusions. It uses an innovative asymmetric rearrangement convolution to retain detailed information in the P2 layer, enhancing the multi-scale target detection capability. It constructs a dual-path feature perception module to improve the multi-scale feature fusion effect while reducing the computational burden. SOD-DETR significantly reduces model parameters and computational load while improving recognition speed and accuracy. Attached Figure Description

[0048] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a flowchart of a lightweight real-time small target recognition method for UAVs based on SOD-DETR according to an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of the small target recognition model according to an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the multi-scale edge information enhancement (MSEIE) module structure according to an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the asymmetric rearrangement convolution structure according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the dual-path feature sensing module structure according to an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the device structure according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of data acquisition according to an embodiment of the present invention. Detailed Implementation

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0057] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0058] Example 1

[0059] This embodiment proposes a lightweight, real-time small target recognition method for UAVs based on SOD-DETR, such as... Figure 1 As shown, the specific steps include:

[0060] A small target detection device is used to acquire a continuous video stream of the area to be detected;

[0061] A continuous video stream is input into a small target recognition model to obtain detection results containing small target location information and category labels. The small target recognition model is a lightweight small target recognition network based on an improved RT-DERT. The lightweight small target recognition network based on the improved RT-DERT uses a convolutional sub-module and a multi-scale edge information enhancement sub-module to construct the feature extraction backbone module; and uses an asymmetric rearranged convolutional sub-module and a dual-path feature perception sub-module to construct the target feature enhancement module.

[0062] Specifically, this embodiment includes: Step 1: Using the camera of the UAV platform to acquire a continuous video stream of the area to be detected, and performing frame sampling and image preprocessing on the video stream. The video stream frame sampling adopts a fixed sampling of 60fps, and the image preprocessing adjusts all image sequences to a uniform resolution of 1080×768 and normalizes the pixel values ​​to the [0,1] range;

[0063] Step 2: Construct a feature extraction backbone network using the ConvNormLayer convolutional module and the Multi-Scale Edge Enhancement (MSEIE) module. The MSEIE module employs a Cross-Stage Partial (CSP) structure. Its processing flow includes multi-scale feature extraction, convolution operations, and edge enhancement at different stages of the network. Finally, feature maps at different scales are aligned to the same size using bilinear interpolation and then stitched together.

[0064] Step 3: Construct SDENet, a small target detection feature enhancement network improved from the path aggregation network PANet. This network extracts feature maps (i.e., detection layers P2, P3, P4, and P5) from the MSEIE modules at positions 3, 5, 7, and 9 of the backbone network. Asymmetric rearranged convolutions are used to process the P2 detection layer. This module replaces the stride convolutional layers and pooling layers of the traditional feature pyramid structure, reducing the feature map size to half while doubling the number of channels. It retains small target feature information and then fuses it with the P3-P5 feature layers. An improved dual-path feature perception module is obtained using the CSP structure and the OmniKernel module. This module is added to the bottom layer of SDENet to receive all features from the P2-P5 detection layers. This module uses the ConvNormLayer convolutional module and the OmniKernel module in different paths, improving the small target feature integration capability while reducing computational cost.

[0065] Step 4: Deploy and run the SOD-DETR algorithm on the drone platform, taking the preprocessed image sequence as input, and outputting the detection results containing small target location information and category labels in real time.

[0066] Furthermore, before inputting the continuous video stream into the small target recognition model, the continuous video stream needs to be preprocessed. This preprocessing includes:

[0067] A continuous video stream is sampled at fixed frames, and the resolution of the sampled continuous video stream is uniformly adjusted to obtain an image sequence.

[0068] Furthermore, the small target recognition model includes: a feature extraction backbone module, a target feature enhancement module, and an output module;

[0069] The feature extraction backbone module is used to perform multi-scale feature extraction, convolution operations, and edge information enhancement processing on image sequences to obtain enhanced edge information feature maps.

[0070] The target feature enhancement module is used to extract and integrate the enhanced edge information to obtain an integrated feature map;

[0071] The output module is used to output the detection results, which include the location information of small targets and category labels.

[0072] Specifically, in step 2, the backbone network first downsamples the input image to 1 / 4 of its size using two consecutive convolutional modules, ConvNormLayer, and then uses MSEIE to extract features as the P2 detection layer for SDENet. Subsequently, three sets of convolutional modules, ConvNormLayer and MSEIE, are used sequentially to obtain the P3, P4, and P5 detection layers, with sizes of 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively.

[0073] Furthermore, the multi-scale edge information enhancement submodule includes: an adaptive average pooling unit, a convolutional unit, and an edge information enhancement unit;

[0074] The adaptive average pooling unit is used for pooling operations on multi-scale features to extract local information of different sizes and capture multi-level features of the image.

[0075] A convolutional unit is used to learn feature maps by performing convolution operations on multi-level features;

[0076] The edge information enhancement unit is used to process the feature map and obtain enhanced edge information feature maps.

[0077] Specifically, MSEIE consists of an Adaptive Average Pooling (AdaptiveAvgPool), a ConvNormLayer (Convolutional module), and an EdgeEnhancer (EdgeEnhancer module), which respectively implement multi-scale feature extraction, convolutional operations, and edge enhancement. Its main purpose is to extract features from different scales, highlight edge information, and then combine these multi-scale features, learning the enhanced features through convolutional operations. The calculation process of the MSEIE output feature map Y1 for a given input X1 is shown below:

[0078]

[0079] Where MSEIE_in is the input feature map of the scale edge information enhancement submodule, MSEIE_out is the output feature map of the multi-scale edge information enhancement submodule, and Split represents the segmentation operation along the channel dimension. This represents the output after input feature segmentation. Let represent the i-th sub-feature after adaptive average pooling, convolution, edge information enhancement, and upsampling. Conv represents the convolution operation, AvgPool represents adaptive average pooling, E represents the edge information enhancement unit, Upsample represents the upsampling operation, and Concat represents the concatenation operation along the channel dimension.

[0080] More specifically, multi-scale feature extraction uses AdaptiveAvgPool to perform pooling operations on multi-scale features, extracting local information of different sizes and capturing multi-level features of the image.

[0081] Convolution operations use the ConvNormLayer module, which consists of a 3×3 convolutional layer Conv2d, a batch normalization layer BatchNorm2d, and an activation function layer SiLU.

[0082] Edge enhancement is achieved using the EdgeEnhancer module. EdgeEnhancer employs residual connections. First, it uses AdaptiveAvgPool to smooth the input feature map, extracting its low-frequency information. Then, it subtracts the smoothed feature map from the original input feature map to obtain the enhanced high-frequency edge information. Finally, it applies convolution operations to further process the enhanced edge information.

[0083] Furthermore, the asymmetric rearrangement convolution submodule is used to rearrange the features in the P2 detection layer of the backbone module and perform asymmetric convolution feature extraction operations. The processed features are then fused with the features in the P3-P5 detection layers to obtain fused features.

[0084] The dual-path feature perception submodule is used to process the fused features using dual-path branches to obtain an integrated feature map.

[0085] Specifically, in step 2, the asymmetric rearrangement convolution divides the input features into two equal parts along the channel dimension, and rearranges the spatial dimensions into channel dimensions along the width and height directions respectively. Then, asymmetric convolution is used to extract features, reducing the number of channels. Compared to ordinary convolution, the asymmetric rearrangement convolution reduces the number of parameters and computational cost by approximately 66.5% and 99.5%, respectively. For a given feature map X3∈ H×W×C The calculation process of the asymmetric rearrangement convolution output feature map Y3 is as follows:

[0086]

[0087] Wherein, given input X, the asymmetric rearrangement convolutional submodule outputs feature map Y, Shuffle means rearranging the input feature map in a specified spatial dimension to a channel dimension, C represents the number of channels, H represents the height, W represents the width, and Conv 3×1Let represent a 3×1 asymmetric convolution kernel, s = (2,1) represent the stride of the convolution in the height and width directions, p = (1,1,0,0) represent the number of padding pixels in the top, bottom, left, and right directions, x1 and x2 represent the first and second parts of the input feature map obtained by segmentation along the channel dimension, x1' and x2' represent the features after different shuffles, and x1" and x2" represent the features after different asymmetric convolutions. H×W×C This represents the spatial dimensions and number of channels of the feature map.

[0088] Furthermore, the dual-path feature perception submodule includes: a first path unit and a second path unit;

[0089] The first pathway unit is used to extract features through the convolutional layer;

[0090] The second pathway unit is used to capture global contextual information, mesoscale features, and local fine-grained features through the OmniKernel layer.

[0091] Specifically, the dual-path feature perception module in step 3 has two branches. First, the input features are fed into the ConvNormLayer convolutional module for feature processing. Then, half of the channel features are taken as path one and fed into the ConvNormLayer convolutional module to continue extracting features. The other half is fed into the OmniKernel (multi-scale convolutional layer) as path two. Neither branch changes the feature map size or the number of channels. Their outputs are concatenated along the channel dimension and then passed through the ConvNormLayer convolutional module again to learn features.

[0092] The OmniKernel module contains a global branch, a large branch, and a local branch, which are used to capture global context information, mesoscale features, and local fine-grained features, respectively. The outputs of the three branches are added together and then passed through a convolutional layer to obtain the final output. The large branch uses a 31×31 depthwise convolution and parallel depthwise convolutions of 1×31 and 31×1 to maintain long-range modeling capabilities while reducing the number of model parameters.

[0093] Example 2

[0094] This embodiment also provides a lightweight real-time small target recognition device for UAVs based on SOD-DETR. The device is mounted on a UAV platform and includes: a device mounting module, a data acquisition module, a lighting assistance module, a power supply module, a data interaction module, a data storage module, an edge computing module, and a communication module.

[0095] The equipment mounting module is located at the bottom of the UAV platform and is connected to the data acquisition module and the lighting assistance module through a three-axis mechanical stabilization gimbal to reduce the impact of attitude disturbances during flight on imaging quality.

[0096] The data acquisition module is used to acquire video stream data;

[0097] The lighting auxiliary module is located on both sides of the three-axis mechanical stabilization gimbal and is used to provide high-brightness LED supplementary lighting at night or in low-light environments.

[0098] The power supply module provides power to all functional modules to ensure stable operation.

[0099] The data interaction module is used to receive video stream data transmitted by the data acquisition module, perform preprocessing operations on the video stream data, and encapsulate the detection results in a structured manner.

[0100] The edge computing module is used to perform calculations on the processed video stream data and output detection results containing location and category information;

[0101] The data storage module is used to store the algorithm model and model parameters;

[0102] The communication module is used to send identification information to the user terminal in real time.

[0103] Specifically, the device is mounted on a drone platform and includes an equipment mounting module, a data acquisition module, a lighting assistance module, a power supply module, a data interaction module, a data storage module, an edge computing module, and a communication module.

[0104] The equipment mounting module is located at the bottom of the UAV platform and is connected to the data acquisition module and the lighting assistance module through a three-axis mechanical stabilization gimbal to reduce the impact of attitude disturbances during flight on imaging quality.

[0105] The data acquisition module includes a binocular visible light camera and an infrared thermal imager, enabling multimodal imaging and distance perception of the target scene;

[0106] The lighting auxiliary module features a small night light with IPX4 waterproof rating. It is mounted on both sides of a three-axis mechanical stabilization gimbal and is used to provide high-brightness LED supplementary lighting at night or in low-light environments.

[0107] The power module, data interaction module, edge computing module, data storage module, and communication module are placed inside the drone.

[0108] The power module provides power to all functional modules to ensure stable operation;

[0109] The data interaction module includes frame extraction, format conversion and result encapsulation functions. It is used to receive video stream data transmitted by the data acquisition module, complete preprocessing operations such as image frame conversion, resolution normalization, and pixel normalization, and perform structured encapsulation, visual annotation and data processing on the detection results output by the edge computing module after recognition is completed.

[0110] The edge computing module uses the NVIDIA Jetson Xavier NX embedded AI computing unit, which internally deploys the SOD-DETR model and combines the ONNX structure with the TensorRT acceleration engine to achieve real-time end-to-end small target detection.

[0111] The data storage module is used to store algorithm models and model parameters, and to save detection logs, detection results and image data;

[0112] The communication module is used to send identification information to the user terminal in real time;

[0113] More specifically, the following workflow is used to complete the lightweight real-time small target recognition task:

[0114] The data acquisition module acquires a continuous video stream of the target area and transmits it to the data interaction module;

[0115] After the data interaction module performs frame extraction, size normalization and format conversion on the video stream, it inputs the image frames to the edge computing module.

[0116] The edge computing module performs real-time small target detection on image frames based on the SOD-DETR algorithm, and outputs detection results containing location and category information.

[0117] The data interaction module encapsulates the detection results in a structured manner, generating encapsulated data with timestamps, corresponding video frame identifiers, target location information, and category information. This encapsulated data is then transmitted to the communication module for real-time image transmission and to the data storage module for historical record saving.

[0118] The communication module sends the encapsulated identification information to the user terminal in real time;

[0119] The following describes Embodiments 1 and 2 in detail with reference to the accompanying drawings:

[0120] The technical solution in this embodiment is as follows: Figure 1As shown: First, the UAV platform acquires a continuous video stream of the area to be detected through its data acquisition unit, and performs frame sampling and image preprocessing on the video stream. Then, a lightweight small target recognition network, SOD-DETR, based on an improved RT-DERT, is constructed. This network: employs a feature extraction backbone network constructed using a convolutional module ConvNormLayer and a multi-scale edge information enhancement module MSEIE; constructs a small target feature enhancement network SDENet, using asymmetric rearranged convolutions to process the P2 detection layer to retain small target feature information; and adds a dual-path feature perception module at the bottom layer of SDENet, accepting all features from the P2-P5 detection layers to improve the small target feature integration capability while reducing computational load. Finally, the SOD-DETR algorithm is deployed and run on the UAV platform, using the preprocessed image sequence as input, and outputting detection results containing small target location information and category labels in real time.

[0121] The specific improvements of the lightweight real-time small target recognition network SOD-DETR based on RT-DETR are as follows: Figure 2 as follows:

[0122] Step 1: Construct a novel feature extraction backbone network using the convolutional module ConvNormLayer and the multi-scale edge information enhancement module MSEIE. The module usage order is ConvNormLayer, ConvNormLayer, MSEIE, ConvNormLayer, MSEIE, ConvNormLayer, MSEIE, ConvNormLayer, MSEIE. This backbone network first rapidly downsamples the input image to 1 / 4 of its size using two consecutive convolutional modules ConvNormLayer, and then uses MSEIE to extract features as the P2 detection layer provided to SDENet. Subsequently, three sets of convolutional modules ConvNormLayer and MSEIE are used sequentially to obtain the P3, P4, and P5 detection layers, with sizes of 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively.

[0123] Step 2: Construct SDENet, an improved small object detection feature enhancement network based on the path aggregation network PANet, to replace the original CCFM. It consists of a feature pyramid structure composed of 4 convolutional modules (ConvNormLayer), 2 upsampling modules (Upsample), 4 channel dimension concatenation modules (Concat), 4 RepC3 modules, 1 asymmetric rearrangement convolution, and 1 dual-path feature perception module. SDENet extracts feature maps from the outputs of modules 3, 5, 7, and 9 (i.e., detection layers P2, P3, P4, and P5) from the backbone network. The P5 detection layer first obtains global attention features through AIFI; then, the P2 detection layer is processed using asymmetric rearrangement convolution. This module reduces the feature map size to half while doubling the number of channels. The output features are then concatenated with the P3-P5 feature layers through the pyramid structure.

[0124] Step 3: Add a dual-path feature-aware module to the bottom layer of SDENet. This module accepts all features from the P2-P5 detection layers. The features fused by SDENet are then input into the IoU-aware query selection and Decoder and Head parts to complete the final detection result output.

[0125] The MSEIE module structure is as follows: Figure 3 As shown, MSEIE adopts a Cross-Stage Partial (CSP) structure, and its processing flow involves dividing the input feature map along the channel dimension. Five parts, the network respectively The four parts employ adaptive average pooling (AdaptiveAvgPool), a convolutional module (ConvNormLayer), an edge enhancement module (Edgeenhancer), and upsampling (Upsample). Upsampling is performed using bilinear interpolation to obtain multi-scale features. The original information is partially preserved using the ConvNormLayer convolutional module. Then, the feature maps at five different scales are concatenated along the channel dimension. Finally, the enhanced features are learned using another ConvNormLayer convolutional module. The calculation process for the MSEIE output feature map MSEIE_out, given the input MSEIE_in, is as follows:

[0126]

[0127] Where Split represents the segmentation operation along the channel dimension. This represents the output after input feature segmentation. Let represent the i-th sub-feature after adaptive S-average pooling, convolution, edge information enhancement, and upsampling. Conv represents the convolution operation, AvgPool represents adaptive average pooling, E represents the edge information enhancement unit, Upsample represents the upsampling operation, and Concat represents the concatenation operation along the channel dimension.

[0128] The EdgeEnhancer module structure is as follows: Figure 3 As shown: Residual connections are used. First, the input feature map is smoothed using AdaptiveAvgPool to extract its low-frequency information. Then, the original input feature map is subtracted from the smoothed feature map to obtain the enhanced high-frequency edge information. Finally, the ConvNormLayer convolutional module is applied to further process the enhanced edge information and add it to the initial input.

[0129] Asymmetric rearranged convolutional structures, such as Figure 4 As shown, this convolution divides the input features into two groups along the channel dimension. Each group of feature maps is shuffled to encode spatial information along the width and height directions into the channel dimension. Then, asymmetric convolutions of size 1×3 and 3×1 are applied to the two groups of features respectively. Finally, the outputs of the two convolutions are concatenated along the channel dimension. For a given feature map X∈ H×W×C The calculation process of the asymmetric rearrangement convolution output feature map Y is as follows:

[0130]

[0131] Where Shuffle means rearranging the input feature map along a specified spatial dimension (width W or height H) to the channel dimension C, Conv 3×1 This represents a 3×1 asymmetric convolution kernel, s=(2,1) represents the stride of the convolution in the height and width directions, and p=(1,1,0,0) represents the number of padding pixels in the top, bottom, left, and right directions.

[0132] The structure of the dual-path feature perception module is as follows: Figure 5 As shown: The input features are first processed by the ConvNormLayer convolutional module and then divided into two parts on an average scale along the channel dimension. The first part is further processed by the ConvNormLayer convolutional module for shallow feature extraction, while the second part is fed into the OmniKernel module to enhance feature perception. Finally, the output is concatenated along the channel dimension and then processed by another ConvNormLayer convolutional module to learn the features.

[0133] The OmniKernel module comprises a global branch, a large branch, and local branches, used to capture global contextual information, mesoscale features, and local fine-grained features, respectively. The local branch employs 1×1 depthwise convolutions; the large branch uses 31×31 depthwise convolutions combined with parallel 1×31 and 31×1 depthwise convolutions; and the global branch uses FSAM (Frequency-Based Spatial Attention Module) and DCAM (Dual-Domain Channel Attention Module). The outputs of the three sub-branches are added to a portion of the initial input along the channel dimension and fused, then further processed through convolutional layers to output an enhanced feature map.

[0134] The device structure of the technical solution in this embodiment is as follows: Figure 6 As shown: This device is based on an unmanned aerial vehicle (UAV) platform and includes an equipment mounting module, a data acquisition module, a lighting assistance module, a power supply module, a data interaction module, a data storage module, an edge computing module, and a communication module. These modules work together to complete image processing, target recognition, result output, and data transmission. The specific workflow is as follows:

[0135] The data acquisition module acquires a continuous video stream of the target area and transmits it to the data interaction module.

[0136] After the data interaction module performs frame extraction, size normalization, and format conversion on the video stream, it inputs the image frames to the edge computing module.

[0137] The edge computing module performs real-time small target detection on image frames based on the SOD-DETR algorithm, and outputs detection results containing location and category information.

[0138] The data interaction module encapsulates the detection results in a structured manner, generating encapsulated data with timestamps, corresponding video frame identifiers, target location information, and category information. This encapsulated data is then transmitted to the communication module for real-time image transmission and to the data storage module for historical record saving.

[0139] The communication module sends the encapsulated identification information to the user terminal in real time.

[0140] The power module provides power to all functional modules to ensure stable operation.

[0141] The lighting auxiliary module is responsible for providing high-brightness LED supplemental lighting at night or in low-light environments.

[0142] Data collection such as Figure 7 As shown: A drone platform equipped with a method and device captures continuous video stream data of small targets, including cars, trains, and pedestrians, by shooting the camera downwards at a 45-degree angle.

[0143] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A SOD-DETR-based unmanned aerial vehicle lightweight real-time small target recognition method, characterized in that, The method comprises the following steps: acquiring a continuous video stream of a detection area to be detected, inputting the continuous video stream into a small target recognition model, and acquiring a detection result containing small target position information and a category label, wherein the small target recognition model is a lightweight small target recognition network of an improved RT-DETR, and the lightweight small target recognition network of the improved RT-DETR adopts a convolutional sub-module and a multi-scale edge information enhancement sub-module to construct a feature extraction backbone module; and adopts an asymmetric rearrangement convolutional sub-module and a double-path feature perception sub-module to construct a target feature enhancement module; the multi-scale edge information enhancement sub-module comprises: an edge information enhancement unit configured to process a feature map output by the convolutional unit to acquire an enhanced edge information feature map; ； wherein MSEIE_in is the input feature map of the multi-scale edge information enhancement sub-module, MSEIE_out is the output feature map of the multi-scale edge information enhancement sub-module, Split represents a split operation along the channel dimension, represents the output after input feature segmentation, represents the sub-feature after the i-th sub-feature undergoes adaptive S average pooling, convolution operation, edge information enhancement, and upsampling, Conv represents a convolution operation, AvgPool represents adaptive average pooling, E represents an edge information enhancement unit, Upsample represents an upsampling operation, and Concat represents a concatenation operation along the channel dimension.

2. The SOD-DETR-based real-time small target recognition method for lightweight unmanned aerial vehicles according to claim 1, characterized in that, The continuous video stream needs to be preprocessed before being input into the small target recognition model, and the preprocessing of the continuous video stream comprises: fixed frame sampling is performed on the continuous video stream, and the resolution of the sampled continuous video stream is uniformly adjusted to acquire an image sequence.

3. The SOD-DETR-based real-time small target recognition method for lightweight UAVs according to claim 2, characterized in that, The small target recognition model comprises a feature extraction backbone module, a target feature enhancement module and an output module; the feature extraction backbone module is configured to perform multi-scale feature extraction, convolution operation and edge information enhancement processing on the image sequence to acquire an enhanced edge information feature map; the target feature enhancement module is configured to perform extraction and integration operations on the enhanced edge information to acquire an integrated feature map; the output module is configured to output a detection result containing small target position information and a category label.

4. The SOD-DETR-based real-time small target recognition method for lightweight unmanned aerial vehicles according to claim 1, characterized in that, The multi-scale edge information enhancement sub-module further comprises an adaptive average pooling unit and a convolutional unit; the adaptive average pooling unit is configured to perform a pooling operation on multi-scale features to extract local information of different sizes and capture multi-level features of an image; the convolutional unit is configured to perform convolution learning on the multi-level features through a convolution operation to acquire a feature map.

5. The SOD-DETR-based real-time small target recognition method for lightweight UAVs according to claim 3, characterized in that, The asymmetric rearrangement convolutional sub-module is configured to perform rearrangement and asymmetric convolution feature extraction operations on features in a P2 detection layer of a feature pair backbone module, fuse the processed features with features in P3-P5 detection layers, and acquire fused features; the double-path feature perception sub-module is configured to process the fused features by using a double-path branch to acquire the integrated feature map.

6. The SOD-DETR-based real-time small target recognition method for lightweight UAVs according to claim 5, characterized in that, The method for acquiring the fused module by the asymmetric rearrangement convolutional sub-module is as follows: ； wherein, given an input X, the asymmetric rearrangement convolution sub-module outputs a feature map Y, Shuffle represents rearranging the input feature map in the specified spatial dimension direction to the channel dimension, C represents the number of channels, H represents the height, W represents the width, Conv 3×1 represents a 3x1 asymmetric convolution kernel, s=(2, 1) represents the step length of convolution in the height and width directions, p=(1, 1, 0, 0) represents the number of padding pixels in the up, down, left and right directions, represents the first and second part features obtained by dividing the input feature map along the channel dimension, represents the features after different Shuffles, represents the features after different asymmetric convolutions, represents the spatial size and the number of channels of the feature map.

7. The SOD-DETR-based real-time small target recognition method for lightweight UAVs according to claim 5, characterized in that, The double-path feature perception sub-module comprises a first path unit and a second path unit; the first path unit is configured to extract features through a convolutional layer; the second path unit is configured to capture global context information, medium-scale features and local fine-grained features through a multi-scale convolutional layer.

8. A SOD-DETR-based unmanned aerial vehicle lightweight real-time small target recognition device realized by the method according to any one of claims 1-7, wherein the device is carried on an unmanned aerial vehicle platform. The method comprises the following steps: a data acquisition module, a data interaction module, a data storage module and an edge computing module; the data acquisition module is configured to acquire video stream data; the data interaction module is configured to receive the video stream data transmitted by the data acquisition module, perform a preprocessing operation on the video stream data, and structureally encapsulate a detection result; The edge computing module is configured to perform computation on the processed video stream data and output a detection result containing position information and category information.

9. The SOD-DETR-based unmanned aerial vehicle lightweight real-time small target recognition device according to claim 8, characterized in that, The small target recognition device further comprises a device mounting module, an illumination assisting module, a power module, a data storage module and a communication module. The device mounting module is arranged at the bottom of the unmanned aerial vehicle platform and is connected to the data acquisition module and the illumination assisting module through a three-axis mechanical stabilization holder, so as to reduce the influence of attitude disturbance on imaging quality during flight. The illumination assisting module is arranged at both sides of the three-axis mechanical stabilization holder and provides high-brightness LED fill light in night or low-illumination environment. The power module is configured to provide power for each functional module to ensure stable operation. The data storage module is configured to store algorithm models and model parameters. The communication module is configured to send recognition information to the user end in real time.

Images