An RT-DETR-based small target detection method for unmanned aerial vehicle aerial images

Abstract

The application discloses a UAV aerial image small target detection method based on RT-DETR, improves an end-to-end target detection algorithm RT-DETR, and proposes an FCW-RTDETR algorithm, including: using adaptive frequency hole convolution to improve a backbone main network, and extracting UAV aerial image features; performing scale-in feature interaction AIFI on a P5 layer in the main network, inputting the obtained features into a cross-scale feature fusion network CCFM; using a context and spatial feature calibration network CSFCN and wavelet pooling to improve the CCFM structure and an upsampling method, respectively, and fusing features of different scales in the CCFM to obtain an image feature sequence; using IoU-aware query selection to select features from the feature sequence output by an encoder as initial target queries of a decoder; and using the decoder to iteratively optimize the target queries through an auxiliary prediction head, generate a bounding box and a confidence score, and complete small target detection. The scheme improves the efficiency and accuracy of small target detection.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of computer vision, and relates to target detection and deep learning, and particularly relates to a small target detection method for unmanned aerial vehicle aerial image based on RT-DETR. BACKGROUND

[0002] Due to the complexity of the traffic scene photographed by the unmanned aerial vehicle, the unmanned aerial vehicle aerial image is usually affected by multiple factors such as photographing height, angle and weather conditions, and the detected target has the problems of small size, mutual occlusion between targets and difficulty in distinguishing from the surrounding background.

[0003] Small target detection has important practical value in the fields of unmanned aerial vehicle traffic road condition inspection, unmanned aerial vehicle emergency disaster relief, traffic flow monitoring and road safety monitoring, and therefore a scheme is needed to detect different targets (pedestrians, cars, bicycles and the like) in different traffic environments (urban and rural areas) and different densities (sparse and crowded scenes) according to the images photographed by the unmanned aerial vehicle. SUMMARY

[0004] In view of the above technical problems, the application provides a small target detection method for unmanned aerial vehicle aerial image based on RT-DETR.

[0005] The application is implemented by using the following technical scheme: a small target detection method for unmanned aerial vehicle aerial image based on RT-DETR, an end-to-end target detection algorithm RT-DETR is improved, and an FCW-RTDETR algorithm is proposed, including the following steps.

[0006] Step S1: the backbone main network is improved by using adaptive frequency hole convolution, and the unmanned aerial vehicle aerial image features are extracted;

[0007] Step S2: the P5 layer in the main network is subjected to scale-in feature interaction AIFI, and the obtained features are input into a cross-scale feature fusion network CCFM;

[0008] Step S3: the CCFM structure is improved by using a context and spatial feature calibration network CSFCN, and the up-sampling method in the CCFM is improved by using wavelet pooling, different scale features are fused in the CCFM, and an image feature sequence is obtained;

[0009] Step S4: IoU-aware query selection is used to select a fixed number of features from the feature sequence output by the encoder as the initial target query of the decoder;

[0010] Step S5: the decoder iteratively optimizes the target query through an auxiliary prediction head, generates a bounding box and a confidence score, and completes small target detection.

[0011] Specifically, the improvement of the step S1 adaptive frequency hole convolution includes adaptive hole rate, adaptive kernel and frequency selection, the adaptive hole rate adjusts the hole rate in space, specifically including the following steps:

[0012] Step A1: using discrete Fourier transform Transform the feature map to the frequency domain, denoted as:

[0013] ;

[0014] wherein, represents a complex array of DFT output; and respectively represent the height and width of the feature map , and and respectively represent the coordinates of the feature map ; The normalized frequency of the feature map on the height and width is given by |u| and |v|, after moving the low-frequency part to the center, u takes value from the set , and v takes value from ;

[0015] Step A2: introduce an adaptive hole rate strategy, assign different hole rates to each pixel, denoted as:

[0016] ;

[0017] Predict through a convolution layer with a parameter θ;

[0018] Step A3: adopt a modulation mechanism and introduce a ReLU layer, maximize the receptive field while minimizing the lost frequency information of each pixel, define a local feature centered at p with window size s as , the receptive field of which is positively correlated with ;

[0019] Step A4: measure the lost frequency information by calculating the high-frequency power , directly optimize , increase the hole rate at a lower position to enhance the large receptive field, and suppress the hole rate at a higher position to reduce the loss of frequency information, denoted as:

[0020] ;

[0021] wherein, and ​​respectively represent the pixels with the highest and lowest high-frequency power.

[0022] Specifically, the adaptive kernel operates on the convolution kernel weights, specifically including the following steps:

[0023] Step B1: Before introducing dynamic weighting adjustment frequency response, the convolution kernel parameters are decomposed into low and high frequency components, for a static convolution kernel, its weights can be decomposed as follows:

[0024] ;

[0025] where, represents the weight of the kernel-by-kernel average , as a low-pass mean filter, with defined parameters 1 × 1 convolution, represents the residual part;

[0026] Step B2: After decomposition, the adaptive kernel dynamically adjusts the high and low frequency components, denoted as:

[0027] ;

[0028] where, , is the dynamic weight of each channel, which is predicted by a global pooling + convolution layer; according to the input context, the proportion of is dynamically adjusted, and the network focuses on a specific frequency band.

[0029] Specifically, the frequency selection balances the frequency power expansion receptive field of the input feature, specifically including the following steps:

[0030] Step C1: Frequency selection decomposes features into different frequency bands by applying different masks in the Fourier domain, denoted as:

[0031] ;

[0032] where, represents the inverse fast Fourier transform; is a binary mask designed to extract the corresponding frequency, expressed as:

[0033] ;

[0034] where, , is obtained from B+1 predefined frequency thresholds ;

[0035] Step C2: spatially dynamically re-weighting the frequency components of different bands, denoted as:

[0036] ;

[0037] wherein, is the frequency equalization feature after frequency selection learning, denotes the selection mapping of the b-th band.

[0038] Specifically, the step S3 context and spatial feature calibration network CSFCN adopts an asymmetric encoder-decoder architecture, introduces a context feature calibration module to construct a private context for each pixel to enhance its discriminability, and uses a spatial feature calibration module for spatial feature calibration; the context feature calibration module specifically includes:

[0039] Given the feature and process to capture highly abstract multi-scale contexts ;

[0040] Calculate the pixel context similarity , wherein and respectively represent the total number of pixels and the total number of contexts, and ;

[0041] Use as a guide to aggregate the context for each pixel, further adjust the response value of each semantic context, generate a fine-grained context, and realize the recalibration of the context. The context calibration is defined as:

[0042] ;

[0043] wherein, respectively represent the input, output, re-calibration factor and context, the value range of , denotes a pair function for calculating the affinity between features.

[0044] Specifically, the spatial feature calibration module specifically includes:

[0045] The feature is reconstructed by feature resampling, and the spatial coordinates of each position on the feature map are defined as , and the learned 2D offset map is ; the calibration function is denoted as:

[0046] ;

[0047] from The sample features of the place are used to derive the output ;

[0048] The calibration semantic features are adaptively fused by a gating strategy and the fine-grained features , bridging the representation gap between the calibration semantic features and the fine-grained features :

[0049] ;

[0050] wherein, and denote the gate mask;

[0051] The calibration and fusion process are integrated in a single spatial feature calibration module.

[0052] Specifically, the single spatial feature calibration module calibration and fusion process specifically includes:

[0053] Given the features and , the channels are unified to the same number C by two convolutional layers;

[0054] The bilinear interpolation is used to up-sample , and the up-sampled and are spliced, and the spliced input is input into a convolutional block to predict two sets of offset maps and , aligning the features of the two levels, , controlling the flow of features information of the two levels;

[0055] The calibrated cross-level features are element-wise summed to obtain the output, and the spatial feature calibration module is represented as:

[0056] ;

[0057] wherein, denotes a bilinear up-sampling function, and denote convolutional layers with BN and ReLU;

[0058] For , , the 1+tanh activation is used, becomes an identity mapping and , and the spatial feature calibration module is represented as:

[0059] .

[0060] Specifically, the improved wavelet pooling downsampling specifically comprises:

[0061] In the wavelet domain, the 2-order decomposition of the pooled features is carried out by fast wavelet transform, and is expressed as:

[0062] ;

[0063] ;

[0064] wherein is an approximation function, is a detail function, , are respectively called approximation coefficients and detail coefficients; and are respectively a time-reversed scale vector and a wavelet vector, n represents a sample in the vector, and j represents a resolution;

[0065] The image is processed by applying the transform to the row and the column respectively, and the detail subbands LH, HL and HH of each decomposition layer and the approximation subband LL of the highest decomposition layer are obtained; after performing the 2-order decomposition, the image features are reconstructed by using the 2-order wavelet subband, and the image features are pooled by 2 times by using the inverse FWT:

[0066] .

[0067] Specifically, the improved wavelet pooling upsampling specifically comprises: reversing the process of downsampling, performing 1-order wavelet decomposition on the upsampled feature map; the detail coefficient subband after the decomposition is upsampled by 2 times to form a new 1st layer decomposition, the initial decomposition becomes a 2nd layer decomposition, the new 2-order wavelet decomposition reconstructs the image features, and the IDWT is used for upsampling.

[0068] The method has the advantages that: the method uses a frequency-adaptive dilated convolution to improve the modules in the backbone network of a benchmark model, introduces a context and spatial feature calibration network in the encoder part of the network to realize context feature calibration and spatial feature calibration, and finally uses a wavelet pooling method to effectively suppress aliasing when ordinary convolution operation is used for downsampling, so that aliasing and performance degradation are avoided. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described and shown in the embodiments of the present application can be arranged and designed in various different configurations.

[0070] Figure 1 RT-DETR structure diagram in an embodiment of the present application;

[0071] Figure 2 RT-DETR flow chart in an embodiment of the present application;

[0072] Figure 3 FCW-RTDETR algorithm model architecture diagram in an embodiment of the present application;

[0073] Figure 4 Frequency adaptive hole convolution schematic diagram in an embodiment of the present application;

[0074] Figure 5 Context space and spatial feature calibration network schematic diagram in an embodiment of the present application;

[0075] Figure 6 Context feature calibration module schematic diagram in an embodiment of the present application;

[0076] Figure 7 Spatial feature calibration module schematic diagram in an embodiment of the present application;

[0077] Figure 8 Wavelet pooling down-sampling schematic diagram in an embodiment of the present application;

[0078] Figure 9 Wavelet pooling up-sampling schematic diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0080] It should be noted that: similar labels and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0081] The following will combine the drawingsFigures 1-9 Some embodiments of the present application will be described in detail. The following examples and features in the examples can be combined with each other in the case of no conflict.

[0082] The present application proposes a UAV aerial image small target detection method based on RT-DETR. In a preferred embodiment, as shown in Figure 2 , the method specifically comprises the following steps:

[0083] Step S1: improve the backbone network using adaptive frequency hole convolution, and extract the UAV aerial image features;

[0084] Step S2: perform scale-in feature interaction AIFI on the P5 layer in the backbone network, and input the obtained features into the cross-scale feature fusion network CCFM;

[0085] Step S3: improve the CCFM structure using the context and spatial feature calibration network CSFCN, and improve the up-sampling method in the CCFM using wavelet pooling, fuse features of different scales in the CCFM, and obtain an image feature sequence;

[0086] Step S4: use IoU-aware query selection to select a fixed number of features from the feature sequence output by the encoder as the initial target query of the decoder;

[0087] Step S5: the decoder iteratively optimizes the target query through the auxiliary prediction head, generates a bounding box and a confidence score, and completes the small target detection.

[0088] In this embodiment, the algorithm architecture of the present scheme is as shown in Figure 1 , and the content is as follows:

[0089] 1. Add adaptive frequency hole convolution to the backbone network ResNet18;

[0090] 2. Add context and spatial feature calibration network to the cross-scale feature fusion CCFM;

[0091] 3. Replace the ordinary pooling operation in the network with wavelet pooling.

[0092] Firstly, YOLO method as the most popular target detection method at present, has achieved a good balance in computing speed and accuracy, but because YOLO algorithm belongs to single-stage target detection method, the post-processing process of non-maximum suppression will not only affect the calculation speed, but also introduce hyperparameters, which will bring uncertainty to the model performance, so the method makes improvements and innovations to a kind of end-to-end target detection algorithm RT-DETR(Real-Time Detection Transformer) based on Transformer, and puts forward FCW-RTDETR algorithm, and the architecture is as Figure 3 Secondly, the traditional convolution has the problems of spatial hierarchical information loss and small object information reconstruction, and the dilated convolution increases the receptive field without reducing the resolution, reduces the information loss of up-sampling and down-sampling, so the method uses frequency-adaptive dilated convolution to improve the module in the backbone network of the benchmark model. Then, in the efficient hybrid encoder part of the network, although the multi-level feature fusion method can effectively improve the performance of small target detection, it cannot specially process the pixel context mismatch and feature misalignment problem, so the application introduces the context and spatial feature calibration network in the encoder part of the network to realize the context feature calibration and spatial feature calibration. Finally, since the ordinary convolution operation is usually ignored when down-sampling, which leads to aliasing and performance degradation, so the method adopts the method of wavelet pooling to effectively suppress aliasing.

[0093] I. Frequency-Adaptive Dilated Convolution

[0094] In this embodiment, frequency-adaptive dilated convolution (FADC) is introduced to improve the BasicBlock in Backbone. As shown in Figure 4 FADC includes three key strategies, namely adaptive dilation rate (AdaDR), adaptive kernel (AdaKern), and frequency selection (FreqSelect), aiming to improve each stage of dilated convolution, as shown in Figure 4 AdaDR adjusts the dilation rate in space, AdaKern operates on the convolution kernel weight, and FreqSelect directly balances the frequency power of the input feature to expand the receptive field.

[0095] 1. Adaptive Dilation Rate

[0096] The currently widely used dilated convolution can be expressed as follows:

[0097]

[0098] wherein, represents the pixel value of the position in the output feature map, represents the kernel size, represents the weight parameter of the kernel,

[0099] represents the pixel value of the position in the input feature map corresponding to , and the offset is . The variable represents the th position of the predefined grid sampling (-1, -1), (-1, 0), (-1, +1),..., (+1, +1). The receptive field can be expanded by increasing the dilation rate .

[0100] Previous studies have found that increasing the dilation rate will result in a decrease in the ability to capture frequency information. Specifically, according to the scaling property of the Fourier transform, when the dilation rate is increased from 1 to , the magnification of the convolution kernel is . Therefore, the response frequency of the convolution kernel is reduced to , causing the frequency response to shift from high frequency to low frequency. The convolution effectively works at a sampling rate of , which means it samples once every elements. However, this also means that it cannot capture frequencies beyond the Nyquist frequency, which is half of the sampling rate ( ). Therefore, dilated convolution may not be able to adequately represent high-frequency information beyond this limit.

[0101] Specifically, the present method first uses the discrete Fourier transform (DFT) to transform the feature map into the frequency domain, which can be expressed as:

[0102]

[0103] wherein represents the complex number array output by the DFT. and represent the height and width of the feature map , respectively, and represent the coordinates of the feature map . The feature map ​The normalized frequencies in height and width are given by |u| and |v|. After moving the low frequency part to the center, u is removed from the set Take the value from Therefore, the high frequency set greater than the Nyquist frequency It cannot be captured accurately, which limits its bandwidth.

[0104] Based on the above analysis, the choice of dilation rate can be seen as a trade-off between a large receptive field and effective bandwidth. Considering that the input feature map varies in space, the optimal dilation rate for each pixel can be different. Therefore, this method introduces an adaptive dilation rate (AdaDR) strategy to achieve a better balance. It assigns a different dilation rate to each pixel:

[0105]

[0106] It can be predicted by a convolutional layer with parameter θ. This method introduces a ReLU layer to ensure the non-negativity of the hole, and also adopts a modulation mechanism. It aims to maximize the receptive field while minimizing the frequency information lost for each pixel. For a local feature with a window size of s and a center of p, this method is called Its receptive field and A set The frequency in the can not be accurately captured. Therefore, the high frequency power can be calculated to measure the lost frequency information. Therefore, The optimization can be written as:

[0107]

[0108] However, due to the frequency set Due to the discrete nature of and the non-differentiable nature of HP, direct optimization may be unrealistic. Therefore, this method chooses to directly optimize , that is, by The lower position increases the void rate to increase the receptive field. Higher positions suppress the void rate to reduce the loss of frequency information. This method can be expressed as:

[0109]

[0110] here, and Represents the pixels with the highest / lowest (e.g., 25%) high-frequency power, respectively.

[0111] 2. Adaptive Kernel

[0112] AdaDR achieves a delicate balance between effective bandwidth and receptive field by assigning a separate hole rate to each pixel, jointly optimizing these two factors. Effective bandwidth is closely related to the weights of the convolution kernel, playing a key role. Traditional convolution kernel learning spans features across different frequency bands, which are essential for understanding complex visual patterns, but they become static once trained. To further enhance effective bandwidth, before introducing dynamic weighting to adjust the frequency response, this method decomposes the convolution kernel parameters into low and high frequency components. This process only adds a small amount of additional parameters and computational overhead. For a static convolution kernel, its weights W can be decomposed as follows:

[0113]

[0114] Here, represents the kernel-wise average . It acts as a low-pass mean filter, then 1 × 1 convolution with defined parameters. Higher mean values decay high frequency components more easily. Denote the residual part with , capturing local differences, extracting high frequency components. After decomposition, AdaKern dynamically adjusts the high and low frequency components and can be formally represented as:

[0115]

[0116] where , is the dynamic weight for each channel, predicted by a simple and lightweight global pooling + convolution layer. According to the input context, the proportion of is dynamically adjusted, allowing the network to focus on specific frequency bands and adapt to the complexity of visual patterns in features. This dynamic frequency adaptive approach enhances the network's ability to capture both low frequency context and high frequency local details. This in turn increases the effective bandwidth, thereby improving performance in segmentation tasks that require different feature extraction at different frequencies.

[0117] 3. Frequency Selection

[0118] Traditional convolution usually acts as a high-pass filter. Therefore, the resulting features tend to exhibit a higher proportion of high frequency components. This tendency leads to the adoption of smaller overall hole rates to maintain a higher effective bandwidth, but unfortunately at the expense of the size of the receptive field. FreqSelect aims to enhance the receptive field by balancing the high and low frequency components in the feature representation.

[0119] Specifically, FreqSelect first decomposes the feature into different frequency bands by applying different masks in the Fourier domain:

[0120]

[0121] where denotes the inverse fast Fourier transform. is a binary mask designed for extracting the corresponding frequency:

[0122]

[0123] Here, , is obtained from B+1 predefined frequency thresholds Subsequently, FreqSelect spatially dynamically re-weights the frequency components in different frequency bands. This can be denoted as:

[0124]

[0125] where is the frequency-balanced feature after FreqSelect learning, denotes the selection map of the b-th frequency band. Specifically, the frequency is decomposed into 4 frequency bands in an octave manner, i.e. .

[0126] II. Contextual space and spatial feature calibration network

[0127] The overall architecture of the contextual space and spatial feature calibration network used by the present application is shown in Figure 5 , which adopts an asymmetric encoder-decoder architecture. The present application uses a lightweight backbone network to extract multi-level features. Then, a context feature calibration module (CFC) is introduced to construct a private context for each pixel to enhance its discriminability. Further, the present method uses a spatial feature calibration module (SFC) for spatial feature calibration to produce strong semantic features with precise boundaries.

[0128] 1. Context Feature Calibration Module

[0129] Contextual information can provide rich scene category priors to correct unexpected misclassifications. However, previous methods aggregate context within a predefined region, ignoring that not all context contributes equally to the classification of a given pixel, which inevitably leads to the problem of context mismatch. Moreover, since large objects contain more pixels, the captured context is heavily biased towards large objects and leads to over-smoothing of small objects. Based on the above insights, our method adopts a Context Feature Calibration module to crop and refine the semantic context for each pixel.

[0130] Typically, large objects dominate an image, and global context tends to be similar to these large objects rather than spatial details. This method matches context to each pixel to capture the context that is most instructive for its classification. That is, this method aggregates context from regions with closer semantic distances rather than regions with closer spatial distances. Specifically, given a feature , first of all Processing to capture highly abstract multi-scale context Then, we calculate the pixel context similarity , which is the spatial attention map, where and Represent the total number of pixels and the total number of contexts respectively. This method uses As a guide, context is aggregated for each pixel to achieve context feature calibration. Finally, this method further adjusts the response value of each semantic context to generate fine-grained context, thereby achieving context recalibration. Mathematically, CFC can be defined as:

[0131]

[0132] In the formula denote input, output, rescaling factor and context respectively, The value range is , Represents a pairwise function used to calculate the affinity between features.

[0133] Figure 6 The CFC module is shown in . In order to pursue efficiency, this method uses the Cascaded Pyramid Pooling (CPP) module to reuse the pooling results of the previous layers, reducing unnecessary redundant calculations. Figure 6 As shown, given a feature , this method first uses a 1×1 convolution layer to generate a dimensionality reduction feature , where C′ is much smaller than C (by default, C′ = 32 and C = 256). Then, the CPP block is used to obtain multi-scale context In particular, when the pooling layer outputs height (Generally, the output width is equal to the height), Default settings . Since the pooling layer performs pooling on a homogeneous spatial grid (e.g., 3×3), this may lead to information redundancy in the pooled features when the aspect ratio of the input is not 1. To this end, our method keeps the aspect ratio of the pooled output size equal to the input. For example, when training on square crops, the minimum output size of our method is 1×1, while for an input of 1024×2048, the minimum output size of our method is 1×2. Finally, our method feeds Z into two convolutional layers (with BN and ReLU) to produce two forms of contextual representations, namely and .

[0134] 2. Spatial Feature Calibration Module

[0135] To compensate for the loss of spatial details caused by downsampling, previous methods use cross-level feature fusion to enhance high-level semantic features with low-level details. and high-resolution features , firstly use standard bilinear interpolation to Upsample, and then and after upsampling The output is obtained by adding or concatenating them. However, due to spatial misalignment and a large representation gap, directly fusing them still fails to achieve satisfactory performance. Moreover, since feature maps encode various semantic information, performing uniform feature alignment along the channel dimension can also hurt performance. To alleviate these issues, this method groups the channel dimension into multiple sub-features and performs alignment operations separately, and seamlessly integrates a gating mechanism to adaptively fuse cross-layer features.

[0136] Regarding feature calibration, this method proposes to reconstruct features by using feature resampling. Specifically, assuming that the spatial coordinates of each position on the feature map are , the learned 2D offset map is The calibration function of this method is It can be expressed as:

[0137]

[0138] from The sample features at the output Since p represents an arbitrary position, the above formula enumerates all integral positions and uses a bilinear interpolation kernel to obtain the characteristics of the sampling position. Intuitively, if Figure 5 As shown in the SFC sampling process, this method only needs to sample the most favorable feature (blue point) at the current position to replace the feature at the current position (green point) for calibration. In addition, in order to perform more precise calibration, this method divides the feature F into G groups along the channel dimension, and then aligns the features in each group separately, as shown in Figure 7 shown.

[0139] because 'usually contains rich spatial details, while 'contains more semantic information, and simply calibrating and fusing them cannot achieve satisfactory results. This is because feature calibration alone cannot handle the huge representation differences between features. Therefore, this method further proposes to adaptively fuse and calibrate semantic features through a gating strategy. and fine-grained features , to bridge the representation gap between them:

[0140]

[0141] In the formula and Represents a gate mask.

[0142] To improve efficiency, this method integrates the calibration and fusion processes into a single spatial feature calibration module. Figure 7 As shown, given the feature and , this method first unifies their channels to the same number C (128 by default) through two convolutional layers. Secondly, this method uses bilinear interpolation to Then, the upsampled and Then, it is input into a convolution block to predict two sets of offset maps and , used to align features of two levels, two gate masks , , which is used to control the flow of feature information between the two levels. Finally, the calibrated cross-level features are summed element by element to obtain the output. In summary, the SFC module can be formally written as:

[0143]

[0144] in represents the bilinear upsampling function, and Represents a convolutional layer with BN and ReLU.

[0145] It is worth noting that the proposed method adopts the idea of residual to alleviate the negative impact of large initial offset and mask prediction error, which makes the proposed method can be inserted into the network without compromising its original performance (if the convolutional block is constructed as zero mapping). That is, the proposed method initializes the weights of the last convolution layer of the convolutional block to zero to learn more accurate offset and mask step by step. In addition, the proposed method selects 1+tanh activation for the gate mask. Therefore, at the beginning becomes an identity mapping and . Now, the SFC can be represented as:

[0146] .

[0147] Three, wavelet pooling

[0148] Compared with general object detection, aliasing has a more significant impact on micro object detection. This is because micro objects themselves have more high-frequency features, and their pixel values will change rapidly in a very small range, making them more sensitive to aliasing effects. Unfortunately, the convolutional neural network relied on by the current backbone network for micro object detection often ignores the Nyquist sampling theorem when performing downsampling operations, which directly leads to serious aliasing problems, thereby affecting the accuracy and reliability of detection. In order to suppress the aliasing effect, the proposed method replaces the upsampling and downsampling methods in the backbone network and neck network with the wavelet pooling method.

[0149] 1. Down-sampling

[0150] The wavelet pooling method adopted pools features by performing 2nd-order decomposition in the wavelet domain according to the fast wavelet transform (FWT), which is a more efficient implementation of the two-dimensional discrete wavelet transform (DWT):

[0151]

[0152]

[0153] where is the approximation function, and ψ is the detail function, , are called approximation coefficients and detail coefficients, respectively. and are the time-reversed scaling and wavelet vectors, n denotes the samples in the vector, and j denotes the resolution. When using FWT to process the image, the method applies it twice (once on the rows and then again on the columns). Through such a combination, the method obtains the detail subbands (LH, HL, HH) of each decomposition layer and the approximation subband (LL) of the highest decomposition layer. After performing 2-order decomposition, the method reconstructs the image feature but only uses the 2-order wavelet subband. The method uses inverse FWT (IFWT) to perform 2-fold pooling on the image feature on the basis of inverse DWT (IDWT):

[0154]

[0155] Figure 8 An algorithm diagram of wavelet pooling downsampling is given.

[0156] 2. Upsampling

[0157] The introduced wavelet pooling algorithm performs upsampling by reversing the process of its downsampling. First, the upsampled feature map is subjected to 1-order wavelet decomposition. The detail coefficient subband after decomposition is upsampled by 2 times to form a new 1st layer decomposition. The initial decomposition becomes the 2nd layer decomposition. Finally, this new 2-order wavelet decomposition reconstructs the image feature so as to use IDWT for further upsampling. Figure 9 An algorithm diagram of wavelet pooling upsampling is introduced.

[0158] In a specific embodiment data, the data set used is the VisDrone2019 data set. The method performs comparative experiments on this data set with representative target detection algorithms to verify the effectiveness of the model proposed by the method.

[0159] The data set is collected by the AISKYEYE team of the Machine Learning and Data Mining Laboratory of Tianjin University. The benchmark data set contains 288 video clips, consisting of 261,908 video frames and 10,209 static images. Among them, the image data used by the method, 6471 are used for training, 548 are used for verification, and 3190 are used for testing. These data come from various unmanned aerial camera platforms, covering a wide range, including different cities (from 14 different cities in China, thousands of kilometers apart), different environments (urban and rural), different objects (pedestrians, vehicles, bicycles, etc.), and different densities (sparse and crowded scenes). It should be noted that the data set is collected using different models of unmanned aerial platforms in different scenes, different weather and lighting conditions.

[0160] The biggest difference between VisDrone2019 dataset and other datasets is that the object scale is small. Table 1 shows the target scale statistics of the dataset. According to the standard of MS COCO dataset, the target with resolution less than 32x32 is small target. As can be seen from the table, 59.6% of the targets in VisDrone2019 dataset are small targets, which is undoubtedly the biggest difficulty and challenge of this dataset.

[0161] Table 1 VisDrone2019 target scale statistics

[0162]

[0163] Visual analysis: In complex environments, small targets are affected by various factors such as different environments, uneven lighting, different UAV aerial heights, and object overlap causing occlusion. This method evaluates the performance of the FCW-RTDETR model in various scenarios. The detection results show that this scheme can accurately identify small targets from UAV aerial images and has good robustness.

[0164] Comparing the detection results of small targets under different conditions, the algorithm proposed in this method shows good adaptability in various environments and can effectively detect small targets in parking lots, parks and other scenes. The detection results have high accuracy and stability. Comparison of detection results under different light intensities shows that light changes have some impact on detection results, but overall, the algorithm shows strong robustness to light changes. Analysis of the impact of aerial height on detection results shows that at different heights, the algorithm can still detect targets well. Detection results under dense occlusion show that the algorithm can detect most of the occluded targets, but a small part of the targets are not correctly identified, indicating that there is still room for further optimization when dealing with severe occlusion problems.

[0165] In an embodiment, the FCW-RTDETR and RT-DETR-R18 models and the YOLOv8m model are compared in small target detection using unmanned aerial vehicle aerial images under various environmental conditions; the FCW-RTDETR is superior to the benchmark model RT-DETR-R18 and the current mainstream target detection model YOLOv8m in small target detection performance. The FCW-RTDETR comprehensively detects the very small pedestrians in the original image, while the RT-DETR-R18 has certain missed detection on pedestrians, and the YOLOv8m hardly detects pedestrians, indicating that the FCW-RTDETR has high sensitivity to micro targets. The RT-DETR-R18 misdetects the car above the original image as a van, misdetects the garbage can at the lower left as a person, and, like the YOLOv8m, misses the motor to the right and above, and the FCW-RTDETR is superior to other models in detection accuracy while having a low miss rate. Overall, the FCW-RTDETR exhibits excellent detection performance on small targets in various complex aerial images, making it suitable for high-altitude unmanned aerial vehicle detection tasks

[0166] For the foregoing embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily necessary for the present application.

[0167] In the above embodiments, the basic principles and main features of the present application and the advantages of the present application are described. Those skilled in the art should understand that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application should be within the protection scope of the claims of the present application.

Claims

1. An RT-DETR-based small target detection method for unmanned aerial vehicle aerial images, characterized in that, The FCW-RTDETR algorithm is improved based on the end-to-end object detection algorithm RT-DETR, including the following steps: Step S1: the backbone network is improved using adaptive frequency hole convolution to extract the UAV aerial image features; Step S2: the P5 layer in the backbone network is subjected to scale-in feature interaction AIFI, and the obtained features are input into the cross-scale feature fusion network CCFM; Step S3: the CCFM structure is improved using the context and spatial feature calibration network CSFCN, and the wavelet pooling is used to improve the up-sampling method in the CCFM, so as to fuse the features of different scales in the CCFM and obtain the image feature sequence; Step S4: the initial target query of the decoder is selected from the feature sequence output by the encoder using the IoU-aware query selection; Step S5: the decoder iteratively optimizes the target query through the auxiliary prediction head to generate the bounding box and the confidence score, and completes the small target detection.

2. The RT-DETR-based small target detection method for UAV aerial images according to claim 1, wherein, The improvement of the adaptive frequency hole convolution in the step S1 includes adaptive hole rate, adaptive kernel and frequency selection, the adaptive hole rate adjusts the hole rate in space, and specifically includes the following steps: Step A1: Use the discrete Fourier transform Transform the feature map into the frequency domain, denoted as: ； wherein, represents a complex array of DFT output; and respectively represent the height and width of a feature map , and respectively represent the coordinates of a feature map ; the normalized frequencies of a feature map in height and width are given by |u| and |v|, u takes values from the set , after moving the low frequency part to the center, v takes values from the set Step A2: an adaptive hole rate strategy is introduced, and different hole rates are allocated for each pixel, which is represented as: ； wherein, represents a pixel value at a position in the output feature map, represents a kernel size, represents a weight parameter of the kernel, represents a pixel value at a position in the input feature map corresponding to , and an offset is , is predicted by a convolutional layer with a parameter of​ Step A3: A modulation mechanism is adopted and a ReLU layer is introduced to maximize the receptive field while minimizing the frequency information lost for each pixel. A local feature with a center of p and a window size of s is defined as , its receptive field and There is a positive correlation; Step A4: Calculate high frequency power Measure the lost frequency information, directly optimize At Lower position to increase the empty rate to enhance large receptive field, in Higher position to inhibit the empty rate to reduce the loss of frequency information, expressed as: ； wherein, and respectively represent pixels with the highest and lowest high frequency power.

3. The RT-DETR-based small target detection method for UAV aerial images according to claim 2, characterized in that, The adaptive kernel operates on the convolution kernel weight, and specifically includes the following steps: Step B1: Before introducing the dynamic weighting adjustment frequency response, decompose the convolution kernel parameters into low and high frequency components, for a static convolution kernel, its weights can be decomposed as follows: ； in, Represents the average weight of each core , as a low-pass Mean filter, using The parameters defined are used for 1 × 1 convolution. represents the residual part; Step B2: after decomposition, the adaptive kernel dynamically adjusts the high-frequency and low-frequency components, which is represented as: ； wherein, , is the dynamic weight of each channel, predicted by a global pooling + convolution layer; dynamically adjusts the proportion of according to the input context, the network focuses on a specific frequency band.

4. The RT-DETR-based small target detection method for UAV aerial images according to claim 2, characterized in that, The frequency selection balances the frequency power expansion receptive field of the input features, and specifically includes the following steps: Step C1: the frequency selection decomposes the features into different frequency bands by applying different masks in the Fourier domain, which is represented as: ； wherein denotes the inverse fast Fourier transform; is a binary mask designed for extracting the corresponding frequency, expressed as: ； wherein , is obtained from B+1 predefined frequency thresholds ; Step C2: the spatial dynamic reweighting is performed on the frequency components of different frequency bands, which is represented as: ； wherein, is a frequency equalization feature after frequency selection learning, denotes the selection mapping of the bth frequency band.

5. The RT-DETR-based small target detection method for UAV aerial images according to claim 1, characterized in that, The context and spatial feature calibration network CSFCN in the step S3 adopts an asymmetric encoder-decoder architecture, introduces a context feature calibration module to construct a private context for each pixel to enhance its discriminability, and uses a spatial feature calibration module for spatial feature calibration; the The context feature calibration module specifically includes: Given features And processing, capturing highly abstract multi-scale context ;​ Computing pixel context similarity wherein and respectively denote the total number of pixels and the total number of contexts, and ; Using As a guide, aggregate the context for each pixel, further adjust the response value of each semantic context, generate fine-grained context, realize the recalibration of the context, and the context calibration is defined as: ； wherein, respectively represent input, output, rescaling factor and context, with a value range of , denotes a pair-wise function for computing affinity between features.

6. A method for detecting small targets in UAV aerial images based on RT-DETR as claimed in claim 5, characterized in that: The spatial feature calibration module specifically includes: The features are reconstructed by feature resampling, and the spatial coordinates of each position on the feature map are defined as The learned 2D offset map is The calibration function is represented as: ； From the sample characteristics at the output ; Adaptively fusing calibration semantic features by gating strategies and fine-grained features , bridging the representation gap between calibration semantic features and fine-grained features ​ ； wherein and represents a gate mask; The calibration and fusion process is integrated in a single spatial feature calibration module.

7. The RT-DETR-based small target detection method for UAV aerial images according to claim 6, characterized in that, The calibration and fusion process of the single spatial feature calibration module specifically includes: Given features and unify the channels to the same number C by two convolutional layers; adopting bilinear interpolation to upsample, splicing the upsampled and inputting into a convolution block, predicting two groups of offset maps and aligning two levels of features, , controlling the flow of two levels of feature information; The cross-level features after calibration are summed element by element to obtain the output, and the spatial feature calibration module is represented as: ； wherein, denotes a bilinear up-sampling function, and denotes a convolutional layer with BN and ReLU; To , with 1 + tanh activation, becomes an identity mapping and , the spatial feature alignment module is represented as: 。 8. The RT-DETR-based small target detection method for UAV aerial images according to claim 1, wherein, The improved wavelet pooling downsampling specifically includes: The pooled features are subjected to 2-order decomposition in the wavelet domain using fast wavelet transform, which is represented as: ； ； wherein is an approximation function, is a detail function, , are called approximation coefficients and detail coefficients, respectively; and are the time-reversed scale and wavelet vectors, respectively, n denotes the samples in the vector, and j denotes the resolution. The image is processed by applying the transform to the rows and columns respectively to obtain the detail subbands LH, HL and HH of each decomposition layer, and the approximation subband LL of the highest decomposition layer; after performing 2-order decomposition, the image features are reconstructed using 2-order wavelet subbands, and the image features are pooled by 2 times using inverse FWT: 。 9. The RT-DETR-based small target detection method for UAV aerial images according to claim 8, wherein, The improved wavelet pooling upsampling specifically comprises: reversing the process of down-sampling, performing 1st-order wavelet decomposition on the up-sampled feature map; the decomposed detail coefficient sub-band is up-sampled by 2 times to form a new 1st layer decomposition, the initial decomposition becomes a 2nd layer decomposition, the new 2nd-order wavelet decomposition reconstructs the image feature, and IDWT is used for up-sampling.

Images