Remote sensing small target detection method, device and equipment based on improved RT-DETR model

Abstract

The application discloses a remote sensing small target detection method, device and equipment based on an improved RT-DETR model, and the method comprises the following steps: S1, acquiring a remote sensing image dataset as a training set, performing multi-scale transformation on remote sensing images in the remote sensing image dataset to obtain remote sensing images with different resolutions, and then performing enhancement processing on the remote sensing images with different resolutions to obtain processed remote sensing images; S2, inputting the processed remote sensing images into the improved RT-DETR model to train the improved RT-DETR model and obtain a trained remote sensing small target detection network model; and S3, performing remote sensing small target detection on inputted remote sensing images by using the remote sensing small target detection network model. Through the improved RT-DETR model, the feature extraction capability of the backbone network on small targets can be improved, and the decoder can be optimized for small target sizes, so that the detection accuracy of remote sensing small targets is improved.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision technology, specifically to a remote sensing small target detection method, apparatus, and device based on an improved RT-DETR model. Background Technology

[0002] Remotely Spaced Object Detection (RSOD) is an important branch of computer vision, aiming to identify and locate targets in remote sensing images acquired from platforms such as satellites and drones. However, remote sensing images are significantly complex, manifested in the following ways: the same target varies greatly in size across images of different resolutions; small targets (such as vehicles and ships) often occupy only a few pixels, making their features blurry and difficult to identify; cloud cover, vegetation cover, and building shadows result in low contrast between the target and the background, and unclear edge features; traditional target detection algorithms (such as YOLO and Faster R-CNN) are prone to missed or false detections when processing low-quality remote sensing images due to insufficient feature extraction.

[0003] The RT-DETR (Real-Time Detection Transformer) algorithm, as an end-to-end detection framework, has received widespread attention and has been applied to the field of remote sensing target detection due to its excellent performance in real-time and accuracy. However, it has the following problems when dealing with small remote sensing targets: the backbone network has insufficient ability to extract features of small targets and lacks an effective dimensionality expansion mechanism; the decoder is not optimized for the size of small targets, and insufficient receptive field leads to the loss of contextual information; the loss function does not highlight the importance of small targets, and the imbalance of samples affects the detection accuracy. Summary of the Invention

[0004] The present invention aims to provide a remote sensing small target detection method, apparatus and equipment based on an improved RT-DETR model to improve the above-mentioned problems.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0006] A remote sensing small target detection method based on an improved RT-DETR model, the method comprising:

[0007] S1. Obtain the remote sensing image dataset as the training set, perform multi-scale transformation on the remote sensing images in the dataset to obtain remote sensing images of different resolutions, and then perform enhancement processing on the remote sensing images of different resolutions to obtain the processed remote sensing images.

[0008] S2, the processed remote sensing image is input into the improved RT-DETR model to train the improved RT-DETR model, resulting in a trained remote sensing small target detection network model; wherein, the backbone network of the improved RT-DETR model uses element-wise multiplication to process the features of the input remote sensing image, mapping the input features to a high-dimensional nonlinear space through element-wise multiplication, thereby enhancing the ability to extract small target features; the decoding process of the decoder of the improved RT-DETR model includes a feature extraction stage, a feature downsampling stage, and a feature fusion stage; in the feature extraction stage, features targeting small targets are introduced. The extraction branch is configured to adapt to small target sizes by adjusting the kernel size and stride of the extraction branch. Dilated convolutions are added to the extraction branch to expand the receptive field and obtain richer contextual information. In the feature downsampling stage, the SPDConv module is introduced. Through the concatenation and convolution of multi-branch sub-region features of the SPDConv module, the details of small targets are preserved while reducing feature resolution. In the feature fusion stage, the CSPOmniKernel module is introduced. Through the feature splitting, multi-directional feature capture, and attention enhancement of the CSPOmniKernel module, the feature extraction capability of small targets is balanced with the computational complexity.

[0009] S3, use the remote sensing small target detection network model to perform remote sensing small target detection on the input remote sensing image.

[0010] Preferably, in step S2, the SPDConv module is implemented as follows:

[0011] During module initialization, the number of input channels inc and the number of output channels ouc are set, and a 3×3 convolutional layer with an input channel count of inc×4 is set internally.

[0012] During the forward propagation process, the input feature map is sampled for non-overlapping sub-regions with a step size of 2, generating 4 complementary sub-features, including: the upper left sub-region of even-numbered rows and even-numbered columns of pixels, the lower left sub-region of odd-numbered rows and even-numbered columns of pixels, the upper right sub-region of even-numbered rows and odd-numbered columns of pixels, and the lower right sub-region of odd-numbered rows and odd-numbered columns of pixels.

[0013] The four sub-features are concatenated along the channel dimension to form an intermediate feature with inc×4 channels. This intermediate feature is then compressed using a 3×3 convolution to the output channel number ouc. The output resolution is equal to the input feature resolution. Figure 1 / 2 characteristics.

[0014] Preferably, in step S2, the CSPOmniKernel module includes 1×1 convolutional layers cv1 and cv2 and an OmniKernel submodule, which is implemented as follows:

[0015] During module initialization, the number of feature channels dim and the split ratio e are set to 0.25. The number of input channels of the OmniKernel submodule is dimo, where dimo is the integer part of dim × e.

[0016] During the forward propagation, the input features are split into two branches proportionally after being convolved by cv1: the features of the dimo channel are processed by the OmniKernel submodule, and the features of the other channels are directly retained as the identity branch.

[0017] The output features of the OmniKernel submodule and the features of the identity branch are concatenated along the channel dimension and then output as fused features via cv1 convolution.

[0018] Preferably, the OmniKernel submodule includes an input convolutional layer with GELU activation, an output convolutional layer, a multi-directional depthwise convolutional group, a frequency domain attention module, a spatial attention module, and a ReLU activation layer; wherein:

[0019] The multi-directional depth convolution group includes four depth-separable convolutions: a 1×31 convolution for capturing long-range features in the horizontal direction, a 31×1 convolution for capturing long-range features in the vertical direction, a 31×31 convolution for capturing global spatial features, and a 1×1 convolution for local feature interactions within the channel.

[0020] The frequency domain attention module generates attention weights through global average pooling and 1×1 convolution, and weights the features in the Fourier domain to suppress high-frequency noise; the spatial attention module generates a spatial attention map through global average pooling and 1×1 convolution, and weights the features processed in the frequency domain to enhance the response of small target regions.

[0021] During forward propagation, the input features are processed by the multi-directional depth convolutional group, the frequency domain attention module, and the spatial attention module after passing through the input convolutional layer. All processing results are added to the original input features, activated by ReLU, and then output through the output convolutional layer.

[0022] Preferably, the element-wise multiplication operation in the backbone network generates high-dimensional features in the following way:

[0023] Single-level element-wise multiplication is represented as:

[0024]

[0025] in, , For the weight vector, , for , transpose, For the input feature vector, For feature dimensions; Represents the weight vector The i-th element, Represents the weight vector The j-th element, These represent the i-th and j-th features of the input feature vector, respectively; after expansion, they generate... The nonlinear feature terms achieve dimensional expansion without increasing computational cost; through layer-by-layer element-wise multiplication, the feature dimension grows exponentially, with the 1st layer... The layer output feature dimension is .

[0026] Preferably, a loss term for small targets is added to the loss function of the improved RT-DETR model, including a small target size loss term and a small target localization loss term; wherein, the small target size loss term adopts a combination of IoU loss and CIoU loss, and the calculation formula is as follows:

[0027]

[0028] in, For the prediction box, For the true frame, The square of the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box. Let be the diagonal length of the smallest closure region containing both the predicted and ground truth boxes. These are the weighting coefficients. Used to measure the aspect ratio difference between the predicted bounding box and the ground truth bounding box.

[0029] Preferably, in step S2, the improved RT-DETR model employs an adaptive learning rate adjustment strategy during training. This strategy is a cosine annealing learning rate adjustment method, with the following formula:

[0030]

[0031] in, The current learning rate, To minimize the learning rate, To achieve the maximum learning rate, This is the current training round number. This represents the maximum number of training rounds.

[0032] Preferably, in step S3, the working principle of the remote sensing small target network model is as follows:

[0033] Multi-scale visual features of remote sensing images are extracted using a backbone network to generate a feature pyramid containing spatial semantic information.

[0034] A set of learnable target query vectors is initialized through the target query vector generation module. These target query vectors are used to encode prior knowledge of the target detection task, including target location, size, and category.

[0035] The feature pyramid is input into the encoder for multi-layer feature enhancement. The encoder uses a multi-head self-attention mechanism and a feedforward neural network to perform cross-regional context modeling of visual features at different levels, thereby enhancing the feature representation of small target regions. At the same time, during the feature pyramid construction stage, the SPDConv module is used to downsample the high-resolution feature layer to preserve the details of small targets.

[0036] The decoder receives the enhanced features and target query vector output by the encoder and establishes a correspondence between the target query and visual features through a cross-attention mechanism. Specifically: First, the visual feature-modulated attention mechanism for the target query calculates the similarity between the target query vector and the visual features, selecting region features related to small targets from the feature pyramid and suppressing background noise interference. Then, the target query-guided attention mechanism for the visual features uses the modulated target query vector to guide the feature aggregation direction, enabling the decoder to focus on the detailed features of small targets. The features after bidirectional attention interaction are input into the attention module to model the relationship between targets, avoiding confusion in the detection of adjacent small targets. At the same time, the CSPOmniKernel module is introduced in the feature fusion stage of the decoder to enhance the discriminative power of small target features.

[0037] The detection results output by the decoder are processed by the Hungarian matching algorithm to perform optimal matching between the predicted bounding boxes and the ground truth bounding boxes. The classification loss and localization loss are calculated: the classification loss adopts the focus loss to solve the problem of imbalance of small target samples; the localization loss adopts the CIoU loss, which combines the overlap area between the predicted bounding box and the ground truth bounding box, the distance between the center point and the aspect ratio difference to improve the localization accuracy of small targets.

[0038] By optimizing network parameters through end-to-end training, the weights of the backbone network, encoder, and decoder are updated synchronously during training, enabling the model to adaptively learn the feature representation and detection logic of remote sensing small targets.

[0039] This invention also provides a remote sensing small target detection device based on an improved RT-DETR model, comprising:

[0040] The image processing unit is used to acquire the remote sensing image dataset as the training set, perform multi-scale transformation on the remote sensing images in the dataset to obtain remote sensing images of different resolutions, and then perform enhancement processing on the remote sensing images of different resolutions to obtain the processed remote sensing images.

[0041] The model training unit is used to input the processed remote sensing image into the improved RT-DETR model for training, resulting in a trained remote sensing small target detection network model. The backbone network of the improved RT-DETR model uses element-wise multiplication to process the features of the input remote sensing image, mapping the input features to a high-dimensional nonlinear space to enhance the extraction capability of small target features. The decoder of the improved RT-DETR model includes a feature extraction stage, a feature downsampling stage, and a feature fusion stage. In the feature extraction stage, features targeting small targets are introduced. The system extracts features from branches and configures them to adapt to small target sizes by adjusting the kernel size and stride of the extraction branches. Dilated convolutions are added to the extraction branches to expand the receptive field and obtain richer contextual information. In the feature downsampling stage, the SPDConv module is introduced. Through the concatenation and convolution of multi-branch sub-region features in the SPDConv module, the detailed information of small targets is preserved while reducing feature resolution. In the feature fusion stage, the CSPOmniKernel module is introduced. Through feature splitting, multi-directional feature capture, and attention enhancement in the CSPOmniKernel module, the feature extraction capability and computational complexity of small targets are balanced.

[0042] The detection unit is used to perform remote sensing small target detection on the input remote sensing image using the remote sensing small target detection network model.

[0043] This invention also provides a remote sensing small target detection device based on an improved RT-DETR model, which includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the remote sensing small target detection method based on the improved RT-DETR model as described above.

[0044] In summary, the improved RT-DETR model in this embodiment has the following advantages compared to the existing RT-DETR model:

[0045] 1. The backbone network uses element-wise multiplication to process the features of the input remote sensing image. By mapping the input features to a high-dimensional nonlinear space through element-wise multiplication, the ability to extract features of small targets is enhanced.

[0046] 2. In the feature extraction stage, an extraction branch for small target features is introduced and configured to adapt to the size of small targets by adjusting the kernel size and stride of the extraction branch. Dilated convolutions are added to the extraction branch to expand the receptive field and obtain richer contextual information.

[0047] 3. In the feature downsampling stage, the SPDConv module is introduced. Through the multi-branch sub-region feature concatenation and convolution fusion of the SPDConv module, the detailed information of small targets is preserved when the feature resolution is reduced.

[0048] 4. In the feature fusion stage, the CSPOmniKernel module is introduced. Through feature splitting, multi-directional feature capture, and attention enhancement of the CSPOmniKernel module, the feature extraction capability of small targets and computational complexity are balanced. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 The diagram shown is a flowchart of the remote sensing small target detection method based on the improved RT-DETR model provided in Embodiment 1 of the present invention.

[0051] Figure 2 The diagram shown is a structural schematic of the improved RT-DETR model provided in Embodiment 1 of the present invention;

[0052] Figure 3 The diagram shown is a schematic diagram of the remote sensing small target detection device based on the improved RT-DETR model provided in Embodiment 2 of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0055] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0056] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0057] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0058] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0059] The use of "first" and "second" in the embodiments is merely to distinguish similar objects and does not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0061] Example 1

[0062] Embodiment 1 of the present invention provides a remote sensing small target detection method based on an improved RT-DETR model, which can be implemented by a remote sensing small target detection device based on an improved RT-DETR model (hereinafter referred to as the detection device). In particular, it is executed by one or more processors within the detection device to implement the following method:

[0063] S1. Obtain the remote sensing image dataset as the training set, perform multi-scale transformation on the remote sensing images in the dataset to obtain remote sensing images of different resolutions, and then perform enhancement processing on the remote sensing images of different resolutions to obtain the processed remote sensing images.

[0064] In this embodiment, the detection device may be an electronic device equipped with a processor, the processor having a computer program for the detection method and the computer program being executable, such as a computer, smartphone, smart tablet, workstation, etc., which are not limited here.

[0065] In this embodiment, specifically, the remote sensing image dataset can be the VisDrone2019 and DOTA-v1.0 datasets, but is not specifically limited to them.

[0066] S2. The processed remote sensing image is input into the improved RT-DETR model to train the improved RT-DETR model and obtain a trained remote sensing small target detection network model.

[0067] Among them, the RT-DETR (Real-Time Detection Transformer) model is a real-time object detection model based on the Transformer architecture. It combines the advantages of CNN and Transformer, aiming to improve the balance between detection accuracy and speed.

[0068] In this embodiment, the improvements of the improved RT-DETR model over the existing RT-DETR model include the backbone network, decoder, loss function, and learning rate adjustment strategy, which are described in detail below.

[0069] 1. Backbone Network

[0070] In this embodiment, the backbone network of the improved RT-DETR model uses element-wise multiplication to process the features of the input remote sensing image. By mapping the input features to a high-dimensional nonlinear space through element-wise multiplication, the ability to extract features of small targets is enhanced.

[0071] Single-level element-wise multiplication is represented as:

[0072]

[0073] in, , For the weight vector, , for , transpose, For the input feature vector, For feature dimensions; Represents the weight vector The i-th element, Represents the weight vector The j-th element, These represent the i-th and j-th features of the input feature vector, respectively; after expansion, they generate... The nonlinear feature terms achieve dimensional expansion without increasing computational cost; through layer-by-layer element-wise multiplication, the feature dimension grows exponentially, with the 1st layer... The layer output feature dimension is .

[0074] 2. Decoder

[0075] In this embodiment, the decoding process of the improved RT-DETR model decoder includes a feature extraction stage, a feature downsampling stage, and a feature fusion stage. In the feature extraction stage, an extraction branch for small target features is introduced and configured to adapt to the small target size by adjusting the kernel size and stride of the extraction branch. Dilated convolutions are added to the extraction branch to expand the receptive field and obtain richer contextual information. In the feature downsampling stage, the SPDConv module is introduced. Through the concatenation and convolution of multi-branch sub-region features of the SPDConv module, the detailed information of small targets is preserved while reducing the feature resolution. In the feature fusion stage, the CSPOmniKernel module is introduced. Through the feature splitting, multi-directional feature capture, and attention enhancement of the CSPOmniKernel module, the feature extraction capability and computational complexity of small targets are balanced.

[0076] Specifically, the SPDConv module is implemented as follows:

[0077] During module initialization, the number of input channels inc and the number of output channels ouc are set, and a 3×3 convolutional layer with an input channel count of inc×4 is set internally.

[0078] During the forward propagation process, the input feature map is sampled for non-overlapping sub-regions with a step size of 2, generating 4 complementary sub-features, including: the upper left sub-region of even-numbered rows and even-numbered columns of pixels, the lower left sub-region of odd-numbered rows and even-numbered columns of pixels, the upper right sub-region of even-numbered rows and odd-numbered columns of pixels, and the lower right sub-region of odd-numbered rows and odd-numbered columns of pixels.

[0079] The four sub-features are concatenated along the channel dimension to form an intermediate feature with inc×4 channels. This intermediate feature is then compressed using a 3×3 convolution to the output channel number ouc. The output resolution is equal to the input feature resolution. Figure 1 / 2 characteristics.

[0080] The CSPOmniKernel module includes 1×1 convolutional layers cv1 and cv2, and an OmniKernel submodule, which is implemented as follows:

[0081] During module initialization, the number of feature channels dim and the split ratio e are set to 0.25. The number of input channels of the OmniKernel submodule is dimo, where dimo is the integer part of dim × e.

[0082] During the forward propagation, the input features are split into two branches proportionally after being convolved by cv1: the features of the dimo channel are processed by the OmniKernel submodule, and the features of the other channels are directly retained as the identity branch.

[0083] The output features of the OmniKernel submodule and the features of the identity branch are concatenated along the channel dimension and then output as fused features via cv1 convolution.

[0084] The OmniKernel submodule includes an input convolutional layer with GELU activation, an output convolutional layer, a multi-directional depthwise convolutional group, a frequency domain attention module, a spatial attention module, and a ReLU activation layer; wherein:

[0085] The multi-directional depth convolution group includes four depth-separable convolutions: a 1×31 convolution for capturing long-range features in the horizontal direction, a 31×1 convolution for capturing long-range features in the vertical direction, a 31×31 convolution for capturing global spatial features, and a 1×1 convolution for local feature interactions within the channel.

[0086] The frequency domain attention module generates attention weights through global average pooling and 1×1 convolution, and weights the features in the Fourier domain to suppress high-frequency noise; the spatial attention module generates a spatial attention map through global average pooling and 1×1 convolution, and weights the features processed in the frequency domain to enhance the response of small target regions.

[0087] During forward propagation, the input features are processed by the multi-directional depth convolutional group, the frequency domain attention module, and the spatial attention module after passing through the input convolutional layer. All processing results are added to the original input features, activated by ReLU, and then output through the output convolutional layer.

[0088] 3. Loss Function

[0089] In this embodiment, a loss term for small targets is added to the loss function of the improved RT-DETR model, including a small target size loss term and a small target localization loss term; wherein, the small target size loss term adopts a combination of IoU loss and CIoU loss, and the calculation formula is as follows:

[0090]

[0091] in, For the prediction box, For the true frame, The square of the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box. Let be the diagonal length of the smallest closure region containing both the predicted and ground truth boxes. These are the weighting coefficients. Used to measure the aspect ratio difference between the predicted bounding box and the ground truth bounding box.

[0092] In this embodiment, the improved RT-DETR model employs an adaptive learning rate adjustment strategy during training. This strategy is a cosine annealing learning rate adjustment method, and the formula is as follows:

[0093]

[0094] in, The current learning rate, To minimize the learning rate, To achieve the maximum learning rate, This is the current training round number. This represents the maximum number of training rounds.

[0095] S3, use the remote sensing small target detection network model to perform remote sensing small target detection on the input remote sensing image.

[0096] In this embodiment, the working principle of the remote sensing small target network model in the small target detection process is as follows:

[0097] Multi-scale visual features of remote sensing images are extracted using a backbone network to generate a feature pyramid containing spatial semantic information.

[0098] A set of learnable target query vectors is initialized through the target query vector generation module. These target query vectors are used to encode prior knowledge of the target detection task, including target location, size, and category.

[0099] The feature pyramid is input into the encoder for multi-layer feature enhancement. The encoder uses a multi-head self-attention mechanism and a feedforward neural network to perform cross-regional context modeling of visual features at different levels, thereby enhancing the feature representation of small target regions. At the same time, during the feature pyramid construction stage, the SPDConv module is used to downsample the high-resolution feature layer to preserve the details of small targets.

[0100] The decoder receives the enhanced features and target query vector output by the encoder and establishes a correspondence between the target query and visual features through a cross-attention mechanism. Specifically: First, the visual feature-modulated attention mechanism for the target query calculates the similarity between the target query vector and the visual features, selecting region features related to small targets from the feature pyramid and suppressing background noise interference. Then, the target query-guided attention mechanism for the visual features uses the modulated target query vector to guide the feature aggregation direction, enabling the decoder to focus on the detailed features of small targets. The features after bidirectional attention interaction are input into the attention module to model the relationship between targets, avoiding confusion in the detection of adjacent small targets. At the same time, the CSPOmniKernel module is introduced in the feature fusion stage of the decoder to enhance the discriminative power of small target features.

[0101] The detection results output by the decoder are processed by the Hungarian matching algorithm to perform optimal matching between the predicted bounding boxes and the ground truth bounding boxes. The classification loss and localization loss are calculated: the classification loss adopts the focus loss to solve the problem of imbalance of small target samples; the localization loss adopts the CIoU loss, which combines the overlap area between the predicted bounding box and the ground truth bounding box, the distance between the center point and the aspect ratio difference to improve the localization accuracy of small targets.

[0102] By optimizing network parameters through end-to-end training, the weights of the backbone network, encoder, and decoder are updated synchronously during training, enabling the model to adaptively learn the feature representation and detection logic of remote sensing small targets.

[0103] In summary, the improved RT-DETR model in this embodiment has the following advantages compared to the existing RT-DETR model:

[0104] 1. The backbone network uses element-wise multiplication to process the features of the input remote sensing image. By mapping the input features to a high-dimensional nonlinear space through element-wise multiplication, the ability to extract features of small targets is enhanced.

[0105] 2. In the feature extraction stage, an extraction branch for small target features is introduced and configured to adapt to the size of small targets by adjusting the kernel size and stride of the extraction branch. Dilated convolutions are added to the extraction branch to expand the receptive field and obtain richer contextual information.

[0106] 3. In the feature downsampling stage, the SPDConv module is introduced. Through the multi-branch sub-region feature concatenation and convolution fusion of the SPDConv module, the detailed information of small targets is preserved when the feature resolution is reduced.

[0107] 4. In the feature fusion stage, the CSPOmniKernel module is introduced. Through feature splitting, multi-directional feature capture, and attention enhancement of the CSPOmniKernel module, the feature extraction capability of small targets and computational complexity are balanced.

[0108] 5. The detection results output by the decoder are processed by the Hungarian matching algorithm to perform optimal matching between the predicted bounding box and the ground truth bounding box, and to calculate the classification loss and localization loss: the classification loss adopts the focal loss to solve the imbalance problem of small target samples; the localization loss adopts the CIoU loss, which combines the overlap area between the predicted bounding box and the ground truth bounding box, the distance between the center point and the aspect ratio difference to improve the localization accuracy of small targets.

[0109] 6. Optimize network parameters through end-to-end training, and update the weights of the backbone network, encoder and decoder synchronously during training, so that the model can adaptively learn the feature representation and detection logic of remote sensing small targets.

[0110] Please see Figure 3The second embodiment of the present invention also provides a remote sensing small target detection device based on an improved RT-DETR model, which includes:

[0111] Image processing unit 210 is used to acquire a remote sensing image dataset as a training set, perform multi-scale transformation on the remote sensing images in the remote sensing image dataset to obtain remote sensing images of different resolutions, and then perform enhancement processing on the remote sensing images of different resolutions to obtain the processed remote sensing images.

[0112] The model training unit 220 is used to input the processed remote sensing image into the improved RT-DETR model to train the improved RT-DETR model and obtain a trained remote sensing small target detection network model. The backbone network of the improved RT-DETR model uses element-wise multiplication to process the features of the input remote sensing image, mapping the input features to a high-dimensional nonlinear space through element-wise multiplication to enhance the ability to extract features of small targets. The decoding process of the decoder of the improved RT-DETR model includes a feature extraction stage, a feature downsampling stage, and a feature fusion stage. In the feature extraction stage, features targeting small targets are introduced. The system extracts target features through branches configured to adapt to small target sizes by adjusting the kernel size and stride of the extraction branches. Dilated convolutions are added to these extraction branches to expand the receptive field and capture richer contextual information. During feature downsampling, the SPDConv module is introduced. By combining multi-branch sub-region feature concatenation and convolutional fusion, the feature resolution of small targets is reduced while preserving detailed information. Finally, the CSPOmniKernel module is introduced during feature fusion. Through feature splitting, multi-directional feature capture, and attention enhancement, the CSPOmniKernel module balances the feature extraction capability and computational complexity for small targets.

[0113] The detection unit 230 is used to perform remote sensing small target detection on the input remote sensing image using the remote sensing small target detection network model.

[0114] The third embodiment of the present invention also provides a remote sensing small target detection device based on an improved RT-DETR model, which includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can realize the remote sensing small target detection method based on the improved RT-DETR model as described above.

[0115] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0117] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A remote sensing small target detection method based on an improved RT-DETR model, characterized in that, The methods include: S1. Obtain the remote sensing image dataset as the training set, perform multi-scale transformation on the remote sensing images in the dataset to obtain remote sensing images of different resolutions, and then perform enhancement processing on the remote sensing images of different resolutions to obtain the processed remote sensing images. S2, the processed remote sensing image is input into the improved RT-DETR model to train the improved RT-DETR model, resulting in a trained remote sensing small target detection network model; wherein, the backbone network of the improved RT-DETR model uses element-wise multiplication operations to process the features of the input remote sensing image, mapping the input features to a high-dimensional nonlinear space; the decoding process of the decoder of the improved RT-DETR model includes a feature extraction stage, a feature downsampling stage, and a feature fusion stage; in the feature extraction stage, an extraction branch for small target features is introduced and configured as follows. It can adapt to small target sizes by adjusting the kernel size and stride of the extraction branches, and add dilated convolutions to the extraction branches to expand the receptive field and obtain richer contextual information. In the feature downsampling stage, the SPDConv module is introduced. Through the concatenation and convolution of multi-branch sub-region features of the SPDConv module, the detailed information of small targets is preserved while reducing feature resolution. In the feature fusion stage, the CSPOmniKernel module is introduced. Through the feature splitting, multi-directional feature capture and attention enhancement of the CSPOmniKernel module, the feature extraction capability of small targets is balanced with the computational complexity. S3, using the aforementioned remote sensing small target detection network model to perform remote sensing small target detection on the input remote sensing image; wherein, in step S2, the SPDConv module is implemented as follows: During module initialization, the number of input channels inc and the number of output channels ouc are set, and a 3×3 convolutional layer with an input channel count of inc×4 is set internally. During the forward propagation process, the input feature map is sampled for non-overlapping sub-regions with a step size of 2, generating 4 complementary sub-features, including: the upper left sub-region of even-numbered rows and even-numbered columns of pixels, the lower left sub-region of odd-numbered rows and even-numbered columns of pixels, the upper right sub-region of even-numbered rows and odd-numbered columns of pixels, and the lower right sub-region of odd-numbered rows and odd-numbered columns of pixels. The four sub-features are concatenated along the channel dimension to form an intermediate feature with inc×4 channels, which is then compressed to the output channel number ouc through a 3×3 convolution. The output resolution is half that of the input feature map. The CSPOmniKernel module contains 1×1 convolutional layers cv1 and cv2 and an OmniKernel sub-module, and its implementation is as follows: During module initialization, the number of feature channels dim and the split ratio e are set to 0.

25. The number of input channels of the OmniKernel submodule is dimo, where dimo is the integer part of dim × e. During the forward propagation, the input features are split into two branches proportionally after being convolved by cv1: the features of the dimo channel are processed by the OmniKernel submodule, and the features of the other channels are directly retained as the identity branch. The output features of the OmniKernel submodule and the features of the identity branch are concatenated along the channel dimension and then output as fused features via cv1 convolution.

2. The remote sensing small target detection method based on the improved RT-DETR model according to claim 1, characterized in that, The OmniKernel submodule includes an input convolutional layer with GELU activation, an output convolutional layer, a multi-directional depthwise convolutional group, a frequency domain attention module, a spatial attention module, and a ReLU activation layer; wherein: The multi-directional depth convolution group includes four depth-separable convolutions: a 1×31 convolution for capturing long-range features in the horizontal direction, a 31×1 convolution for capturing long-range features in the vertical direction, a 31×31 convolution for capturing global spatial features, and a 1×1 convolution for local feature interactions within the channel. The frequency domain attention module generates attention weights through global average pooling and 1×1 convolution, and weights the features in the Fourier domain to suppress high-frequency noise; the spatial attention module generates a spatial attention map through global average pooling and 1×1 convolution, and weights the features processed in the frequency domain to enhance the response of small target regions. During forward propagation, the input features are processed by the multi-directional depth convolutional group, the frequency domain attention module, and the spatial attention module after passing through the input convolutional layer. All processing results are added to the original input features, activated by ReLU, and then output through the output convolutional layer.

3. The remote sensing small target detection method based on the improved RT-DETR model according to claim 1, characterized in that, The high-dimensional features are generated by element-wise multiplication operations in the backbone network as follows: Single-level element-wise multiplication is represented as: in, , For the weight vector, , for , transpose, For the input feature vector, For feature dimensions; Represents the weight vector The i-th element, Represents the weight vector The j-th element, These represent the i-th and j-th features of the input feature vector, respectively; after expansion, they generate... The nonlinear feature terms achieve dimensional expansion without increasing computational cost; through layer-by-layer element-wise multiplication, the feature dimension grows exponentially, with the 1st layer... The layer output feature dimension is R represents the real number field.

4. The remote sensing small target detection method based on the improved RT-DETR model according to claim 1, characterized in that, The improved RT-DETR model's loss function includes a loss term for small targets, comprising a small target size loss term and a small target localization loss term. The small target size loss term is calculated using a combination of IoU and CIoU losses, as shown in the following formula: in, For the prediction box, For the true frame, The square of the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box. Let be the diagonal length of the smallest closure region containing both the predicted and ground truth boxes. These are the weighting coefficients. Used to measure the aspect ratio difference between the predicted bounding box and the ground truth bounding box.

5. The remote sensing small target detection method based on the improved RT-DETR model according to claim 1, characterized in that, In step S2, the improved RT-DETR model employs an adaptive learning rate adjustment strategy during training. This strategy is a cosine annealing learning rate adjustment method, and the formula is as follows: in, The current learning rate, To minimize the learning rate, To achieve the maximum learning rate, This is the current training round number. This represents the maximum number of training rounds.

6. The remote sensing small target detection method based on the improved RT-DETR model according to claim 5, characterized in that, In step S3, the working principle of the remote sensing small target detection network model is as follows: Multi-scale visual features of remote sensing images are extracted using a backbone network to generate a feature pyramid containing spatial semantic information. A set of learnable target query vectors is initialized through the target query vector generation module. These target query vectors are used to encode prior knowledge of the target detection task, including target location, size, and category. The feature pyramid is input into the encoder for multi-layer feature enhancement. The encoder uses a multi-head self-attention mechanism and a feedforward neural network to perform cross-regional context modeling of visual features at different levels, thereby enhancing the feature representation of small target regions. At the same time, during the feature pyramid construction stage, the SPDConv module is used to downsample the high-resolution feature layer to preserve the details of small targets. The decoder receives the enhanced features and target query vector output by the encoder and establishes a correspondence between the target query and visual features through a cross-attention mechanism. Specifically: First, the visual feature-modulated attention mechanism for the target query calculates the similarity between the target query vector and the visual features, selects region features related to small targets from the feature pyramid, and suppresses background noise interference. Then, the target query-guided attention mechanism for the visual features uses the modulated target query vector to guide the feature aggregation direction, enabling the decoder to focus on the detailed features of small targets. Features after bidirectional attention interaction are input into the attention module to model the relationship between targets, avoiding confusion in the detection of adjacent small targets; at the same time, the CSPOmniKernel module is introduced in the feature fusion stage of the decoder to enhance the discriminative power of small target features; The detection results output by the decoder are processed by the Hungarian matching algorithm to perform optimal matching between the predicted bounding boxes and the ground truth bounding boxes. The classification loss and localization loss are calculated: the classification loss adopts the focus loss to solve the problem of imbalance of small target samples; the localization loss adopts the CIoU loss, which combines the overlap area between the predicted bounding box and the ground truth bounding box, the distance between the center point and the aspect ratio difference to improve the localization accuracy of small targets. By optimizing network parameters through end-to-end training, the weights of the backbone network, encoder, and decoder are updated synchronously during training, enabling the model to adaptively learn the feature representation and detection logic of remote sensing small targets.

7. A remote sensing small target detection device based on an improved RT-DETR model, characterized in that, include: The image processing unit is used to acquire the remote sensing image dataset as the training set, perform multi-scale transformation on the remote sensing images in the dataset to obtain remote sensing images of different resolutions, and then perform enhancement processing on the remote sensing images of different resolutions to obtain the processed remote sensing images. The model training unit is used to input the processed remote sensing image into the improved RT-DETR model to train the improved RT-DETR model and obtain a trained remote sensing small target detection network model. The backbone network of the improved RT-DETR model uses element-wise multiplication to process the features of the input remote sensing image, mapping the input features to a high-dimensional nonlinear space. The decoding process of the decoder of the improved RT-DETR model includes a feature extraction stage, a feature downsampling stage, and a feature fusion stage. In the feature extraction stage, an extraction branch for small target features is introduced and configured to adjust the extraction... The kernel size and stride of the branches are adapted to the size of small targets. Dilated convolutions are added to the extraction branches to expand the receptive field and obtain richer contextual information. The SPDConv module is introduced in the feature downsampling stage. Through multi-branch sub-region feature concatenation and convolution fusion of the SPDConv module, the detailed information of small targets is preserved while reducing feature resolution. The CSPOmniKernel module is introduced in the feature fusion stage. Through feature splitting, multi-directional feature capture, and attention enhancement of the CSPOmniKernel module, the feature extraction capability and computational complexity of small targets are balanced. The implementation of the SPDConv module is as follows: During module initialization, the number of input channels (inc) and the number of output channels (ouc) are set, and a 3×3 convolutional layer with an input channel count of inc×4 is set internally. During forward propagation, the input feature map is sampled for non-overlapping sub-regions with a stride of 2, generating four complementary sub-features, including: the upper left sub-region of even-numbered rows and even-numbered columns, the lower left sub-region of odd-numbered rows and even-numbered columns, the upper right sub-region of even-numbered rows and odd-numbered columns, and the lower right sub-region of odd-numbered rows and odd-numbered columns. The four sub-features are concatenated along the channel dimension to form an intermediate feature with an input channel count of inc×4, which is then compressed to the output channel count (ouc) through a 3×3 convolution, resulting in an output feature with a resolution of 1 / 2 of the input feature map. The CSPO The `mniKernel` module contains 1×1 convolutional layers `cv1`, `cv2`, and the `OmniKernel` submodule. Its implementation is as follows: During module initialization, the number of feature channels `dim` and the splitting ratio `e` are set to 0.

25. The number of input channels for the `OmniKernel` submodule is `dimo`, where `dimo` is the integer part of `dim×e`. During forward propagation, the input features are convolved by `cv1` and split proportionally into two branches: the features of the `dimo` channels are processed by the `OmniKernel` submodule, while the features of the remaining channels are directly retained as the identity branch. The output features of the `OmniKernel` submodule and the features of the identity branch are concatenated along the channel dimension and then convolved by `cv1` to output the fused features. The detection unit is used to perform remote sensing small target detection on the input remote sensing image using the remote sensing small target detection network model.

8. A remote sensing small target detection device based on an improved RT-DETR model, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the remote sensing small target detection method based on the improved RT-DETR model as described in any one of claims 1-6.

Images