A remote sensing target detection method and system based on improved RT-DETR

Abstract

The application discloses a remote sensing target detection method and system based on an improved RT-DETR. The method comprises the following steps: step 1, collecting remote sensing image data; step 2, performing preprocessing such as denoising, enhancing contrast and clarity on the collected remote sensing image data; step 3, making the picture into a remote sensing data set after preprocessing, and dividing the remote sensing data set into a training set and a verification set according to a proportion; step 4, training the remote sensing data set by using an RT-DETR model to obtain a remote sensing target detector; step 5, optimizing various parameters of the target detector on the verification set by using a hyperparameter search verification method; and step 6, detecting a test image or a video by using the remote sensing target detector obtained after optimization in step 5. Compared with the original RT-DETR model, the application reduces the amount of calculation and accelerates the network convergence speed on the premise of maintaining the original accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing target detection based on deep learning, and particularly relates to a remote sensing target detection method and system based on improved RT-DETR. BACKGROUND

[0002] With the rapid development of remote sensing technology and the wide application of remote sensing platforms, the observation capability of the earth's surface has been significantly improved, and remote sensing images have provided rich data for various fields. Remote sensing target detection technology, as a key to automatically identify and locate specific targets, is becoming a research hotspot. This technology not only shows great potential in urban planning, environmental monitoring, disaster warning and other fields, but also provides accurate spatial information and target analysis, and effectively supports government decision-making, scientific research and public services.

[0003] Traditional remote sensing target detection technology is mainly based on threshold analysis, pixel classification, template matching and other methods. However, these methods have some limitations, such as limited accuracy, poor generalization ability, high computational cost and other problems. In contrast, remote sensing target detection methods based on deep learning have higher flexibility, lower cost and higher detection accuracy, making remote sensing target detection methods achieve great improvement in detection accuracy and robustness.

[0004] DETR is an end-to-end target detector based on Transformer. Compared with other deep learning models, DETR does not need to preset anchor boxes or candidate regions, simplifies the detection process, eliminates post-processing operations, realizes end-to-end training, and improves the overall performance and efficiency of the model. It regards target detection as a set prediction problem, which can handle targets of different numbers and sizes, and improves the flexibility of the model. RT-DETR, while maintaining the advantages of DETR, further improves the real-time performance and accuracy of target detection by optimizing the encoder design and introducing a new query selection mechanism. However, the existing RT-DETR model still has defects. It is difficult to balance speed and accuracy. Since it emphasizes real-time performance, RT-DETR usually sacrifices accuracy to some extent. In order to achieve faster speed, it reduces the investment of computing resources, resulting in a decrease in detection accuracy. Therefore, the network structure of the current RT-DETR has a large amount of calculation when processing large size images, which leads to low efficiency on embedded hardware, especially in mobile devices or edge computing scenarios. SUMMARY

[0005] In view of the defects in the prior art, the present application provides a remote sensing target detection method and system based on improved RT-DETR, which can better realize the detection of remote sensing targets. The present application replaces the backbone module of the existing RT-DETR with FasterNet constructed by partial convolution (PConv) and point-wise convolution (PWConv) as the main module, and replaces the Deformable Attention self-attention used in the decoder part with AgentAttention self-attention, which realizes significant speed improvement while maintaining high accuracy, solves the problems of slow remote sensing target detection speed, low detection accuracy, complex background interference and other problems, and can meet the requirements of more real-time detection scenes.

[0006] The present application adopts the following technical solutions:

[0007] A remote sensing target detection method based on improved RT-DETR, comprising the following steps:

[0008] Step 1, collecting remote sensing image data;

[0009] Step 2, preprocessing the collected remote sensing image data;

[0010] Step 3, making the preprocessed remote sensing image data into a remote sensing data set, and dividing it into a training set and a validation set in proportion;

[0011] Step 4, training the data set using an improved RT-DETR model to obtain a remote sensing target detector;

[0012] Step 5, using hyperparameter search verification to optimize the parameters of the target detector on the validation set;

[0013] Step 6, using the remote sensing target detector obtained after optimization in step 5 to detect test images or videos.

[0014] Further, in step 1, the remote sensing image data collection method includes: image data obtained by shooting the ground surface through a UAV camera; and image data in the related open source remote sensing data set RSOD.

[0015] Further, in step 2, median filtering and histogram equalization are used for image preprocessing to denoise, enhance contrast and clarity of the collected vehicle road image data.

[0016] Further, the detailed process of step 3 includes:

[0017] Step 301, using LabelImg to manually label the detection target of each image in the data set, and taking the labeled minimum bounding rectangle as the true box.

[0018] Step 302, randomly divide the data set into training set and validation set in the ratio of 8:2.

[0019] Further, in step 4, the improved RT-DETR model includes: replacing the backbone of RT-DETR with FasterNet, which uses novel partial convolution (PConv) and point-wise convolution (PWConv) as the main building block. The whole architecture is divided into four levels, each of which starts with an embedding layer or a merging layer for spatial down-sampling and channel expansion. Each level contains multiple FasterNet blocks, each of which contains a PConv layer followed by two PWConv layers, in the form of an inverted residual block, with an expanded number of channels in the middle layer and a shortcut connection to reuse input features. In order to maintain feature diversity and reduce latency, only normalization and activation layers are placed after each intermediate PWConv, and batch normalization (BN) is used instead of other alternatives. It more effectively extracts spatial features by simultaneously reducing redundant calculations and memory access.

[0020] Further, in step 4, the improved RT-DETR model includes: replacing the decoder part with Deformable Attention self-attention with Agent Attention self-attention. Agent Attention introduces additional agent tokens into the traditional attention module, which first serve as proxies for query tokens to aggregate information from keys and values, and then broadcast information back to query tokens. It maintains the accuracy of the softmax operation while achieving the efficiency of linear attention.

[0021] The application is preferably configured as follows in step 4:

[0022] Step 401, replace the backbone of RT-DETR with FasterNet, which uses novel partial convolution (PConv) and point-wise convolution (PWConv) as the main building blocks. The entire architecture is divided into four levels, each of which starts with an embedding layer or a merge layer for spatial down-sampling and channel expansion. Each level contains multiple FasterNet blocks, each of which contains a PConv layer followed by two PWConv layers, taking the form of an inverted residual block, where the middle layer has an expanded number of channels and has a shortcut connection to reuse input features. To maintain feature diversity and reduce latency, only normalization and activation layers are placed after each intermediate PWConv, and batch normalization (BN) is used instead of other alternatives. It more effectively extracts spatial features by simultaneously reducing redundant calculations and memory access.

[0023] Step 402, after backbone network feature extraction, three different scale effective feature maps S3, S4, S5 are obtained, Tokenize the last layer feature map S5 and perform attention calculation, while the S3, S4 layer is extracted by convolution form, and finally the scale characteristics between the feature maps output by the encoder are fused to obtain multi-scale features, while greatly reducing the amount of calculation.

[0024] Step 403, replace the Deformable Attention self-attention used in the decoder part with AgentAttention self-attention. AgentAttention introduces additional agent tokens into the traditional attention module, which first serve as proxies for query tokens to aggregate information from keys and values, and then broadcast the information back to the query tokens. It maintains the accuracy of the softmax operation while achieving the efficiency of linear attention.

[0025] Step 404, after the calculation of the Decoder is completed, the prediction result is matched with the target through the Hungarian matching method, and the loss calculation is carried out at the same time. The Hungarian algorithm is a classic algorithm for solving the maximum matching of bipartite graphs in polynomial time. In RT-DETR, the predicted bounding box and the real bounding box can be regarded as two vertex sets of a bipartite graph, and the matching relationship between them can be represented by constructing a cost matrix. Each element in the cost matrix represents the matching cost between a predicted bounding box and a real bounding box. First, according to the IoU, L1 loss between the predicted bounding box and the real bounding box, the matching cost between them is calculated, and the cost matrix is constructed. Then, the cost matrix is taken as the input of the Hungarian algorithm, and the algorithm outputs an optimal matching scheme, that is, a matching set that makes the total matching cost minimum. Finally, according to the output result of the Hungarian algorithm, it is determined which predicted bounding box and real bounding box are successfully matched, and which are not matched. The matching result can be used to calculate the loss function in the training process, thereby guiding the optimization of the model.

[0026] Preferably, the hyperparameter search verification method in step 5 represents different combinations of parameters (batch size, learning rate, etc.) are tried, and the validation set is used to optimize the performance of the detector model.

[0027] Further, in step 6, the detector obtained in step 5 is used to detect test images or videos, and further, the detection effect is evaluated.

[0028] The application also discloses an improved RT-DETR-based remote sensing target detection system for executing the above method, comprising the following modules:

[0029] An image acquisition module is arranged to acquire remote sensing image data.

[0030] A preprocessing module is arranged to preprocess the acquired remote sensing image data.

[0031] A data set making module is arranged to make the preprocessed remote sensing image data into a remote sensing data set, which is divided into a training set and a validation set in proportion.

[0032] A training module is arranged to train the remote sensing data set by using the RT-DETR model to obtain a remote sensing target detector.

[0033] An optimization module is arranged to optimize the parameters of the remote sensing target detector on the validation set by using a hyperparameter search verification method.

[0034] A detection module is arranged to detect test images or videos by using the optimized remote sensing target detector.

[0035] The application has the following beneficial effects:

[0036] (1) Through the image preprocessing process, the image quality is effectively improved, and the detection effect is enhanced.

[0037] (2) The RT-DETR model is improved, the backbone is replaced by FasterNet, the architecture introduces partial convolution (PConv) and point-wise convolution (PWConv), more effectively utilizes the information of all channels, and adopts a new network architecture, which not only realizes significant speed improvement, but also maintains high accuracy.

[0038] (3) The RT-DETR model is improved, the Deformable Attention of the decoder part is replaced by Agent Attention. By introducing agent tokens, the computing efficiency is significantly improved while maintaining the global context modeling capability. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 The overall flowchart of a remote sensing target detection method based on an improved RT-DETR according to a preferred embodiment of the present application is shown in the figure.

[0041] Figure 2 The structure diagram of the improved backbone module according to the present application is shown in the figure.

[0042] Figure 3 The structure diagram of the improved Agent Attention module according to the present application is shown in the figure.

[0043] Figure 4 The block diagram of a remote sensing target detection system based on an improved RT-DETR according to a preferred embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0044] The embodiments of the present application will be described below through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure of the present specification. The present application can also be implemented or applied by other different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] The preferred embodiment of the present application provides a remote sensing target detection method based on improved RT-DETR, as shown in the formula (1), comprising the following steps: Figure 1

[0046] Step 1, collecting remote sensing image data;

[0047] Step 2, preprocessing the collected remote sensing image data;

[0048] Step 3, making the preprocessed remote sensing image data into a remote sensing data set;

[0049] Step 4, training the data set using an improved RT-DETR model to obtain a remote sensing target detector;

[0050] Step 5, using hyperparameter search verification to optimize the parameters of the target detector on the validation set;

[0051] Step 6, using the remote sensing target detector obtained after optimization in step 5 to detect test images or videos and evaluate the detection effect.

[0052] In step 1 of the embodiment, the remote sensing image data collection method includes image data obtained by shooting the ground surface through a UAV camera, and image data in the related open source remote sensing data set RSOD.

[0053] In step 2 of the embodiment, median filtering and histogram equalization are used for image preprocessing to denoise, enhance contrast and clarity of the remote sensing image data collected in step 1.

[0054] The detailed process of step 3 of the embodiment includes:

[0055] Step 301, using LabelImg to manually label the detection target of each image in the data set, and using the minimum circumscribed rectangle of the labeled target as the real box.

[0056] Step 302, randomly dividing the data set into a training set and a validation set in a ratio of 8:2.

[0057] In step 4 of the embodiment, the RT-DETR model is built, which is composed of four parts: a backbone network, an encoder, a decoder, and a detection head; this step is as follows:

[0058] Step 401, as shown in the formula (2), the backbone network is used to extract the feature of the input image, and the feature is sent to the encoder and decoder for further processing. Figure 2 ​As shown, FasterNet is used to replace the backbone of RT-DETR, which is divided into four stages (Stage), each stage contains a series of FasterNet blocks (FasterNet Block), and there is an embedding or merging layer before each stage. The last three layers are used for feature classification. Inside each FasterNet block, the PConv operation is adopted, followed by two PWConv operations. This structure effectively utilizes all channel information, forming a T-shaped convolution structure, making the model pay more attention to the features of the center position. After the PConv operation, only the normalization and activation layer is added after the middle layer to maintain the diversity of the features and reduce the calculation delay. After the image passes through the Backbone, the deep convolution features are extracted, and the S3, S4, S5 features in the Backbone are taken as the input of the encoder.

[0059] Step 402, after feature extraction by the backbone network, three effective feature maps of different scales S3, S4, S5 are obtained, and the last layer feature map S5 is tokenized and attention calculation is performed (calculation process: first, each pixel block in the input sequence is converted into an embedding vector in a high-dimensional space. For each embedding vector in the input sequence, three new vectors are generated through three different linear transformations (i.e. multiplied by three different weight matrices): query vector (Query, Q), key vector (Key, K) and value vector (Value, V). These three vectors will play different roles in subsequent attention calculations. Next, for each pixel block in the sequence, the score (Score) between it and all other pixel blocks in the sequence is calculated. This score is obtained by the dot product operation of the query vector (Q) of the current pixel block and the key vector (K) of all other pixel blocks. This score reflects the correlation or importance between the current pixel block and other pixel blocks. Since the result of the dot product operation can be very large, in order to ensure numerical stability and the effectiveness of gradient propagation, the score is divided by a scaling factor (usually the square root of the key vector dimension). Then, the scaled score is applied to the Softmax function to ensure that the sum of all scores is 1, thereby obtaining a probability distribution, i.e. attention weight. Finally, the value vector (V) of all pixel blocks in the sequence is weighted and summed using the attention weight obtained in the previous step. The result of this weighted sum is a new vector that contains the contribution information of all pixel blocks in the sequence to the current pixel block. This new vector can be used as the output representation of the current pixel block after the self-attention layer), while S3 and S4 layers extract features through convolution, and finally fuse the features between different scales with the encoder output, thereby obtaining multi-scale features while greatly reducing the computational complexity.

[0060] Step 403, asFigure 3 As shown, the Deformable Attention self-attention applied by the decoder part is replaced with Agent Attention self-attention. Agent Attention introduces additional agent tokens into the traditional attention module, which first aggregate information from keys and values as proxies for query tokens, and then broadcast the information back to the query tokens. It maintains the accuracy of the softmax operation while achieving the efficiency of linear attention.

[0061] Step 404, after completing the calculation of the decoder, the prediction results are matched with the target by the Hungarian matching method, and the loss calculation is performed.

[0062] In step 5 of the embodiment, the hyperparameter search verification mode indicates that different combinations of parameters (batch size, learning rate, etc.) are tried, and the validation set is used to optimize the detector model performance to achieve the best effect.

[0063] With the above technical solution, compared with the original RT-DETR model, the calculation amount is reduced and the network convergence speed is accelerated while maintaining the original accuracy.

[0064] As shown in Figure 4 The embodiment discloses a remote sensing target detection system based on an improved RT-DETR, which is used to execute the above method, and includes the following modules:

[0065] An image acquisition module is configured to acquire remote sensing image data.

[0066] A preprocessing module is configured to preprocess the acquired remote sensing image data.

[0067] A data set making module is configured to make the preprocessed remote sensing image data into a remote sensing data set, which is divided into a training set and a validation set in proportion.

[0068] A training module is configured to train the remote sensing data set by using an RT-DETR model to obtain a remote sensing target detector.

[0069] An optimization module is configured to optimize the parameters of the remote sensing target detector on the validation set by using a hyperparameter search verification mode.

[0070] A detection module is configured to detect a test image or video by using the optimized remote sensing target detector.

[0071] Other contents of the embodiment can refer to the above method embodiment.

[0072] The above-described embodiments are merely intended to describe the preferred embodiments of the present application, and are not intended to limit the scope of the present application. Various modifications and improvements to the present application made by those skilled in the art are intended to fall within the scope of the present application.

Claims

1. An improved RT-DETR-based remote sensing target detection method, characterized in that, The method comprises the following steps: Step 1, collecting remote sensing image data; Step 2, preprocessing the collected remote sensing image data; Step 3, making the preprocessed remote sensing image data into a remote sensing data set, and dividing the remote sensing data set into a training set and a validation set; Step 4, training the remote sensing data set using an RT-DETR model to obtain a remote sensing target detector; in this step, the RT-DETR model: replaces the existing RT-DETR backbone module with FasterNet, which includes partial convolution PConv and point-wise convolution PWConv, and the FasterNet architecture is divided into four levels, each level starts with an embedding layer or a merging layer for spatial down-sampling and channel expansion; each level contains multiple FasterNet blocks, each FasterNet block contains a PConv layer followed by two PWConv layers in the form of an inverted residual block, wherein the middle layer has an expanded channel number and has a shortcut connection to reuse the input features; The RT-DETR model: replaces the Deformable Attention self-attention used in the decoder part with Agent Attention self-attention, which introduces agent tokens that first aggregate information from keys and values as proxies for query tokens, and then broadcast the information back to the query tokens; Step 5, optimizing the parameters of the remote sensing target detector on the validation set using hyperparameter search validation; Step 6, using the remote sensing target detector obtained after optimization in step 5 to detect test images or videos.

2. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: In step 1, the remote sensing image data is collected by a UAV camera to obtain image data; or, the image data in the open source remote sensing data set RSOD is used.

3. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: In step 2, the remote sensing image data collected in step 1 is denoised, enhanced in contrast and clarity.

4. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: Step 3 is as follows: Step 301, using LabelImg to label the detection targets of each image in the data set, and using the minimum bounding rectangle of the labeled target as the true box; Step 302, randomly dividing the data set into a training set and a validation set in a ratio of 8:

2.

5. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 401, using FasterNet, which is divided into four stages, each stage contains multiple FasterNet blocks, and there is an embedding or merging layer before each stage; the last three layers are used for feature classification; within each FasterNet block, PConv operation is used, followed by two PWConv operations; after the PConv operation, only the normalization and activation layer is added after the middle layer; after the image passes through the Backbone, the deep convolution features are extracted, and the S3, S4 and S5 features in the Backbone are used as the input of the encoder; Step 402: After backbone network feature extraction, three effective feature maps S3, S4 and S5 of different scales are obtained. Tokenization is performed on the last layer feature map S5 and attention calculation is performed. The S3 and S4 layers are extracted by convolution, and finally the scale characteristics are fused with the feature map output by the encoder to obtain multi-scale features. Step 403: Agent Attention self-attention is used. The self-attention introduces an additional agent token. The agent token first acts as a proxy for the query token to aggregate information from the key and value, and then broadcasts the information back to the query token. Step 404: The prediction result is matched with the target by the Hungarian matching method, and the loss is calculated.

6. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: In step 5, the hyperparameter search verification method is used to try different combinations of parameters, and the validation set is used to optimize the performance of the detector.

7. An improved RT-DETR based remote sensing target detection system for performing the method of any one of claims 1-6. The method comprises the following modules: An image acquisition module is configured to acquire remote sensing image data. A preprocessing module is configured to preprocess the acquired remote sensing image data. A dataset production module is configured to produce a remote sensing dataset from the preprocessed remote sensing image data, and divide the remote sensing dataset into a training set and a validation set according to a proportion. A training module is configured to train the remote sensing dataset by using an RT-DETR model to obtain a remote sensing target detector. An optimization module is configured to optimize the parameters of the remote sensing target detector on the validation set by using a hyperparameter search verification method. A detection module is configured to detect a test image or a test video by using the optimized remote sensing target detector.

Images