Remote sensing target detection method and system based on improved RT-DETR

Abstract

The invention discloses a remote sensing target detection method and system based on improved RT-DETR. The method comprises the following steps: step 1, collecting remote sensing image data; step 2, carrying out preprocessing such as denoising, contrast enhancement and definition enhancement on the acquired remote sensing image data; 3, making the preprocessed pictures into a remote sensing data set, and dividing the remote sensing data set into a training set and a verification set in proportion; 4, training the remote sensing data set by adopting an RT-DETR model to obtain a remote sensing target detector; step 5, optimizing various parameters of the target detector on the verification set by adopting a hyper-parameter search verification mode; and step 6, detecting a test image or video by using the remote sensing target detector optimized in the step 5. Compared with an original RT-DETR model, on the premise that original precision is kept, the calculation amount is reduced, and the network convergence speed is increased.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing target detection based on deep learning, and specifically relates to a remote sensing target detection method and system based on improved RT-DETR. Background Art

[0002] With the rapid development of remote sensing technology and the widespread application of remote sensing platforms, the observation capabilities of the earth's surface have been significantly improved, and remote sensing images have provided rich data for various fields. Remote sensing target detection technology, as the key to automatically identifying and locating specific targets, is becoming a research hotspot. This technology not only shows great potential in urban planning, environmental monitoring, disaster warning and other fields, but also strongly supports government decision-making, scientific research and public services by providing accurate spatial information and target analysis.

[0003] Traditional remote sensing target detection technology is mainly based on threshold analysis, pixel classification, template matching and other methods. However, these methods have some limitations, such as limited accuracy, poor generalization ability, high computational cost and other problems. In contrast, remote sensing target detection methods based on deep learning have higher flexibility, lower cost and higher detection accuracy, which has greatly improved the detection accuracy and robustness of remote sensing target detection methods.

[0004] DETR is an end-to-end target detector based on Transformer. Compared with other deep learning models, DETR does not need to preset anchor boxes or candidate regions, simplifies the detection process, eliminates post-processing operations, realizes end-to-end training, and improves the overall performance and efficiency of the model; and regards target detection as a set prediction problem, which can handle targets of different numbers and sizes, and improves the flexibility of the model. While maintaining the advantages of DETR, RT-DETR further improves the real-time and accuracy of target detection by optimizing the encoder design and introducing a new query selection mechanism. However, the existing RT-DETR model still has defects. It is reflected in: the current RT-DETR is difficult to achieve a trade-off between speed and accuracy. Due to the emphasis on real-time, RT-DETR usually sacrifices accuracy to a certain extent. In order to achieve faster speed, the investment in computing resources will be reduced, resulting in a decrease in detection accuracy. Therefore, the current network structure of RT-DETR has a large amount of calculation when processing large-size images, which leads to low efficiency on embedded hardware, especially in mobile devices or edge computing scenarios. Summary of the invention

[0005] In view of the defects in the prior art, the present invention provides a remote sensing target detection method and system based on improved RT-DETR, which can better realize the detection of remote sensing targets. The present invention replaces the backbone module of the existing RT-DETR with FasterNet, which is mainly built by partial convolution (PConv) and point-by-point convolution (PWConv), and replaces the Deformable Attention self-attention used in the decoder part with AgentAttention self-attention, thereby achieving a significant improvement in operating speed while maintaining a high accuracy, solving the problems of slow remote sensing target detection speed, low detection accuracy, complex background interference, etc., and can meet the requirements of more real-time detection scenarios.

[0006] The present invention adopts the following technical scheme:

[0007] A remote sensing target detection method based on improved RT-DETR includes the following steps:

[0008] Step 1: Collect remote sensing image data;

[0009] Step 2: preprocessing the collected remote sensing image data;

[0010] Step 3: The preprocessed remote sensing image data is made into a remote sensing data set, which is divided into a training set and a validation set in proportion;

[0011] Step 4: Use the improved RT-DETR model to train the data set to obtain a remote sensing target detector;

[0012] Step 5: Use hyperparameter search verification to optimize various parameters of the target detector on the verification set;

[0013] Step 6: Use the remote sensing target detector optimized in step 5 to detect the test image or video.

[0014] Furthermore, in step 1, the remote sensing image data collection method includes: image data obtained by photographing the surface through a drone camera; and image data in the relevant open source remote sensing dataset RSOD.

[0015] Furthermore, in step 2, median filtering and histogram equalization are used for image preprocessing to remove noise, enhance contrast and clarity of the collected vehicle road image data.

[0016] Furthermore, the detailed process of step 3 includes:

[0017] Step 301: Use LabelImg to manually label the detection target of each image in the data set, and use the minimum bounding rectangle of the labeled target as the real box.

[0018] Step 302: randomly divide the data set into a training set and a validation set in a ratio of 8:2.

[0019] Further, in step 4, the improved RT-DETR model includes: replacing the backbone of RT-DETR with FasterNet, which uses novel partial convolution (PConv) and point-by-point convolution (PWConv) as the main building blocks. The entire architecture is divided into four levels, each of which starts with an embedding layer or a merging layer for spatial downsampling and channel number expansion. Each level contains multiple FasterNet blocks, each FasterNet block contains a PConv layer followed by two PWConv layers, in the form of an inverted residual block, where the intermediate layer has an expanded number of channels and a shortcut connection to reuse input features. In order to maintain feature diversity and reduce latency, normalization and activation layers are placed only after each intermediate PWConv, and batch normalization (BN) is used instead of other alternatives. It extracts spatial features more efficiently by simultaneously cutting redundant calculations and memory accesses.

[0020] Furthermore, in step 4, the improved RT-DETR model includes: replacing the DeformableAttention self-attention used in the decoder part with Agent Attention self-attention. Agent attention introduces additional agent tokens into the traditional attention module. These agent tokens first act as agents for query tokens to aggregate information from keys and values, and then broadcast the information back to the query tokens. It maintains the accuracy of the softmax operation while achieving the high efficiency of linear attention.

[0021] The preferred configuration of the present invention is that step 4 is specifically as follows:

[0022] Step 401, replace the backbone of RT-DETR with FasterNet, which uses novel partial convolution (PConv) and point-by-point convolution (PWConv) as the main building blocks. The entire architecture is divided into four levels, each of which starts with an embedding layer or a merging layer for spatial downsampling and channel number expansion. Each level contains multiple FasterNet blocks, each FasterNet block contains a PConv layer followed by two PWConv layers, in the form of an inverted residual block, where the intermediate layer has an expanded number of channels and a shortcut connection to reuse input features. In order to maintain feature diversity and reduce latency, normalization and activation layers are placed only after each intermediate PWConv, and batch normalization (BN) is used instead of other alternatives. It extracts spatial features more efficiently by simultaneously cutting redundant calculations and memory accesses.

[0023] Step 402: After the backbone network feature extraction, three effective feature maps of different scales, S3, S4, and S5, are obtained. The last layer of feature map S5 is tokenized and attention calculation is performed, while the S3 and S4 layers are feature extracted by convolution. Finally, the inter-scale characteristics are fused with the feature map output by the encoder to obtain multi-scale features while greatly reducing the amount of calculation.

[0024] Step 403: Replace the Deformable Attention used in the decoder with AgentAttention. Agent attention introduces additional agent tokens into the traditional attention module. These agent tokens first act as agents for query tokens to aggregate information from keys and values, and then broadcast the information back to the query tokens. It maintains the accuracy of the softmax operation while achieving the high efficiency of linear attention.

[0025] Step 404, after completing the calculation of the decoder, the prediction result is matched with the target by the Hungarian matching method, and the loss is calculated at the same time. The Hungarian algorithm is a classic algorithm for solving the maximum matching of a bipartite graph in polynomial time. In RT-DETR, the predicted bounding box and the true bounding box can be regarded as two vertex sets of a bipartite graph, and the matching relationship between them can be represented by constructing a cost matrix. Each element in the cost matrix represents the matching cost between a predicted bounding box and a true bounding box. First, according to the IoU and L1 losses between the predicted bounding box and the true bounding box, the matching cost between them is calculated, and the cost matrix is ​​constructed. Then, the cost matrix is ​​used as the input of the Hungarian algorithm, and the algorithm outputs an optimal matching solution, that is, a matching set with the minimum total matching cost. Finally, according to the output results of the Hungarian algorithm, it is determined which predicted bounding boxes are successfully matched with the true bounding boxes and which are not matched. The matching results can be used to calculate the loss function in the training process, thereby guiding the optimization of the model.

[0026] The present invention is preferably configured such that the hyperparameter search verification method in step 5 represents trying different combinations of parameters (batch size, learning rate, etc.) and using a verification set to optimize the detector model performance.

[0027] Furthermore, in step 6, the detector obtained in step 5 is used to detect the test image or video, and further, the detection effect is evaluated.

[0028] The present invention also discloses a remote sensing target detection system based on improved RT-DETR, which is used to execute the above method and includes the following modules:

[0029] Image acquisition module: collect remote sensing image data;

[0030] Preprocessing module: preprocess the collected remote sensing image data;

[0031] Dataset making module: Make the pre-processed remote sensing image data into a remote sensing dataset, and divide it into a training set and a validation set in proportion;

[0032] Training module: Use the RT-DETR model to train the remote sensing data set to obtain the remote sensing target detector;

[0033] Optimization module: uses hyperparameter search and verification to optimize the parameters of the remote sensing target detector on the verification set;

[0034] Detection module: Use the optimized remote sensing target detector to detect the test image or video.

[0035] The beneficial effects of the present invention are:

[0036] (1) The image quality can be effectively improved through image preprocessing, which is beneficial to enhancing the detection effect.

[0037] (2) The present invention improves the RT-DETR model by replacing the backbone with FasterNet. This architecture more effectively utilizes the information of all channels by introducing partial convolution (PConv) plus point-by-point convolution (PWConv). At the same time, it adopts a new network architecture, which not only achieves a significant improvement in operating speed but also maintains a high accuracy.

[0038] (3) The present invention improves the RT-DETR model by replacing the Deformable Attention in the decoder part with Agent Attention. By introducing agent tokens, the computational efficiency is significantly improved while maintaining the global context modeling capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 The present invention is a preferred embodiment of a remote sensing target detection method based on improved RT-DETR overall flow chart;

[0041] Figure 2 The improved backbone module structure diagram of the present invention;

[0042] Figure 3 The structure diagram of the Agent Attention module improved by the present invention;

[0043] Figure 4 The figure is a block diagram of a remote sensing target detection system based on improved RT-DETR according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention through specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] The preferred embodiment of the present invention provides a remote sensing target detection method based on improved RT-DETR, such as Figure 1 As shown, the following steps are included:

[0046] Step 1: Collect remote sensing image data;

[0047] Step 2: preprocessing the collected remote sensing image data;

[0048] Step 3, the pre-processed remote sensing image data is made into a remote sensing data set;

[0049] Step 4: Use the improved RT-DETR model to train the data set to obtain a remote sensing target detector;

[0050] Step 5: Use hyperparameter search verification to optimize various parameters of the target detector on the verification set;

[0051] Step 6: Use the remote sensing target detector optimized in step 5 to detect the test image or video and evaluate the detection effect.

[0052] In step 1 of this embodiment, the remote sensing image data acquisition method includes: image data obtained by photographing the ground surface through a drone camera; and image data in the relevant open source remote sensing dataset RSOD.

[0053] In step 2 of this embodiment, median filtering and histogram equalization are used to perform image preprocessing, and the remote sensing image data collected in step 1 is subjected to denoising, contrast enhancement and clarity enhancement.

[0054] The detailed process of step 3 of this embodiment includes:

[0055] Step 301: Use LabelImg to manually label the detection target of each image in the data set, and use the minimum bounding rectangle of the labeled target as the real box.

[0056] Step 302: randomly divide the data set into a training set and a validation set in a ratio of 8:2.

[0057] In step 4 of this embodiment, an RT-DETR model is built, which consists of four parts: a backbone network, an encoder, a decoder, and a detection head; the specific steps are as follows:

[0058] Step 401: Figure 2As shown in the figure, FasterNet is used to replace the backbone of RT-DETR. It is divided into four stages (Stage), each stage contains a series of FasterNet blocks (FasterNet Block), and there is an embedding (Embedding) or merging (Merging) layer before each stage. The last three layers are used for feature classification. Inside each FasterNet block, the PConv operation is used, followed by two PWConv operations. This structure effectively utilizes the information of all channels to form a T-shaped convolution structure, which makes the model pay more attention to the features at the center position. After the PConv operation, normalization and activation layers are added only after the middle layer to maintain the diversity of features and reduce calculation delays. After the image passes through the Backbone, the deep convolution features are extracted, and the three-layer features of S3, S4, and S5 in the Backbone are used as the input of the encoder.

[0059] Step 402: After the backbone network feature extraction, three effective feature maps of different scales, S3, S4, and S5, are obtained. The last layer feature map S5 is tokenized and attention calculation is performed (calculation process: First, each pixel block in the input sequence is converted into an embedding vector in a high-dimensional space. For each embedding vector in the input sequence, three new vectors are generated through three different linear transformations (i.e., multiplication by three different weight matrices): query vector (Query, Q), key vector (Key, K), and value vector (Value, V). These three vectors will play different roles in subsequent attention calculations. Next, for each pixel block in the sequence, the score (Score) between it and all other pixel blocks in the sequence is calculated. This score is obtained by performing a dot product operation on the query vector (Q) of the current pixel block and the key vectors (K) of all other pixel blocks. This score is reflected It reflects the correlation or importance between the current pixel block and other pixel blocks. Since the result of the dot product operation may be very large, for the sake of numerical stability and the effectiveness of gradient propagation, the score is divided by a scaling factor (usually the square root of the key vector dimension). Then, the Softmax function is applied to the scaled score to ensure that the sum of all scores is 1, thereby obtaining a probability distribution, namely the attention weight. Finally, the value vectors (V) of all pixel blocks in the sequence are weighted and summed using the attention weights obtained in the previous step. The result of this weighted summation is a new vector that contains the contribution information of all pixel blocks in the sequence to the current pixel block. This new vector can be used as the output representation of the current pixel block after the self-attention layer), while the S3 and S4 layers extract features through convolution, and finally fuse the scale features with the feature map output by the encoder to obtain multi-scale features while greatly reducing the amount of calculation.

[0060] Step 403: Figure 3 As shown in the figure, the Deformable Attention used in the decoder is replaced by Agent Attention. Agent Attention introduces additional agent tokens into the traditional attention module. These agent tokens first act as agents for query tokens to aggregate information from keys and values, and then broadcast the information back to the query tokens. It maintains the accuracy of the softmax operation while achieving the high efficiency of linear attention.

[0061] Step 404: After the calculation of the decoder is completed, the prediction result and the target are matched by the Hungarian matching method, and the loss is calculated at the same time.

[0062] In step 5 of this embodiment, the hyperparameter search and verification method means trying different combinations of parameters (batch size, learning rate, etc.) and using a verification set to optimize the detector model performance to achieve the best effect.

[0063] By adopting the above technical solution, compared with the original RT-DETR model, the present invention reduces the amount of calculation and accelerates the network convergence speed while maintaining the original accuracy.

[0064] like Figure 4 As shown, this embodiment discloses a remote sensing target detection system based on improved RT-DETR, which is used to execute the above method, and includes the following modules:

[0065] Image acquisition module: collect remote sensing image data;

[0066] Preprocessing module: preprocess the collected remote sensing image data;

[0067] Dataset making module: Make the pre-processed remote sensing image data into a remote sensing dataset, and divide it into a training set and a validation set in proportion;

[0068] Training module: Use the RT-DETR model to train the remote sensing data set to obtain the remote sensing target detector;

[0069] Optimization module: uses hyperparameter search and verification to optimize the parameters of the remote sensing target detector on the verification set;

[0070] Detection module: Use the optimized remote sensing target detector to detect the test image or video.

[0071] For other contents of this embodiment, please refer to the above method embodiment.

[0072] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope of the present invention.

Claims

1. A remote sensing target detection method based on improved RT-DETR, characterized in that: The following steps are involved: Step 1: Collect remote sensing image data; Step 2: preprocessing the collected remote sensing image data; Step 3, converting the preprocessed remote sensing image data into a remote sensing data set, and dividing the remote sensing data set into a training set and a validation set; Step 4: Use the RT-DETR model to train the remote sensing data set to obtain a remote sensing target detector; Step 5: Use hyperparameter search verification method to optimize the parameters of remote sensing target detector on the verification set; Step 6: Use the remote sensing target detector optimized in step 5 to detect the test image or video.

2. A remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: In step 1, the remote sensing image data can be collected by: using a drone camera to photograph the surface to obtain image data; or using image data in the open source remote sensing dataset RSOD.

3. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: In step 2, the remote sensing image data collected in step 1 is subjected to denoising, contrast enhancement and clarity enhancement.

4. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: Step 3 is as follows: Step 301: Use LabelImg to label the detection target of each image in the data set, and use the minimum bounding rectangle of the labeled target as the real frame; Step 302: randomly divide the data set into a training set and a validation set in a ratio of 8:

2.

5. A remote sensing target detection method based on improved RT-DETR according to any one of claims 1 to 4, characterized in that: In step 4, the RT-DETR model: uses FasterNet, FasterNet includes partial convolution PConv and point-by-point convolution PWConv, and the FasterNet architecture is divided into four levels, each level starts with an embedding layer or a merging layer for spatial downsampling and channel number expansion; each level contains multiple FasterNet blocks, each FasterNet block contains a PConv layer, followed by two PWConv layers, in the form of an inverted residual block, where the intermediate layer has an expanded number of channels and has a shortcut connection to reuse the input features.

6. The remote sensing target detection method based on improved RT-DETR according to claim 5, characterized in that: In step 4, the RT-DETR model: adopts Agent Attention, which introduces agent tokens. The agent tokens first aggregate information from keys and values ​​as agents of query tokens, and then broadcast the information back to the query tokens.

7. The remote sensing target detection method based on improved RT-DETR according to claim 6, characterized in that: Step 4: The specific process is as follows: Step 401, using FasterNet, divided into four stages, each stage contains multiple FasterNet blocks, and there is an embedding or merging layer before each stage; the last three layers are used for feature classification; in each FasterNet block, a PConv operation is used, followed by two PWConv operations; after the PConv operation, normalization and activation layers are added only after the middle layer; the image is extracted after Backbone to obtain deep convolutional features, and the three-layer features S3, S4, and S5 in Backbone are used as the input of the encoder; Step 402: After the backbone network feature extraction, three effective feature maps of different scales S3, S4, and S5 are obtained. The last layer feature map S5 is tokenized and attention calculation is performed, while the S3 and S4 layers are feature extracted in the form of convolution. Finally, the inter-scale characteristics are fused with the feature map output by the encoder to obtain multi-scale features. Step 403: Use Agent Attention. Self-attention introduces an additional agent token. The agent token first acts as a proxy for the query token to aggregate information from the key and value, and then broadcasts the information back to the query token. Step 404: Match the prediction result and the target by the Hungarian matching method, and calculate the loss at the same time.

8. The remote sensing target detection method based on improved RT-DETR according to claim 1, characterized in that: In step 5, the hyperparameter search verification method is used to try different combinations of parameters and use the verification set to optimize the performance of the detector.

9. A remote sensing target detection system based on improved RT-DETR, used to perform the method according to any one of claims 1 to 8, characterized in that: Includes the following modules: Image acquisition module: collect remote sensing image data; Preprocessing module: preprocess the collected remote sensing image data; Dataset making module: Make the pre-processed remote sensing image data into a remote sensing dataset, and divide it into a training set and a validation set in proportion; Training module: Use the RT-DETR model to train the remote sensing data set to obtain the remote sensing target detector; Optimization module: uses hyperparameter search and verification to optimize the parameters of the remote sensing target detector on the verification set; Detection module: Use the optimized remote sensing target detector to detect the test image or video.

Images