A remote sensing image target detection method based on rectangular deformable convolution

Abstract

The present invention discloses a remote sensing image target detection method based on rectangular deformable convolution, which belongs to the field of target detection technology, including using a remote sensing image target detection data set to train an improved Oriented RCNN model; wherein, a rectangular deformable convolution is designed to replace the general convolution in the convolutional neural network, a high-level and low-level feature fusion module is added to the feature pyramid network and a sub-pixel convolution is applied to generate a high-resolution feature map, two shared fully connected layers of the detection head are deleted and an adaptation network is used in the classification and regression branches respectively to obtain an improved Oriented RCNN model; the improved Oriented RCNN model is used in remote sensing image target detection. The present invention uses rectangular deformable convolution to better adapt to the geometric transformation of remote sensing image targets, and the feature fusion module is used to retrieve targets with smaller scales. The present invention can obtain better target detection accuracy.

Description

Technical Field

[0001] The invention belongs to the technical field of target detection, and in particular relates to a remote sensing image target detection method based on rectangular deformable convolution. Background Art

[0002] The development of microelectronics hardware and software, the update of navigation and communication technology, and the breakthrough of materials and energy technology have made aviation platforms a hot research topic in the field of international remote sensing. With the vigorous development of detection tasks, horizontal frames can no longer meet the needs of researchers in the field of remote sensing target detection. Remote sensing image targets, such as airplanes, cars, and ships, are relatively narrow and long, and generally present a variety of rotation angles, which requires the model to have a strong recognition ability for targets of any rotation angle.

[0003] Rotated object detection is an extended branch of object detection. It can detect objects at any angle in the image, which is of great significance for scenes such as remote sensing images. In the past decade, with the development of deep learning, the field of object detection has made significant progress. Deep learning has been widely used in the field of remote sensing image object detection due to its powerful feature learning and generalization capabilities and has received widespread attention. However, remote sensing image objects have the characteristics of multi-scale, multi-rotation angle, and complex background. Traditional deep learning-based object detection methods, such as Faster R-CNN, rely on horizontal bounding boxes to locate objects, but this method cannot accurately locate rotated objects because the use of horizontal detection boxes will cause the detection box to contain a lot of background information or other target interference information, resulting in poor final classification and positioning effects, and it is difficult to obtain ideal results.

[0004] At the same time, the scale of the target from the drone's perspective will appear in different sizes on the image depending on the shooting position or height, which brings challenges to image analysis and target detection tasks. For example, targets of the same category may have different shapes, textures, and structural features at different resolutions, or targets in the same image may have different scales due to factors such as perspective, distance, and projection. Multi-scale feature fusion can effectively alleviate the problem of large changes in target scale.

[0005] The most common method of multi-scale fusion in general target detection is feature pyramid network (FPN), which can achieve higher detection accuracy for small objects. FPN adds top-down and lateral connection structures in deep convolutional neural networks to fuse high-level semantic information with low-level image detail information to form a feature pyramid with high-level semantic information, and then uses the features of each layer for target detection. However, it is still not suitable for data characteristics of large remote sensing image size and small targets.

[0006] In summary, there are still problems in aerial remote sensing images such as small target size, unstable angle, variable direction and great detection difficulty, and the existing methods are not well adapted to the detection of aerial images containing a large number of small targets. Summary of the invention

[0007] In order to solve the above-mentioned deficiencies in the existing technology, the present invention provides a remote sensing image target detection method based on deformable convolution, which aims to solve the problems of small size, non-fixed angle, changeable direction and great detection difficulty of aerial remote sensing image targets.

[0008] To achieve the above object, the present invention provides a remote sensing image target detection method based on rectangular deformable convolution, comprising:

[0009] Use remote sensing image target detection dataset to train the improved Oriented RCNN model;

[0010] The improved Oriented RCNN model refers to replacing the general convolution in the convolutional neural network with a rectangular deformable convolution, adding a high-level and low-level feature fusion module to the feature pyramid network, deleting the two shared fully connected layers of the detection head and using adaptation networks in the classification and regression branches respectively, to obtain an improved Oriented RCNN model;

[0011] The network model is trained and the parameters are updated according to the training set, and then the image features are extracted and tested on the test set to obtain the category, confidence score and corresponding target box coordinate information of each detected target in the test image.

[0012] The invention discloses a remote sensing image target detection method based on rectangular deformable convolution, which uses widely used public remote sensing image target detection datasets DOTA v1.0 and FAIR1M 1.0 as training data and test data of the model.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) Existing target detection methods cannot accurately locate rotating targets, and the final classification and positioning effects are poor, making it difficult to obtain ideal results. The present invention designs a rectangular deformable convolution suitable for rotating target detection tasks, which can better adapt to the geometric transformation of targets in remote sensing images, accurately locate and extract remote sensing image features, thereby improving target detection accuracy.

[0015] (2) To address the problem that remote sensing images are large in size and targets are small in size, occupying few pixels in the entire image, making it difficult to distinguish them from the surrounding background; the present invention designs a low-level and high-level feature fusion module, integrates low-level and high-level features of different granularities, and uses sub-pixel convolution for upsampling to generate high-resolution feature maps to detect small-scale objects, which to a certain extent solves the problem of detecting smaller targets in large-scale remote sensing images.

[0016] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention and do not constitute an improper limitation of the present invention. Moreover, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0018] Figure 1 It is a model structure diagram of a remote sensing image target detection method based on rectangular deformable convolution of the present invention;

[0019] Figure 2 is a comparison diagram of the general deformable convolution and the rectangular deformable convolution of the first embodiment;

[0020] Figure 3 This is a process diagram of generating offset by rectangular deformable convolution in the first embodiment;

[0021] Figure 4 is a structural diagram of the improved PANet of the first embodiment;

[0022] Figure 5 is a structural diagram of the high-level and low-level feature fusion modules of the first embodiment;

[0023] Figure 6 is a structural diagram of a feature adaptation network of the first embodiment;

[0024] Figure 7 is a schematic diagram of the comparison between the first embodiment and the existing method on the DOTA v1.0 dataset;

[0025] Figure 8 is a schematic diagram of the comparison between the first embodiment and the existing method on the FAIR1M 1.0 dataset;

[0026] Fig. 9 is a schematic diagram of the validity verification of each module in the second embodiment;

[0027] Fig.10 It is a flow chart of a remote sensing image target detection method based on rectangular deformable convolution. Implementation

[0028] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art. Example

[0029] like Figure 1 As shown, this embodiment discloses a remote sensing image target detection method based on rectangular deformable convolution, including:

[0030] Step 1: Acquire and process remote sensing image data sets, and divide the remote sensing image data sets;

[0031] Step 2: Improve the Oriented RCNN model;

[0032] Step 3: Use the remote sensing image dataset to train the improved Oriented RCNN model;

[0033] Step 4: Extract image features from the trained network model and test it on the test set to obtain the category, confidence score, and corresponding target box coordinate information of each detected target in the test image, and calculate the mean average precision (mAP).

[0034] Among them, the detection targets include: bridges, ports, ships, airplanes, vehicles, stadiums, oil storage tanks, swimming pools, roundabouts and other targets.

[0035] In step 1, step 101, obtaining and processing a remote sensing image dataset, includes:

[0036] Download the DOTA v1.0 and FAIR1M 1.0 datasets from the DOTA and FAIR1M official websites respectively;

[0037] Convert the FAIR1M 1.0 dataset to DOTA format annotations.

[0038] Step 102: Processing the original image of the data set, including:

[0039] The original image is cropped into 1024×1024 patches; the cropping stride is set to 824, i.e., the pixel overlap between two adjacent patches is 200.

[0040] In step 2, the Oriented RCNN model is improved, including:

[0041] Step 201, designing a rectangular deformable convolution to replace the general convolution in the convolutional neural network;

[0042] Deformable convolution introduces a learnable two-dimensional offset for each point on the basis of traditional convolution kernel. (offset in the x and y axis directions), so that the convolution kernel can perform convolution operations on an irregular grid instead of a fixed square grid. The deformable convolution is defined as

[0043]

[0044] in Each point of the feature map corresponds to the center point of the convolution kernel. is the position of each convolution kernel.

[0045] The rectangular deformable convolution designed by the present invention improves the deformable convolution kernel; by modifying the generated learnable two-dimensional offset , so that the final convolution kernel shape is a rotated rectangle, rather than an arbitrary irregular shape; the general deformable convolution and the rectangular deformable convolution of the present invention are compared. Figure 2 shown.

[0046] For a convolution with a kernel size of 3, let the set of 9 sampling points be for,

[0047]

[0048] First generate The points at the upper left and upper right corners, i.e. , 2 learnable offsets, and then The offset is added to Lower left corner ( ),Will The offset is added to Lower right corner ( ) makes it a parallelogram ABCD, such as Figure 3 (a); then the parallelogram is stretched along its shorter diagonal to become a rectangle, and the coordinates of the two vertices E and F of the newly generated rectangle and the coordinates of the center point of the rectangle are calculated. ,like Figure 3 (b) As shown; re-adjust The four corner points around , , , ) so that the offset becomes the four corner points A, E, C, and F of the rectangle; adjust the center point of the convolution kernel offset to the center of the rectangle ,like Figure 3 (c) shows, calculate The offsets of the four points from top to bottom, left to right to the midpoints of the four sides of the rectangle are as follows: Figure 2 As shown in (b); for a convolution with a kernel size of 3, the network only needs to learn 4 offset values ​​instead of 18, and the rest can be calculated, thereby reducing the number of deformable convolution parameters.

[0049] The rectangular deformable convolution in the present invention is added to the visual backbone network, and the rectangular deformable convolution is used to replace the 3×3 convolution of the bottleneck block in the second, third, and fourth stages of ResNet50 and ResNet101 to enhance the feature expression ability and receptive field; the rectangular convolution can adaptively adjust the shape and position of the convolution kernel to adapt to different input features, thereby improving the diversity and robustness of the features.

[0050] Step 202: adding high-level and low-level feature fusion modules to the feature pyramid network and applying sub-pixel convolution to generate a high-resolution feature map;

[0051] Feature pyramid network (FPN) is a basic component for detecting objects of different scales in target detection tasks. It has a horizontally connected top-down architecture and builds high-level semantic feature maps at all scales. PANet adds a bottom-up path after the top-down path of FPN to make up for the lack of low-level feature details in FPN high-level features. Based on PANet, the present invention uses three high-level and low-level feature fusion modules (HLFFM) to replace the top-down path of PANet. The structure diagram of the improved PANet is shown in FIG. Figure 4 shown.

[0052] The high-level and low-level feature fusion module (HLFFM) designed by the present invention is as follows Figure 5 In order to effectively retrieve high-resolution feature maps, this module integrates low-level and high-level features to extract features of different granularities and uses sub-pixel convolution for upsampling to generate high-resolution feature maps; first, high-level features are Perform nearest neighbor interpolation upsampling; apply 1×1 convolution to low-level features ; Among them, the high-level features are the outputs of the deeper layers of the network or the outputs of HLFFM, and the low-level features are the outputs of the shallower layers of the network; then the high-level and low-level features are connected, and sub-pixel convolution is used to generate a higher resolution feature map; finally, 3×3 convolution is used to refine the output. The calculation process is expressed as

[0053]

[0054] where represents the output of HLFFM.

[0055] Assume that the features of the image obtained through the backbone network are as follows from shallow to deep layers: , the calculation process of the three HLFFMs can be expressed as

[0056]

[0057] in represents the output of the i-th HLFFM.

[0058] Step 203, delete the two shared fully connected layers of the detection head and use the adaptation network in the classification and regression branches respectively;

[0059] The present invention deletes the two shared fully connected layers after RoI Align in the Oriented RCNN model, and uses adaptation networks in the classification and regression branches respectively; the purpose is that when performing classification or regression, the two branches do not use the same feature, but learn more suitable features for the two branches respectively through the adaptation network, so as to further enhance the discriminative power of the features of the region of interest; and there is no increase in the number of parameters and the amount of calculation compared with the original model.

[0060] The adaptation network usually consists of a fully connected layer for dimensionality reduction, a nonlinear activation function, a fully connected layer for dimensionality increase, and a residual connection. The structure is as follows Figure 6 As shown in the figure, the features previously obtained by aligning the image features and the candidate region frame by rotating the region of interest are recorded as ,Will Input classification adaptation network separately and regression adaptation network , and then the new classification features obtained and regression features They are used for subsequent category classification and detection box regression respectively. They are expressed as follows:

[0061]

[0062]

[0063] in , , , To adapt the learnable parameters in the network.

[0064] The dimension reduction and dimension increase operations of the fully connected layer of the feature adaptation network can help the network focus on more important and discriminative features and reduce the interference of redundant information and noise. In this way, the network can better learn the key features of the target, thereby improving performance. At the same time, the residual connection of the feature adaptation network allows the network to directly transfer information across layers, which helps to alleviate the gradient vanishing problem and improve information flow. Through the residual connection, the network can better utilize the information of the previous layer, avoid the loss of information during the propagation process, and help the network better learn the subtle features and contextual information in the image.

[0065] The dimension reduction and dimension increase operations of the fully connected layer of the feature adaptation network can help the network focus on more important and discriminative features and reduce the interference of redundant information and noise. In this way, the network can better learn the key features of the target, thereby improving performance. At the same time, the residual connection of the feature adaptation network allows the network to directly transfer information across layers, which helps to alleviate the gradient vanishing problem and improve information flow. Through the residual connection, the network can better utilize the information of the previous layer, avoid the loss of information during the propagation process, and help the network better learn the subtle features and contextual information in the image.

[0066] In step 3, the specific steps of training the improved Oriented RCNN model using the remote sensing image dataset are:

[0067] Step 301, use a batch size of 2 for training on a single Nvidia A40 GPU, use ResNet50 and ResNet101 as the backbone of the network, use ImageNet pre-trained parameters to initialize the network, use horizontal and vertical flipping as data enhancement during training, set the learning rate to 0.0025, use the SGD algorithm to optimize the entire network, set the momentum to 0.9, and the weight decay to 0.0001;

[0068] Step 302: Divide the data set into a training set and a validation set;

[0069] Step 303: Load pre-training weights;

[0070] Step 304: Perform model training and update weights;

[0071] Step 305: Use the validation set to validate the model, and save the network parameters when the model detection effect reaches the best for testing;

[0072] In step 4, the specific steps of extracting image features from the trained network model on the test set and testing it are as follows:

[0073] Step 401, loading pre-training weights;

[0074] Step 402: Input the test set into the network to predict the category, confidence score and corresponding target frame coordinate information of each detected target in the test image;

[0075] Step 403: Calculate the mean average precision according to the prediction information. The calculation method is as follows:

[0076]

[0077] Where N represents the number of target categories, N=15 on the DOTA v1.0 dataset, and N=37 on the FAIR1M 1.0 dataset.

[0078] In order to better illustrate the effectiveness of the present invention, the improved Oriented RCNN is compared with the existing methods in the detection effects of DOTAv1.0 and FAIR1M 1.0. The results are as follows: Figure 7 and Figure 8 As shown,

[0079] On the DOTA v1.0 dataset, the proposed method surpasses all the compared methods. Using ResNet50 as the backbone network for single-scale training, a mAP of 77.18% was obtained.

[0080] In addition, the accuracy of small target categories is compared, and four small target categories are selected: ship (SH), plane (PL), helicopter (HC), and small vehicle (SV). The detection accuracy of the method of the present invention in the above four categories is better than that of the existing methods, which shows that the method of the present invention can better detect small-scale targets.

[0081] The method of the present invention is compared with other 6 existing methods on the FAIR1M 1.0 dataset. The results show that the performance of the method of the present invention is better than the existing methods, reaching 46.72%.

[0082] Embodiment 2

[0083] This embodiment verifies the effectiveness of various parts of the improvement of the Oriented R-CNN model in the invention method; the experiment is conducted on the DOTA v1.0 dataset. The experiment adopts single-scale training, uses ResNet50 as the backbone network of the image encoder, selects Oriented R-CNN as the basic method (without adding any modules), and adds different modules on it for experiments. The experimental results are compared with the experimental results of the Oriented R-CNN method and the method with all modules added (improved complete model) to verify the effectiveness of each module. The results are as follows: Fig. 9 shown.

[0084] From the results, we can see that the overall effect of the model is always better than the results of each group of ablation experiments, proving the effectiveness of each module; when only rectangular deformable convolution is used, the performance reaches 76.87%, and the model is better than Oriented R-CNN ( Fig. 9 The first row) is improved by 1.18%, which proves the effectiveness of the rectangular deformable convolution of the present invention; when there is only an adaptation network, the performance is improved by 1.09%, indicating that the fully connected layers and residual connections of dimensionality reduction and dimensionality increase can help the network better learn the subtle features and contextual information in the image, thereby improving the performance; when there is only a feature fusion module, the performance is also improved by 0.52%, indicating that the module can improve the category pixel positioning and restore the object details by feature fusion and generating high-resolution feature maps, thereby improving the accuracy of target detection.

[0085] At the same time, the number of parameters of the model was compared, compared with the model without adding any modules ( Fig. 9 Line 1), add the model of all modules ( Fig. 9 In the 5th row, the performance is improved by 1.49% with only an 8% increase in the number of parameters. Comparing the experimental results in the 1st and 2nd rows, by using the rectangular deformable convolution model with basically unchanged number of parameters, the performance is improved by 1.18%, because it can reduce invalid information (such as background or foreground areas) to improve performance. It can be seen that the proposed rectangular deformable convolution can better adapt to the geometric transformation of the target.

[0086] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0087] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A remote sensing target detection method based on rectangular deformable convolution, characterized in that: include: Acquire and process a remote sensing image dataset, and divide the remote sensing image dataset; use the remote sensing image dataset to train an improved Oriented RCNN model; the network structure of the improved Oriented RCNN model refers to replacing the general convolution in the Oriented RCNN convolutional neural network with a rectangular deformable convolution, adding high-level and low-level feature fusion modules to the feature pyramid network, deleting two shared fully connected layers of the detection head, and using adaptation networks in the classification and regression branches respectively; Deformable convolution introduces a learnable two-dimensional offset for each feature point based on the traditional convolution kernel. , represents the offset in the x and y axis directions, so that the convolution kernel can perform convolution operations on an irregular grid instead of a fixed square grid. ,in Each point of the feature map corresponds to the center point of the convolution kernel. is each position of the convolution kernel; the rectangular deformable convolution generates a learnable two-dimensional offset by modifying , so that the final convolution kernel shape is a rotated rectangle instead of an arbitrary irregular shape; The high-level and low-level feature fusion module refers to integrating low-level and high-level features to extract features of different granularities and using sub-pixel convolution for upsampling to generate high-resolution feature maps; performing nearest neighbor interpolation upsampling on the high-level feature h; applying 1×1 convolution to the low-level feature l; wherein the high-level feature is the output of a deeper layer of the network or the output of the HLFFM, and the low-level feature is the output of a shallower layer of the network; then connecting the high-level and low-level features, using sub-pixel convolution to generate a higher-resolution feature map; finally, using 3×3 convolution to refine the output; The adaptation network is usually composed of a fully connected layer with reduced dimension, a nonlinear activation function, a fully connected layer with increased dimension and a residual connection. The features previously obtained by aligning the image features and the candidate region box by rotating the region of interest are recorded as ,Will Input classification adaptation network separately and regression adaptation network , and then the new classification features obtained and regression features They are used for subsequent category classification and detection box regression respectively.

2. The remote sensing target detection method based on rectangular deformable convolution as claimed in claim 1, characterized in that: For the frequently used remote sensing image target detection datasets DOTA v1.0 and FAIR1M 1.0, the datasets are divided according to the configuration files.

3. The remote sensing target detection method based on rectangular deformable convolution as claimed in claim 1, characterized in that: For the backbone network with convolutional neural network ResNet as the backbone network for extracting remote sensing image features, a rectangular deformable convolution suitable for the task of rotated target detection is designed, and it is used to replace the general convolution in the bottleneck blocks of the 2nd, 3rd and 4th stages of the backbone network. By offsetting the position of each convolution kernel, a more flexible and adaptive convolution operation is achieved. Unlike traditional convolution, the sampling points of the deformable convolution kernel can be determined by learning, which has stronger spatial adaptability.

4. The remote sensing target detection method based on rectangular deformable convolution as claimed in claim 1, characterized in that: The feature pyramid network of the model is used with PANet and three high-level and low-level feature fusion modules are added to it. The purpose is to solve the problem of diverse target sizes and shapes in detection tasks and to enhance the semantic information of features.

5. The remote sensing target detection method based on rectangular deformable convolution as claimed in claim 4, characterized in that: The acquisition steps of the high-level and low-level feature fusion module include: upsampling the high-level feature, modifying the number of channels of the low-level feature, connecting the two features and using sub-pixel convolution, and finally refining the output by convolution.

6. A remote sensing target detection method based on rectangular deformable convolution as claimed in claim 5, characterized in that: Sub-pixel convolution is used to generate high-resolution feature maps for detecting smaller objects using high-level and low-level fused features.

7. The remote sensing target detection method based on rectangular deformable convolution as claimed in claim 1, characterized in that: The two shared fully connected layers of the detection head are deleted and the adaptation network is used in the classification and regression branches respectively to enhance the discriminative power of the features of the region of interest; The improved Oriented RCNN model includes: an improved backbone network, an improved feature pyramid network and a detection head.

Images