An enhanced remote sensing image target detection method and system

Abstract

The present invention provides an enhanced remote sensing image target detection method and system, comprising inputting a high-resolution image dataset with annotations; setting an enhanced target detection model, including constructing an enhanced target detection model using embedded connections and skip connections, achieving a feature upsampling process at different spatial resolutions through embedded connections, and simultaneously utilizing skip connections to supplement lost detail information to enhance the model's feature representation capabilities, thereby providing a first-order target detection framework to achieve high-resolution remote sensing image target detection tasks; and training the obtained enhanced target detection model, and performing target detection on the input image based on the trained model to obtain the results. The enhanced remote sensing image target detection method provided by the present invention can not only effectively improve the performance of traditional methods, but can also be applied to on-orbit target real-time detection tasks in constrained environments onboard satellites.

Description

Technical Field

[0001] The present invention belongs to the field of satellite remote sensing image target detection, and particularly relates to a target detection scheme for high-resolution remote sensing images. An enhanced target detection network is proposed to realize the task of remote sensing image target recognition under high resolution. Background Art

[0002] In recent years, my country has completed a series of remote sensing satellite launches, building a space-based observation system with high spatial and temporal resolution. Consequently, acquiring high-resolution remote sensing imagery has become increasingly accessible. For high-resolution remote sensing imagery target detection tasks, such as aircraft detection and model identification at airports, ship detection at sea, and high-speed vehicle detection, the ability to quickly and accurately extract targets of interest is crucial.

[0003] With the continuous development of artificial intelligence technology, deep learning (DL) has been widely used in the field of computer vision. The core of deep learning is to build neural network models to achieve target detection tasks. Currently, the more mature target detection networks can be divided into first-order detection models and second-order detection models. First-order detection models are mainly represented by the YOLO series. Their main advantage is fast detection speed, which can meet the real-time target detection tasks in orbit. However, the detection accuracy is not high enough to meet the actual accuracy requirements. Second-order detection models are represented by the RCNN series of models. Their advantage is high detection accuracy, but the speed is slow, which is not suitable for real-time target detection and recognition tasks in orbit.

[0004] Remote sensing images differ from natural scene images. The objects to be detected generally vary greatly in size. Objects of the same type can produce significant differences in image quality due to factors such as viewing angle and resolution. Object distribution is also uneven, with some scenes being densely populated and others sparsely populated. Remote sensing images are significantly affected by lighting, cloud cover, and sensor imaging methods. Using standard object detection models directly makes it difficult to effectively extract key information, resulting in poor accuracy in detecting objects of interest. Summary of the Invention

[0005] Aiming at the current problems of difficult extraction of effective information and poor detection accuracy in target detection of high-resolution remote sensing images, the present invention provides an enhanced remote sensing image target detection solution to increase the model's feature extraction capability and detection accuracy.

[0006] The technical solution provided by the present invention is an enhanced remote sensing image target detection method, comprising the following steps:

[0007] Step 1: Input high-resolution image dataset with annotations;

[0008] Step 2: Setting up an enhanced target detection model, including constructing an enhanced target detection model using embedded connections and skip connections. Embedded connections are used to achieve feature upsampling at different spatial resolutions, while skip connections are used to supplement lost detail information, thereby enhancing the model's feature representation capabilities. This provides a first-order target detection framework for high-resolution remote sensing image target detection tasks.

[0009] Step 3: Model training, including training the enhanced target detection model obtained in step 2;

[0010] Step 4: Output the result, which is the result of target detection on the input image based on the model trained in step 3.

[0011] Moreover, in step 1, high-resolution remote sensing images from my country's Gaofen-2 satellite and Google Earth were used as data sources, and a high-resolution remote sensing image target detection dataset was constructed by manually annotating targets.

[0012] Moreover, in the enhanced target detection model, let the given input image be X, Represents the convolution operation, where W and H represent the width and height of the feature map respectively, and i is used to represent the level of the convolution layer block.

[0013]

[0014]

[0015]

[0016]

[0017] in, represents downsampling operation, Δ represents upsampling operation, represents the feature map concatenation operation, χ i ,i∈[0,9] represents the input feature map, f spp Represents the feature map output by the spatial pyramid pooling layer;

[0018] The method of upsampling or downsampling feature maps of different spatial resolutions into features of the same spatial resolution and then splicing the feature maps is called an embedded connection structure. The splicing between feature maps of the same spatial resolution is called a skip connection.

[0019] Moreover, during model training in step 3, a total of 300 epochs were trained, 32 images were loaded into the model for batch training each time, the initial learning rate was set to 0.01, and the learning rate was reduced every 50 epochs, each time reduced to 1 / 10 of the original value.

[0020] Moreover, the training results in step 3 are displayed. By inputting high-resolution remote sensing images, the detected target categories and the coordinates of the bounding box surrounding the target are output.

[0021] Moreover, it is used for real-time detection of on-orbit targets in constrained environments on board satellites.

[0022] The present invention provides an enhanced remote sensing image target detection system for implementing the enhanced remote sensing image target detection method described above.

[0023] Furthermore, the following modules are included,

[0024] The first module is used to input a high-resolution image dataset with annotations;

[0025] The second module is used to set up an enhanced target detection model, including constructing an enhanced target detection model using embedded connections and skip connections. The embedded connections are used to achieve feature upsampling at different spatial resolutions, while skip connections are used to supplement lost detail information, thereby enhancing the model's feature representation capabilities. This provides a first-order target detection framework for high-resolution remote sensing image target detection tasks.

[0026] The third module is used for model training, including training the enhanced object detection model obtained in the second module;

[0027] The fourth module is used to output the results, which are the results obtained by performing target detection on the input image based on the model trained in the third module.

[0028] Alternatively, it includes a software environment, a central processing unit, a memory, and a graphics image processor. The software environment is used to configure the basic modules and dependent environments for system execution, the memory is used to store the required data sets, and the graphics image processor is used to execute the enhanced remote sensing image target detection method as described above.

[0029] This paper provides an enhanced target detection model solution that can effectively improve the detection network's ability to quickly and accurately detect targets of interest in complex environments. This solution has high detection accuracy and speed, and can meet the real-time target detection tasks in constrained environments onboard satellites. Compared with existing target detection technologies, it has the following advantages:

[0030] (1) The enhanced target detection framework provided can effectively make up for the shortcomings of existing methods in high-resolution remote sensing image detection.

[0031] (2) The method proposed in this invention provides a new solution for achieving real-time on-orbit target detection on satellites, and is suitable for multi-target detection tasks in various complex environments.

[0032] (3) It is very practical. It not only maintains a high detection speed, but also effectively improves the detection accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of an enhanced remote sensing image target detection method according to an embodiment of the present invention.

[0034] Figure 2 Schematic diagram of an enhanced remote sensing image target detection system according to an embodiment of the present invention.

[0035] Figure 3 Schematic diagram of an enhanced target detection method according to an embodiment of the present invention.

[0036] Figure 4 Schematic diagram of the basic feature extraction unit structure of an embodiment of the present invention.

[0037] Figure 5 Schematic diagram of the spatial pyramid pooling structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0039] See also Figure 1 The embodiment of the present invention is a target detection network model method based on high-resolution remote sensing images. It can be used for experiments using the Python programming language, and can also be used for engineering applications using the C / C++ programming language.

[0040] The embodiment of the present invention provides an enhanced remote sensing image target detection method, see Figure 1 , including the following steps:

[0041] Step 1, data construction, includes inputting annotated high-resolution remote sensing image dataset;

[0042] This step can be used to process remote sensing image data, including automatic cropping and data preprocessing of high-resolution remote sensing images.

[0043] This example uses my country's Gaofen-2 panchromatic image data and some high-resolution remote sensing image data collected on Google Earth to construct a remote sensing image target detection dataset. The spatial resolution of the images is 0.5m to 0.8m, and the size ranges from 600×600 to 1372×1024. In practice, the collected images can be pre-labeled using the open-source LabelMe software. A total of 5,977 images were annotated. 4,781 images were used as a training set, 598 images as a validation set, and 598 images as a test set.

[0044] Step 2: Setting up the enhanced model includes setting up the enhanced target detection model structure to implement the first-order target detection framework.

[0045] The enhanced target detection model described in the embodiment adopts two methods, embedded connection and skip connection, to enhance the feature representation ability of the model, thereby effectively improving the detection performance of the model. The embedded connection mainly adopts a multi-path upsampling method, performing upsampling operations on the basis of different spatial resolutions, and then obtaining feature maps of the same spatial resolution through identity mapping and skip connection and performing feature map vector splicing operations, see Figure 3 .

[0046] The enhanced target detection model is as follows:

[0047] The enhanced target detection network mainly uses the nested connection structure and the skip connection method to realize the mining of deep features and the compensation of lost information. Suppose the given input image is X, Represents the convolution operation, where W and H represent the width and height of the feature map, respectively. The specific values are determined according to the convolution module, for example, W = w, H = h, W = w / 2, H = h / 2, etc.; i is used to represent the level of the convolution module. The convolution module is implemented using a basic feature extraction unit consisting of a convolution layer, a normalization layer, an activation layer, and an interpolation layer. Then we have:

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] in, represents downsampling operation, Δ represents upsampling operation, represents the feature map concatenation operation, χ i ,i∈[0,9] represents the input feature map, f spp Represents the feature map output by the spatial pyramid pooling layer. Feature maps of different spatial resolutions are upsampled or downsampled to the same spatial resolution and then concatenated together in an embedded connection structure. Concatenating feature maps of the same spatial resolution is called a skip connection.

[0054] The specific implementation of the enhanced target detection model is as follows:

[0055] (1) Input image processing

[0056] The input image must first be converted into a tensor before GPU-accelerated computations can be performed. Assume the dimensions of the input image X are C × W × H, where C represents the number of channels, W represents the width, and H represents the height. Since X has three dimensions, we need to add a dimension, X′, to X, bringing the dimensions of X′ to B × C × W × H.

[0057] (2) Convolution operation

[0058] Use the convolution layer to extract features from the input tensor X′, see Figure 3 , here the target detection model is divided into 11 different convolutional modules. The direction of the arrow indicates the data flow from one module to another module; for example, The arrow points to express The output is Input; for The input of the module is composed of and The outputs are composed of .

[0059] The first convolutional module is Indicates that the size of the feature map in this module is w×h.

[0060] The second convolutional module is Indicates the first module After one downsampling, the feature map of the second module is obtained. The input is The output of . The calculation formula is as follows:

[0061]

[0062] The third convolution module is The input is Output;

[0063] The fourth convolutional module is The input is Output;

[0064] The fifth convolutional module is The input is and Output;

[0065] The sixth convolutional module is The input is Output;

[0066] The seventh convolution module is The input is Output;

[0067] The eighth convolutional module is The input is Output;

[0068] The ninth convolutional module is The input is Output of SPP module;

[0069] The tenth convolutional module is The input is Output;

[0070] The eleventh convolution module is The input is Output.

[0071] In addition, set the SPP module, the input of the SPP module is Output.

[0072] For the second, third, and fourth convolution modules and the sixth convolutional module Basically the same, the mathematical expression is as follows:

[0073]

[0074] The fifth convolutional module Different from the previous four, this module is relative to the third convolution module There is no downsampling operation, but a direct identity mapping, and the module also integrates the fourth convolution module and the seventh convolutional module The up-sampled feature map is expressed as follows:

[0075]

[0076] The seventh convolutional module and the eighth convolutional module With the same fusion method, the mathematical expression is as follows:

[0077]

[0078] The ninth convolutional module The input comes from the sixth convolution module The output of , is not downsampled. Because, the fifth convolution module The output feature map is the input image after four downsampling operations. The features of many small targets have become very unclear. Therefore, the eighth convolution module and the spatial pyramid pooling module (Spatial Pyramid Pooling, SPP) are used to extract deep features of these small targets.

[0079]

[0080] Among them, f spp Represents the feature map output by the spatial pyramid pooling layer.

[0081] The tenth convolution operation module The shallow texture information features and deep semantic information features are integrated respectively. The mathematical expressions are as follows:

[0082]

[0083] Large targets often have larger scales, which is more conducive to the detection of large targets from a global perspective. Therefore, the feature map after four downsampling is used for coordinate prediction and target recognition tasks. The eleventh convolution module The mathematical expression is as follows:

[0084]

[0085] The above eleven convolutional modules are used to extract shallow and deep texture and semantic information respectively, thereby increasing the feature representation ability of the model and thus improving the detection performance of the model.

[0086] The present invention preferably proposes a first-order target detection network framework, which includes three modules: a backbone module (Backbone), a neck module (Neck) and a detection head module (Head);

[0087] The backbone module includes the first convolution module, the second convolution module, the third convolution module, the fourth convolution module, and the sixth convolution module;

[0088] The neck module (Neck) includes the fifth convolution module, the seventh convolution module, the ninth convolution module and the SPP module;

[0089] The detection head module (Head) includes an eighth convolution module, a tenth convolution module and an eleventh convolution module.

[0090] The input of the backbone module is an RGB image, and the output is a high-dimensional feature map, which is composed of five feature blocks, representing feature maps of different spatial resolutions. Each feature block is composed of basic feature extraction units (see Figure 4 ), the input of the first feature block is the RGB image, and in the following four feature blocks, the output of the previous feature block is the input of the next feature block;

[0091] The neck module includes feature maps of three spatial resolutions and an SPP module. The input of the convolutional layer where each resolution feature map is located is the feature map output by the backbone module at three different resolutions. The feature maps of different resolutions are all composed of basic feature extraction units. The input of the SPP module is the output of the sixth module.

[0092] The inputs of the detection head module are connected to the three-resolution outputs of the neck module. The module includes three detection heads as the target detection outputs. From top to bottom, they represent the detection of large, medium, and small targets, respectively. Each detection head is composed of basic feature extraction units.

[0093] In fact, each convolution module is a basic feature extraction unit, see Figure 4 , each convolution module is composed of three convolution layers (CONV_1, CONV_2 and CONV_3), Bottleneck, CAT and BN / Leaky_relu, where CONV_1, Bottleneck and CONV_2 are connected in sequence, the input of CAT includes the input of CONV_1 and the output of CONV_2, the output of CAT is connected to the input of BN / Leaky_relu, and the output of BN / Leaky_relu is connected to the input of CONV_3. Among them, Bottleneck is composed of two convolutions of 3×3, CAT represents feature vector concatenation, BN and Leaky_relu represent batch normalization and activation function respectively. The spatial pyramid pooling module can effectively capture representation information at different scales. For this purpose, the embodiment adopts the spatial pyramid pooling module to increase the robustness of the model. The location of this module participates Figure 3 SPP module.

[0094] Spatial Pyramid Pooling Module SPP see Figure 5, is a global maximum pooling of the input feature map by three different global maximum pooling layers (pooling kernel size POOL is 5, 9, and 13 respectively), obtaining feature maps of three different scales, 5×5, 9×9, and 13×13 respectively; then these three feature maps are upsampled and feature vector concatenated to obtain the output feature map. The embodiment uses the spatial pyramid pooling method to extract feature information at different scales (receptive fields), and uses three detection heads to achieve the target detection task, see Figure 5 The SPP module in .

[0095] Step 3: Model training settings, including preprocessing of input images and training implementation details.

[0096] Due to the inconsistency of the input image sizes, batch loading of data into the network model for training is not possible, and considering the limitations of the GPU memory of the hardware device. To this end, the embodiment uniformly scales and crops the input images to a size of 480×480. This step uses the training set and validation set obtained in step 1 for model training. The training set is used for model parameter update and training; the validation set is used for training and verification. The verification results are best obtained during the training, and the training parameters are saved.

[0097] When training the model, Ubuntu 16.04LTS system is required, and the system environment requires Pytorch 1.6, Python3.6 or above. The hardware platform requires an RTX2080Ti graphics card as the main computing platform, while the CPU memory is not less than 8G and the solid-state drive is not less than 256G. The preferred setting adopted by the embodiment when training the model is to train a total of 300 epochs (batches, one epochs = 150 iterations), each time loading 32 images into the model for batch training, the initial learning rate is set to 0.01, and the learning rate is reduced every 50 epochs, each time reduced to 1 / 10 of the original.

[0098] Step 4: Test based on the test set obtained in step 1, including testing the optimized model on the created dataset.

[0099] The embodiment of the present invention performs target detection tasks based on high-resolution remote sensing images, obtains semantic features at different resolutions through embedded connections, and uses skip connections to integrate multiple features, thereby enhancing the feature representation ability of the model and improving the detection performance of the model.

[0100] The steps for enhanced remote sensing image target detection are as follows: (The following part is the testing step, which starts after the model training is completed)

[0101] (1) Image processing and cropping

[0102] Most remote sensing images exceed 10,000 × 10,000 pixels. Scaling them directly to 480 × 480 pixels results in significant information loss. This paper addresses this issue by using a sliding window to crop large images in blocks while maintaining a 25% overlap. The cropped images are then scaled to 1,000 × 1,000 pixels and then scaled to 480 × 480 pixels for target detection.

[0103] (2) Block input to model detection

[0104] According to the above cropped images, the blocks are input into the designed enhanced target detection model for detection, and the detection results are output.

[0105] (3) Non-maximum suppression of repeated borders

[0106] Since step (1) performs a 25% overlapping cropping operation on the large-scale image, the overlapping targets appear in both images at the same time. To prevent them from being predicted as two targets, the present invention adopts a non-maximum suppression method to filter out the repeatedly predicted target frames. If the intersection-over-union ratio of any two target frames exceeds a certain threshold, they are determined to be repeatedly predicted frames and deleted.

[0107] (4) Output results

[0108] After step (3), the repeatedly predicted frames will be effectively filtered out, the remaining target detection frames will be output, and the target information will be marked on the original image.

[0109] To facilitate understanding of the technical effects of the present invention, a comparison between the present invention and the traditional method is provided below:

[0110] Table 1 Performance comparison

[0111]

[0112] Table 1 shows an experimental comparison using a dataset created by the present invention. The method provided by the present invention demonstrates superior performance. Compared to the second-order object detection network model Faster RCNN, the method provided by the present invention effectively improves mAP from 72.1% to 97.5%. Compared to the CNN-SOSF method, the mAP is improved by 16.4%.

[0113] Table 2 Performance comparison on open source datasets

[0114]

[0115] To further validate the effectiveness of our method, we used an internationally available open-source dataset for comparison. Table 2 shows the results, showing that our method achieved the highest mAP score and also improved performance for each target.

[0116] Table 3 Detection speed comparison

[0117]

[0118] Table 3 shows the comparison of detection speed when the input image is 10000×10000. It can be seen that the method provided by the present invention can effectively improve the performance of the model. At the same time, the detection time is within the allowable range, which meets the target detection task in the on-orbit environment.

[0119] By comparing the above two data sets, the enhanced target detection network provided by the present invention has stronger detection performance and can meet the real-time target detection task in the on-orbit environment.

[0120] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0121] In some possible embodiments, an enhanced remote sensing image target detection system is provided, comprising the following modules:

[0122] The first module is used to input a high-resolution image dataset with annotations;

[0123] The second module is used to set up an enhanced target detection model, including constructing an enhanced target detection model using embedded connections and skip connections. The embedded connections are used to achieve feature upsampling at different spatial resolutions, while skip connections are used to supplement lost detail information, thereby enhancing the model's feature representation capabilities. This provides a first-order target detection framework for high-resolution remote sensing image target detection tasks.

[0124] The third module is used for model training, including training the enhanced object detection model obtained in the second module;

[0125] The fourth module is used to output the results, which are the results obtained by performing target detection on the input image based on the model trained in the third module.

[0126] In some possible embodiments, an enhanced remote sensing image target detection system is provided, including a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute an enhanced remote sensing image target detection method as described above.

[0127] In some possible embodiments, an enhanced remote sensing image target detection system is provided, including a readable storage medium having a computer program stored thereon. When the computer program is executed, an enhanced remote sensing image target detection method as described above is implemented.

[0128] In some possible embodiments, an enhanced remote sensing image target detection system is provided, including a software environment, a central processing unit, a memory, and a graphics image processor. The software environment is used to configure the basic modules and dependent environments executed by the system, the memory is used to store the required data sets, and the graphics image processor is used to execute the enhanced remote sensing image target detection method as described above.

[0129] The enhanced remote sensing image target detection method provided by the present invention can not only effectively improve the performance of traditional methods, but also be applied to the real-time detection task of on-orbit targets in a constrained environment on board a satellite.

[0130] The specific examples described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific examples without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. An enhanced remote sensing image target detection method, characterized in that: The following steps are involved: Step 1: Input high-resolution image dataset with annotations; Step 2: Setting up an enhanced target detection model, including constructing an enhanced target detection model using embedded connections and skip connections. In the enhanced target detection model, eleven convolutional modules are set to extract shallow and deep texture and semantic information respectively, and the feature upsampling process of different spatial resolutions is realized through embedded connections. At the same time, skip connections are used to supplement the lost detail information, thereby enhancing the model's feature representation capability, thereby providing a first-order target detection framework to achieve high-resolution remote sensing image target detection tasks; In the enhanced target detection model, let the given input image be X, Represents the convolution operation, where W and H represent the width and height of the feature map respectively, and i is used to represent the level of the convolution layer block. in, represents downsampling operation, Δ represents upsampling operation, represents the feature map concatenation operation, χ i ,i∈[0,9] represents the input feature map, f spp Represents the feature map output by the spatial pyramid pooling layer; The method of upsampling or downsampling feature maps of different spatial resolutions into features of the same spatial resolution and then splicing them is called an embedded connection structure. The splicing between feature maps of the same spatial resolution is called a skip connection. Step 3: Model training, including training the enhanced target detection model obtained in step 2; Step 4: Output the result, which is the result of target detection on the input image based on the model trained in step 3.

2. The enhanced remote sensing image target detection method according to claim 1, characterized in that: In step 1, high-resolution remote sensing images from my country's Gaofen-2 satellite and Google Earth were used as data sources, and a high-resolution remote sensing image target detection dataset was constructed by manually annotating targets.

3. The enhanced remote sensing image target detection method according to claim 1, characterized in that: During model training in step 3, a total of 300 epochs were trained. 32 images were loaded into the model for batch training each time. The initial learning rate was set to 0.01, and the learning rate was reduced every 50 epochs, each time to 1 / 10 of the original value.

4. The enhanced remote sensing image target detection method according to claim 1, characterized in that: The training results in step 3 are displayed. By inputting high-resolution remote sensing images, the detected target categories and the coordinates of the bounding box surrounding the target are output.

5. The enhanced remote sensing image target detection method according to claim 1, 2, 3 or 4, characterized in that: Used for real-time detection of on-orbit targets in constrained environments on board satellites.

6. An enhanced remote sensing image target detection system, characterized by: Used to implement an enhanced remote sensing image target detection method as described in claims 1-5.

7. The enhanced remote sensing image target detection system according to claim 6, characterized in that: Includes the following modules, The first module is used to input a high-resolution image dataset with annotations; The second module is used to set up an enhanced target detection model, including constructing an enhanced target detection model using embedded connections and skip connections. The embedded connections are used to achieve feature upsampling at different spatial resolutions, while skip connections are used to supplement lost detail information, thereby enhancing the model's feature representation capabilities. This provides a first-order target detection framework for high-resolution remote sensing image target detection tasks. The third module is used for model training, including training the enhanced object detection model obtained in the second module; The fourth module is used to output the results, which are the results obtained by performing target detection on the input image based on the model trained in the third module.

8. The enhanced remote sensing image target detection system according to claim 6, characterized in that: The system comprises a software environment, a central processing unit, a memory, and a graphics image processor. The software environment is used to configure the basic modules and dependent environments executed by the system. The memory is used to store the required data sets. The graphics image processor is used to execute the enhanced remote sensing image target detection method as described in claims 1-5.

Images