Remote sensing image salient target detection method based on deep learning
Through the remote sensing image significant object detection method based on deep learning, and using technologies such as ResNet34 and hollow convolution, the significant object detection problem in complex scenarios of remote sensing images is solved, efficient feature extraction and information fusion are achieved, and detection accuracy is improved.
Patent Information
- Application Number
- CN202411098079.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The existing technology lacks an effective remote sensing image significant object detection model. It is inappropriate to directly apply the edge segmentation model of natural scene images to remote sensing images, and it is impossible to effectively process complex remote sensing image scenes.
A method of significant object detection of remote sensing images based on deep learning was designed, using ResNet34 backbone network to extract multi-level features, combining hollow convolution and edge extraction modules, and fusing semantic features and edge features through advanced semantic information processing modules and information fusion modules. , perform object detection.
Accurate and significant object detection of remote sensing images is achieved, model performance is improved, and the accuracy and reliability of detection results are improved through multi-level feature fusion and supervision of losses.
Smart Images

Figure CN119091273A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for detecting salient objects in remote sensing images based on deep learning. Background Art
[0002] Human visual attention mechanism tries to find the most attractive part of a photo or a video, and salient object detection is inspired by this. In recent years, salient object detection has received widespread attention because it has achieved good results in many fields, such as image / video segmentation, video summarization, image quality assessment, image classification, etc.
[0003] In recent years, various salient models have been designed to process photos or videos of natural scenery. It is worth noting that with the continuous development and improvement of deep learning technology, its performance effect on salient target detection models has become increasingly prominent. Unfortunately, there are indeed relatively few models for salient target detection in remote sensing images. In fact, remote sensing images are very useful. In the agricultural field, remote sensing images can be used to detect crop growth and help increase crop yields. In the military field, they can be used to collect intelligence, target identification, etc. With the continuous development of technology, remote sensing plays an increasingly important role in new fields and applications. Whether it is environmental protection, resource management, disaster response or scientific research, remote sensing images have played a very important role.
[0004] Remote sensing images are mainly divided into aerial photographs and satellite photos, which are usually obtained by high-altitude aircraft or satellites. Therefore, remote sensing images often present complex scenes, such as the size of various objects, different directions, cluttered backgrounds, and even no significant targets, etc. Moreover, natural scene images are usually taken by handheld cameras, so it is obviously inappropriate to directly apply the significant target detection model based on edge segmentation of natural scene images to remote sensing images. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides a method for detecting salient targets in remote sensing images based on deep learning.
[0006] The method of the present invention comprises the following steps:
[0007] Step (1). Acquire optical remote sensing images and construct a data set.
[0008] Step (2). In the encoding stage, the data in the dataset is extracted by building a backbone network with multi-level features. The specific method is as follows:
[0009] In the encoder stage, the backbone network is mainly designed based on ResNet34 to extract multi-level features. The backbone network includes five encoding modules connected in sequence, Encoder-1, Encoder-2, Encoder-3, Encoder-4, and Encoder-5, which output five levels of features, F1, F2, F3, F4, and F5, respectively.
[0010] The Encoder-1 includes Conv1 and Conv2_x of ResNet34, and adds a maximum pooling layer after Conv1; the subsequent Encoder-2, Encoder-3 and Encoder-4 respectively use Conv3_x, Conv4_x and Conv5_x in ResNet34; finally, Encoder-5 contains a maximum pooling layer and three residual modules.
[0011] Through research, it can be concluded that shallow low-level features retain the spatial information of the reconstructed target boundary, and deep high-level features retain the semantic information of the target. The semantic information contained in the high-level features plays an important role in camouflaged target detection.
[0012] Step (3). Construct a high-level semantic information processing module GFM through dilated convolution and obtain semantic features based on multi-level features. The specific method is as follows:
[0013] The proposed model designs a GFM module, the main body of which is a dilated convolution. There are five branches in the GFM module structure, including four convolution kernels with different expansion rates to process F5 respectively, and then a convolution operation is used to retain the original information. The results obtained from the five branches are subjected to feature fusion through a concatenation operation, and the fused feature map is subjected to a convolution operation and element-by-element addition of F5 to perform feature fusion; finally, the output F is obtained through a CBR unit, i.e., convolution, batch normalization and activation function activation operation. * 5, and adding loss supervision ls0 at the output of GFM improves the performance.
[0014] Step (4). Construct an edge extraction module EEM to extract edge features based on multi-level features. The specific method is as follows:
[0015] The edge extraction module EEM converts F * 5 After upsampling and concatenating features with F3, it is then concatenated with F1 after downsampling, and finally the edge feature E is obtained through the CBR unit. * ; and add a supervision loss after the EEM module.
[0016] Step (5). In the decoding stage, decoding is performed based on multi-level features, and an information fusion module EFM is constructed to combine semantic features and edge features for feature fusion, and output the target detection results. The specific method is as follows:
[0017] The model proposed in this invention designs the EFM module, F * i-1 The specific operations obtained by fusion of upper and lower level features are as follows:
[0018]
[0019] Where conv represents the convolution operation, the convolution kernel size is 1×1, and [.,.] represents the concatenated feature fusion operation. When i is equal to 5, the operation is as follows:
[0020]
[0021] For the five encoding modules, five corresponding decoding modules Decoder-B1, Decoder-B2, Decoder-B3, Decoder-B4, and Decoder-B5 are constructed respectively. The input of Decoder-B5 is F * 5, the output is D5; D5 and F * 4 Input information fusion module EFM to get the corresponding output D * 5. Then D * 5 and F4 are input into Decoder-B4 to get D4, and D4 and F * 3 After passing through the EFM module, D is obtained * 4. The implementation process of Decoder-B3, Decoder-B2, and Decoder-B1 is the same as that of Decoder-B4. The output D1 of Decoder-B1 is the target detection result.
[0022] Through the EFM module, we can get {D * 2, D * 3, D * 4.D * 5}, the entire process of the EFM module is described as follows:
[0023]
[0024] Where u / d(.) indicates that downsampling or upsampling is selected according to different levels to keep the height and width of the feature map consistent. * 2, D * 3, D * 4.D * 5} both add a supervision and calculate the loss to improve the performance of the model, so {D* 2, D * 3, D * 4.D * 5} All of them need to go through convolution and upsampling operations and finally use the sigmoid activation function to obtain the saliency map, calculate the comparison loss, and finally obtain D1, which is the final output of the network.
[0025] The beneficial effects of the present invention are as follows:
[0026] The main advantages of the method of the present invention are as follows: Specifically, the model designs an encoder based on ResNet-34 to extract features at all levels, and designs two modules to better utilize high-level semantic information and edge information. The high-level semantic information is processed by a dilated convolution method that can increase the receptive field, and edge supervision is added to more efficiently utilize edge information. In the decoding stage, a module is designed to act on each layer to fuse edge information and upper and lower level feature fusion information. The inputs of the modules designed in the decoding stage are the output of the previous level, edge information, and upper and lower level feature fusion information, respectively, so as to obtain the final accurate salient target prediction map. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a framework diagram of the method of the present invention;
[0028] Figure 2 This is a schematic diagram of GFM implementation;
[0029] Figure 3 This is a result diagram of the method of the present invention. DETAILED DESCRIPTION
[0030] The present invention will be further described below in conjunction with the accompanying drawings.
[0031] like Figure 1 As shown, the present invention provides a remote sensing image salient target detection method based on deep learning as follows:
[0033] Step (1). Obtain the data set. The specific method is as follows:
[0034] In order to thoroughly verify the feasibility of the model, a relatively basic dataset in optical remote sensing images, the Extended Optical Remote Sensing Image Salient Object Detection (EORSSD) dataset, was used. Specifically, the EORSSD dataset contains a total of 2,000 images, of which 1,400 images are used to train the model and 600 images are used for testing. It is worth mentioning that this dataset is an extension of the ORSSD dataset. This new dataset is larger and more diverse than the original dataset. It contains 2,000 images and corresponding pixel-level ground truth, which contains many semantically meaningful but challenging images.
[0035] Step (2). Feature extraction, the specific method is as follows:
[0036] In the encoder stage, the backbone network is mainly designed based on ResNet34 to extract multi-level features. However, unlike the traditional ResNet34 network, the present invention replaces the convolution kernel with a size of 7×7, 64 channels, and a step size of 2 in the traditional ResNet34 network Conv1 with a convolution kernel with a size of 3×3, 64 channels, and a step size of 1, and adds a maximum pooling layer afterwards. Therefore, the Encoder-1 in the present invention includes the changed Conv1 and the Conv2_x in the original ResNet34, and the subsequent Encoder-2, Encoder-3 and Encoder-4 respectively use Conv3_x, Conv4_x and Conv5_x in ResNet34. In addition, in order to expand the receptive field of the model of the present invention, an Encoder-5 is designed after Encoder-4, which includes a pooling convolution kernel with a size of 2×2 and a step size of 2, and three residual modules with a channel number of 512, such as Figure 1 , through these five modules we can get {F1, F2, F3, F4, F5}.
[0037] Step (3). Advanced semantic information processing module, the specific method is as follows:
[0038] In order to better utilize high-level semantic information, the proposed model of the present invention designs a GFM module, such as Figure 2 The main body shown uses dilated convolution, which is also called expanded convolution. It inserts extra zero values between the elements of the convolution kernel, so as to increase the size of the convolution kernel receptive field without adding extra parameters or using pooling layers. In dilated convolution, a very important parameter is the dilation rate, which is defined as the spacing between values when the convolution kernel processes data. In the GFM module structure, four different dilation rates are used to process F5 separately. At the same time, a 1×1 convolution kernel is introduced to retain the original information. The results obtained from the five branches are concatenated for feature fusion. At the same time, in order to better utilize high-level information, a "+" operation, i.e., add operation, is introduced. Therefore, a convolution kernel of size 1×1 is required to transform the number of channels of the result after feature fusion. The fused feature map is added element by element with F5 through a convolution operation to perform feature fusion. Finally, the final F is obtained through CBR, i.e., convolution, batch normalization, and activation function activation operations. * 5. A loss supervision is added at the output of GFM to improve model performance, so the obtained F *5 A saliency map is obtained by convolution with a convolution kernel of size 1×1, upsampling and sigmoid activation function activation operations for comparison and calculation of loss.
[0039] Step (4). Edge extraction module, the specific method is as follows:
[0040] For the detailed processing of edge information, an edge extraction module, namely the EEM module, is designed. Here, three levels of features are used for feature fusion, among which F * 5 is the output of the module GFM, which is then upsampled and fused with F3, and then F1 is downsampled and fused with it. Finally, the final E is obtained through CBR operations, i.e. convolution, batch normalization and activation. * Edge information. The whole process is described as follows:
[0041]
[0042] Where f represents convolution, batch normalization and activation operations, up(.) represents upsampling operations, and down(.) represents downsampling operations. In order to guide the extraction and accuracy of edge information, a supervision is added after the EEM module, and the loss is le.
[0043] Step (5). Information fusion module, the specific method is as follows:
[0044] In order to better integrate the edge information obtained above and the processed high-level semantic information into the decoding layer, the model proposed in this invention designs an EFM module. “×” represents the element-wise multiplication operation, F * i-1 The specific operations obtained by fusion of upper and lower level features are as follows:
[0045]
[0046] Where conv represents the convolution operation, the convolution kernel size is 1×1, and [.,.] represents the concatenated feature fusion operation. When i is equal to 5, the operation is as follows:
[0047]
[0048] For the five encoding modules, five corresponding decoding modules Decoder-B1, Decoder-B2, Decoder-B3, Decoder-B4, and Decoder-B5 are constructed respectively. The input of Decoder-B5 is F * 5, the output is D5; D5 and F * 4 Input information fusion module EFM to get the corresponding output D * 5. Then D *5 and F4 are input into Decoder-B4 to get D4, and D4 and F * 3 After passing through the EFM module, D is obtained * 4. The implementation process of Decoder-B3, Decoder-B2, and Decoder-B1 is the same as that of Decoder-B4. The output D1 of Decoder-B1 is the target detection result.
[0049] Finally, we get {D * 2, D * 3, D * 4.D * 5}, the entire process of the EFM module is described as follows:
[0050]
[0051] Among them, u / d(.) indicates that downsampling or upsampling is selected according to different levels to keep the height and width of the feature map consistent. Figure 1 In {D * 2, D * 3, D * 4.D * 5} both add a supervision and calculate the loss to improve the performance of the model, so {D * 2, D * 3, D * 4.D * 5} all need to pass through the convolution kernel of size 1×1 and the upsampling operation, and finally use the sigmoid activation function to obtain the saliency map, and calculate the comparison loss. The final network output is D1
[0052] Step (5). Result error analysis, the specific method is as follows:
[0053] exist Figure 3 The result diagram of the method of the present invention can be seen in the figure, and it can be seen that the effect is good. In order to quantitatively compare with other significant target detection models on the data set EORSSD, the following evaluation indicators are used, including maxF-measure (maxF), S-measure (S), max E-measure (maxE) and MAE. Among them, the above indicators of the present invention are: 0.8318, 0.9022, 0.9456 and 0.0125, respectively. It also proves the feasibility of the present invention.
Claims
1. A method for detecting salient objects in remote sensing images based on deep learning, characterized in that: The following steps are involved: Step 1. Obtain optical remote sensing images and build a data set; Step 2. In the encoding phase, the data in the dataset is extracted with multi-level features by building a backbone network; Step 3. Construct a high-level semantic information processing module GFM through dilated convolution to obtain semantic features based on multi-level features; Step 4. Construct an edge extraction module EEM to extract edge features based on multi-level features; Step 5. In the decoding stage, decoding is performed based on multi-level features, and an information fusion module EFM is constructed to combine semantic features and edge features for feature fusion, and output the target detection results.
2. The method for detecting salient objects in remote sensing images based on deep learning according to claim 1, characterized in that: The specific implementation process of the backbone network is as follows: The backbone network consists of five encoding modules, Encoder-1, Encoder-2, Encoder-3, Encoder-4, and Encoder-5, which are connected in sequence and output five levels of features, F1, F2, F3, F4, and F5, respectively; The Encoder-1 includes Conv1 and Conv2_x of ResNet34, and adds a maximum pooling layer after Conv1; the subsequent Encoder-2, Encoder-3 and Encoder-4 respectively use Conv3_x, Conv4_x and Conv5_x in ResNet34; finally, Encoder-5 contains a maximum pooling layer and three residual modules.
3. The method for detecting salient targets in remote sensing images based on deep learning according to claim 2, characterized in that: The specific implementation process of the advanced semantic information processing module GFM is as follows: The GFM adopts dilated convolution as the main body. There are five branches in the GFM module structure, including four convolution kernels with different expansion rates to process F5 respectively, and then use a convolution operation to retain the original information. The results obtained from the five branches are fused by concatenation operation, and the fused feature map is added element by element with F5 through a convolution operation to perform feature fusion; finally, the output F is obtained through CBR unit, i.e., convolution, batch normalization and activation function activation operation. * 5, and adding loss supervision at the output of GFM improves the performance.
4. The method for detecting salient targets in remote sensing images based on deep learning according to claim 3, characterized in that: The specific implementation process of constructing the edge extraction module EEM is as follows: Use three levels of features for feature fusion, and transform F * 5 After upsampling and concatenating features with F3, it is then concatenated with F1 after downsampling, and finally the edge feature E is obtained through the CBR unit. * ; and add a supervision loss after the EEM module.
5. The method for detecting salient targets in remote sensing images based on deep learning according to claim 4, characterized in that: The specific implementation process of step 5 is as follows: F is obtained by fusing the upper and lower level features * i-1 The specific operations are as follows: Where conv represents the convolution operation, down(.) represents the downsampling operation, and [.,.] represents the concatenated feature fusion operation; when i is equal to 5, the operation is as follows: Construct five corresponding decoding modules Decoder-B1, Decoder-B2, Decoder-B3, Decoder-B4, and Decoder-B5 for the five encoding modules respectively; The input of Decoder-B5 is F * 5, the output is D5; D5 and F * 4 Input information fusion module EFM to get the corresponding output D * 5. Then D * 5 and F4 are input into Decoder-B4 to get D4, and D4 and F * 3 After passing through the EFM module, D is obtained * 4. The implementation process of Decoder-B3, Decoder-B2, and Decoder-B1 is the same as that of Decoder-B4. The output D1 of Decoder-B1 is the target detection result.
6. The method for detecting salient targets in remote sensing images based on deep learning according to claim 5, characterized in that: The information fusion module EFM is specifically implemented as follows: Among them, × represents the element-wise multiplication operation, and u / d(.) means choosing downsampling or upsampling according to different levels to keep the height and width of the feature map consistent; at the same time, the contrast loss is calculated after EFM.
Citation Information
Patent Citations
Optical remote sensing image saliency target detection method based on attention edge interaction
CN116129289A
Multi-scale remote sensing image target detection method based on enhanced small target feature extraction
CN117809200A
Multi-modal saliency target detection method and system based on lightweight three-branch coding and decoding network
CN118247486A
Boundary-optimized remote sensing image semantic segmentation method and apparatus, and device and medium
WO2023077816A1
