A method for salient object detection in remote sensing images based on deep learning

Abstract

The present invention discloses a method for detecting significant targets in remote sensing images based on deep learning. The method first acquires an optical remote sensing image and constructs a [network-wide, unspecified, or 'semantic'] network. Secondly, in the encoding stage, the data set data is used to extract multi-level features by constructing a backbone network. Then, a high-level semantic information processing module (GFM) is constructed through dilated convolution, and semantic features are obtained based on multi-level features. An edge extraction module (EEM) is constructed to extract edge features. Finally, in the decoding stage, decoding is performed based on multi-level features, and an information fusion module (EFM) is constructed to perform feature fusion by combining semantic features and edge features, and output target detection results. The present invention can better utilize high-level semantic information and edge information to obtain accurate significant target detection results.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a method for detecting salient objects in remote sensing images based on deep learning. Background Art

[0002] Salient object detection is inspired by the human visual attention mechanism, which attempts to find the most attractive part of a photo or video. In recent years, salient object detection has received widespread attention because it has achieved good results in many fields, such as image / video segmentation, video summarization, image quality assessment, and image classification.

[0003] In recent years, various saliency models have been designed to process photos or videos of natural scenery. Notably, with the continuous development and improvement of deep learning technology, its performance in salient object detection models has become increasingly prominent. Unfortunately, there are relatively few models for salient object detection in remote sensing images. Remote sensing images are widely used. In agriculture, they can be used to monitor crop growth, helping to increase yields. In the military, they can be used for intelligence gathering and target identification. With the continuous advancement of technology, remote sensing is playing an increasingly important role in new fields and applications. Whether in environmental protection, resource management, disaster response, or scientific research, remote sensing images play a vital role.

[0004] Remote sensing images are mainly divided into aerial photographs and satellite images, which are usually obtained from high-altitude aircraft or satellites. Therefore, remote sensing images often present complex scenes, such as various object sizes, different orientations, cluttered backgrounds, and even the absence of salient objects. Moreover, natural scene images are usually captured with handheld cameras, so it is obviously inappropriate to directly apply salient object detection models based on edge segmentation in natural scene images to remote sensing images. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides a method for detecting salient targets in remote sensing images based on deep learning.

[0006] The method of the present invention comprises the following steps:

[0007] Step (1). Acquire optical remote sensing images and construct a data set.

[0008] Step (2). In the encoding stage, the data in the dataset is extracted by building a backbone network to extract multi-level features. The specific method is as follows:

[0009] In the encoder stage, the backbone network is mainly designed based on ResNet34 to extract multi-level features. The backbone network consists of five encoding modules connected in sequence: Encoder-1, Encoder-2, Encoder-3, Encoder-4, and Encoder-5, which output five levels of features F1, F2, F3, F4, and F5 respectively.

[0010] The Encoder-1 includes Conv1 and Conv2_x of ResNet34, and adds a maximum pooling layer after Conv1; the subsequent Encoder-2, Encoder-3 and Encoder-4 respectively use Conv3_x, Conv4_x and Conv5_x in ResNet34; finally, Encoder-5 contains a maximum pooling layer and three residual modules.

[0011] Through research, it can be concluded that shallow low-level features retain the spatial information of the reconstructed target boundary, while deep high-level features retain the semantic information of the target, and the semantic information contained in high-level features plays an important role in camouflaged target detection.

[0012] Step (3). Construct a high-level semantic information processing module GFM through dilated convolution and obtain semantic features based on multi-level features. The specific method is as follows:

[0013] The proposed model designs a GFM module, the main body of which is the use of dilated convolution. There are five branches in the GFM module structure, including the use of four different expansion rate convolution kernels to process F5 respectively, and then use a convolution operation to retain the original information. The results obtained by the five branches are fused by the concatenation operation, and the fused feature map is added element by element with F5 through a convolution operation to perform feature fusion; finally, the output F is obtained through the CBR unit, i.e., convolution, batch normalization and activation function activation operation. * 5, and adding loss supervision ls0 at the output of GFM improves the performance.

[0014] Step (4). Construct an edge extraction module EEM to extract edge features based on multi-level features. The specific method is as follows:

[0015] Edge extraction module EEM, F * 5 After upsampling and concatenating features with F3, it is then concatenated with F1 after downsampling, and finally the edge feature E is obtained through the CBR unit. * ; and add a supervision loss after the EEM module.

[0016] Step (5). In the decoding stage, decoding is performed based on multi-level features, and an information fusion module (EFM) is constructed to combine semantic features and edge features for feature fusion and output the target detection results. The specific method is as follows:

[0017] The model proposed in this invention designs the EFM module, F * i-1 The specific operations obtained by fusing upper and lower level features are as follows:

[0018]

[0019] Where conv represents the convolution operation, the convolution kernel size is 1×1, and [.,.] represents the concatenated feature fusion operation. When i is 5, the operation is as follows:

[0020]

[0021] For the five encoding modules, five corresponding decoding modules Decoder-B1, Decoder-B2, Decoder-B3, Decoder-B4, and Decoder-B5 are constructed. The input of Decoder-B5 is F * 5, the output is D5; D5 and F * 4 Input information fusion module EFM to get the corresponding output D * 5, then D * 5 and F4 are input into Decoder-B4 to get D4, and D4 and F * 3 After passing through the EFM module, D is obtained * 4. The implementation process of Decoder-B3, Decoder-B2, and Decoder-B1 is the same as that of Decoder-B4. The output D1 of Decoder-B1 is the target detection result.

[0022] Through the EFM module, we can get {D * 2, D * 3, D * 4, D * 5}, the entire process of the EFM module is described as follows:

[0023]

[0024] Where u / d(.) indicates that downsampling or upsampling is selected according to different levels to keep the height and width of the feature map consistent. * 2, D * 3, D * 4, D * 5} both add a supervision and calculate the loss to improve the performance of the model, so {D* 2, D * 3, D * 4, D * 5} requires convolution and upsampling operations, and finally a sigmoid activation function to obtain a saliency map, calculate the contrast loss, and finally obtain D1, which is the final output of the network.

[0025] The beneficial effects of the present invention are as follows:

[0026] The main advantages of the method of the present invention are as follows: Specifically, the model designs an encoder based on ResNet-34 to extract features at all levels, and designs two modules to better utilize high-level semantic information and edge information. The high-level semantic information is processed by the method of void convolution that can increase the receptive field, and edge supervision is added to more efficiently utilize edge information. In the decoding stage, a module is designed to act on each layer to fuse edge information and upper and lower-level feature fusion information. The inputs of the modules designed in the decoding stage are the output of the previous level, edge information, and upper and lower-level feature fusion information, so as to obtain the final accurate salient target prediction map. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A framework diagram of the method of the present invention;

[0028] Figure 2 This is a schematic diagram of GFM implementation;

[0029] Figure 3 This is a result diagram of the method of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below with reference to the accompanying drawings.

[0031] like Figure 1 As shown, the present invention provides a method for detecting salient targets in remote sensing images based on deep learning as follows:

[0032] Step (1). Obtain the dataset. The specific method is as follows:

[0033] In order to thoroughly verify the feasibility of the model, a relatively basic dataset in optical remote sensing images, the Extended Optical Remote Sensing Image Salient Object Detection (EORSSD) dataset, was used. Specifically, the EORSSD dataset contains a total of 2,000 images, of which 1,400 images are used to train the model and 600 images are used for testing. It is worth mentioning that this dataset is an extension of the ORSSD dataset. This new dataset is larger and more diverse than the original dataset. It contains 2,000 images and corresponding pixel-level ground truth, which contains many semantically meaningful but challenging images.

[0034] Step (2). Feature extraction, the specific method is as follows:

[0035] In the encoder stage, the backbone network is mainly designed based on ResNet34 to extract multi-level features. However, unlike the traditional ResNet34 network, the present invention replaces the convolution kernel of the traditional ResNet34 network Conv1 with a size of 7×7, a number of channels of 64, and a step size of 2 with a convolution kernel of a size of 3×3, a number of channels of 64, and a step size of 1, and adds a maximum pooling layer after it. Therefore, the Encoder-1 in the present invention includes the changed Conv1 and the Conv2_x in the original ResNet34, and the subsequent Encoder-2, Encoder-3 and Encoder-4 respectively use Conv3_x, Conv4_x and Conv5_x in ResNet34. In addition, in order to expand the receptive field of the model of the present invention, an Encoder-5 is designed after Encoder-4, which includes a pooling convolution kernel of a size of 2×2 and a step size of 2, and three residual modules with a channel number of 512, such as Figure 1 , through these five modules we can get {F1, F2, F3, F4, F5}.

[0036] Step (3). Advanced semantic information processing module, the specific method is as follows:

[0037] In order to better utilize high-level semantic information, the proposed model of this invention designs a GFM module, such as Figure 2 The main body shown uses dilated convolution, which is also called expanded convolution. It inserts extra zero values ​​between the elements of the convolution kernel, thereby increasing the size of the convolution kernel receptive field without adding extra parameters or using pooling layers. In dilated convolution, a very important parameter is the dilation rate, which is defined as the spacing between values ​​when the convolution kernel processes data. In the GFM module structure, four different dilation rates are used to process F5 separately. At the same time, a 1×1 convolution kernel is introduced to retain the original information. The results obtained from the five branches are concatenated for feature fusion. At the same time, in order to better utilize high-level information, the "+" operation, i.e., the add operation, is introduced. Therefore, a convolution kernel of size 1×1 is required to transform the number of channels of the result after feature fusion. The fused feature map is subjected to a convolution operation and added element-by-element with F5 for feature fusion. Finally, the final F is obtained through CBR, i.e., convolution, batch normalization, and activation function activation operations. * 5. A loss supervision is added at the output of GFM to improve the model performance, so the obtained F *5. A saliency map is obtained by convolution with a 1×1 convolution kernel, upsampling, and sigmoid activation function for comparison and loss calculation.

[0038] Step (4). Edge extraction module, the specific method is as follows:

[0039] For the detailed processing of edge information, an edge extraction module, namely the EEM module, is designed. Here, three levels of features are used for feature fusion, where F * 5 is the output of the module GFM, which is then upsampled and fused with F3. F1 is then downsampled and fused with it. Finally, the final E is obtained through CBR operations, namely convolution, batch normalization and activation. * Edge information. The whole process is described as follows:

[0040]

[0041] Where f represents convolution, batch normalization, and activation operations, up(.) represents upsampling, and down(.) represents downsampling. To guide the extraction and accuracy of edge information, a supervision is added after the EEM module, and the resulting loss is le.

[0042] Step (5). Information fusion module, the specific method is as follows:

[0043] In order to better integrate the edge information obtained above and the processed high-level semantic information into the decoding layer, the model proposed in this invention designs an EFM module. “×” represents the element-wise multiplication operation, F * i-1 The specific operations obtained by fusing upper and lower level features are as follows:

[0044]

[0045] Where conv represents the convolution operation, the convolution kernel size is 1×1, and [.,.] represents the concatenated feature fusion operation. When i is 5, the operation is as follows:

[0046]

[0047] For the five encoding modules, five corresponding decoding modules Decoder-B1, Decoder-B2, Decoder-B3, Decoder-B4, and Decoder-B5 are constructed. The input of Decoder-B5 is F * 5, the output is D5; D5 and F * 4 Input information fusion module EFM to get the corresponding output D * 5, then D *5 and F4 are input into Decoder-B4 to get D4, and D4 and F * 3 After passing through the EFM module, D is obtained * 4. The implementation process of Decoder-B3, Decoder-B2, and Decoder-B1 is the same as that of Decoder-B4. The output D1 of Decoder-B1 is the target detection result.

[0048] Finally, we get {D * 2, D * 3, D * 4, D * 5}, the entire process of the EFM module is described as follows:

[0049]

[0050] Among them, u / d(.) indicates that downsampling or upsampling is selected according to different levels to keep the height and width of the feature map consistent. Figure 1 In {D * 2, D * 3, D * 4, D * 5} both add a supervision and calculate the loss to improve the performance of the model, so {D * 2, D * 3, D * 4, D * 5} need to pass the convolution kernel of size 1×1 and upsampling operation, and finally pass the sigmoid activation function to obtain the saliency map and calculate the comparison loss. The final network output is D1

[0051] Step (5). Result error analysis, the specific method is as follows:

[0052] exist Figure 3 The results of the proposed method are shown in Figure 2, which demonstrates good performance. To quantitatively compare the proposed method with other salient object detection models on the EORSSD dataset, we used several evaluation metrics, including maxF-measure (maxF), S-measure (S), maxE-measure (maxE), and MAE. The values ​​of the proposed method are 0.8318, 0.9022, 0.9456, and 0.0125, respectively. This demonstrates the feasibility of the proposed method.

Claims

1. A method for detecting salient objects in remote sensing images based on deep learning, characterized in that: The following steps are involved: Step 1. Acquire optical remote sensing images and construct a dataset; Step 2. In the encoding phase, the data in the dataset is extracted with multi-level features by constructing a backbone network. The specific implementation process of the backbone network is as follows: The backbone network consists of five encoding modules, Encoder-1, Encoder-2, Encoder-3, Encoder-4, and Encoder-5, which are connected in sequence and output five-level features, F1, F2, F3, F4, and F5, respectively. The Encoder-1 includes Conv1 and Conv2_x of ResNet34, and adds a maximum pooling layer after Conv1; the subsequent Encoder-2, Encoder-3 and Encoder-4 respectively use Conv3_x, Conv4_x and Conv5_x in ResNet34; finally, Encoder-5 contains a maximum pooling layer and three residual modules; Step 3. Construct a high-level semantic information processing module (GFM) through dilated convolution to obtain semantic features based on multi-level features. The specific implementation process of the advanced semantic information processing module GFM is as follows: The GFM adopts dilated convolution as the main body. There are five branches in the GFM module structure, including four convolution kernels with different expansion rates to process F5 respectively, and then use a convolution operation to retain the original information. The results obtained from the five branches are fused by concatenation operation, and the fused feature map is added element by element with F5 through a convolution operation to perform feature fusion. Finally, the output F is obtained through CBR unit, i.e., convolution, batch normalization and activation function activation operation. * 5. Add loss supervision at the output of GFM to improve performance; Step 4. Construct an edge extraction module EEM to extract edge features based on multi-level features; Step 5. In the decoding stage, decoding is performed based on multi-level features, and an information fusion module (EFM) is constructed to fuse features by combining semantic features and edge features, and output target detection results.

2. The method for detecting salient objects in remote sensing images based on deep learning according to claim 1, characterized in that: The specific implementation process of constructing the edge extraction module EEM is as follows: Use three levels of features to perform feature fusion, and transform F * 5 After upsampling and concatenating features with F3, it is then concatenated with F1 after downsampling, and finally the edge feature E is obtained through the CBR unit. * ; and add a supervision loss after the EEM module.

3. The method for detecting salient objects in remote sensing images based on deep learning according to claim 2, characterized in that: The specific implementation process of step 5 is as follows: F is obtained by fusing upper and lower level features * i-1 The specific operations are as follows: Where conv represents the convolution operation, down(.) represents the downsampling operation, and [.,.] represents the concatenated feature fusion operation. When i is equal to 5, the operation is as follows: Construct five corresponding decoding modules Decoder-B1, Decoder-B2, Decoder-B3, Decoder-B4, and Decoder-B5 for the five encoding modules respectively; The input of Decoder-B5 is F * 5, the output is D5; D5 and F * 4 Input information fusion module EFM to get the corresponding output D * 5, then D * 5 and F4 are input into Decoder-B4 to get D4, and D4 and F * 3 After passing through the EFM module, D is obtained * 4. The implementation process of Decoder-B3, Decoder-B2, and Decoder-B1 is the same as that of Decoder-B4. The output D1 of Decoder-B1 is the target detection result.

4. The method for detecting salient objects in remote sensing images based on deep learning according to claim 3, wherein: The information fusion module EFM is specifically implemented as follows: Among them, × represents the element-wise multiplication operation, and u / d(.) indicates the selection of downsampling or upsampling according to different levels to keep the height and width of the feature map consistent; at the same time, the contrast loss is calculated after EFM.

Images