An enhanced-receptive-field-based optical remote sensing image salient target detection method
By introducing a bifurcation jump connection module and a distributed ensemble module into optical remote sensing image detection, the receptive field is enhanced and deep supervised training is performed, thus solving the accuracy and efficiency problems of optical remote sensing image detection and achieving more efficient target detection.
Patent Information
- Application Number
- CN202310154291.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-16
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-02-16
AI Technical Summary
Existing technologies struggle to effectively improve the accuracy and efficiency of salient target detection in optical remote sensing images, especially in images with cluttered backgrounds. Existing methods neglect channel attention cues and the influence of high-level semantics on low-level features during attention transmission, and also suffer from high computational complexity.
An encoder-decoder architecture employing a bifurcated jump connection module and a distributed ensemble module is used. The receptive field is enhanced by expanding convolutions, and the features are refined by combining attention-guided cascade structures. The model is trained through deep supervision using the distributed ensemble module.
It significantly improves the accuracy and efficiency of salient target detection in optical remote sensing images, reduces computational complexity, and is adaptable to detection objects of different sizes and shapes.
Smart Images
Figure CN116052005B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image processing, and particularly relates to a significant target detection method for optical remote sensing images based on enhanced receptive field. BACKGROUND
[0002] Visual attention mechanism aims to capture the most attractive regions in a scene, which plays an important role in human visual system. In the field of computer vision, people have been committed to modeling this mechanism, which can be generally divided into two important themes: fixation prediction and salient object detection. The former predicts the visual saliency of regions, while the latter highlights the salient regions of objects. The salient object detection has many successful applications in the field of computer vision, such as object segmentation, image quality assessment, image reorientation, etc.
[0003] Unlike hyperspectral images containing more spectral band information, optical remote sensing images refer to color images taken by satellites and aerial sensors in the range of 400-760 nm, only three optical bands (RGB). The salient object detection of optical remote sensing images aims to highlight the planes, islands, ships, buildings and rivers attracting human attention at the pixel level of optical remote sensing images. In the field of computer vision, great progress has been made in salient object detection of natural scene images, and many well-performing methods have emerged, including GateNet, MINet, SUCA and PA-KRN. In contrast, salient object detection of optical remote sensing images is still a challenging emerging topic. Since the scale, illumination and structure of salient object detection of optical remote sensing images are significantly different from those of salient object detection of natural scene images, it is difficult to directly migrate the method from salient object detection of natural scene images to salient object detection of optical remote sensing images. Therefore, how to effectively improve the accuracy and efficiency of scene analysis and target detection of background cluttered optical remote sensing images needs to be further explored.
[0004] Recently, some methods related to salient object detection of optical remote sensing images have been proposed, such as: end-to-end dense attention flow network DAFNet fuses low-level attention cues to deeper levels to guide the generation of high-level attention maps. However, in the process of attention transmission, the influence of channel attention cues and high-level semantics on low-level features is usually ignored, and the refinement of image edges is also lacking. Multi-content complementary network MCCNet combines foreground, edge and background features, as well as global image-level content, to explore the complementarity of multiple semantics in salient object detection of optical remote sensing images. Although this method has greatly improved the accuracy, the computational complexity is quite high, and the number of parameters is inevitably large. SUMMARY
[0005] The present application overcomes the above-mentioned shortcomings of the prior art and provides a significant target detection method for optical remote sensing images based on an enhanced receptive field, which adds a bifurcated skip connection module and a dispersed integration module on the architecture of an encoder-decoder. First, the bifurcated skip connection module uses dilated convolution to expand the receptive field and captures multi-scale context information through bifurcated skip connection. Second, the dispersed integration module seeks finer features by refining features in a coarse-to-fine manner through an attention-guided cascade structure. Finally, the model training is supervised in a deep supervision manner to achieve good detection effect.
[0006] To achieve the above object, the technical scheme adopted by the present application is as follows:
[0007] A significant target detection method for optical remote sensing images based on an enhanced receptive field, comprising:
[0008] extracting five scale encoding features by using Res2Net for feature extraction of the optical remote sensing image to be predicted;
[0009] inputting the extracted encoding features into the bifurcated skip connection module respectively to output features with enhanced receptive field;
[0010] inputting the features with enhanced receptive field into the dispersed integration module respectively to further enrich the texture information and output refined features;
[0011] inputting the refined features into the decoder respectively to output the final prediction result.
[0012] Further, the bifurcated skip connection module performs the following operations:
[0013] compressing the channel number of the encoding features through a 1x1 convolution layer;
[0014] inputting the features after channel compression into three consecutive 3x3 convolution layers to extract the first layer feature f t,1 , the second layer feature f t,2 and the third layer feature f t,3 ;
[0015] performing dilated convolution operation with expansion rates of 1 and 2 on the outputs of the first layer and the second layer in the three consecutive 3x3 convolution layers respectively to obtain features and
[0016] performing channel connection operation on the third layer feature f t,3 in the three consecutive 3x3 convolution layers and the feature after dilated convolution operation to obtain features with rich context information
[0017] capture spatial attention information of the feature
[0018] perform element-wise addition operation on the feature and its spatial attention information to reduce redundant information and obtain a receptive field enhanced feature
[0019] Further, the dispersion integration module performs the following operations:
[0020] respectively perform downsampling operation on the receptive field enhanced feature by using max-pooling and 1x1 convolution layer to obtain three different features
[0021] capture channel attention information of the feature
[0022] capture spatial attention information of the channel attention information
[0023] perform upsampling operation on the spatial attention information and perform channel connection operation with the feature to obtain a feature
[0024] capture channel attention information of the feature capture spatial attention information of the channel attention information by using spatial global max-pooling and element-wise multiplication operation
[0025] perform upsampling operation on the spatial attention information and perform channel connection operation with the feature to obtain a feature
[0026] capture channel attention information of the feature capture spatial attention information of the channel attention information by using spatial global max-pooling and element-wise multiplication operation to obtain a refined feature
[0027] Further, the decoder performs the following operations:
[0028] the fifth scale corresponding to the refined feature input two layers of 3x3 convolution layers and a layer of 2x2 deconvolution layer, to obtain the feature
[0029] the fourth scale corresponding to the refined feature and the feature perform a channel connection operation, and input two layers of 3x3 convolution layers and a layer of 2x2 deconvolution layer, to obtain the feature
[0030] the third scale corresponding to the refined feature and the feature perform a channel connection operation, and input two layers of 3x3 convolution layers and a layer of 2x2 deconvolution layer, to obtain the feature
[0031] the second scale corresponding to the refined feature and the feature perform a channel connection operation, and input two layers of 3x3 convolution layers and a layer of 2x2 deconvolution layer, to obtain the feature
[0032] the first scale corresponding to the refined feature and the feature perform a channel connection operation, and input two layers of 3x3 convolution layers and a layer of 2x2 deconvolution layer, to obtain the feature
[0033] the feature input a layer of 1x1 convolution layer to reduce the feature channel to 1, to obtain the detection result.
[0034] Further, the significant target detection method based on the enhanced receptive field of the optical remote sensing image further comprises:
[0035] the feature t∈{1,2,3,4,5} respectively input a layer of 1x1 convolution layer to reduce the feature channel to 1, to obtain the corresponding output feature P t ;
[0036] the output feature P t , t∈{1,2,3,4,5} and the actual label are used to calculate the loss, and the model is trained by using the calculated loss, and the calculation formula of the loss is as follows:
[0037]
[0038] wherein represents the BCE loss, IoU loss, F-m loss, G represents the actual label.
[0039] The application provides a significant target detection method for optical remote sensing images based on enhanced receptive field, which uses a bifurcated jump connection module to expand the receptive field, so that the model can better adapt to different sizes, shapes and quantities of the detected objects. The dispersion integration module contains cross-scale fusion and attention mechanism, which guides the learning of features through a three-layer cascade structure, so that the output features obtain more rich context information and texture information. Compared with other methods, the method provided by the application significantly improves the accuracy of significant target detection. BRIEF DESCRIPTION OF DRAWINGS
[0040] Figure 1 A flowchart of the significant target detection method for optical remote sensing images based on enhanced receptive field of the application is shown in the figure.
[0041] Figure 2 A schematic diagram of the detection model of the application is shown in the figure.
[0042] Figure 3 A schematic diagram of the bifurcated jump connection module of the embodiment of the application is shown in the figure.
[0043] Figure 4 A schematic diagram of the dispersion integration module of the embodiment of the application is shown in the figure.
[0044] Figure 5 A schematic diagram of the decoder of the embodiment of the application is shown in the figure. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.
[0046] The significant target detection method for optical remote sensing images based on enhanced receptive field of the application uses a detection model to detect the significant targets of optical remote sensing images by constructing and training the detection model, fully utilizes the advantages of the method of receptive field enhancement and multi-size information interaction and fusion, and realizes accurate significant target detection of optical remote sensing images.
[0047] Specifically, as shown in the figure, Figure 1 The significant target detection method for optical remote sensing images based on enhanced receptive field provided by the embodiment includes:
[0048] Step S1, using Res2Net to extract features from the optical remote sensing image to be predicted, and extracting five scales of encoded features.
[0049] The detection model constructed by the application is shown in the figure. Figure 2As shown, it includes an encoder (Res2Net), a bifurcated jump connection module, a scattered integration module, and a decoder.
[0050] After the optical remote sensing image is collected, the optical remote sensing image of 256x256x3 is obtained by unified cropping. In order to extract as many low-level features and high-level features of the optical remote sensing image as possible, the optical remote sensing image to be predicted of 256x256x3 is input into the encoder of the model. In this embodiment, the Res2Net is used to extract image features.
[0051] The encoder extracts five scale coding features of the optical remote sensing image, and the feature sizes are 128x128x64, 64x64x256, 32x32x512, 16x16x1024, and 8x8x2048.
[0052] In this embodiment, Res2Net is a relatively mature technology in the field, which will not be described here.
[0053] In step S2, the extracted coding features are input into the bifurcated jump connection module, and the receptive field enhanced features are output.
[0054] In order to better obtain the context information of the features, in this embodiment, the five scale coding features extracted by the encoder are input into the bifurcated jump connection module. As shown in Figure 2 In this embodiment, the five scale coding features extracted in step S1 are input into the bifurcated jump connection module. On the basis of maintaining the original resolution, the channel number of the features is reduced, the calculation amount of the model is reduced, and the receptive field enhanced features are extracted through three layers of continuous 3x3 convolution, dilated convolution, channel connection operation and attention mechanism. As shown in Figure 3 The module can better adapt to the changes of the size, shape and number of the detected objects in the optical remote sensing image.
[0055] In this embodiment, the bifurcated jump connection module performs the following operations:
[0056] In S21, the channel number of the coding feature is compressed through one layer of 1x1 convolution.
[0057] The coding features extracted by Res2Net are Where t represents different scales, and the scales are 128x128x64, 64x64x256, 32x32x512, 16x16x1024, and 8x8x2048, respectively, which are represented as and Input bifurcation jump connection module. First, a 1×1 convolution layer is used to compress the number of feature channels to reduce the computational cost of the model. The feature scales after channel compression are 128×128×32, 64×64×64, 32×32×128, 16×16×256, and 8×8×512.
[0058] S22. Input the compressed channel features into three consecutive 3×3 convolutional layers to extract the first layer features f. t,1 Second layer features f t,2 and the third layer feature f t,3 .
[0059] S23. Perform dilation convolution operations with dilation rates of 1 and 2 on the outputs of the first and second layers of the three consecutive 3×3 convolutional layers to obtain the features. and
[0060] In this embodiment, the compressed channel features are input into three consecutive 3×3 convolutional layers to extract features f. t,i (i∈{1,2,3}). Then, the outputs of the first and second layers of the three consecutive 3×3 convolutional layers are subjected to dilation convolution operations with dilation rates of 1 and 2, respectively, as calculated in the following formulas:
[0061]
[0062] Among them, DConv 3×3 (*,*) represents the dilated convolution operation, where the kernel size is 3×3 and r i It is the dilation rate of the dilated convolution. For example... Figure 2 As shown, in a specific embodiment, the expansion rate corresponding to the first layer features is 1, and the expansion rate corresponding to the second layer features is 2.
[0063] S24. The feature f from the third layer of a three-layer continuous 3×3 convolutional layer... t,3 Features after dilated convolution operation Perform channel connection operations to obtain features with rich contextual information.
[0064] The calculation formula is as follows:
[0065]
[0066] Among them, Conv 3×3 (*) is a convolution operation with a 3×3 kernel, and Concat(*) represents a channel concatenation operation.
[0067] S25. Capturing features through spatial global max pooling and element-wise multiplication operations. Spatial attention information
[0068] The calculation formula is as follows:
[0069]
[0070] wherein, Conv 1×1 (*) is a convolution operation with a 1x1 convolution kernel, GMP s (*) represents a spatial global maximum pooling operation, represents an element-level multiplication operation.
[0071] S26, the feature and its spatial attention information perform an element-level addition operation, reduce redundant information, and obtain a receptive field enhanced feature
[0072] The calculation formula is as follows:
[0073]
[0074] wherein, represents an element-level addition operation.
[0075] In this example, the branch jump connection module makes the feature have rich context information, but the feature still contains redundant information.
[0076] Step S3, input the receptive field enhanced feature into the dispersion integration module respectively, further enrich the texture information, and output the refined feature.
[0077] In order to obtain more rich texture information, in this embodiment, the feature processed by the branch jump connection module is input into the dispersion integration module, as shown in Figure 2 .
[0078] In this embodiment, the feature processed in step S3 is input into the dispersion integration module respectively. This module can capture both spatial clues of low-level features and semantic clues of high-level features, extract features with more rich detail information, and the dispersion integration module is as shown in Figure 4 .
[0079] S31, respectively using maximum pooling and 1x1 convolution layer to perform down-sampling operation on the receptive field enhanced feature , three different features
[0080] The feature processed in step S3 of this embodiment has a size of H t x W t x C t , which is input into the dispersion integration module.
[0081] In the dispersed collection module, first, the maximum pooling and 1x1 convolution layer are respectively used to perform down-sampling operation on the feature to obtain three different features The feature sizes are Figure 4 The maximum pooling operation is used in the middle down-sampling process.
[0082] S32, capture the channel attention information of the feature by channel global maximum pooling and channel-level multiplication operation
[0083] The calculation formula is as follows:
[0084]
[0085] Where, GMP c (*) represents the channel global maximum pooling operation, and represents the channel-level multiplication operation.
[0086] S33, capture the spatial attention information of the channel attention information by spatial global maximum pooling and element-level multiplication operation
[0087] The calculation formula is as follows:
[0088]
[0089] Where, Conv 1×1 (*) is a convolution operation with a 1x1 convolution kernel, GMP s (*) represents the spatial global maximum pooling operation, and represents the element-level multiplication operation.
[0090] S34, up-sample the spatial attention information and perform channel connection operation with the feature to obtain the feature
[0091] This step up-samples the spatial attention information to enlarge the feature size to and performs channel connection operation with The calculation formula is as follows:
[0092]
[0093] Where, up(*) indicates up-sampling operation.
[0094] S35, capture the feature channel attention information and capture channel attention information by using spatial global max pooling and element-wise multiplication operation spatial attention information
[0095] The calculation formula is as follows:
[0096]
[0097]
[0098] S36, the spatial attention information is up-sampled, and is connected with the feature by channel connection operation, to obtain the feature .
[0099] This step up-samples the spatial attention information , enlarges the feature size to , and is connected with by channel connection operation, to obtain the feature The calculation formula is as follows:
[0100]
[0101] S37, capture the channel attention information of the feature by using channel global max pooling and channel-wise multiplication operation and capture the spatial attention information of the feature by using spatial global max pooling and element-wise multiplication operation, to obtain the refined feature
[0102] The calculation formula is as follows:
[0103]
[0104]
[0105] For the five scale encoding features, after the operation of the above steps, the corresponding refined features are respectively represented as and
[0106] Step S4, input the refined features into the decoder respectively, and output the final detection result.
[0107] The decoder in this embodiment includes five decoder units, and each decoder unit is as follows: Figure 5As shown, including two layers of continuous 3x3 convolution layer and a layer of 2x2 deconvolution layer. Since the model of the present application captures multi-scale information, the decoder infers the salient object by gradually restoring the resolution, performs the following operations:
[0108] S41, the fifth scale corresponding to the refined feature Input two layers of continuous 3x3 convolution layer and a layer of 2x2 deconvolution layer to obtain the feature
[0109] This step will refine the five scales corresponding to the feature Input two layers of continuous 3x3 convolution layer and a layer of 2x2 deconvolution layer to obtain the feature The size is 16x16x512.
[0110] S42, the fourth scale corresponding to the refined feature The feature Channel connection operation is performed, and two layers of continuous 3x3 convolution layer and a layer of 2x2 deconvolution layer are input to obtain the feature The size is 32x32x256.
[0111] S43, the third scale corresponding to the refined feature The feature Channel connection operation is performed, and two layers of continuous 3x3 convolution layer and a layer of 2x2 deconvolution layer are input to obtain the feature The size is 64x64x128.
[0112] S44, the second scale corresponding to the refined feature The feature Channel connection operation is performed, and two layers of continuous 3x3 convolution layer and a layer of 2x2 deconvolution layer are input to obtain the feature The size is 128x128x64.
[0113] S45, the first scale corresponding to the refined feature The feature Channel connection operation is performed, and two layers of continuous 3x3 convolution layer and a layer of 2x2 deconvolution layer are input to obtain the feature The size is 256x256x32.
[0114] S46, the feature Input a layer of continuous 1x1 convolution layer to reduce the feature channel to 1 to obtain the detection result.
[0115] The above steps describe the detection process of the salient object detection method based on enhanced receptive field optical remote sensing image of the present application. In training the model of the present applicationFigure 2 As shown in the model, the decoder calculates the loss of each layer output feature and the ground truth label by deep supervision, and uses the loss to supervise the model training, including:
[0116] The feature t∈{1,2,3,4,5} is input into a 1x1 convolution layer to reduce the feature channel to 1, and the size is restored to 256x256x1 to obtain the corresponding output feature P t The output feature P t with a size of 256x256x1 is calculated with the actual label to calculate the loss, and the calculated loss is used to supervise the model training, and the calculation formula of the loss is as follows:
[0117]
[0118] Wherein represents the BCE loss, represents the IoU loss, represents the F-m loss, and G represents the actual label.
[0119] The present embodiment designs a bifurcated jump connection module and a scattered integration module. The bifurcated jump connection module mainly uses dilated convolution to enhance the receptive field of the feature, so that the model can more freely adapt to different sizes, shapes and quantities of the detected objects. The scattered integration module includes cross-scale fusion and attention mechanism, which guides feature learning to obtain more rich context information and more detailed texture information.
[0120] The experiments on the EORSSD and ORSSD optical remote sensing image data sets show that the method provided in the present application has the following test results on the EORSSD data set:
[0121] S α , 0.9432, MAE, 0.0061,
[0122] The test results of each evaluation index on the ORSSD data set are as follows:
[0123] S α , 0.9445, MAE, 0.0084,
[0124] Compared with other methods, the method provided in the present application significantly improves the accuracy of significant target detection.
[0125] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.
Claims
1. A method for detecting salient objects in optical remote sensing images based on enhanced receptive field, characterized in that, The method comprises the following steps: Res2Net is used to extract features of the to-be-predicted optical remote sensing image, and five kinds of scale coding features are extracted; The extracted coding features are respectively input into a bifurcated jump connection module, and features with enhanced receptive fields are output; The features with enhanced receptive fields are respectively input into a dispersion integration module, and the texture information is further enriched, and refined features are output; The refined features are respectively input into a decoder, and a final prediction result is output; The bifurcated jump connection module performs the following operations: By one layer Convolution, compress the number of channels of encoded features; The features after compression channels are input into three continuous convolution layers In the convolution layers, first layer features , second layer features and third layer features are extracted The three continuous layers are expanded convolutional layers The outputs of the first layer and the second layer in the convolutional layer are respectively subjected to expansion convolutional operation with expansion rates of 1 and 2 to obtain features and ; Three layers of continuous Third layer features in the convolutional layer Features after the dilated convolution operation , Perform a channel connection operation to obtain features with rich context information ; Capturing spatial attention information of features by spatial global max pooling and element-wise multiplication operations ; characteristics with its spatial attention information performing element-wise addition operations, reducing redundant information, and obtaining a feature with enhanced receptive field ; The dispersion integration module performs the following operations: The field-of-view enhanced features are respectively subjected to max-pooling and a 1x1 convolution layer Down-sampling operations are performed to obtain three different features , , ; Capturing features through channel global max pooling and channel-wise multiplication operations Channel attention information of the features ; Capturing channel attention information using spatial global max pooling and element-wise multiplication operations spatial attention information ; spatial attention information up-sampling operation is performed, and the feature a channel connection operation is performed to obtain a feature ; Features are captured using channel-based global max pooling and channel-level multiplication operations. Channel Attention Information It employs spatial global max pooling and element-wise multiplication operations to capture channel attention information. Spatial attention information ; spatial attention information up-sampling operation is performed, and the feature a channel connection operation is performed to obtain a feature ; Channel global max pooling and channel-wise multiplication operations are employed to capture the channel attention information Spatial global max pooling and element-wise multiplication operations are employed to capture the spatial attention information . 2. The enhanced-receptive-field-based optical remote sensing image salient target detection method according to claim 1, characterized in that, The decoder performs the following operations: the fifth scale corresponding to the refined feature input two layers of successive convolutional layers and one layer of deconvolutional layers to obtain features ; corresponding to the fourth scale after refinement with the feature Perform a channel connection operation, and input two layers of continuous convolution layers and one layer of deconvolution layers to obtain a feature ; corresponding to the third scale after refinement with the feature Perform channel connection operation, and input two layers of continuous convolution layer and one layer of deconvolution layer, to obtain feature ; corresponding to the second scale after refinement with the feature Perform channel connection operation, and input two layers of continuous convolution layer and a layer of deconvolution layer, to obtain feature ; The first scale corresponds to the refined feature With the feature Perform a channel connection operation, and input two layers of continuous Convolution layer and a layer Deconvolution layer, to obtain the feature ; The features Input a continuous The convolutional layer reduces the feature channels to 1 to get the detection result.
3. The method of claim 2, wherein the method is based on an enhanced receptive field. The method further comprises the following steps: characteristics input a layer The convolutional layer reduces the feature channel to 1 to obtain the corresponding output feature ; The output features The loss is calculated with the actual label, and the detection model is supervised and trained with the calculated loss. The calculation formula of the loss is as follows: ； wherein denotes the BCE loss, denotes the IoU loss, denotes the F-m loss, G stands for the actual label.