Optical remote sensing image saliency target detection model and detection method

Abstract

The invention discloses an optical remote sensing image saliency target detection model and detection method, and belongs to the technical field of target detection. The saliency target detection model comprises a PVT backbone network module, an edge information extraction and fusion module BG, a feature enhancement module HAM, two space attention modules, a feature interaction module KTM and a feature fusion module. According to the invention, by introducing an edge detection module (BG) and a feature enhancement module (HAM), the target detection precision under a complex background is effectively improved. Compared with the existing CNN-based method, the method provided by the invention can better process targets with different scales and direction changes, reduces the probability of missing detection and false detection, and enables saliency target detection to be more complete and accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and more specifically, relates to a salient target detection model and a detection method for an optical remote sensing image. Background Art

[0002] Visual object detection (SOD) is mainly used to identify and locate the most prominent object area in an image. In recent years, it has been widely used in agriculture, forestry, and environmental science. With the advancement of satellite and aerial photography technology, the resolution of optical remote sensing images has been continuously improved, which has promoted the development of optical remote sensing salient object detection (ORSI-SOD) technology.

[0003] Among the existing optical remote sensing salient object detection methods, convolutional neural networks (CNNs) are widely used in mainstream models such as VGG and ResNet due to their powerful feature extraction capabilities, and are usually used as the backbone network of these methods. CNN can effectively capture the basic features of optical remote sensing images by extracting local features of images in layers.

[0004] However, although CNN-based salient object detection methods have made some progress in practical applications, they still have obvious limitations. For example, CNN can only extract features within the local receptive field and lacks the ability to model global information, which makes it prone to problems such as direction insensitivity, incomplete salient object detection, and missed detection when facing optical remote sensing images with wide scenes, complex backgrounds, and noise. In addition, since objects in optical remote sensing images have diversity in scale and direction, CNN performs poorly in dealing with these changes and is difficult to fully capture salient objects in the image. Summary of the invention

[0005] The purpose of the present invention is to provide a salient target detection model and detection method for optical remote sensing images, so as to effectively solve the existing significant target detection methods for optical remote sensing images, which have great limitations in practical applications, especially the existing methods based on convolutional neural networks (CNNs) are prone to defects such as direction insensitivity, missed detection, and incomplete significant target detection when dealing with complex backgrounds and noise. At the same time, the present invention can also effectively capture the global information in the optical remote sensing image, and has good robustness for detecting objects of different scales and directions, so that the detection accuracy can be effectively guaranteed.

[0006] In order to achieve the above object, the technical solution provided by the present invention is:

[0007] A first aspect of the present invention provides a salient object detection model for an optical remote sensing image, comprising:

[0008] PVT backbone network module, which is used to extract features from the input optical remote sensing image to be detected and obtain four-level global basic features The obtained four-level global basic features are channel normalized to obtain the normalized four-level global basic features. Among them, i∈{1, 2, 3, 4} is the corresponding level, C is the number of channels, and h×w is the spatial dimension of the corresponding feature map;

[0009] Edge information extraction and fusion module BG, which is used to extract the edge information from the normalized underlying features f t 1 and the top-level feature f t 4 Extract the corresponding edge information, and fuse the respective edge information into the respective feature maps to obtain the feature maps embedded with the edge information;

[0010] Feature enhancement module HAM, the feature enhancement module HAM is used to process the two intermediate features f after normalization t 2 and f t 3 , extracting features of different scales through multi-scale convolution branches, and combining the channel attention mechanism CAM to enhance important channel features, the residual connection retains the original features and further enhances them, thereby improving the model's ability to capture multi-scale and channel features, and enhancing feature expression and semantic information;

[0011] Two spatial attention modules are used to further extract features from the bottom-level feature maps and the top-level feature maps after embedding edge information;

[0012] Feature interaction module KTM, the feature interaction module KTM is used to process the intermediate layer features f after being processed by the feature enhancement module t 2 and f t 3 to integrate;

[0013] The feature fusion module is used to fuse the feature maps output by the two spatial attention modules and the feature interaction module to finally generate a saliency image.

[0014] In view of the problems and shortcomings of the existing optical remote sensing image salient target detection methods in practical applications, on the one hand, the present invention introduces an edge information extraction and fusion module, which can help the model better capture the edge structure of the target by enhancing the edge details of the salient target, and solve the problem of low accuracy of edge feature extraction of optical remote sensing images under complex backgrounds. It is particularly suitable for scenes with blurred edges of salient targets or complex backgrounds.

[0015] The spatial attention module focuses on spatial weighted feature extraction, mainly locating the features of salient areas based on the spatial distribution of the image. After embedding edge information, the spatial attention module can more accurately focus on key edges and detail areas. This processing method not only effectively highlights and enhances edge features, but also significantly improves the accuracy of the model in detecting salient targets in complex backgrounds.

[0016] On the other hand, the present invention first uses the feature enhancement module HAM to normalize the two intermediate features f t 2 and f t 3 The HAM module can enhance the input features through multi-scale convolution and channel attention mechanism, so that important features of different scales are amplified and noise is effectively suppressed, which helps to provide higher quality and more discriminative features before entering the KTM module, thereby improving the effect of subsequent contextual interaction.

[0017] In addition, the HAM module extracts feature information at different levels through its multi-scale convolution branches and retains the original features through residual connections, which ensures that when KTM performs contextual interaction, the transmitted feature information covers multiple scales from local details to global semantics, helping KTM to more effectively capture co-occurring saliency information. In other words, processing by the HAM module first and then by the KTM module can ensure that the features are fully enhanced before the contextual interaction, thereby improving the accuracy and robustness of the final detection.

[0018] Furthermore, the PVT backbone network module comprises four transformer encoder blocks and a convolutional layer, wherein the four transformer encoder blocks are used to extract features of the input optical remote sensing image to be detected to generate four-level global basic features. The convolution layer is used to perform channel normalization processing on the generated four-level global basic features; the size of the input optical remote sensing image to be detected is 3×352×352, and the number of channels of the generated four-level global basic features is C i ∈{64, 128, 320, 512}, the number of channels C of the four-level global basic features after normalization is unified to 32.

[0019] As a further improvement of any technical solution of the first aspect of the present invention, the processing process of the edge information extraction and fusion module includes: processing the bottom feature f of the four-level global basic features after normalization t 1 and the top-level feature f t 4Convolution operation is performed to extract the corresponding edge information; then the bottom-level features and the highest-level features are normalized respectively, and the respective edge information is fused into the respective feature maps through the modulation of the γ and β coefficients. The processing operation formula is as follows:

[0020]

[0021] In the above formula, BGM represents the feature map embedded with edge information, ASM i represents the normalized feature map output from the backbone network, CB represents 3×3 convolution and batch normalization; γ represents the scaling parameter, β represents the offset parameter, and BM represents the extracted edge information. represents element-wise multiplication, Stands for element-wise addition.

[0022] As a further improvement of any technical solution of the first aspect of the present invention, the multi-scale convolution branch is used to extract features of different scales, specifically using a ((2b-1)×(2b-1)) convolution layer and a 3×3 convolution layer for feature extraction respectively.

[0023] As a further improvement of any technical solution of the first aspect of the present invention, the feature enhancement module performs normalization on the two intermediate features f t 2 and f t 3 The processing process includes:

[0024] (1) Calculate the CAM of the original feature A, reshape A into QC×N, then perform matrix multiplication, multiply A by its transpose, and obtain the channel attention map P∈QC×C through softmax. The specific calculation formula is as follows:

[0025]

[0026] Among them: nm represents the influence of the mth channel on the nth channel, C is the number of channels of the feature map, Q is the coefficient, and N = h × w represents the flattened size of the spatial dimension of the input feature map;

[0027] (2) Multiply the transpose of P by A and reshape the result into Qh×w×C. Then, multiply the result by the parameter α and perform an element-wise addition operation with A to obtain the final output C∈Qh×w×C:

[0028]

[0029] In the above formula, α is a trainable parameter with an initial value of 0.

[0030] As a further improvement of any technical solution of the first aspect of the present invention, the spatial attention module includes D-SWSAM and SWSAM, wherein SWSAM adopts a weighted spatial attention module to extract the highest level features; D-SWSAM adopts a spatial weighted attention module with directional convolution to extract the underlying features.

[0031] As a further improvement of any technical solution of the first aspect of the present invention, the feature interaction module adopts a context interaction and knowledge transfer module KTM of cross-level features.

[0032] A second aspect of the present invention provides a method for detecting salient objects in an optical remote sensing image, comprising:

[0033] The PVT backbone network module is used to extract features from the input optical remote sensing image to be detected, and four-level global basic features are obtained. The obtained four-level global basic features are channel normalized to obtain the normalized four-level global basic features. Among them, i∈{1, 2, 3, 4} is the corresponding level, C is the number of channels, and h×w is the spatial dimension of the corresponding feature map;

[0034] Through edge information extraction and fusion module, the underlying features f t 1 and the top-level feature f t 4 Extract the corresponding edge information, and fuse the respective edge information into the respective feature maps to obtain the feature maps embedded with the edge information;

[0035] The feature enhancement module is used to enhance the two normalized intermediate features f t 2 and f t 3 Processing, extracting features of different scales through multi-scale convolution branches, and combining the channel attention mechanism CAM to enhance important channel features, residual connection retains the original features and further enhances them;

[0036] The two spatial attention modules are used to further extract features from the underlying feature map and the highest-level feature map after embedding edge information.

[0037] The feature interaction module is used to process the intermediate layer features f processed by the feature enhancement module. t 2 and f t 3 to integrate;

[0038] The feature fusion module is used to fuse the feature maps output by the two spatial attention modules and the feature interaction module, and finally a saliency image is generated.

[0039] As a further improvement of any technical solution of the second aspect of the present invention, the edge information extraction and fusion module extracts the bottom-level features f from the normalized t 1 and the top-level feature f t 4 The corresponding edge information is extracted from the four-level global basic features after normalization, and the respective edge information is fused into the respective feature maps to obtain the feature maps embedded with the edge information. t 1 and the top-level feature f t 4 Convolution operation is performed to extract the corresponding edge information; then the bottom-level features and the highest-level features are normalized respectively, and the respective edge information is fused into the respective feature maps through the modulation of the γ and β coefficients. The processing operation formula is as follows:

[0040]

[0041] In the above formula, BGM represents the feature map embedded with edge information, ASM i represents the normalized feature map output from the backbone network, CB represents 3×3 convolution and batch normalization; γ represents the scaling parameter, β represents the offset parameter, and BM represents the extracted edge information. represents element-wise multiplication, Stands for element-wise addition.

[0042] As a further improvement of any technical solution of the second aspect of the present invention, the multi-scale convolution branch is used to extract features of different scales, specifically using a ((2b-1)×(2b-1)) convolution layer and a 3×3 convolution layer for feature extraction respectively.

[0043] As a further improvement of any technical solution of the second aspect of the present invention, the feature enhancement module is used to enhance the two normalized intermediate features f t 2 and f t 3 Processing includes:

[0044] (1) Calculate the CAM of the original feature A, reshape A into QC×N, and then perform matrix multiplication to replace the reshaped matrix A with its transposed matrix A T Multiply them together to get the relationship matrix between channels P = A × A T, and obtain the channel attention map P∈QC×C through softmax. The specific calculation formula is as follows:

[0045]

[0046] Among them: nm represents the influence of the mth channel on the nth channel, and A represents the two intermediate features f after normalization. t 2 and f t 3 , C is the number of channels of the feature map, Q is the coefficient, and N = h × w represents the flattened size of the spatial dimension of the input feature map;

[0047] (2) Multiply the transpose of P by A and reshape the result into Qh×w×C. Then, multiply the result by the parameter α and perform an element-wise addition operation with A to obtain the final output C∈Qh×w×C:

[0048]

[0049] In the above formula, α is a trainable parameter with an initial value of 0.

[0050] As a further improvement of any technical solution of the second aspect of the present invention, the feature fusion module uses a part of the decoder as a saliency predictor, and the resolution of the input features of this part of the decoder is respectively f dswsam (32×11×11),f ktm (32×22×22), f swsam (32×88×88); then the low-resolution feature maps are gradually upsampled to make them consistent with the highest resolution (32×88×88); finally, the merged feature maps are convolved to generate a preliminary saliency map s∈[0,1] 1×88×88 ;

[0051] Then, a 4-fold upsampling operation is performed to restore the resolution of the saliency map to be consistent with the input optical remote sensing image, thereby finally obtaining the required saliency map s∈[0,1] 1×352×352 .

[0052] Compared with the prior art, the technical solution provided by the present invention can achieve the following beneficial effects:

[0053] (1) The present invention uses PVT as the backbone network and utilizes its powerful global modeling capabilities to reduce the redundant extraction of a large number of local features, so that the model can complete the calculation at a faster speed when detecting complex optical remote sensing images. Compared with the traditional CNN model, the detection efficiency of the present invention is significantly improved, the processing speed is faster, and it is suitable for the rapid processing of large-scale remote sensing data sets.

[0054] (2) The present invention effectively improves the accuracy of target detection in complex backgrounds by introducing edge detection module (BG) and channel attention module (HAM). Compared with the existing CNN-based methods, the present invention can better handle targets of different scales and directions, reduce the probability of missed detection and false detection, and make the detection of significant targets more complete and accurate.

[0055] (3) The present invention reduces the consumption of computing resources by unifying the number of channels and using multi-scale convolution branches to extract features of different scales. Under the same conditions, the model requires less storage and computing resources, thereby reducing hardware requirements and energy consumption. It is suitable for salient target detection tasks in optical remote sensing images in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of the process of the method for detecting salient objects in optical remote sensing images in an embodiment of the present application.

[0057] Figure 2 Schematic diagram of the process of dynamically adjusting edge information in an embodiment of the present application.

[0058] Figure 3 It is a flow chart of the feature enhancement HAM module in the embodiment of the present application.

[0059] Figure 4 This is an F-measure curve graph generated by the model of an embodiment of the present application on the ORSSD dataset.

[0060] Figure 5 This is the PR graph generated by the model of an embodiment of the present application on the ORSSD dataset.

[0061] Figure 6 This is an F-measure curve graph generated by the model of an embodiment of the present application on the EORSSD dataset.

[0062] Figure 7 This is the PR graph generated by the model of the embodiment of the present application on the EORSSD dataset.

[0063] Figure 8 This is an F-measure curve graph generated by the model of an embodiment of the present application on the ors-4199 dataset.

[0064] Fig. 9 This is a PR graph generated by the model of an embodiment of the present application on the ors-4199 dataset. DETAILED DESCRIPTION

[0065] The embodiment of the present invention provides a salient target detection model for optical remote sensing images, which can effectively cope with the challenges in complex optical remote sensing images, significantly improve the detection accuracy, and especially pay special attention to improving the common missed detection and false detection problems in the prior art, so as to achieve higher reliability and accuracy and be suitable for practical applications. Figure 1 , the salient object detection model of the embodiment of the present invention includes:

[0066] PVT backbone network module, which is used to extract features from the input optical remote sensing image to be detected and obtain four-level global basic features The obtained four-level global basic features are channel normalized to obtain the normalized four-level global basic features. Among them, i∈{1, 2, 3, 4} is the corresponding level, C is the number of channels, and h×w is the spatial dimension of the corresponding feature map;

[0067] Edge information extraction and fusion module BG, which is used to extract the edge information from the normalized underlying features f t 1 and the top-level feature f t 4 Extract the corresponding edge information, and fuse the respective edge information into the respective feature maps to obtain the feature maps embedded with the edge information;

[0068] Feature enhancement module HAM, which is used to process the two intermediate features f after normalization t 2 and f t 3 , extract features of different scales through multi-scale convolution branches, and combine the channel attention mechanism CAM to enhance important channel features, and the residual connection retains the original features and further enhances them;

[0069] Two spatial attention modules D-SWSAM and SWSAM are used to further extract features from the bottom-level feature maps and the top-level feature maps after embedding edge information, respectively;

[0070] Feature interaction module KTM, which is used to process the intermediate layer features f after being processed by the feature enhancement module t 2 and f t 3 to integrate;

[0071] The feature fusion module is used to fuse the feature maps output by the two spatial attention modules and the feature interaction module to finally generate a saliency image.

[0072] Specifically, the embodiment of the present invention selects the GeleNet model as the benchmark model for improvement. The backbone network of the GeleNet model adopts PVT instead of the traditional CNN. PVT shows stronger feature extraction ability when processing optical remote sensing images. In addition, the D-SWSAM and SWSAM attention mechanism modules in GeleNet are more suitable for processing optical remote sensing images with complex backgrounds and multiple scales than most traditional attention mechanism templates. Among them, the SWSAM module focuses on enhancing the channel interaction of global features and improving the effectiveness of spatial attention, so as to highlight the salient areas more accurately; on this basis, by referring to the D-SWSAM (Directional SWSAM) module, it can adapt to the different directions of salient targets in ORSI, and effectively solve the series of challenges faced by the existing attention mechanism in the detection of salient targets in optical remote sensing images.

[0073] In order to effectively solve the great limitations of existing optical remote sensing image salient target detection methods in practical applications, especially the problems that most existing convolutional neural network (CNN)-based methods are prone to defects such as direction insensitivity, missed detection and incomplete salient target detection when dealing with complex backgrounds and noise, the present invention further introduces an edge information extraction and fusion module BG on the basis of the existing GeleNet model, through which the edge information extraction and fusion module BG is extracted from the normalized underlying feature f t 1 and the top-level feature f t 4 The corresponding edge information is extracted from the image and the respective edge information is fused into the respective feature maps to enhance the edge details of the salient target. This can help the model better capture the edge structure of the target and solve the problem of low accuracy in edge feature extraction of optical remote sensing images under complex backgrounds. It is especially suitable for scenes with blurred edges of salient targets or complex backgrounds.

[0074] Since SWSAM focuses on spatial weighted feature extraction and locates the features of salient areas mainly based on the spatial distribution of the image, after embedding edge information, SWSAM can more accurately focus on key edges and detail areas. This processing method not only effectively highlights and enhances edge features, but also significantly improves the accuracy of the model in detecting salient targets in complex backgrounds.

[0075] On the other hand, the present invention uses a feature enhancement module to enhance the two normalized intermediate features f t 2 and f t 3After feature enhancement processing, it enters the KTM module for fusion. The HAM module enhances the input features through multi-scale convolution and channel attention mechanism, so that important features of different scales are amplified and noise is suppressed. This helps to provide higher quality and more discriminative features before entering the KTM module, and improve the effect of subsequent contextual interaction. At the same time, multi-scale information is also retained: the HAM module extracts feature information at different levels through its multi-scale convolution branch and retains the original features through residual connection. This ensures that when KTM performs contextual interaction, the transmitted feature information covers multiple scales from local details to global semantics, which can help KTM more effectively capture the co-occurrence saliency information. In general, processing by the HAM module first and then the KTM module can ensure that the features are fully enhanced before the contextual interaction, thereby improving the accuracy and robustness of the final detection.

[0076] Specifically, the PVT backbone network module in the embodiment of the present invention includes four transformer encoder blocks and a convolution layer, wherein the four transformer encoder blocks are used to extract features of the input optical remote sensing image to be detected to generate four-level global basic features. The size of the input optical remote sensing image to be detected is 3×352×352, and the number of channels of the generated four-level global basic features is C i ∈{64, 128, 320, 512}, h i ∈{88, 44, 22, 11}, w i ∈{88, 44, 22, 11}; the four-level global basic features generated are normalized by the convolution layer, and the four-level global basic features obtained after normalization are The number of channels C is unified as 32, where f t 1 The space size is 88×88, f t 2 The space size is 22×22, f t 3 The space size is 22×22, f t 4 The space size is 11×11.

[0077] like Figure 2 As shown, the edge information extraction and fusion module BG connects the lowest layer feature f of the PVT backbone network t 1 And the top feature f t 4, and uses convolution operations to generate a boundary map, which will be supervised by the true boundary map (GT). The feature map generated by the PVT (Pyramid Vision Transformer) backbone network is passed to the Conditional Batch Normalization (CBN) module, and the boundary map supervised by the GT boundary map is obtained through convolution operations.

[0078] In the traditional batch normalization (BN) process, the parameters of the affine operation (i.e., the scaling parameter γ and the offset parameter β) are difficult to learn enough information without prior knowledge, which will limit the performance of the model. To solve this problem, this application uses boundary mapping to learn these affine parameters. Specifically, the boundary prediction is used as a condition to embed spatial information into the feature map, so that the original feature map can better capture the boundary features. The processing process of the edge information extraction and fusion module BG includes: the bottom feature f in the four-level global basic features after normalization t 1 and the top-level feature f t 4 Convolution operation is performed to extract the corresponding edge information; then the bottom-level features and the highest-level features are normalized respectively, and the respective edge information is fused into the respective feature maps through the modulation of the γ and β coefficients. The calculation formula is defined as:

[0079]

[0080] In the above formula, BGM represents the feature map embedded with edge information, ASM i represents the normalized feature map output from the backbone network, CB represents 3×3 convolution and batch normalization; γ and β are affine parameters (γ represents the scaling parameter, β represents the offset parameter), each of which contains a 3×3 convolution layer for encoding boundary map information and expanding the channel to 64, the same as the coarse feature map; BM is the extracted edge information, represents element-wise multiplication, Stands for element-wise addition.

[0081] Furthermore, the spatial attention module in the embodiment of the present invention includes D-SWSAM and SWSAM, wherein SWSAM uses a weighted spatial attention module to extract the highest level features; and D-SWSAM uses a spatial weighted attention module with directional convolution to extract the underlying features. Both low-level features and high-level features are processed by the BG module, and then feature extraction is performed by D-SWSAM and SWSAM respectively. This process not only enhances the model's detection accuracy and sensitivity to details of the target edge during feature extraction, especially in important features related to the edge, but also improves the accuracy of feature extraction, thereby reducing the possibility of false detection and missed detection.

[0082] Combination Figure 3 The feature enhancement module HAM captures multi-scale information through multi-branch convolution operations, that is, more branches are used, and a multi-branch convolution is constructed by utilizing the complementarity of receptive fields of different sizes. Each branch captures different features of the input data through different convolution operations. By reducing the number of output channels and using convolution kernels of different sizes, the model's ability to capture the diversity and complexity of input data features is enhanced. Specifically, in the embodiments of the present invention, ((2b-1)×(2b-1)) convolution layers and 3×3 convolution layers are used respectively, combined with the (2b-1) expansion rate {b=5,...,1} for feature extraction.

[0083] However, if the features output by the above branches are directly spliced ​​together, the final spliced ​​features will be inconsistent in space, because the features from different receptive fields cannot accurately distinguish subtle features. To solve this problem, the present invention further introduces a channel attention mechanism (CAM). The feature enhancement module HAM performs a special attention mechanism on the two normalized intermediate features f t 2 and f t 3 The specific processing process includes:

[0084] (1) Calculate the CAM of the original feature A, reshape A into QC×N, and then perform matrix multiplication to replace the reshaped matrix A with its transposed matrix A T Multiply them together to get the relationship matrix between channels P = A × A T , and obtain the channel attention map P∈QC×C through softmax, so each element P of P nm It is determined by the inner product of the nth channel feature and the mth channel feature. The specific calculation formula is as follows:

[0085]

[0086] Among them: nmrepresents the influence of the mth channel on the nth channel, and A represents the two intermediate features f after normalization. t 2 and f t 3 , C is the number of channels of the feature map, Q is the coefficient, and N = h × w represents the flattened size of the spatial dimension of the input feature map;

[0087] (2) Multiply the transpose of P by A and reshape the result into Qh×w×C. Then, multiply the result by the parameter α and perform an element-wise addition operation with A to obtain the final output C∈Qh×w×C:

[0088]

[0089] In the above formula, α is a trainable parameter with an initial value of 0.

[0090] Through this process, the final feature of each channel is the result of weighting all channel features and original features, which enhances semantic relevance and generates more stable features. The channel attention mechanism further optimizes the feature representation, enabling the model to better capture details and global information. Through the splicing and fusion of multi-scale features, the HAM module generates richer and more stable feature representations. In addition, short connections are added to the HAM module to make gradient transfer smoother without increasing computational overhead, thereby improving training efficiency and convergence speed. The design of the HAM module is aimed at improving the performance of the model in various complex scenarios, enabling it to better handle image information of different scales and different levels of detail, and enhancing the robustness and generalization ability of the model.

[0091] By combining multi-scale convolution and channel attention mechanism, the HAM module provides an effective solution for remote sensing image processing. Furthermore, the feature map processed by HAM is input into the cross-layer feature context interaction and knowledge transfer module KTM for processing. KTM performs cross-layer context interaction on it, which will generate richer and more accurate discriminant features, thereby improving the accuracy of feature matching.

[0092] In order to more effectively utilize the output features of the DSWSAM, KTM and SWSAM modules, the present invention introduces an efficient partial decoder as a saliency predictor to generate a saliency image. The partial decoder first adjusts the resolution of the input features to (32×22×22), (32×44×44) and (32×88×88). Based on these adjustments, the saliency predictor is able to generate a preliminary saliency map s∈[0,1] 1×88×88 ; Then, a 4-fold upsampling operation is performed to restore the resolution of the saliency map to be consistent with the input optical remote sensing image, so as to finally obtain the required saliency map s∈[0,1] 1×352×352 .

[0093] The embodiment of the present invention further provides a method for detecting salient targets in an optical remote sensing image, using any of the above detection models of the present invention, including:

[0094] The PVT backbone network module is used to extract features from the input optical remote sensing image to be detected, and four-level global basic features are obtained. The obtained four-level global basic features are channel normalized to obtain the normalized four-level global basic features. Among them, i∈{1, 2, 3, 4} is the corresponding level, C is the number of channels, and h×w is the spatial dimension of the corresponding feature map;

[0095] Through edge information extraction and fusion module, the underlying features f t 1 and the top-level feature f t 4 Extract the corresponding edge information, and fuse the respective edge information into the respective feature maps to obtain the feature maps embedded with the edge information;

[0096] The feature enhancement module is used to enhance the two normalized intermediate features f t 2 and f t 3 Processing, extracting features of different scales through multi-scale convolution branches, and combining the channel attention mechanism CAM to enhance important channel features, residual connection retains the original features and further enhances them;

[0097] The two spatial attention modules are used to further extract features from the underlying feature map and the highest-level feature map after embedding edge information.

[0098] The feature interaction module is used to process the intermediate layer features f processed by the feature enhancement module. t 2 and f t 3 to integrate;

[0099] The feature fusion module is used to fuse the feature maps output by the two spatial attention modules and the feature interaction module, and finally a saliency image is generated.

[0100] In addition, in order to target the characteristics of the data set, especially the data set with fewer training samples, the present invention further preferably designs a multi-scale training strategy in the training process of the model (to facilitate more sufficient training). Specifically, by training at different scale ratios, the model can better adapt to the scale changes of various targets. This embodiment specifically adopts a ratio list of 0.75, 1, and 1.25 times, which means that the input image is scaled at different scales, and a loop is used to traverse the training data loader to obtain a training data packet. Each data packet contains an input image and its corresponding annotation map. For each training data packet, a loop is performed again to apply different scale ratios, and each loop uses one of the ratios to scale the input data.

[0101] By adopting the above multi-scale training strategy to train at different scale magnifications, the model can better adapt to the scale changes of various targets. Even when the training samples are limited, the present invention can still significantly enhance the detection ability of targets of different scales, while effectively reducing the false detection and missed detection caused by scale differences. Therefore, this multi-scale training strategy helps to improve the overall performance of the model in the detection of salient targets in optical remote sensing images.

[0102] In summary, the present invention adopts PVT (Pyramid Vision Transformer) as the backbone network, making full use of its global characteristics to improve the detection performance. At the same time, by introducing edge detection module, attention module and feature enhancement module, it can effectively reduce the false detection and missed detection phenomenon in the detection process, significantly improve the detection accuracy, and overcome the shortcomings of the existing methods that perform poorly under complex background and noise conditions.

[0103] Specific application experiment of the optical remote sensing image salient object detection model of the embodiment of the present invention:

[0104] 1) The experiment was conducted on the ORSSD, EORSSD and ors-4199 datasets respectively. The ORSSD dataset is the first public dataset of ORSI-SOD, containing 800 images and corresponding pixel-level GT, of which 600 are used for training and 200 for testing. The EORSSD dataset contains 2000 images and corresponding GT, of which 1400 are used for training and 600 are used for testing. The ors-4199 dataset is the largest dataset of ORSI-SOD, containing 4199 images and corresponding GT, of which 2000 are used for training and 2199 are used for testing. For the ors-4199 dataset, the present invention also adds a challenge dataset for testing. The challenge dataset is divided into nine categories, namely: large salient objects (BSO), incomplete salient objects (ISO), off-center salient objects (OC), complex scenes (CS), complex salient objects (CSO), low-contrast salient objects (LSO), narrow salient objects (NSO), small salient objects (SSO) and multiple salient objects (MSO). These categories are designed to evaluate the performance of the model in different situations. In addition, in order to train the FGNet model, the present invention also performs data augmentation on the image, including rotation and mirroring operations, trains the FGNet model on the training set of each data set, and tests it on the corresponding test set.

[0105] 2) Network Implementation: All experiments were conducted on PyTorch using an NVIDIA GeForce RTX 2080Ti graphics card. In order to balance effectiveness and efficiency, PVT-v2-b2 was used as the backbone and initialized with pre-trained parameters. Rotation and a combination of flipping and rotation were used for data augmentation, and the input image and GT were resized to 352×352. The FGNet model was trained for 65 epochs using the Adam optimizer, with a batch size of 8 and a basic learning rate of 1e-4, which decayed to 1 / 10 every 30 epochs. In the training phase, a hybrid loss function was used to optimize the proposed model, including intersection-over-union (IoU) loss and binary cross entropy (BCE) loss.

[0106] 3) Evaluation indicators

[0107] The present invention adopts some widely used evaluation indicators to quantitatively evaluate the performance of the model of the present invention, and compares all methods on three datasets, including Smeasure for evaluating the structural similarity between the predicted segmentation results and the true segmentation (S α ), mean absolute error MAE is the average of the absolute values ​​of the difference between the predicted value and the true value (M), and adaptive E-measure is a structural similarity index (E ξ adp), Mean E-measure is the average value of the adaptive E-measure over the entire image (E ξ mean ), Maximum E-measure is the maximum value of the adaptive E-measure on the entire image (E ξ max ), Adaptive F-measure is a comprehensive evaluation indicator that combines the adaptive weights of precision and recall (F β adp ), Mean F-measure is the average value of the adaptive F-measure over the entire image (F β mean ), Maximum F-measure is the maximum value of the adaptive F-measure on the entire image (F β max ), in order to facilitate evaluation, the embodiment of the present invention uses version 2 of the Matlab evaluation tool to calculate these indicators.

[0108] Then, based on the above technical solution, experiments were conducted on three data sets (ORSSD, EORSSD, and ors-4199), and then the method was compared with the current mainstream methods. According to the experimental results, the FGNet model is significantly superior to the current most advanced methods in terms of accuracy and reliability.

[0109] like Figure 4 , Figure 5 As shown in the figure, on the ORSSD dataset, compared with other mainstream methods, the present invention is superior to the second and third place of other mainstream methods in S α The improvement was 0.88% and 0.65% in F β adp The increase was 2.24% and 2.36% in E ξ mean The proposed method improves the performance by 0.78% and 0.83% in S and 0.09% and 0.16% in MAE(M) respectively. α and F β max 、F β mean 、F β adp 、E ξ max 、E ξ adp The indicators have been improved to varying degrees, among which S α and F β adp and Fβ mean The improvements in these three indicators were the most significant, increasing by 0.3%, 0.46% and 0.23% respectively.

[0110] like Figure 6 , Figure 7 As shown in the figure, on the EORSSD dataset, the present invention is compared with the second and third place of other mainstream methods in S α The improvement was 0.67% and 0.78% respectively on F β adp The results were 3.64% and 3.37% higher in E ξ mean The proposed method improves the performance by 1.16% and 1.18% respectively, and reduces the MAE(M) by 0.18% and 0.22% respectively. α and F β max 、F β mean 、F β adp There are different degrees of improvement in indicators such as S α and F β mean and F β adp The improvements in these three indicators are the most significant, increasing by 0.33%, 0.56%, and 0.37% respectively, and MAE(M) has decreased by 0.05%.

[0111] like Figure 8 , Fig. 9 As shown, on the ors-4199 dataset, the present invention is compared with the second and third place of other mainstream methods in S α The increase was 1.27% and 1.4% in F β adp The increase was 1.45% and 3.86% in E ξ mean The proposed method improves the performance by 1.31% and 1.66% in S and 0.36% and 0.72% in MAE(M) respectively. α 、F β max 、F β mean 、E ξ adpThe indicators have been improved to varying degrees, increasing by 0.04%, 0.1%, 0.15%, and 0.13% respectively. The test results on the ors-4199 challenge dataset show that the method of the present invention has obvious advantages in all nine scenarios, and the average score exceeds the second place among other methods by 0.011 points. This effectively proves that the method of the present invention is superior to other ORSI-SOD methods in dealing with challenging scenarios, demonstrating its strong stability.

Claims

1. A salient object detection model for optical remote sensing images, characterized in that: include: PVT backbone network module, which is used to extract features from the input optical remote sensing image to be detected and obtain four-level global basic features The obtained four-level global basic features are channel normalized to obtain the normalized four-level global basic features. Among them, i∈{1, 2, 3, 4} is the corresponding level, C is the number of channels, and h×w is the spatial dimension of the corresponding feature map; Edge information extraction and fusion module, which is used to extract the edge information from the normalized underlying features f t 1 and the top-level feature f t 4 Extract the corresponding edge information, and fuse the respective edge information into the respective feature maps to obtain the feature maps embedded with the edge information; Feature enhancement module, which is used to process the two intermediate features f after normalization. t 2 and f t 3 , extract features of different scales through multi-scale convolution branches, and combine the channel attention mechanism CAM to enhance important channel features, and the residual connection retains the original features and further enhances them; Two spatial attention modules are used to further extract features from the bottom-level feature maps and the top-level feature maps after embedding edge information; Feature interaction module, which is used to process the intermediate layer features f after being processed by the feature enhancement module t 2 and f t 3 to integrate; The feature fusion module is used to fuse the feature maps output by the two spatial attention modules and the feature interaction module to finally generate a saliency image.

2. The optical remote sensing image salient object detection model according to claim 1, characterized in that: The PVT backbone network module includes four transformer encoder blocks and a convolutional layer, wherein the four transformer encoder blocks are used to extract features of the input optical remote sensing image to be detected to generate four-level global basic features. The convolution layer is used to perform channel normalization processing on the generated four-level global basic features; the size of the input optical remote sensing image to be detected is 3×352×352, and the number of channels of the generated four-level global basic features is C i ∈{64, 128, 320, 512}, the number of channels C of the four-level global basic features after normalization is unified to 32.

3. The optical remote sensing image salient object detection model according to claim 1, characterized in that: The processing process of the edge information extraction and fusion module includes: the bottom feature f of the four-level global basic features after normalization t 1 and the top-level feature f t 4 Convolution operation is performed to extract the corresponding edge information; then the bottom-level features and the highest-level features are normalized respectively, and the respective edge information is fused into the respective feature maps through the modulation of the γ and β coefficients. The processing operation formula is as follows: In the above formula, BGM represents the feature map embedded with edge information, ASM i represents the normalized feature map output from the backbone network, CB represents 3×3 convolution and batch normalization; γ represents the scaling parameter, β represents the offset parameter, and BM represents the extracted edge information. represents element-wise multiplication, Stands for element-wise addition.

4. The optical remote sensing image salient object detection model according to any one of claims 1 to 3, characterized in that: The multi-scale convolution branch is used to extract features of different scales, specifically, a ((2b-1)×(2b-1)) convolution layer and a 3×3 convolution layer are used for feature extraction respectively.

5. The optical remote sensing image salient object detection model according to claim 4, characterized in that: The feature enhancement module performs normalization on the two intermediate features f t 2 and f t 3 The processing process includes: (1) Calculate the CAM of the original feature A, reshape A into QC×N, and then perform matrix multiplication to replace the reshaped matrix A with its transposed matrix A T Multiply them together to get the relationship matrix between channels P = A × A T , and obtain the channel attention map P∈QC×C through softmax, so each element P n It is determined by the inner product of the nth channel feature and the mth channel feature. The specific calculation formula is as follows: Among them: nm represents the influence of the mth channel on the nth channel, and A represents the two intermediate features f after normalization. t 2 and f t 3 , C is the number of channels of the feature map, Q is the coefficient, and N = h × w represents the flattened size of the spatial dimension of the input feature map; (2) Multiply the transpose of P by A and reshape the result into Qh×w×C. Then, multiply the result by the parameter α and perform an element-wise addition operation with A to obtain the final output C∈Qh×w×C: In the above formula, α is a trainable parameter with an initial value of 0.

6. The optical remote sensing image salient object detection model according to any one of claims 1 to 3, characterized in that: The spatial attention module includes D-SWSAM and SWSAM, wherein SWSAM adopts a weighted spatial attention module for extracting the highest level features; and D-SWSAM adopts a spatial weighted attention module with directional convolution for extracting the underlying features.

7. The optical remote sensing image salient object detection model according to any one of claims 1 to 3, characterized in that: The feature interaction module adopts contextual interaction of cross-level features and a knowledge transfer module KTM.

8. The optical remote sensing image salient object detection model according to any one of claims 1 to 3, characterized in that: During the training process, the model adopts a multi-scale training strategy, that is, training is performed at different scale multiples.

9. A method for detecting salient objects in optical remote sensing images, characterized in that: The detection model according to any one of claims 1 to 8 specifically comprises: The PVT backbone network module is used to extract features from the input optical remote sensing image to be detected, and four-level global basic features are obtained. The obtained four-level global basic features are channel normalized to obtain the normalized four-level global basic features. Among them, i∈{1, 2, 3, 4} is the corresponding level, C is the number of channels, and h×w is the spatial dimension of the corresponding feature map; Through edge information extraction and fusion module, the underlying features f t 1 and the top-level feature f t 4 Extract the corresponding edge information, and fuse the respective edge information into the respective feature maps to obtain the feature maps embedded with the edge information; The feature enhancement module is used to enhance the two normalized intermediate features f t 2 and f t 3 Processing, extracting features of different scales through multi-scale convolution branches, and combining the channel attention mechanism CAM to enhance important channel features, residual connection retains the original features and further enhances them; The two spatial attention modules are used to further extract features from the underlying feature map and the highest-level feature map after embedding edge information. The feature interaction module is used to process the intermediate layer features f processed by the feature enhancement module. t 2 and f t 3 to integrate; The feature fusion module is used to fuse the feature maps output by the two spatial attention modules and the feature interaction module, and finally a saliency image is generated.

10. The optical remote sensing image salient object detection method according to claim 9, characterized in that: The feature fusion module uses a partial decoder as a saliency predictor, which first adjusts the resolution of the input features to (32×22×22), (32×44×44) and (32×88×88), respectively, to generate a preliminary saliency map s∈[0,1] 1×88×88 ; Then, a 4-fold upsampling operation is performed to restore the resolution of the saliency map to be consistent with the input optical remote sensing image, thereby finally obtaining the required saliency map s∈[0,1] 1×352×352 .

Images