Optical remote sensing salient target detection method and system based on heterogeneous feature collaboration

Abstract

The present invention relates to an optical remote sensing salient target detection method and system based on heterogeneous feature collaboration, and belongs to the field of remote sensing image processing and deep learning technology. It includes: performing heterogeneous feature extraction on a given optical remote sensing image, including: using a convolutional neural network to extract features from a given optical remote sensing image, and obtaining local image detail features extracted by different convolutional layers; introducing a Transformer network layer in the deep layer of the convolutional neural network, and the global semantic features of the image; fusing the extracted global semantic features of the image and the local image detail features at different stages; and fusing the features at different stages by constructing a saliency-guided attention enhancement decoder to obtain the final salient target detection map. The present invention realizes accurate positioning and fine segmentation of salient targets.

Description

Technical Field

[0001] The present invention relates to an optical remote sensing salient target detection method and system based on heterogeneous feature collaboration, and belongs to the technical field of remote sensing image processing and deep learning. Background Art

[0002] In recent years, with the rapid development of space technology and aeronautical science, the technologies related to intelligent interpretation of satellite remote sensing images have attracted much attention. The application demand of these technologies in the fields of natural resource management, disaster monitoring and assessment, urban planning and management has grown rapidly.

[0003] The task of salient object detection in optical remote sensing images (ORSI-SOD) is to use computer vision and image processing technology to automatically identify and highlight the most significant targets or areas from optical remote sensing images. These targets may be geographic information elements such as buildings, roads, vehicles, vegetation, water bodies, or specific types of land objects, diseases, environmental changes, etc. Therefore, optical remote sensing images play an important role in geological exploration, agriculture and forestry, urban planning, environmental monitoring, disaster response and military reconnaissance.

[0004] Existing optical remote sensing image salient target detection technologies are mainly divided into: 1) Machine learning-based detection technology, which uses traditional machine learning methods such as support vector machines (SVMs) and random forests, combined with manually extracted features for target detection; 2) Deep learning-based detection technology, which uses end-to-end networks such as convolutional neural networks to directly learn features from data without the need for complex manual feature design. Although existing technologies have made significant progress, there are still some defects and challenges. Specifically, due to the wide coverage of remote sensing images, the complex natural landscapes and man-made features covered, and their susceptibility to weather and lighting conditions, the salient targets in the images usually present irregular topological structures and are accompanied by complex and interfering backgrounds, which makes it difficult for current technologies to achieve ideal detection effects.

[0005] Detection technology based on deep learning usually uses deep learning models to learn complex feature representations from a large amount of data, thereby automatically extracting features from images and performing effective target detection. In the ORSI-SOD task, the deep learning model not only needs to extract global semantic information in the image to locate salient targets, but also needs to extract local detail information to separate the complete outline of the target from the image. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides an optical remote sensing salient target detection method based on heterogeneous feature collaboration;

[0007] The present invention also provides an optical remote sensing salient target detection system based on heterogeneous feature collaboration.

[0008] The present invention first constructs a hybrid heterogeneous encoder to extract image features, mainly using a convolutional neural network to extract local detail information, and introduces a Transformer network layer in the deep layer of the convolutional neural network to assist in extracting global semantic information. Secondly, the present invention proposes an adaptive global-local information aggregation module, which uses feature alignment methods in global and local dimensions to fuse complementary heterogeneous features extracted from the convolutional neural network and the Transformer to achieve effective aggregation of global and local information. Finally, the present invention constructs a saliency-guided attention enhancement decoder, which uses potential saliency information to guide the shallow decoder of the network to pay more attention to important areas while suppressing cluttered backgrounds, thereby fully detecting salient targets in optical remote sensing images.

[0009] Terminology explanation:

[0010] ASPP (Atrous Spatial Pyramid Pooling) module: is a neural network component designed to improve the ability of convolutional neural networks to capture image features at different scales. By using a series of dilated convolutions with different sampling rates, the module can effectively expand the receptive field and capture a wider range of contextual information without adding additional computational cost or the number of model parameters.

[0011] The technical solution of the present invention is as follows:

[0012] The optical remote sensing salient target detection method based on heterogeneous feature collaboration includes:

[0013] Perform heterogeneous feature extraction on a given optical remote sensing image, including: using a convolutional neural network to extract features from a given optical remote sensing image, obtaining local detail features of the image extracted by different convolutional layers; introducing a Transformer network layer in the deep layer of the convolutional neural network, obtaining global semantic features of the image extracted by different Transformer network layers;

[0014] Fusion of the global semantic features of the image and the local detail features of the image extracted at different stages;

[0015] By constructing a saliency-guided attention-enhanced decoder, the features of different stages are fused to obtain the final salient object detection map.

[0016] As a further preferred scheme, the above-mentioned optical remote sensing salient target detection method based on heterogeneous feature collaboration is implemented through a trained heterogeneous feature collaboration network model; the heterogeneous feature collaboration network model includes a hybrid heterogeneous encoder, an adaptive global-local information aggregation module, and a saliency-guided attention enhancement decoder.

[0017] As a further preferred solution, a hybrid heterogeneous encoder is constructed to extract heterogeneous features from a given optical remote sensing image; including:

[0018] The five feature extraction layers of the VGG-16 network model, namely the five VggBlock modules, use a 3*3 convolution kernel to respond to subtle changes in the input image with a smaller receptive field, thereby accurately capturing local detail information in the image features and obtaining a set of visual features recorded as ;

[0019] In the third to fifth stages of the VGG-16 network model, the last three Transformer network layers of the Swin-Transformer network model are introduced to achieve a comprehensive analysis of the entire image area through the self-attention mechanism, thereby capturing the global semantic information in the image features and obtaining another set of visual features recorded as .

[0020] As a further preferred solution, an Adaptive Global-Local Integration module (AGLI) is constructed to fuse the global semantic features of the image and the local detail features of the image extracted at different stages; including:

[0021] Perform feature alignment of the global dimension on two sets of heterogeneous features;

[0022] Perform feature alignment of local dimensions on two sets of heterogeneous features;

[0023] The complementary heterogeneous features are fused after feature alignment.

[0024] As a further preferred solution, firstly, the pooling operation is used to perform a highly generalized representation of the global information of the two sets of heterogeneous features, namely, a global representation, as shown in the following formula:

[0025] ;

[0026] ;

[0027] in, Represents the basic convolution module, which includes a 3×3 convolution layer, a batch normalization layer and a RELU activation function; Represents the global average pooling operation on the channel dimension; and They are the global representations of the features extracted by the convolutional neural network and the Transformer network layers respectively;

[0028] Using mean square error loss to achieve features and Alignment on a global dimension to exclude irrelevant features; global level loss The definition of is as follows:

[0029] ;

[0030] in, represents the mean square error loss function;

[0031] The global feature representation after feature alignment is used to guide the original visual features, as shown in the following formula:

[0032] ;

[0033] ;

[0034] in, For splicing operation; and are heterogeneous features after global alignment;

[0035] The correlation between the two sets of heterogeneous feature pixels and pixels after the original visual features are guided is calculated as shown in the following formula:

[0036] ;

[0037] ;

[0038] ;

[0039] in, represents the matrix multiplication operation, represents a function, Represents the operation of flattening a two-dimensional matrix into a one-dimensional vector;

[0040] The true value G is downsampled to obtain , and then apply matrix multiplication to determine the pixel correlation of the true value , as shown below:

[0041] ;

[0042] The binary cross entropy loss function is used to calculate the pixel correlation of the true value G. Pixel correlation with heterogeneous features Consistency constraints are imposed to align heterogeneous features in local dimensions to eliminate the noise caused by feature heterogeneity; the pixel-level loss in this process The definition of is as follows:

[0043] ;

[0044] in, is the binary cross entropy loss function;

[0045] The original visual features are guided by the correlation of aligned pixels, as shown in the following formula:

[0046] ;

[0047] ;

[0048] Aligned features and As the input of the next encoder, continue to extract features; at the same time, the features after feature alignment and The fused output is used as the input to the saliency-guided attention-enhanced decoder:

[0049] ;

[0050] The output of the last adaptive global-local information aggregation module Send to ASPP module to get features ,The ASPP module sets five convolution branches in parallel, and realizes the extraction of multi-scale features by changing the expansion rate, and further locates the significant target objects.

[0051] As a further preferred solution, by constructing a saliency-guided attention enhanced decoder (SGAED), the features of different stages are fused to obtain the final salient object detection map; including:

[0052] The channel enhancement module is applied to the output features of the current coding layer and the output features of the previous decoder layer respectively to filter out the important channel features in the visual features and suppress the irrelevant channel features;

[0053] Apply spatial enhancement modules to the output features of the current encoding layer and the output features of the previous decoder layer, respectively, to highlight the important spatial positions in the visual features and suppress the unimportant background;

[0054] The two enhanced features are fused.

[0055] As a further preferred solution, for the i-th saliency-guided attention-enhanced decoder, given the output features of the i-th hybrid heterogeneous encoder layer , the output features of the i+1th decoder layer And the saliency prediction result of the i+1th layer encoder , respectively, using the channel enhancement module to enhance the features Enhanced ;

[0056] Analyze the significant prediction results output by the i+1th layer encoder With the output features of the i-th hybrid heterogeneous encoder layer The correlation between the channels , as shown below:

[0057] ;

[0058] use Generate the enhanced channel attention map as shown below:

[0059] ;

[0060] in, represents the global maximum pooling operation, is a multi-layer perceptron;

[0061] Using the enhanced channel attention map , filter out important channel features in visual features, and suppress irrelevant channel features, and enhance the features It is expressed as:

[0062] ;

[0063] Among them, ⊙ represents the Hadamard product, represents an operation that repeats a tensor along a spatial dimension, Represents the sigmoid function;

[0064] Use the Spatial Enhanced Module (SEM) to further enhance the features in the spatial dimension; use the saliency prediction results Generate enhanced spatial attention map :

[0065] ;

[0066] Using the enhanced spatial attention map , highlighting the important spatial positions in visual features, as shown in the following formula:

[0067] ;

[0068] in, Represents an operation that repeats a tensor along the channel dimension;

[0069] Enhance the output features of the i+1th layer decoder , and obtain the enhanced features ;

[0070] The enhanced features are fused by performing two basic convolution operations to obtain the output of the first layer decoder :

[0071] ;

[0072] in, is an upsampling operation.

[0073] Predicting significant results using enhanced features :

[0074] ;

[0075] in, represents a 3×3 convolution operation, Select as the final result.

[0076] As a further preferred solution, the heterogeneous feature collaborative network model adopts a deep supervision mechanism during the training process. The loss function combines the pixel-level binary cross entropy (BCE) loss and the intersection over union (IoU) loss based on the overall mapping. The global level loss for heterogeneous features is also introduced in the loss function. and pixel-level loss , the overall loss of the heterogeneous feature collaborative network model As shown below:

[0077] ;

[0078] ;

[0079] Among them, G represents the true result, represents the loss function based on intersection-over-union.

[0080] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of an optical remote sensing salient target detection method based on heterogeneous feature collaboration are implemented.

[0081] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an optical remote sensing salient target detection method based on heterogeneous feature collaboration.

[0082] The optical remote sensing salient target detection system based on heterogeneous feature collaboration includes:

[0083] The heterogeneous feature extraction module is configured to: perform heterogeneous feature extraction on a given optical remote sensing image, including: extracting features from the given optical remote sensing image using a convolutional neural network to obtain local detail features of the image extracted by different convolutional layers; introducing a Transformer network layer in the deep layer of the convolutional neural network to obtain global semantic features of the image extracted by different Transformer network layers;

[0084] The module for acquiring image global semantic features and image local detail features is configured to: fuse the image global semantic features and image local detail features extracted at different stages;

[0085] The salient object detection map acquisition module is configured to: fuse features of different stages by constructing a saliency-guided attention-enhanced decoder to obtain a final salient object detection map.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] 1. The hybrid heterogeneous encoder constructed by the present invention captures local detail information by deploying a convolutional neural network, and introduces a Transformer layer in the deep layer of the network to assist in obtaining global semantic information, thereby achieving accurate positioning and fine segmentation of salient targets.

[0088] 2. The present invention proposes an adaptive global-local aggregation module, which utilizes an innovative global and local feature alignment method to aggregate global information and local information, thereby effectively perceiving the irregular topological structure of salient targets.

[0089] 3. The saliency-guided attention enhancement decoder proposed in the present invention innovatively introduces a channel enhancement module and a spatial enhancement module to gradually filter redundant information, enhance useful saliency information, and effectively eliminate the interference of complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a schematic diagram of the overall model framework of the optical remote sensing salient target detection method based on heterogeneous feature collaboration of the present invention;

[0091] Figure 2 Schematic diagram of the internal structure of the adaptive global-local information aggregation module of the present invention;

[0092] Figure 3 A schematic diagram of the internal structure of the saliency-guided attention enhancement decoder of the present invention;

[0093] Figure 4 It is a schematic diagram comparing the effect diagrams of the present invention with other existing models. DETAILED DESCRIPTION

[0094] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0095] Example 1

[0096] The optical remote sensing salient target detection method based on heterogeneous feature collaboration includes:

[0097] Perform heterogeneous feature extraction on a given optical remote sensing image, including: using a convolutional neural network to extract features from a given optical remote sensing image, obtaining local detail features of the image extracted by different convolutional layers; introducing a Transformer network layer in the deep layer of the convolutional neural network, obtaining global semantic features of the image extracted by different Transformer network layers;

[0098] Fusion of global semantic features and local detail features of images extracted at different stages to achieve complementary advantages;

[0099] By constructing a saliency-guided attention-enhanced decoder, the features of different stages are fused to obtain the final salient object detection map.

[0100] Example 2

[0101] The difference between the optical remote sensing salient target detection method based on heterogeneous feature collaboration described in Example 1 is that:

[0102] The optical remote sensing salient target detection method based on heterogeneous feature collaboration is implemented through a trained heterogeneous feature collaboration network model; the heterogeneous feature collaboration network model includes a hybrid heterogeneous encoder, an adaptive global-local information aggregation module, and a saliency-guided attention enhancement decoder.

[0103] By constructing a hybrid heterogeneous encoder, heterogeneous features are extracted from a given optical remote sensing image; including:

[0104] The five feature extraction layers of the VGG-16 network model (convolutional neural network model VGG-16), namely the five VggBlock modules (i.e. Figure 1VggBlock-1, VggBlock-2, VggBlock-3, VggBlock-4, VggBlock-5 in the image), using a 3*3 convolution kernel to respond to subtle changes in the input image with a smaller receptive field, thereby accurately capturing local detail information in the image features and obtaining a set of visual features recorded as ;

[0105] The last three Transformer network layers of the Swin-Transformer network model are introduced in the third to fifth stages of the VGG-16 network model, namely Figure 1 The SwinBlock-3, SwinBlock-4, and SwinBlock-5 modules in the image are used to analyze the entire image area through the self-attention mechanism, thereby capturing the global semantic information in the image features and obtaining another set of visual features recorded as .

[0106] By constructing an Adaptive Global-Local Integration module (AGLI), the global semantic features of the image and the local detail features of the image extracted at different stages are integrated; including:

[0107] The global feature alignment module (GFAM) is used to align the two sets of heterogeneous features in the global dimension.

[0108] The pixel-wise feature alignment module (PFAM) is used to align the two sets of heterogeneous features in the local dimension.

[0109] The complementary heterogeneous features are fused after feature alignment.

[0110] like Figure 2 As shown, firstly, the pooling operation is used to perform a highly generalized representation of the global information of the two sets of heterogeneous features, namely the global representation, as shown in the following formula:

[0111] ;

[0112] ;

[0113] in, Represents the basic convolution module, which includes a 3×3 convolution layer, a batch normalization layer and a RELU activation function; Represents the global average pooling operation on the channel dimension; and They are the global representations of the features extracted by the convolutional neural network and the Transformer network layers respectively;

[0114] Although the specific representations of the two heterogeneous features are different, their overall tone is consistent. Therefore, the mean square error loss is used to achieve feature and Alignment on a global dimension to exclude irrelevant features; global level loss The definition of is as follows:

[0115] ;

[0116] in, represents the mean square error loss function;

[0117] The global feature representation after feature alignment is used to guide the original visual features, as shown in the following formula:

[0118] ;

[0119] ;

[0120] in, For splicing operation; and are heterogeneous features after global alignment;

[0121] The correlation between the two sets of heterogeneous feature pixels and pixels after the original visual features are guided is calculated as shown in the following formula:

[0122] ;

[0123] ;

[0124] ;

[0125] in, represents the matrix multiplication operation, represents a function, Represents the operation of flattening a two-dimensional matrix into a one-dimensional vector;

[0126] The true value G is downsampled to obtain , and then apply matrix multiplication to determine the pixel correlation of the true value , as shown below:

[0127] ;

[0128] The pixel correlation of heterogeneous features should be consistent with the pixel correlation of the true value. Therefore, the binary cross entropy loss function is used to calculate the pixel correlation of the true value G. Pixel correlation with heterogeneous features Consistency constraints are imposed to align heterogeneous features in local dimensions to eliminate the noise caused by feature heterogeneity; the pixel-level loss in this process The definition of is as follows:

[0129] ;

[0130] in, is the binary cross entropy loss function;

[0131] The original visual features are guided by the correlation of aligned pixels, as shown in the following formula:

[0132] ;

[0133] ;

[0134] Aligned features and As the input of the next encoder, continue to extract features; at the same time, the features after feature alignment and The fused output is used as the input to the saliency-guided attention-enhanced decoder:

[0135] ;

[0136] The output of the last adaptive global-local information aggregation module Send to ASPP module to get features ,The ASPP module sets five convolution branches in parallel, and realizes the extraction of multi-scale features by changing the expansion rate, and further locates the significant target objects.

[0137] In this method, by constructing a saliency-guided attention enhanced decoder (SGAED), the features of different stages are fused to obtain the final salient object detection map; including:

[0138] The channel enhancement module is applied to the output features of the current coding layer and the output features of the previous decoder layer respectively to filter out the important channel features in the visual features and suppress the irrelevant channel features;

[0139] Apply spatial enhancement modules to the output features of the current encoding layer and the output features of the previous decoder layer, respectively, to highlight the important spatial positions in the visual features and suppress the unimportant background;

[0140] The two enhanced features are fused.

[0141] The architecture of the saliency-guided attention-enhanced decoder is shown in Figure 3 As shown in Figure 1, for the i-th saliency-guided attention-enhanced decoder, its output features are used to predict the saliency of each pixel in the image, that is, the possibility that the pixel belongs to the object of interest, which is recorded as the saliency prediction result; its input features include: the output features of the i-th hybrid heterogeneous encoder layer , the output features of the i+1th layer saliency-guided attention-enhanced decoder and significant prediction results ; The decoder uses the newly designed Channel Enhanced Module (CEM) and Spatial Enhanced Module (SEM) to enhance the image features separately and then fuse them to obtain more effective data representation.

[0142] Specifically, if Figure 3 As shown in Figure 2, the channel enhancement module aims to filter out important channels in the features. Analyze the significant prediction results of the i+1th layer encoder output With the output features of the i-th hybrid heterogeneous encoder layer The correlation between the channels , as shown below:

[0143] ;

[0144] use Generate the enhanced channel attention map as shown below:

[0145] ;

[0146] in, represents the global maximum pooling operation, is a multi-layer perceptron;

[0147] Using the enhanced channel attention map , filter out important channel features in visual features, and suppress irrelevant channel features, and enhance the features It is expressed as:

[0148] ;

[0149] Among them, ⊙ represents the Hadamard product, represents an operation that repeats a tensor along a spatial dimension, Represents the sigmoid function;

[0150] The spatial enhancement module (Spatial Enhanced Module, SEM) is used to further enhance the features in the spatial dimension; Figure 3 As shown in the lower right part, the results of significant prediction are Generate enhanced spatial attention map :

[0151] ;

[0152] Using the enhanced spatial attention map , highlighting the important spatial positions in visual features, as shown in the following formula:

[0153] ;

[0154] in, Represents an operation that repeats a tensor along the channel dimension;

[0155] The same operation is used to enhance the output features of the i+1th layer decoder , and obtain the enhanced features ;

[0156] The enhanced features are fused by performing two basic convolution operations to obtain the output of the first layer decoder :

[0157] ;

[0158] in, is an upsampling operation.

[0159] Predicting significant results using enhanced features :

[0160] ;

[0161] in, represents a 3×3 convolution operation, Select as the final result. It is a two-dimensional matrix of the same size as the original image, in which the value corresponding to each pixel represents the possibility or significance of the pixel belonging to a salient target in the image.

[0162] During the training process of the heterogeneous feature collaborative network model, in order to ensure the effective learning and optimization of the model, a deep supervision mechanism is adopted to carefully monitor all intermediate saliency maps. The loss function combines the pixel-level binary cross entropy (BCE) loss and the intersection over union (IoU) loss based on the overall mapping to achieve a comprehensive evaluation of the model's detection accuracy. In addition, in order to further improve the model's ability to handle heterogeneous features, a global level loss for heterogeneous features is also introduced in the loss function. and pixel-level loss , the overall loss of the heterogeneous feature collaborative network model As shown below:

[0163] ;

[0164] ;

[0165] Among them, G represents the true result, represents the loss function based on intersection-over-union.

[0166] Table 1 International standard datasets for evaluating performance;

[0167] ;

[0168] Table 2 Evaluation indexes for evaluating performance of the present invention;

[0169] ;

[0170] Table 3 Introduction of comparative models of the present invention;

[0171] ;

[0172] Table 4 Comparison of detection accuracy of the model of the present invention;

[0173] ;

[0174] Table 5 Performance analysis comparison of the model of the present invention on different attribute data;

[0175] ;

[0176] Corresponding to Table 5, Figure 4 It is a schematic diagram comparing the effect diagrams of the present invention with other existing models.

[0177] By Tables 1 to 5, and Figure 4, it can be known that: the method of the present invention achieves accurate positioning and fine segmentation of salient targets. The method of the present invention can effectively perceive the irregular topological structure of salient targets. The method of the present invention can gradually filter redundant information, enhance useful salient information, and effectively eliminate the interference of complex background.

[0178] Example 3

[0179] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the optical remote sensing salient target detection method based on heterogeneous feature collaboration described in embodiment 1 or 2 are implemented.

[0180] Example 4

[0181] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the optical remote sensing salient target detection method based on heterogeneous feature collaboration described in Example 1 or 2.

[0182] Example 5

[0183] The optical remote sensing salient target detection system based on heterogeneous feature collaboration includes:

[0184] The heterogeneous feature extraction module is configured to: perform heterogeneous feature extraction on a given optical remote sensing image, including: extracting features from the given optical remote sensing image using a convolutional neural network to obtain local detail features of the image extracted by different convolutional layers; introducing a Transformer network layer in the deep layer of the convolutional neural network to obtain global semantic features of the image extracted by different Transformer network layers;

[0185] The module for acquiring image global semantic features and image local detail features is configured to: fuse the image global semantic features and image local detail features extracted at different stages;

[0186] The salient object detection map acquisition module is configured to: fuse features of different stages by constructing a saliency-guided attention-enhanced decoder to obtain a final salient object detection map.

Claims

1. An optical remote sensing salient target detection method based on heterogeneous feature collaboration, characterized in that: include: Perform heterogeneous feature extraction on a given optical remote sensing image, including: using a convolutional neural network to extract features from a given optical remote sensing image, obtaining local detail features of the image extracted by different convolutional layers; introducing a Transformer network layer in the deep layer of the convolutional neural network, obtaining global semantic features of the image extracted by different Transformer network layers; Fusion of the global semantic features of the image and the local detail features of the image extracted at different stages; By constructing a saliency-guided attention-enhanced decoder, the features of different stages are fused to obtain the final salient object detection map; The optical remote sensing salient target detection method based on heterogeneous feature collaboration is implemented by a trained heterogeneous feature collaboration network model; the heterogeneous feature collaboration network model includes a hybrid heterogeneous encoder, an adaptive global-local information aggregation module, and a saliency-guided attention enhancement decoder; By constructing a hybrid heterogeneous encoder, heterogeneous features are extracted from a given optical remote sensing image; including: The VGG-16 network model is used to capture local information in image features and obtain different features in five stages, that is, a set of visual features of different scales is recorded as f v ; The five feature extraction layers of the VGG-16 network model, namely the five VggBlock modules, use a 3*3 convolution kernel to respond to subtle changes in the input image, capture local detail information in the image features, and obtain a set of visual features denoted as f v ; In the third to fifth stages of the VGG-16 network model, the last three Transformer network layers of the Swin-Transformer network model are introduced to achieve a comprehensive analysis of the entire image area through the self-attention mechanism, thereby capturing the global semantic information in the image features and obtaining another set of visual features denoted as f s ; By building an adaptive global-local information aggregation module, the global semantic features of the image and the local detail features of the image extracted at different stages are fused; including: First, perform feature alignment of the two sets of heterogeneous features in the global dimension; Then, the two sets of features aligned in the global dimension are aligned in the local dimension; Fusing the complementary heterogeneous features after feature alignment; First, the pooling operation is used to perform a highly generalized representation of the global information of the two sets of heterogeneous features, namely the global representation, as shown in the following formula: f v,g =GAP c (Conv(f v )); f s,g =GAP c (Conv(f s )); Among them, Conv(·) represents the basic convolution module, which includes a 3×3 convolution layer, a batch normalization layer and a RELU activation function; GAPc(·) represents the global average pooling operation in the channel dimension; f v,g and f s,g They are the global representations of the features extracted by the convolutional neural network and the Transformer network layers respectively; Using mean square error loss to achieve feature f v,g and f s,g Alignment in the global dimension to exclude irrelevant features; global level loss L g The definition of is as follows: L g =L MSE (in v,g ,f s,g ); Among them, L MSE (·) represents the mean square error loss function; The global feature representation after feature alignment is used to guide the original visual features, as shown in the following formula: f ga,v =Conv(Cat(f v,g ,f s ))+f v ; f ga,s =Conv(Cat(f s,g ,f s ))+f s ; Among them, Cat(·) is the splicing operation; f ga,v and f ga,s are heterogeneous features after global alignment; Calculate the correlation A between the two sets of heterogeneous feature pixels and pixels after the original visual features are guided v,s , as shown below: A v,s =A v→s +A s→v ; in, represents the matrix multiplication operation, softmax(·) represents the softmax function, and flatten(·) represents the operation of flattening a two-dimensional matrix into a one-dimensional vector; The ground truth G is downsampled to obtain G′, and then matrix multiplication is applied to determine the pixel correlation A of the ground truth. gt , as shown below: The binary cross entropy loss function is used to calculate the pixel correlation A of the true value G. gt and pixel correlation of heterogeneous features A v,s Consistency constraints are imposed to align heterogeneous features in the local dimension to eliminate the noise caused by feature heterogeneity; the pixel-level loss L in this process p The definition of is as follows: L p =L BCE (A gt ,A v,s ); Among them, L BCE (·) is the binary cross entropy loss function; The original visual features are guided by the correlation of aligned pixels, as shown in the following formula: The aligned feature f a,v and f a,s As the input of the next encoder, continue to extract features; at the same time, the feature f after feature alignment a,v and f a,s The fusion output is f he , used as input to the saliency-guided attention-enhanced decoder: f he =Conv(Cat(f a,v ,f a,s )); The output of the last adaptive global-local information aggregation module Send it to the ASPP module to get the feature f aspp ,The ASPP module sets five convolution branches in parallel, and realizes the extraction of multi-scale features by changing the expansion rate to locate the significant target objects; By constructing a saliency-guided attention-enhanced decoder, the features of different stages are fused to obtain the final salient object detection map; including: Apply the channel enhancement module to the output features of the current coding layer and the output features of the previous decoder layer, respectively, to filter out the important channel features in the visual features and suppress the irrelevant channel features; obtain the channel-enhanced encoder output features and the channel-enhanced previous decoder output features respectively; Applying spatial enhancement modules to the channel-enhanced encoder output features and the channel-enhanced upper-layer decoder output features respectively, highlighting the important spatial positions in the visual features and suppressing the unimportant background; and then obtaining the channel-enhanced and spatially enhanced encoder output features and the channel-enhanced and spatially enhanced upper-layer decoder output features respectively; The encoder output features after channel and spatial enhancement are fused with the previous layer decoder output features after channel and spatial enhancement.

2. The optical remote sensing salient target detection method based on heterogeneous feature collaboration according to claim 1 is characterized in that: For the i-th saliency-guided attention-enhanced decoder, given the output features f of the i-th hybrid heterogeneous encoder layer he i , the output features of the i+1th decoder layer and the saliency prediction result S of the i+1th layer encoder i+1 , respectively, using the channel enhancement module to enhance the features Enhanced Analyze the significant prediction result S output by the i+1th layer encoder i+1 With the output features of the i-th hybrid heterogeneous encoder layer The correlation between the channels As shown below: use Generate enhanced channel attention map CA i , as shown below: Among them, GMP(·) represents the global maximum pooling operation, and MLP(·) is a multi-layer perceptron; Using the enhanced channel attention map CA i , filter out important channel features in visual features, and suppress irrelevant channel features, and enhance the features It is expressed as: Among them, ⊙ represents the Hadamard product, repeat s (·) represents the operation of repeating the tensor along the spatial dimension, and sig(·) represents the sigmoid function; Use the spatial enhancement module to further enhance the features in the spatial dimension; use the saliency prediction result S i+1 Generate enhanced spatial attention map SA i : Using the enhanced spatial attention map SA i , highlighting the important spatial positions in visual features, as shown in the following formula: Among them, repeat c (·) represents the operation of repeating a tensor along the channel dimension; Enhance the output features of the i+1th layer decoder The enhanced features The enhanced features are fused by performing two basic convolution operations to obtain the output of the i-th layer decoder Among them, Up(·) is the up-sampling operation.

3. The optical remote sensing salient target detection method based on heterogeneous feature collaboration according to claim 2 is characterized in that: Use the enhanced features to predict the significant results S i : Among them, Conv 3×3 (·) represents a 3×3 convolution operation, and selects S 1 As the final result.

4. The optical remote sensing salient target detection method based on heterogeneous feature collaboration according to claim 1, characterized in that: During the training process, the heterogeneous feature collaborative network model adopts a deep supervision mechanism. The loss function combines the pixel-level binary cross entropy loss and the intersection-over-union loss based on the overall mapping. The global level loss L for heterogeneous features is also introduced in the loss function. g and pixel-level loss L p , the overall loss L of the heterogeneous feature collaborative network model all As shown below: L all =L+L g +L p ; Among them, G represents the true value, L IoU (·) represents the loss function based on intersection-over-union.

5. An optical remote sensing salient target detection system based on heterogeneous feature collaboration, characterized in that: include: The heterogeneous feature extraction module is configured to: perform heterogeneous feature extraction on a given optical remote sensing image, including: extracting features from the given optical remote sensing image using a convolutional neural network to obtain local detail features of the image extracted by different convolutional layers; introducing a Transformer network layer in the deep layer of the convolutional neural network to obtain global semantic features of the image extracted by different Transformer network layers; The module for acquiring image global semantic features and image local detail features is configured to: fuse the image global semantic features and image local detail features extracted at different stages; The salient object detection map acquisition module is configured to: fuse the features of different stages by constructing a saliency-guided attention-enhanced decoder to obtain the final salient object detection map; The optical remote sensing salient target detection method based on heterogeneous feature collaboration is implemented by a trained heterogeneous feature collaboration network model; the heterogeneous feature collaboration network model includes a hybrid heterogeneous encoder, an adaptive global-local information aggregation module, and a saliency-guided attention enhancement decoder; By constructing a hybrid heterogeneous encoder, heterogeneous features are extracted from a given optical remote sensing image; including: The VGG-16 network model is used to capture local information in image features and obtain different features in five stages, that is, a set of visual features of different scales is recorded as f v ; The five feature extraction layers of the VGG-16 network model, namely the five VggBlock modules, use a 3*3 convolution kernel to respond to subtle changes in the input image, capture local detail information in the image features, and obtain a set of visual features denoted as f v ; In the third to fifth stages of the VGG-16 network model, the last three Transformer network layers of the Swin-Transformer network model are introduced to achieve a comprehensive analysis of the entire image area through the self-attention mechanism, thereby capturing the global semantic information in the image features and obtaining another set of visual features denoted as f s ; By building an adaptive global-local information aggregation module, the global semantic features of the image and the local detail features of the image extracted at different stages are fused; including: First, perform feature alignment of the two sets of heterogeneous features in the global dimension; Then, the two sets of features aligned in the global dimension are aligned in the local dimension; Fusing the complementary heterogeneous features after feature alignment; First, the pooling operation is used to perform a highly generalized representation of the global information of the two sets of heterogeneous features, namely the global representation, as shown in the following formula: f v,g =GAP c (Conv(f v )); f s,g =GAP c (Conv(f s )); Among them, Conv(·) represents the basic convolution module, which includes a 3×3 convolution layer, a batch normalization layer and a RELU activation function; GAPc(·) represents the global average pooling operation in the channel dimension; f v,g and f s,g They are the global representations of the features extracted by the convolutional neural network and the Transformer network layers respectively; Using mean square error loss to achieve feature f v,g and f s,g Alignment in the global dimension to exclude irrelevant features; global level loss L g The definition of is as follows: L g =L MSE (in v,g ,f s,g ); Among them, L MSE (·) represents the mean square error loss function; The global feature representation after feature alignment is used to guide the original visual features, as shown in the following formula: f ga,v =Conv(Cat(f v,g ,f s ))+f v ; f ga,s =Conv(Cat(f s,g ,f s ))+f s ; Among them, Cat(·) is the splicing operation; f ga,v and f ga,s are heterogeneous features after global alignment; Calculate the correlation A between the two sets of heterogeneous feature pixels and pixels after the original visual features are guided v,s , as shown below: A v,s =A v→s +A s→v ; in, represents the matrix multiplication operation, softmax(·) represents the softmax function, and flatten(·) represents the operation of flattening a two-dimensional matrix into a one-dimensional vector; The ground truth G is downsampled to obtain G′, and then matrix multiplication is applied to determine the pixel correlation A of the ground truth. gt , as shown below: The binary cross entropy loss function is used to calculate the pixel correlation A of the true value G. gt and pixel correlation of heterogeneous features A v,s Consistency constraints are imposed to align heterogeneous features in the local dimension to eliminate the noise caused by feature heterogeneity; the pixel-level loss L in this process p The definition of is as follows: L p =L BCE (A gt ,A v,s ); Among them, L BCE (·) is the binary cross entropy loss function; The original visual features are guided by the correlation of aligned pixels, as shown in the following formula: The aligned feature f a,v and f a,s As the input of the next encoder, continue to extract features; at the same time, the feature f after feature alignment a,v and f a,s The fusion output is f he , used as input to the saliency-guided attention-enhanced decoder: f he =Conv(Cat(f a,v ,f a,s )); The output of the last adaptive global-local information aggregation module Send it to the ASPP module to get the feature f aspp ,The ASPP module sets five convolution branches in parallel, and realizes the extraction of multi-scale features by changing the expansion rate to locate the significant target objects; By constructing a saliency-guided attention-enhanced decoder, the features of different stages are fused to obtain the final salient object detection map; including: Apply the channel enhancement module to the output features of the current coding layer and the output features of the previous decoder layer, respectively, to filter out the important channel features in the visual features and suppress the irrelevant channel features; obtain the channel-enhanced encoder output features and the channel-enhanced previous decoder output features respectively; Applying spatial enhancement modules to the channel-enhanced encoder output features and the channel-enhanced upper-layer decoder output features respectively, highlighting the important spatial positions in the visual features and suppressing the unimportant background; and then obtaining the channel-enhanced and spatially enhanced encoder output features and the channel-enhanced and spatially enhanced upper-layer decoder output features respectively; The encoder output features after channel and spatial enhancement are fused with the previous layer decoder output features after channel and spatial enhancement.

Images