Optical Remote Sensing Image Salient Object Detection Method Based on Dual-Branch Progressive Interaction and Multi-Directional Feature Enhancement

Abstract

The present invention discloses a method for detecting salient objects in optical remote sensing images based on dual-branch progressive interaction and multi-directional feature enhancement. The method includes: preprocessing the original dataset; inputting the preprocessed image into a dual-branch progressive interaction encoder for feature extraction, inserting a large windmill convolution in the CNN branch to capture complex shape features, and using a progressive interaction mechanism to achieve cross-branch interaction and weight adjustment; inputting the extracted features into a semantic association enhancement module, generating semantic association weights through a context-aware attention mechanism to enhance the semantic consistency of salient objects; inputting the enhanced features into a multi-scale adaptive fusion module, dynamically adjusting the feature fusion ratio, and introducing a deformable pooling operation to avoid the loss of key features; finally, inputting the fused features into a decoder to restore the image resolution through cross-layer connection and generate the salient object detection result. The present invention effectively solves the problems of insufficient feature fusion and poor adaptability to multi-scale objects.

Description

Technical Field

[0001] The present invention relates to an optical remote sensing image salient object detection method based on dual-branch progressive interaction and multi-directional feature enhancement, belonging to the technical field of optical remote sensing image processing and analysis. Background Technique

[0002] Optical remote sensing image salient object detection, hereinafter referred to as ORSI-SOD, aims to accurately detect the most salient objects or regions in optical remote sensing images, and can provide an important preprocessing method for other optical remote sensing image tasks. However, since optical remote sensing images are taken from a downward perspective by cameras on remote sensing satellites or airplanes and the imaging is affected by the climate, ORSI-SOD faces challenges such as blurred object boundaries, complex backgrounds, and large differences in object scales. Therefore, the detected salient objects often have problems of being inaccurate and incomplete.

[0003] In 2019, Li et al. published the first paper on deep learning in remote sensing saliency detection, constructed the first open-source dataset ORSSD, and proposed LVNet. In this model, the L-shaped dual-stream pyramid module is responsible for multi-scale feature extraction, and the V-shaped nested encoder-decoder module highlights the salient objects and suppresses the background. Zhang et al. proposed attention dense propagation for more efficient information transmission, expanded the first dataset and named it EORSSD, introduced edge supervision, and proposed DAFNet. This model realizes the attention information transmission from shallow to deep, infers the saliency map and the edge map simultaneously, and enhances the edge information. Tu et al. published a more comprehensive RSI dataset, namely ORSI-4199. They creatively designed a multi-scale joint boundary and region model. Due to the development of deep learning and the accessibility of extensive datasets such as ORSSD, EORSSD, and ORSI-4199, significant progress has been made in the field of ORSI-SOD. Cong et al. constructed a relational inference model for ORSI-SOD using a parallel multi-scale attention mechanism. Zhou et al. utilized edge information to generate object location cues at each decoding step. In addition, some networks were inspired by Transformer and integrated the long-range self-attention mechanism into ORSI-SOD, thereby enhancing the representation of global information. Gao et al. designed an adaptive spatial tokenization Transformer encoder to extract global-local features. Bao et al. took a different route and proposed a dual-branch architecture combining CNN and Transformer, promoting the interactive guided aggregation of multi-level global and local features. In addition, some studies aimed to construct lightweight models such as FSMINet, CorrNet, and SeaNet.

[0004] Although significant progress has been made in the field of salient object detection in remote sensing images in recent years. However, the existing methods still have the following problems when dealing with complex backgrounds and multi-scale objects:

[0005] Insufficient feature extraction and fusion: Traditional CNNs are difficult to capture the global features of objects, while Transformers are weak in local detail extraction. In addition, when existing methods extract features, they often cannot make full use of global context information and have insufficient adaptability to complex backgrounds. At the same time, existing methods fail to effectively fuse multi-scale features, resulting in a decrease in detection accuracy.

[0006] Multi-scale object detection problem: Objects in optical remote sensing images may be at different scales. Traditional convolutional neural networks have limitations in capturing multi-scale information and are difficult to effectively process objects of different scales. Summary of the Invention

[0007] The present invention aims to solve the above problems and proposes an optical remote sensing image salient object detection method based on dual-branch progressive interaction and multi-directional feature enhancement, which specifically includes the following steps:

[0008] S1: Cut the original dataset of optical remote sensing image salient object detection into non-overlapping sub-images according to a specified size, and divide it into a training set, a validation set, and a test set; perform data augmentation before inputting into the network; the specific process is as follows:

[0009] S1.1: Cut the original images in the dataset into pictures of size 224×224 to adapt to the network input size;

[0010] S1.2: Use the stratified sampling method to divide it into a training set, a validation set, and a test set according to different ratios;

[0011] S1.3: Perform random horizontal flipping, random vertical flipping, random rotation, and random cropping operations on the training set images. When randomly rotating, use bilinear interpolation for resampling. When randomly cropping, ensure that the overlap rate between the central area of the cropped image and the original target area is not less than 50%;

[0012] S1.4: Normalize the images and map the pixel values to the interval [0, 1];

[0013] S2: Input the input images into a dual-branch progressive interaction encoder for feature extraction; insert large windmill convolutions in the backbone network to dynamically adjust the convolution kernels and increase the receptive field; use the progressive interaction mechanism to perform cross-branch interaction and dynamic weight adjustment on the features of the two branches;

[0014] S3: Input the features extracted by the dual-branch progressive interaction encoder into the semantic association enhancement module, calculate the similarity between local and global features, generate semantic association weights, and enhance the semantic consistency of salient objects; add a directional convolution unit to enhance the perception ability of the direction of salient objects; introduce a context-aware attention mechanism to highlight salient object features;

[0015] S4: Input the features output by the semantic association enhancement module into the multi-scale adaptive fusion module to dynamically adjust the fusion ratio of features at different scales; introduce deformable pooling operation to adaptively adjust the pooling kernel and avoid the loss of key features; add a directional convolution unit to refine the object boundary;

[0016] S5: Input the features output by the multi-scale adaptive fusion module into the decoder, and fuse the features of the encoder and decoder through cross-layer connection;

[0017] S6: Train the model on the training set and save the best model parameters; input the test set images into the model to obtain the salient object detection results.

[0018] Further, the dual-branch progressive interaction encoder is specifically as follows: Insert a windmill convolution in the backbone network of the CNN branch to extract shallow features in the convolutional layer stage; the windmill convolution adopts an asymmetric padding strategy to capture the distribution characteristics of the target; by dynamically adjusting the shape and size of the convolutional kernel, the receptive field is significantly increased; the Transformer branch is used to extract global context features; cross-branch interaction nodes are set in the CNN branch and the Transformer branch for feature interaction; among them, the CNN branch contains five layers, and the Transfomer branch contains three layers. Feature interaction starts from the third layer of the CNN branch and is carried out layer by layer. First, use the progressive interaction mechanism to perform cross-branch interaction and dynamic weight adjustment on the features of the two branches; secondly, the fourth layer of the CNN branch interacts with the second layer of the Transfomer branch. At this time, the operation of the first time needs to be repeated; finally, the fifth layer of the CNN branch interacts with the third layer of the Transfomer branch. At this time, no dynamic weight adjustment is required, and the features are directly input into the semantic association enhancement module.

[0019] Further, the progressive interaction mechanism is specifically as follows: Introduce a cross-attention mechanism in the cross-branch interaction node, generate attention weights by calculating the similarity between the features of the CNN branch and the Transfomer branch, and dynamically adjust the feature fusion ratio; introduce a dynamic branch weight adjustment mechanism, analyze the global features of the image through the global feature evaluation module, generate a weight adjustment signal, and dynamically adjust the weights of the CNN branch and the Transfomer branch.

[0020] Furthermore, the semantic association enhancement module is specifically as follows: adding a directional convolution unit, which extracts directional features and enhances the feature expression ability through its multi-directional parallel convolution operation; enhancing the perception ability of the target direction by aggregating convolution features in different directions; introducing a context-aware attention mechanism, which incorporates context semantic information while considering the spatial relationship between pixels, takes a 5×5 neighborhood of pixels centered on the current pixel, calculates the sum of the absolute values of the differences, and obtains the semantic association weight through dot product operation and Softmax function, focusing on significant target features and suppressing background interference.

[0021] Furthermore, the multi-scale adaptive fusion module is specifically as follows: dynamically adjusting the fusion ratio of features at different scales according to the feature map resolution and the average activation intensity of the target. For an image containing targets of multiple scales, different scale features are combined through a weighted fusion mechanism; during the feature fusion process, a deformable pooling operation is introduced, and the shape and position of the pooling area are adaptively adjusted by learning the offset to avoid the loss of key features and provide a more accurate feature representation; adding a directional convolution unit, which extracts the target boundary features through convolution kernels in different directions, refines the target boundary, and provides features with refined boundaries and high distinctiveness for the decoder.

[0022] Compared with the existing technologies, the present invention has the following beneficial effects:

[0023] The present invention proposes a dual-branch progressive interaction encoder structure based on dual-branch progressive interaction and multi-directional feature enhancement. By combining the large windmill convolution of the CNN branch and the self-attention mechanism of the Transformer branch, it not only utilizes the powerful ability of the CNN to capture local detailed features but also gives full play to the effective perception advantage of the Transformer for global context information. Through cross-branch interaction and dynamic weight adjustment, it can adaptively balance the fusion of local and global features in different scenarios, achieve multi-scale feature capture from tiny target details to large-area scene patterns, and significantly improve the feature extraction ability for various targets in complex optical remote sensing images.

[0024] The present invention constructs a semantic association enhancement module, which effectively improves the semantic correlation between local and global features. This module calculates the similarity between local and global features through the attention mechanism and contrast learning, generates semantic association weights, focuses on significant target features, and suppresses background interference. At the same time, a context-aware attention mechanism is added, which takes the neighborhood pixels centered on the current pixel to calculate the sum of the absolute values of the differences, and obtains the semantic association weight through dot product operation and Softmax function, further highlighting the significant target features. This series of operations ensures the semantic consistency between features at different scales and levels, reduces the semantic gap between features, and improves the understanding and detection accuracy of the model for significant targets.

[0025] The multi-scale adaptive fusion module designed in the present invention realizes the dynamic fusion and processing of features at different scales. This module uses the resolution of the feature map and the average activation intensity of the target to judge the size of the target scale, and dynamically adjusts the fusion ratio of features at different scales. For images containing multiple-scale targets, different-scale features are combined through a weighted fusion mechanism, giving full play to the structural representation ability of large-scale features for large target regions and the ability of small-scale features to depict small target details. In addition, the deformable pooling operation incorporated adapts the shape and position of the pooling region by learning the offset amount, avoiding the loss of key features, providing a more accurate feature representation, enhancing the model's detection ability for multi-scale and irregularly shaped targets, and improving the adaptability and detection accuracy in complex scenarios.

[0026] The present invention combines multiple training strategies, effectively improving the performance and generalization ability of the model. End-to-end training combines cross-entropy loss, Dice loss, and IoU loss to construct a total loss function, comprehensively considering multiple aspects of object detection and optimizing model training. The AdamW optimizer is selected and combined with the cosine annealing learning rate strategy to achieve a balance between fast convergence and fine optimization during training. At the same time, adversarial training enhances the robustness of the model, multi-task collaborative training improves the accuracy and integrity of detection, and self-supervised learning uses a large amount of unlabeled data to learn general image features during the pre-training stage, enabling the model to better adapt to different optical remote sensing image scenarios, enhancing the generalization ability of the model, and reducing the dependence on specific datasets. Brief Description of the Drawings

[0027] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present invention will become more apparent:

[0028] Figure 1 It is a flowchart of the steps of an optical remote sensing image salient object detection method based on dual-branch progressive interaction and multi-directional feature enhancement provided by the present invention;

[0029] Figure 2 It is a network structure diagram of an optical remote sensing image salient object detection method based on dual-branch progressive interaction and multi-directional feature enhancement provided by the present invention;

[0030] Figure 3 It is a structure diagram of the semantic association enhancement module;

[0031] Figure 4 It is a structure diagram of the multi-scale adaptive fusion module. Detailed Description of the Embodiment

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be described in detail below in conjunction with specific embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] As Figure 1 shown, it is a step flowchart of an optical remote sensing image salient object detection method based on dual-branch progressive interaction and multi-directional feature enhancement provided by the present invention, including steps S1-S6:

[0034] S1, Cut the original dataset of optical remote sensing image salient object detection into non-overlapping sub-images according to a specified size, and divide it into a training set, a validation set, and a test set. Data augmentation is performed before inputting into the network.

[0035] In the example of the present invention, step S1 specifically includes steps S11-S13:

[0036] S11, Cut the original images of the ORSSD, EORSSD, and ORSI-4199 datasets into non-overlapping sub-images according to a specified size, with a size of 224×224 pixels, to adapt to the network input size;

[0037] S12, Divide the cut images into a training set, a validation set, and a test set according to different ratios. For the ORSSD dataset, divide it into a training set, a validation set, and a test set according to the ratio of 600 / 100 / 100; for the EORSSD dataset, divide it into a training set, a validation set, and a test set according to the ratio of 1500 / 250 / 250; for the ORSI-4199 dataset, divide it into a training set, a validation set, and a test set according to the ratio of 3000 / 600 / 599;

[0038] S13, Perform random horizontal flipping, random vertical flipping, random rotation, that is, the angle range is [-30°, 30°], and random cropping operations on these datasets before inputting into the network. When randomly cropping, ensure that the overlap rate between the central area of the cropped image and the original target area is not less than 50%. Finally, normalize the image and map the pixel values to the [0, 1] interval.

[0039] S2, Input the preprocessed image into the dual-branch progressive interaction and multi-directional feature enhancement network, and use the progressive interaction mechanism to perform cross-branch interaction and dynamic weight adjustment on the features of the two branches;

[0040] As Figure 2 shown, it is a structure diagram of a dual-branch progressive interaction and multi-directional feature enhancement network provided by the present invention. The backbone network is divided into a CNN branch and a Transformer branch. The specific steps of step S2 include S21 - S24:

[0041] S21, CNN Branch Feature Extraction: The first layer is at the beginning of ResNet50 (after the conv1 layer) and performs a specially designed windmill convolution. First, two consecutive windmill convolution operations are carried out. The convolution kernel size of each windmill convolution layer is 5×5, the stride is 1, the padding is 2, and the dilation rate is 1. The windmill convolution operation can be expressed as:

[0042] ,

[0043] where, w m,n is the weight of the original convolution kernel, is the weight of the convolution kernel after rotating 45 degrees. In this way, the process of the windmill convolution extracting features from multiple directions is simulated.

[0044] After two windmill convolutions, the image resolution is reduced from 224×224 to 112×112, and the number of channels is increased from 3 to 64. The windmill convolution can better capture complex features such as edges and textures, and can extract the unique information of the target more effectively than ordinary convolutions.

[0045] The second layer is processed using two ordinary convolutions with a convolution kernel size of 3. These two convolution layers act in sequence, reducing the feature map from 112×112 to 56×56 and increasing the number of channels from 64 to 128. Further extract the features of the image, refine and integrate the features extracted by the previous layer of windmill convolution, and provide a more representative feature map for the subsequent layers.

[0046] The third layer is at the middle layer of ResNet50, that is, the conv3_x position, and performs two windmill convolution operations. The convolution kernel size, stride, padding, and dilation rate are the same as those of the windmill convolution in the first layer. The feature map is reduced from 56×56 to 28×28, and the number of channels is increased from 128 to 256. For the feature map after the first two layers of convolution processing, the windmill convolution further captures the features of complex-shaped targets and enhances the model's perception ability of target details.

[0047] The fourth layer uses two ordinary convolutions with a convolution kernel size of 3 to reduce the feature map from 28×28 to 14×14 and increase the number of channels from 256 to 512, further integrating the features and deeply integrating various features extracted before. The fifth layer uses two ordinary convolutions with a convolution kernel size of 3 again to output a feature map with high dimension and high resolution, providing rich information for subsequent cross-branch interaction and feature fusion.

[0048] S22, Transformer Branch Feature Extraction: Use Swin Transformer as the backbone network of the Transformer branch and process the input 224×224 image as follows: In the first layer, perform a dimensionality increase operation on the feature map using a convolutional operation with a convolutional kernel size of 7, padding of 3, and stride of 4. Let the convolutional kernel be Kconv1, then the convolutional operation can be expressed as:

[0049] ,

[0050] After this convolution, the dimension size of the feature map becomes 64, and the feature map scale is 56×56. Then, perform layer normalization on the feature map to stabilize the distribution of the features. The layer normalization formula is:

[0051] ,

[0052] Among them, and are the mean and variance of the feature map in the channel dimension respectively, is a very small constant to prevent the denominator from being 0, and are learnable parameters.

[0053] Perform multi-head self-attention operation. In the multi-head self-attention mechanism, the formula for calculating the attention score is:

[0054] ,

[0055] Among them, Q = W Q F LN , K = W K F LN , V = W V F LN , W Q , W K , W V are linear transformation matrices, F LN is the feature map after layer normalization, d kis the dimension of the key. Calculate the correlation between the features at each position and the features at all other positions to obtain global context information. Add the resulting feature map after the operation to the feature map without multi-head self-attention. This residual connection helps to retain important information in the original features and avoid losing key details during the self-attention operation. That is:

[0056] ,

[0057] Then, perform layer normalization and multi-layer perceptron operations in sequence. The multi-layer perceptron can be expressed as:

[0058] ,

[0059] where W 1 , W 2 are weight matrices, b 1 , b 2 are bias terms. In this way, the features are further transformed and integrated to enhance the expressive power of the features.

[0060] The second layer uses a convolution with a kernel size of 3 and a padding of 1 to change the feature map from 56×56 to 28×28 and increase the number of channels from 64 to 128. During this process, the convolution operation not only changes the size and number of channels of the feature map but also further extracts feature information at different scales.

[0061] The third layer uses a convolution with a kernel size of 3 and a padding of 1 to change the feature map from 28×28 to 14×14 and increase the number of channels from 128 to 256. Further process the feature map to obtain a more refined feature representation. Then perform multi-head self-attention operation, and use the self-attention mechanism again to capture global information, strengthen the correlation between features, and improve the model's understanding ability of complex scenarios.

[0062] S23, Cross-branch interaction node: In the convolutional layers of the CNN branch, namely conv3_x, conv4_x, and the first and second layers of the Transformer branch, cross-branch interaction nodes are set, and feature interaction is performed for each layer. A cross-attention mechanism is introduced in the cross-branch interaction node to calculate the similarity between the features of the CNN branch and the Transformer branch, generate attention weights, and dynamically adjust the feature fusion ratio. The specific calculation process is as follows: The feature maps from the two branches are respectively linearly transformed to obtain Q, K, and V vectors. The attention weights are obtained by calculating the dot product of Q and K and performing the Softmax operation, and then the attention weights are weighted and summed with V to achieve dynamic adjustment of the feature fusion ratio. In this way, the features of the two branches complement each other and give full play to their respective advantages.

[0063] S24, Dynamic weight adjustment: A dynamic branch weight adjustment mechanism is introduced. The global feature evaluation module analyzes the global features of the image to generate a weight adjustment signal and dynamically adjusts the weights of the CNN branch and the Transformer branch. The global feature evaluation module consists of 3 convolutional layers, ReLU activation, and 2 fully connected layers, and is used to extract the scene complexity index and target scale information. When calculating the texture complexity using the gray-level co-occurrence matrix, for a grayscale image I , the gray-level co-occurrence matrix represents the frequency of occurrence of pixel pairs with gray values d and at a distance of i and direction of j . The texture complexity T can be calculated by the following formula:

[0064] ,

[0065] where L is the number of gray levels.

[0066] According to the evaluation results, a weight adjustment signal is generated and transmitted to the weight control nodes of the CNN branch and the Transformer branch respectively to dynamically adjust the branch weights. When detecting an image containing multiple-scale targets, according to the target scale information, the weight of the CNN branch, which is more beneficial for detecting small targets, is increased, or the weight of the Transformer branch, which is more effective for large target and global scene perception, is enhanced, so that the model can more efficiently utilize the features of the two branches in different scenarios.

[0067] S3, Input the features extracted by the dual-branch progressive interaction encoder into the semantic association enhancement module.

[0068] Such as Figure 3As shown in the figure, the present invention provides a structural diagram of a semantic association enhancement module for an embodiment, and the specific steps of step S3 include S31 - S33.

[0069] S31, Directional convolution to enhance feature representation: Add a directional convolution unit to extract and enhance the directional features of the double-branch salient object features. For directional convolution, assume the input feature map is F , the k th directional convolution kernel is K k , then the output of the directional convolution is:

[0070] ,

[0071] Perform convolution operations on the feature map through these convolution kernels in different directions to capture the direction-sensitive features of the salient object and enhance the feature representation ability; splice the 8-direction feature maps of the double-branch, and compress the number of channels through 1×1 convolution;

[0072] S32, Generate Q, K, V vectors, and generate attention vectors based on the fused features;

[0073] S33, Context-aware attention mechanism: Introduce a context-aware attention mechanism, take a 5×5 neighborhood pixel centered on the current pixel, calculate the sum of the absolute values of the differences, and obtain the semantic association weight through dot product operation and Softmax function. Assume the current pixel value is, the neighborhood pixel value is p , the neighborhood pixel value is p ij , The sum of the absolute values of the differences D is:

[0074] ,

[0075] Assume the global feature vector is G , the semantic association weight β is obtained through the following calculation:

[0076] ,

[0077] In this way, focus on the salient object features, suppress background interference, and enhance the semantic relevance between local and global features.

[0078] S4, Input the features output by the semantic association enhancement module into the multi-scale adaptive fusion module for multi-scale feature fusion.

[0079] Such as Figure 4As shown, the present invention provides a structural diagram of a multi-scale adaptive fusion module for an embodiment, and the specific steps of step S4 include S41 - S45.

[0080] S41, target scale judgment: Using the feature map resolution: the width and height can be directly obtained; the target average activation intensity: calculating the average pixel value of the target area, and the target area is obtained by segmenting through a threshold of 0.5, to judge the size of the target scale. Let the feature map resolution be (H, W) , and the target area average activation intensity be A , and define the scale judgment function S(H, W, A) :

[0081] ,

[0082] Among them, T 1 , T 2 , T 3 , T 4 are thresholds set according to experiments. By analyzing the resolution of the feature map, the approximate scale range of the target can be initially determined; combined with the target average activation intensity, the judgment of the target scale can be further refined. For areas with low resolution but high average activation intensity, they may correspond to small-scale but significant targets; while areas with high resolution and moderate average activation intensity may contain large-scale targets. In this way, it provides a basis for subsequent scale feature fusion.

[0083] S42, scale feature fusion: Dynamically adjust the fusion ratio of different scale features. For an image containing multiple scale targets, combine different scale features through a weighted fusion mechanism. Let the large-scale feature map be F large , and the small-scale feature map be F small . According to the target scale judgment result, the fused feature map F fusion is:

[0084] ,

[0085] Among them, ω 1 and ω 2 are adaptively generated through an attention mechanism, and ω 1 + ω 2= 1. According to the target scale judgment result, different weights are assigned to feature maps of different scales. For large-scale targets, the weight of large-scale feature maps is increased to highlight their overall structural information; for small-scale targets, the weight of small-scale feature maps is increased to retain their detailed features. Through this weighted fusion method, the advantages of features of different scales are fully utilized to improve the detection accuracy of multi-scale targets.

[0086] S43, deformable pooling operation: During the feature fusion process, a deformable pooling operation is incorporated. Deformable pooling adaptively adjusts the shape and position of the pooling region by learning the offset, avoiding the loss of key features and providing a more accurate feature representation. The sampling point offset of the deformable pooling module is predicted by a small convolutional network consisting of 2 convolutional layers and ReLU activation. Let the predicted offset be and For the input feature map F the output of the deformable pooling F def is:

[0087] ,

[0088] When performing the pooling operation on feature maps of different scales according to the predicted offset, the pooling region can be adaptively adjusted to better adapt to the shape and size changes of the target, effectively retaining the key features of the target. Deformable pooling can dynamically adjust the pooling region according to the boundary shape of the image, avoiding the loss of edge details.

[0089] S44, directional convolution to refine the boundary: Add a directional convolution unit to perform convolution operations on the fused feature map through these directional convolution kernels to further highlight the target boundary features and refine the target boundary. The directional convolution kernels detect and strengthen the target boundary from multiple directions, providing features with refined boundaries and high discrimination for the decoder, which helps to improve the accuracy and clarity of the target boundary; add a residual connection after the directional convolution to avoid the loss of feature information;

[0090] S45, output the fused feature map: Output the feature map after multi-scale adaptive fusion, deformable pooling, and directional convolution processing for subsequent decoder and output layer processing. These fused feature maps contain multi-scale, directional, and boundary information, providing strong support for the decoder to accurately recover the target image.

[0091] S5, Input the features output by the multi-scale adaptive fusion module into the decoder, and fuse the features of the encoder and decoder through cross-layer connections to generate significant target detection results. Step S5 specifically includes steps S51 - S54.

[0092] S51, Decoder Upsampling and Convolution Operations: The decoder consists of 4 layers of upsampling and transposed convolution operations to gradually restore the image resolution. A directional convolution unit is added after each layer of upsampling. Through upsampling and transposed convolution operations, the low-resolution feature map is gradually restored to a high-resolution image; the directional convolution unit further refines the target boundaries and features, enabling the restored image to more accurately present the details of the target;

[0093] S52, Cross-Layer Connection Feature Fusion: Cross-layer connections are set in the decoder to fuse the features of different levels in the encoder with the corresponding-level features in the decoder. Through cross-layer connections, the features extracted at different levels in the encoder are concatenated with the corresponding-level features in the decoder in the channel dimension and then fused through a 1×1 convolutional layer. This cross-layer connection method can make full use of information at different levels, enabling the decoder to better reconstruct the shape and structure of the target during the process of restoring the image resolution, and improving the accuracy and integrity of target detection;

[0094] S53, Multi-Task Output: The output layer uses multi-task output to output significant target detection results and target category information using convolutional layers. By setting different convolutional kernels and classifiers in the output layer, the fused feature map is processed to output the position, contour information, and the category to which the target belongs;

[0095] S54, Result Output and Saving: The significant target detection results obtained from the output layer are visually displayed and saved. The detection results are displayed in the form of an image using an image processing library to intuitively present the detection effect of the model; at the same time, the detection results are saved as files for subsequent analysis and evaluation.

[0096] S6, Train the model on the training set and save the best model parameters. Input the test set images into the model to obtain significant target detection results.

[0097] In the example of the present invention, an end-to-end training method is adopted, and the total loss function is constructed by combining cross-entropy loss, Dice loss, and IoU loss:

[0098] ,

[0099] The AdamW optimizer is selected, and the initial learning rate is set to 7×10 -5, the weight decay coefficient is 0.0001. The cosine annealing learning rate strategy is used to dynamically adjust the learning rate during training. An adversarial network is constructed. The generator consists of 5 transposed convolutional layers, that is, convolutional kernels of 4×4, stride 2, padding 1, ReLU activation, and the last layer with Tanh activation. The discriminator consists of 5 convolutional layers, LeakyReLU activation, and the last layer with Sigmoid activation. The Wasserstein GAN loss is used for adversarial training to enhance the robustness of the model. An object classification task head is added, that is, it contains 2 fully connected layers, with 256 and 128 nodes in the hidden layer, and Softmax activation; an edge detection task head, that is, it contains 3 convolutional layers and 1 transposed convolutional layer, ReLU activation, and the last layer with Sigmoid activation. The object classification loss and the edge detection loss use cross-entropy loss and Dice loss respectively to achieve multi-task collaborative training and improve the accuracy and integrity of detection. A self-supervised learning task is introduced in the pre-training stage, and a large amount of unlabeled data is used to improve the feature representation ability of the model. During the training process, the model is evaluated on the validation set every 5 epochs, the performance metrics of the model are recorded, and the model with the best performance is selected for saving.

[0100] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be combined arbitrarily without conflict.

Claims

1. An optical remote sensing image salient object detection method based on dual-branch progressive interaction and multi-directional feature enhancement, characterized in that It includes the following steps: S1: Cut the original dataset of optical remote sensing image salient object detection into non-overlapping sub-images according to a specified size, and divide it into a training set, a validation set, and a test set; perform data augmentation before inputting it into the network. The specific process is as follows: S1.1: Cut the original images in the dataset into pictures of size 224×224 to adapt to the network input size; S1.2: Use the stratified sampling method to divide it into a training set, a validation set, and a test set according to different ratios; S1.3: Perform random horizontal flipping, random vertical flipping, random rotation, and random cropping operations on the training set images. When randomly rotating, use bilinear interpolation for resampling. When randomly cropping, ensure that the overlap rate between the central area of the cropped image and the original target area is not less than 50%; S1.4: Normalize the images, mapping the pixel values to the interval [0, 1]; S2: Input the input images into a dual-branch progressive interaction encoder for feature extraction; insert large windmill convolutions in the backbone network to dynamically adjust the convolution kernels and increase the receptive field; use the progressive interaction mechanism to perform cross-branch interaction and dynamic weight adjustment on the features of the two branches; S3: Input the features extracted by the dual-branch progressive interaction encoder into the semantic association enhancement module, calculate the similarity between local and global features, generate semantic association weights, and enhance the semantic consistency of salient objects; add a direction convolution unit to enhance the perception ability of the direction of salient objects; introduce a context-aware attention mechanism to highlight salient object features; The specific steps of S3 include: S31, Directional Convolution to Enhance Feature Representation: Add a directional convolution unit to extract and enhance the directional features of the double-branch salient object features; for directional convolution, assume the input feature map is F , the k -th directional convolution kernel is K k , then the output of the directional convolution is: ， Perform convolution operations on the feature maps through these convolution kernels in different directions to capture the direction-sensitive features of salient objects and enhance the feature expression ability; splice the 8-direction feature maps of the two branches and compress the number of channels through 1×1 convolution; S32, generate Q, K, V vectors, and generate an attention vector based on the fused features; S33, Context-Aware Attention Mechanism: Introduce the context-aware attention mechanism. Take the 5×5 neighborhood pixels centered on the current pixel, calculate the sum of the absolute differences, and obtain the semantic association weights through dot product operation and Softmax function. Let the current pixel value be, the neighborhood pixel value be p , the neighborhood pixel value be p ij , The sum of the absolute differences D is:[[]] ， Let the global feature vector be G , and the semantic association weight β is obtained through the following calculation: ， In this way, focus on salient object features, suppress background interference, and enhance the semantic correlation between local and global features; S4: Input the features output by the semantic association enhancement module into the multi-scale adaptive fusion module to dynamically adjust the fusion ratio of features at different scales; introduce deformable pooling operations to adaptively adjust the pooling kernels to avoid loss of key features; add a direction convolution unit to refine the object boundaries; The specific steps of S4 include: S41, Target scale judgment: Using the feature map resolution: The width and height can be directly obtained; The average activation intensity of the target: Calculate the average pixel value of the target area, and the target area is obtained by segmenting through a threshold of 0.5, to judge the size of the target scale; Let the feature map resolution be (H,W) , and the average activation intensity of the target area be A , define the scale judgment function S(H,W,A) : ， Among them, T 1 , T 2 , T 3 , T 4 are thresholds set according to experiments; by analyzing the resolution of the feature map, the approximate scale range of the target can be initially determined; combined with the average activation intensity of the target, the judgment of the target scale can be further refined; for regions with low resolution but high average activation intensity, they may correspond to small-scale but significant targets; while regions with high resolution and moderate average activation intensity may contain large-scale targets; in this way, a basis is provided for subsequent scale feature fusion; S42, Scale Feature Fusion: Dynamically adjust the fusion ratio of features at different scales. For an image containing objects of multiple scales, combine features at different scales through a weighted fusion mechanism. Let the large-scale feature map be F large , and the small-scale feature map be F small . According to the target scale judgment result, the fused feature map F fusion is: ， Among them, ω 1 and ω 2 are adaptively generated by the attention mechanism, and ω 1 + ω 2 = 1; according to the target scale judgment result, different weights are assigned to feature maps of different scales; for large-scale targets, the weight of large-scale feature maps is increased to highlight their overall structural information; for small-scale targets, the weight of small-scale feature maps is increased to retain their detailed features; through this weighted fusion method, the advantages of different-scale features are fully utilized to improve the detection accuracy of multi-scale targets; S43, Deformable Pooling Operation: Incorporate the deformable pooling operation during feature fusion; Deformable pooling adaptively adjusts the shape and position of the pooling region by learning the offsets, avoiding the loss of key features and providing a more accurate feature representation; The sampling point offsets of the deformable pooling module are predicted by a small convolutional network consisting of 2 convolutional layers and ReLU activation; Let the predicted offsets be and For the input feature map F The output of the deformable pooling F def is: ， When performing pooling operations on feature maps at different scales according to the predicted offset, it can adaptively adjust the pooling area, better adapt to the shape and size changes of the target, and effectively retain the key features of the target; deformable pooling can dynamically adjust the pooling area according to the boundary shape of the image to avoid losing edge details; S44, direction convolution to refine the boundary: Add a direction convolution unit, perform convolution operations on the fused feature maps through these direction convolution kernels, further highlight the object boundary features, and refine the object boundaries; the direction convolution kernels detect and strengthen the object boundaries from multiple directions, providing features with refined boundaries and high distinctiveness for the decoder, which helps to improve the accuracy and clarity of the object boundaries; add a residual connection after the direction convolution to avoid loss of feature information; S45, Output the fused feature map: Output the feature map after multi-scale adaptive fusion, deformable pooling, and orientation convolution processing for subsequent decoder and output layer processing; these fused feature maps contain multi-scale, orientation, and boundary information, providing strong support for the decoder to accurately recover the target image; S5: Input the features output by the multi-scale adaptive fusion module into the decoder, and fuse the features of the encoder and decoder through cross-layer connection; S6: Train the model on the training set and save the best model parameters; input the test set images into the model to obtain the significant object detection results.

2. The method for detecting prominent objects in optical remote sensing images based on dual-branch progressive interaction and multi-directional feature enhancement according to claim 1, wherein, Construct the double-branch progressive interaction encoder, including: Insert the windmill convolution in the backbone network of the CNN branch, and use the convolutional layer to extract shallow features; the windmill convolution adopts an asymmetric padding strategy to capture the distribution characteristics of the target; by dynamically adjusting the shape and size of the convolutional kernel, the receptive field is significantly increased; the Transformer branch is used to extract global context features; Set cross-branch interaction nodes in the CNN branch and the Transformer branch for feature interaction; where the CNN branch contains five layers and the Transfomer branch contains three layers. The feature interaction starts from the third layer of the CNN branch and is carried out layer by layer. First, use the progressive interaction mechanism to perform cross-branch interaction and dynamic weight adjustment on the features of the two branches; secondly, the fourth layer of the CNN branch interacts with the second layer of the Transfomer branch, and at this time, the operation of the first time needs to be repeated; finally, the fifth layer of the CNN branch interacts with the third layer of the Transfomer branch, and at this time, no dynamic weight adjustment is required, and the features are directly input into the semantic association enhancement module.

3. The method for detecting salient objects in optical remote sensing images based on dual-branch progressive interaction and multi-directional feature enhancement according to claim 1, wherein, The progressive interaction mechanism specifically refers to cross-branch interaction and dynamic weight adjustment of the double-branch features, including: Introduce the cross-attention mechanism in the cross-branch interaction node, generate attention weights by calculating the similarity of the features of the CNN branch and the Transformer branch, and dynamically adjust the feature fusion ratio; Introduce the dynamic branch weight adjustment mechanism, analyze the global features of the image through the global feature evaluation module, generate weight adjustment signals, and dynamically adjust the weights of the CNN branch and the Transformer branch.

4. The method for detecting salient objects in optical remote sensing images based on dual-branch progressive interaction and multi-directional feature enhancement according to claim 1, wherein, Construct the semantic association enhancement module, including: Add an orientation convolution unit, extract directional features through its multi-direction parallel convolution operation to enhance the feature expression ability; enhance the perception ability of the target direction by aggregating convolution features in different directions; Introduce the context-aware attention mechanism, integrate context semantic information while considering the spatial relationship between pixels, take a 5×5 neighborhood pixel centered on the current pixel, calculate the sum of the absolute values of the differences, and obtain the semantic association weight through dot product operation and Softmax function to focus on the significant object features and suppress background interference.

5. The method for detecting salient objects in optical remote sensing images based on dual-branch progressive interaction and multi-directional feature enhancement according to claim 1, wherein Construct the multi-scale adaptive fusion module, including: Dynamically adjust the fusion ratio of features at different scales according to the feature map resolution and the average activation intensity of the target. For images containing targets of multiple scales, combine features at different scales through a weighted fusion mechanism; During the feature fusion process, a deformable pooling operation is introduced to adaptively adjust the shape and position of the pooling region by learning the offsets, avoiding the loss of key features and providing a more accurate feature representation; A directional convolution unit is added to extract the target boundary features through convolutional kernels in different directions, refine the target boundary, and provide features with refined boundaries and high distinctiveness for the decoder.

Images