A salient target detection system and method based on the Swin transformer

Abstract

The application discloses a salient object detection system and method based on a Swin transformer, aiming to solve the problems of insufficient cross-modal feature fusion and inaccurate target boundary positioning in existing methods. The application innovatively designs a cross-modal bidirectional fusion network (CENet), constructs a complete encoder-feature fusion-decoder framework, and focuses on cross-modal deep fusion and edge detail refinement. The framework designs two key modules: a cross-modal bidirectional fusion module (CMBF): through a symmetric cross-attention mechanism, it realizes the bidirectional deep interaction and efficient complementation of RGB and Depth features. An edge-aware refinement module (EAR): through a double-pooling mechanism and coordinate attention, it deeply fuses the low-level features containing accurate boundary information with high-level semantic features and then introduces them into the middle layer of the decoder, significantly improving the target edge positioning capability.

Description

Technical Field

[0001] This invention relates to the field of salient target detection, and in particular to a salient target detection system and method based on the Swin transformer. Background Technology

[0002] The human visual system efficiently processes complex scenes through selective attention mechanisms, prioritizing salient regions rather than uniformly analyzing all pixels. Inspired by this, salient object detection technology has emerged, aiming to automatically identify the most visually appealing target regions in an image.

[0003] The core objective of this technology is to automatically locate and extract the target regions that best represent the core semantics or are most visually attractive from complex images. This provides more focused and efficient feature representations for subsequent image analysis, understanding, and high-level vision tasks, including object segmentation and recognition, medical image segmentation, and image retrieval, thereby improving the overall processing performance of the vision system. In recent years, salient object detection technology has made significant progress, and related algorithms have been widely applied in many computer vision tasks such as autonomous driving and robot navigation. However, when facing complex scenes with high visual similarity between foreground and background, severe target occlusion, complex backgrounds, or extreme lighting conditions, relying solely on a single RGB image modality often fails to achieve robust salient object detection. With the development of depth imaging technology and the reduction in hardware costs, depth sensors have become widespread, making the convenient acquisition of depth maps possible. This technology effectively overcomes many limitations in traditional image processing, providing a new perspective and solution for computer vision research. Against this backdrop, salient object detection based on RGB-D (RGB-Depth) images has gradually become a research hotspot in the field of computer vision, and many researchers have proven that introducing depth images can significantly improve the performance of salient object detection.

[0004] Compared to traditional RGB-based SOD, RGB-D salient object detection, by introducing depth modality, can provide additional spatial and geometric structural information. This information not only helps solve the problem of difficulty in distinguishing foreground from background when color contrast is insufficient or illumination changes drastically, but also assists in object segmentation at the 3D structural level, thus exhibiting stronger robustness in complex scenes. For example, in cases with cluttered backgrounds, complex textures, or foreground and background colors that are similar, RGB features are often insufficient to support accurate saliency inference, while depth information can provide a color-independent discrimination criterion through the spatial differences between the object and the background. Studies by Piao et al. and Liu et al. have shown that RGB images and depth images can provide complementary information; however, effectively utilizing RGB-D complementarity still faces two key limitations: First, existing fusion strategies mostly use simple stitching or weighted averaging, failing to capture the complex cross-modal dependencies between appearance and geometric features. Second, current methods mainly focus on region-level accuracy while neglecting boundary quality, resulting in inaccurate object contours.

[0005] To address these issues, researchers have proposed various solutions, including early feature fusion, multi-level feature aggregation, and attention-based cross-modal fusion. Although these methods alleviate modal differences and feature redundancy to some extent, they still have two shortcomings: (1) insufficient cross-modal interaction makes it difficult to achieve efficient complementarity and utilization of RGB and Depth information; (2) insufficient utilization of boundary information leads to problems such as blurred edges of significant targets.

[0006] Therefore, there is an urgent need to design a detection framework that can effectively enhance cross-modal feature interaction while also taking into account boundary details. Summary of the Invention

[0007] In view of the problems of insufficient cross-modal fusion and blurred target edges in the existing salient target detection technology, this invention proposes a salient target detection system and method based on the Swin transformer.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a salient object detection method based on the Swin transformer includes: Step 1: Construct a two-stream encoder: Use a two-stream Swing Transformer encoder to extract features from the input RGB image and Depth image respectively, to obtain a multi-scale RGB feature set. and Depth feature set ; Step 2: Feature Fusion: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] and Input is used to perform cross-modal bidirectional fusion, generating a multi-scale cross-modal fusion feature set. ; Step 3: Construct a multi-level decoder: As input, the resolution of the feature map is restored step by step. In at least one intermediate layer of the decoder, edge-guided thinning is introduced to enhance and refine the fused features at that level, and the final saliency prediction map is output. Step 4: Loss function calculation: The binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function are used to supervise the network. The binary cross-entropy (BCE) loss function measures the difference between the saliency map predicted by the model and the true saliency map, while the intersection-over-union (IoU) loss function is used to better optimize the prediction results from the perspective of overlap and spatial structure relationship. Step 5: Iterative training: Iteratively train the model built in Steps 1-3. In each training iteration, perform forward propagation to calculate the multi-task loss, then update the model parameters through backpropagation, select the optimal model parameters through validation set performance evaluation, and then input the test image into the trained model to generate a prediction map to achieve salient object detection. Step 6: Validation and Evaluation: Five evaluation metrics are used to evaluate the performance of the model method, including PR curve, S-measure, mean absolute error, F-measure, and E-measure. Step 7: Generalization performance verification: Retrain and test the model built in Step 1-3 on the RGB-Thermal saliency object detection dataset to verify the generalization performance of the model.

[0009] Furthermore, in step 1, a Swin transformer is used as the backbone to extract features from the input RGB and Depth images layer by layer. The low-level features of the RGB and Depth images are first input into the Feature Pyramid (FPN) to obtain four features. , , and The remaining levels obtain features , , and .

[0010] Furthermore, in step 2, the following will be done: and As input, RGB features and depth features are used to act as query, key, and value, respectively. A fine-grained feature correspondence is established through cross-attention operations. After generating bidirectional attention features, attention mechanisms are applied to each branch, and the features are then concatenated and fused to generate a multi-scale cross-modal fusion feature set. .

[0011] Furthermore, in step 2, the following will be done: and As input, RGB features and depth features are used as query, key, and value to each other. A fine-grained feature correspondence is established through cross-attention operations, including: For RGB features and depth features First, attention is obtained from RGB to depth. In this path, depth features are used as the query vector. The RGB features are used as key vectors. Sum value vector Through cross-attention operations, each element of the RGB feature is assigned an attention weight, which reflects its correlation with each element in the deep feature; ultimately, an attention-weighted RGB feature is generated. In another approach, attention is obtained down to the RGB level, with RGB features serving as the query vector. Deep features serve as key vectors. Sum value vector This generates an attention-weighted deep feature. ; indicates as: in, Indicates the source mode, Indicates the target mode, , , Represents the learnable parameter matrix, and represents the cross-attention operation. Indicates the scaling factor. This represents the attention feature calculated using RGB modal information as the query and depth information as the key and value. This represents the attention features calculated using deep modality information as the query and RGB information as the key and value.

[0012] Furthermore, in step 2, an attention mechanism is applied to each branch, and the branches are then spliced ​​and fused to generate a multi-scale cross-modal fusion feature set. include: After generating bidirectional attention features, an attention mechanism is applied to each branch; for the two branches... and First, a 1×1 convolutional layer is used to process and generate... and Then, global average pooling is used to compress the spatial dimension to obtain the channel-level feature vector for each branch. Subsequently, the two resulting vectors are fed into a sigmoid activation function to generate channel attention weight vectors. and The weight vector and Features after 1×1 convolution and Cross-multiplication is performed separately to achieve adaptive weighting of the feature channels. After attention weighting, the features from the two branches are concatenated along the channel dimension, as follows: in This represents a two-dimensional convolution operation with a kernel size of 1×1. and These represent the global average pooling operation and the sigmoid activation function, respectively. This indicates a feature-by-feature multiplication operation. Indicates a splicing operation; The concatenated features are fed into a multi-layer convolutional block consisting of two 3×3 convolutional layers and one 1×1 convolutional layer. The 3×3 convolutional layers perform local feature fusion and information integration in the spatial dimension, while the final 1×1 convolutional layer further adjusts the number of channels and completes the final feature fusion, generating the final fused features. , is represented as: in, This represents a two-dimensional convolution operation with a kernel size of 3×3. This represents a two-dimensional convolution operation with a kernel size of 1×1.

[0013] Furthermore, in step 3, the two feature maps from different levels are... and As input, and For low-level features containing rich boundary details, firstly, bilinear interpolation is used to... Upsampling to Using the same spatial dimensions to achieve feature alignment, the two features are then subjected to convolution for dimensionality adjustment and feature extraction, performing preliminary feature enhancement and dimensionality unification on the original features, as shown below: Wherein represents the upsampling operation; Max pooling highlights the strongest responses in local regions, capturing significant geometric features of edges and corners. Average pooling, on the other hand, maintains the overall distribution of features through global averaging, preserving semantic coherence. For each feature, both pooling operations are applied simultaneously to capture different types of spatial information, as shown below: By fusing these two complementary spatial information through element-wise multiplication, we can simultaneously model local saliency and global consistency. Subsequently, a preliminary fusion of information from two levels is achieved through feature addition. An edge-guided gating mechanism is employed, which adaptively adjusts the gating based on the varying contributions of different feature components to the final edge prediction, highlighting the most critical feature information for edge discrimination. This is represented as follows: in a and b These represent the spatial information obtained from low-level features. Represents the edge information weight graph. This represents the feature information obtained through coordinate attention calculation; Using coordinate attention, spatial structural features of the target are captured by encoding positional information in the horizontal and vertical directions respectively, as follows: in, Indicates the input feature map, This represents the pooling feature along the height direction. This represents the pooling feature along the width direction. This indicates the features after splicing.

[0014] Furthermore, in step 4, the BCE loss function is used to measure the difference between the saliency map predicted by the model and the true saliency map, and the IoU loss function is used to better optimize the prediction results from the perspective of overlap and spatial structure relationship. The final loss function is defined as follows: in The model represents the first time. The predicted saliency map is generated at each scale, and its value is typically between 0 and 1, representing the probability that each pixel is a salient object. This represents a true saliency graph.

[0015] Furthermore, in step 5, during the training and testing phases, the input RGB image and depth image are adjusted to a resolution of 256×256. During training, random flipping, rotation, and boundary cropping are used for data augmentation to prevent overfitting of the training images. The parameters of the backbone network are initialized using two Swin-B pre-trained models, while the remaining parameters use the default settings of PyTorch. The Adam optimizer is used during training, with a batch size of 16 and an initial learning rate of 5e-5. The learning rate is decayed to 1 / 10 of its original value every 50 epochs, and the model converges within 100 epochs.

[0016] Secondly, a detection system employing the Swin transformer-based salient target detection method described above, the detection system comprising a feature pyramid (FPN), a cross-modal bidirectional fusion module (CMBF), and an edge-guided refinement module (EAR); Feature Pyramid Network (FPN) captures low-level feature information, enabling low-level features to express high-level semantic information while being rich in texture and edge information, thus making the feature expression more complete. The cross-modal bidirectional fusion module CMBF is connected to the output of the feature pyramid FPN. A symmetrical bidirectional cross-attention mechanism is designed so that the two modal features can enhance each other. The deep fusion of RGB and depth features is achieved through the design of the bidirectional attention interaction mechanism. Fine feature correspondence is established through cross-attention operation to achieve cross-modal information integration. The Edge-Guided Refinement (EAR) module is connected to the output of the Cross-Modal Bidirectional Fusion (CMBF) module. By deeply fusing low-level features with high-level semantic features and introducing them into the intermediate layer of the decoder, it effectively preserves and utilizes the boundary details in low-level features while maintaining the high-level semantic understanding capability. In addition, considering that the target may exhibit different feature performances at different spatial scales, multiple pooling strategies are adopted to capture multi-scale information and enhance the model's adaptability to complex scenes.

[0017] The beneficial effects of the technical solution of the present invention are as follows: This invention addresses the problem of accurate detection of salient targets by proposing a two-stream network (CENet) based on the Swing Transformer, constructing a complete encoder-fusion-decoder framework. Experiments on multiple publicly available RGB-D saliency detection datasets demonstrate that the proposed method exhibits excellent performance in both accuracy and robustness.

[0018] This invention proposes a novel cross-modal bidirectional fusion module that achieves deep interaction between RGB and depth features through cross-attention, effectively mining intermodal complementarity; The present invention proposes an edge-guided refinement module, which significantly improves the target edge localization accuracy by introducing the low-level features containing accurate boundary information into the intermediate layer of the decoder after deep fusion of the low-level features.

[0019] The overall network architecture of this invention, through the synergistic effect of multiple innovative modules, effectively solves problems such as insufficient feature fusion and blurred boundaries while maintaining high-precision segmentation, providing reliable technical support for salient object detection. This invention not only improves detection accuracy but also enhances the interpretability and scalability of the model. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the salient target detection system and method based on the Swin transformer of the present invention; Figure 2 This is a schematic diagram of the cross-modal bidirectional fusion module (CMBF) of the present invention; Figure 3 This is a schematic diagram of the edge guidance refinement module (EAR) of the present invention; Figure 4 This is a comparison chart of the PR curves of the present invention and other methods; Figure 5 For the present invention and other methods Curve comparison chart; Figure 6 This is a comparison chart of the prediction effects of the present invention and other methods; Figure 7 This is a comparison chart of the prediction performance of the present invention on the RGB-T dataset with other methods. Detailed Implementation

[0021] To make the objectives, technical solutions, and effects of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific examples. It should be emphasized that the examples described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] This invention selects the Swin transformer as the backbone network, enhancing feature extraction and global perception capabilities through its hierarchical design. It also innovatively proposes a cross-modal bidirectional fusion mechanism and an edge-guided refinement mechanism to fully utilize information from both modalities and accurately locate edge information, thus presenting an innovative framework for salient object detection. This innovative application not only improves the accuracy of salient object detection but also opens up new research directions in the field.

[0023] This paper proposes a novel RGB-D salient target detection method, such as... Figure 1 As shown, the proposed method effectively addresses the shortcomings of existing methods in feature fusion and edge blurring through two core modules: cross-modal bidirectional fusion and edge-guided refinement. Experimental results demonstrate that the proposed method achieves excellent performance on multiple benchmark datasets, exhibiting particularly strong robustness in complex scenes. The cross-modal generalization experiment shows that the proposed RGB-D saliency detection model maintains excellent salient target perception and robustness when transferred to the RGB-T task, reflecting the strong feature representation ability and generalization performance of this method.

[0024] Efficient feature fusion via CMBF ensures region-level accuracy and robustness, while progressive edge refinement via EAR guarantees the precision of boundary details. CENet serves as a complete encoder-fusion-decoder framework. Enhanced robustness: It can robustly integrate the appearance information of RGB with the geometric structure information of Depth, ensuring the integrity of the salient area and avoiding the problem of missing or blurred target areas; Performance improvements: Excellent performance across quantitative evaluation metrics on multiple RGB-D datasets; Generalization ability: This general fusion and refinement mechanism has also been verified to be effective in the generalization experiment on the RGB-T dataset, demonstrating the network's strong adaptability to different auxiliary modal information.

[0025] A salient object detection method based on the Swin transformer is shown below: Dual-stream encoder: Using a Swing transformer as the backbone, feature extraction is performed layer by layer on the input RGB and Depth images. The low-level features of the RGB and Depth images are first input into a Feature Pyramid (FPN) to obtain four features. , , and The remaining levels obtain features , , and The obtained features are used as input to the subsequent cross-modal bidirectional fusion module (CMBF); Feature fusion: For the RGB and depth features at the current scale, we first compute the RGB-to-depth attention pathway. In this pathway, the depth features are used as the query vector. The RGB features are used as key vectors. Sum value vector Through a cross-attention operation, each element of the RGB feature is assigned an attention weight, which reflects its correlation with each element in the deep feature. Ultimately, this generates an attention-weighted RGB feature. This allows RGB features to effectively focus on the geometric structure or spatial context provided by depth information, thereby enhancing their representational power in saliency detection tasks. Conversely, in another approach, depth-to-RGB attention is computed, with RGB features acting as the query vector. Deep features serve as key vectors. Sum value vector This generates an attention-weighted deep feature. This process allows deep features to be enhanced using the rich texture and semantic information of the RGB modality, compensating for their shortcomings in detail and semantic understanding. The calculation process is shown below.

[0026] in Indicates the source mode, Indicates the target mode, , , Represents the learnable parameter matrix, and represents the cross-attention operation. This represents the scaling factor.

[0027] After generating bidirectional attention features, the module applies the attention mechanism to each branch separately. For the two branches... and First, a 1×1 convolutional layer is used to process and generate... and Then, global average pooling is used to compress the spatial dimension, obtaining the channel-level feature vector for each branch. Subsequently, the two resulting vectors are fed into a sigmoid activation function (represented by colored squares in the figure) to generate channel attention weight vectors. and The weight vector and Features after 1×1 convolution and Perform cross-multiplication separately ( This process adaptively weights the feature channels. After attention-weighted processing, the features from the two branches are concatenated along the channel dimension. This step aims to adaptively adjust the weight of each channel based on the reliability of the features, highlighting the channel information that contributes more to the task. The calculation process is shown below.

[0028] in This represents a two-dimensional convolution operation with a kernel size of 1×1. and These represent the global average pooling operation and the sigmoid activation function, respectively. This indicates a feature-by-feature multiplication operation. This indicates a splicing operation.

[0029] The concatenated features are fed into a multi-layer convolutional block. This block consists of two 3×3 convolutional layers and one 1×1 convolutional layer. The 3×3 convolutional layers are used for local feature fusion and information integration in the spatial dimension. The final 1×1 convolutional layer is used to further adjust the number of channels and complete the final feature fusion, generating the final fused features. The calculation process above is shown below.

[0030] in, This represents a two-dimensional convolution operation with a kernel size of 3×3. This represents a two-dimensional convolution operation with a kernel size of 1×1.

[0031] Multi-level decoder: The Edge Guided Refinement (EAR) module receives two feature maps from different levels. and As input, and This is for low-level features containing rich boundary details. First, bilinear interpolation is used to... Upsampling to The same spatial dimensions are used to achieve feature alignment. Subsequently, the two features are subjected to convolution for dimensionality adjustment and feature extraction, performing preliminary feature enhancement and dimensionality unification on the original features. The specific implementation is shown below: This indicates the upsampling operation.

[0032] To effectively capture different types of spatial information, this invention designs a dual-pooling feature extraction mechanism. Max pooling highlights the strongest response in local regions, making it particularly suitable for capturing significant geometric features such as edges and corners; while average pooling maintains the overall distribution characteristics of features through global averaging, helping to preserve semantic coherence. For each feature, both pooling operations are applied simultaneously: By fusing these two complementary spatial information elements through element-wise multiplication, simultaneous modeling of local saliency and global consistency is achieved. The core advantage of this dual-pooling mechanism lies in its ability to preserve sensitivity to local details such as edges while maintaining the global stability of the feature distribution, thus providing a richer and more balanced feature representation for subsequent feature fusion.

[0033] Subsequently, a preliminary fusion of information from the two levels is achieved through feature addition. To further enhance the model's adaptability, this paper introduces an edge-guided gating mechanism. The motivation behind this mechanism is that different feature components contribute differently to the final edge prediction; therefore, an adaptive control strategy is needed to highlight the most critical feature information for edge discrimination. The calculation process is shown below.

[0034] in a and b These represent the spatial information obtained from low-level features. Represents the edge information weight graph. This represents the feature information obtained through coordinate attention calculation.

[0035] Features are gating using learned edge factors. This residual gating design ensures effective information transfer while allowing the network to learn adjustment strategies sensitive to edge information. Finally, a coordinate attention mechanism is integrated to further enhance feature representation capabilities.

[0036] Traditional channel attention mechanisms ignore positional information, while coordinate attention, by encoding positional information in the horizontal and vertical directions respectively, can better capture the spatial structural features of the target. This is especially important for edge localization tasks. The specific calculation process of coordinate attention is as follows: in, Indicates the input feature map, This represents the pooling feature along the height direction. This represents the pooling feature along the width direction. This indicates the features after splicing.

[0037] By performing pooling operations along both the height and width directions, coordinate attention can embed positional information into the calculation of channel attention, enabling the network to simultaneously model the importance and spatial distribution characteristics of features. This position-aware mechanism provides important spatial prior information for accurate edge localization.

[0038] The Edge-Guided Refinement (EAR) module achieves multi-dimensional feature enhancement through the aforementioned hierarchical design. The dual-pooling mechanism captures complementary information of local saliency and global consistency during feature extraction, the adaptive gating mechanism intelligently adjusts feature importance during feature fusion, and the coordinate attention mechanism enhances the network's positional awareness during spatial modeling. This multi-layered architecture effectively integrates the boundary details of low-level features with the semantic expressive power of high-level features, thereby improving overall detection performance.

[0039] Loss function calculation: The BCE loss function measures the difference between the saliency map predicted by the model and the true saliency map, while the IoU loss function is used to better optimize the prediction results from the perspectives of overlap and spatial structure relationships. The final loss function is defined as follows: in The model represents the first time. The predicted saliency map is generated at each scale, and its value is usually between 0 and 1, representing the probability that each pixel is a salient object. This represents a true saliency graph.

[0040] Iterative training: During training and testing, the input RGB and depth images were resized to 256×256 resolution. Data augmentation strategies such as random flipping, rotation, and boundary cropping were employed during training to prevent overfitting of the training images. Two Swin-B pre-trained models were used to initialize the backbone network parameters, while the remaining parameters used PyTorch's default settings. The Adam optimizer was used during training with a batch size of 16 and an initial learning rate of 5e-5, which was decayed to 1 / 10 of its original value every 50 epochs. All experiments were conducted on a workstation equipped with an NVIDIA RTX 3090 GPU (24GB VRAM), an Intel(R) Xeon(R) E5-2680 v4 @ 2.40GHz × 7-core CPU, and 30GB of RAM. Under this environment, the model converged within 100 epochs.

[0041] Verification and evaluation: Of the five evaluation metrics used, the PR curve, as a fundamental and crucial quantitative indicator, plays a vital role in quantitatively evaluating algorithm performance. Used to evaluate the structural similarity between the real-valued saliency map and the true value; calculate the mean absolute error between the pixel-level predicted saliency map and the ground truth (GT); Used to better evaluate performance through precision and recall; It is used to comprehensively consider local and global information.

[0042] Generalization performance verification: Experiments were conducted on three RGB-T datasets: VT821, VT1000, and VT5000. 2500 pairs of data were selected from VT5000 as the training set, and the remaining data were used as the test set. Ten state-of-the-art methods were compared.

[0043] Output result: The test image is input into the trained model to generate a prediction map that clearly shows the regions and contours of saliency detection.

[0044] Figure 1 This is a schematic diagram of the salient object detection system and method based on the Swin Transformer. The overall framework adopts an encoder-fusion-decoder architecture, extracts hierarchical features through a dual-stream Swin Transformer, applies a cross-modal bidirectional fusion module at multiple scales, and enhances boundary quality through an edge-aware refinement module during the decoding stage.

[0045] refer to Figure 1 and Figure 2The cross-modal bidirectional fusion design is based on the observation that RGB features are rich in texture details but lack geometric constraints, while depth features provide spatial structure but have limited semantic information. Traditional unidirectional fusion cannot fully utilize this complementarity. To address this issue, a symmetrical bidirectional cross-attention mechanism is designed to allow the two modal features to mutually enhance each other, achieving deep fusion of RGB and depth features through this bidirectional attention interaction mechanism. The core idea of ​​this module is to allow the two modal features to act as queries, keys, and values, establishing fine-grained feature correspondences through cross-attention operations, thereby achieving more effective cross-modal information integration.

[0046] refer to Figure 1 and Figure 3 The edge-aware refinement module significantly improves boundary localization capabilities while maintaining region-level detection accuracy. Low-level features, although relatively weak in semantic information, contain rich spatial details and boundary information; while high-level features have strong semantic expressive power but lower spatial resolution. By deeply fusing low-level and high-level semantic features and introducing them into the intermediate layer of the decoder, the boundary details in low-level features are effectively preserved and utilized while maintaining the high-level semantic understanding capability. Furthermore, considering that targets may exhibit different feature representations at different spatial scales, a FeaturePyramid Network and various pooling strategies are employed to capture this multi-scale information, enhancing the model's adaptability to complex scenes.

[0047] refer to Figure 4 Our method achieves curves closer to the upper right corner on most datasets, demonstrating excellent precision and recall. Particularly on datasets with complex scenes or boundary details, such as DUT, LFSD, and SSD, our method generally outperforms the comparison methods. Furthermore, it maintains relatively stable performance on the NLPR, STERE, NJU2K, and SIP datasets.

[0048] refer to Figure 5 The proposed method maintains high performance and stability on most datasets. The value is significantly better than the comparison methods, especially on complex scenario datasets such as DUT, LFSD, and SSD.

[0049] refer to Figure 6 This method, along with other methods, provides qualitative analysis in a variety of complex scenarios.

[0050] refer to Figure 7This method applies the network architecture and fusion strategy originally designed for RGB-D tasks to RGB-T scenarios to verify whether the proposed method has the universality to process different types of auxiliary modal information, that is, to extend from processing geometric structure information (depth) to processing temperature contrast information (thermal infrared), thus proving the modality independence of the network design and the universality of the fusion mechanism.

[0051] Tables 1 and 2 systematically present the performance of the method of this invention on seven RGB-D saliency target detection datasets: DUT, LFSD, SSD, NLPR, STERE, NJU2K, and SIP. The evaluation metrics cover... , , and .in, , and A higher value indicates better structural and semantic consistency of the salient region, while The lower the value, the smaller the difference between the predicted saliency map and the true label. The first, second, and third results are marked in red, green, and blue, respectively.

[0052] Overall, the proposed method achieves state-of-the-art performance on the DUT, LFSD, and SSD datasets, demonstrating significant advantages. Specifically, compared to suboptimal methods, on the DUT dataset, , and They increased by 0.5%, 0.8%, and 0.7% respectively, while Significantly reduced by 0.4%; on the LFSD dataset, the model improved by 0.3%, 0.1%, and 0.1% on the three improvement metrics, respectively, and It decreased by 0.3%; on the SSD dataset, the three improvement metrics increased by 0.7%, 0.8%, and 0.1% respectively, while The decrease of 0.1% indicates that the proposed method performs well on these differentiated datasets, demonstrating advantages in modal complementarity and effectively fusing RGB features with deep geometric information. It exhibits good cross-scene adaptability and generalization. This improvement is mainly attributed to: (1) the bidirectional fusion mechanism effectively utilizes RGB-D complementarity, resulting in greater robustness in complex scenes; and (2) the edge refinement module significantly improves boundary quality. The improvement in indicators is particularly noticeable.

[0053] On the NLPR dataset ,and All metrics showed optimal performance. It achieved the best result among other models, and compared with the next best result, It improved by 0.2% and reached the optimal level, and decreased by 0.2%. Achieving the second result on the STERE dataset. and All achieved excellent results. It improved by 0.2% compared to the second place. They also demonstrated good performance. and They ranked second and third respectively. On the NJU2K and SIP datasets, our method is close to the state-of-the-art methods in all four metrics, further demonstrating our generalization ability in multiple scenarios.

[0054] Table 1. Quantitative comparison and evaluation of the method's performance on the DUT, LSFD, NJU2K, and NLPR datasets. Table 2. Quantitative comparison and evaluation of the method's performance on the SIP, SSD, and STERE datasets. In terms of qualitative analysis, from Figure 6 As can be observed, some methods, such as SPNet, DCF, and MoADNet, tend to produce significant target boundary blurring or region incompleteness under conditions such as incomplete depth information, complex background interference, or complex target shape, as shown in rows 1 and 7; other methods, such as DeNet, SSL, and C... 2 While algorithms like DFNet can improve the integrity of salient regions to some extent, they still have shortcomings in preserving fine details (such as the butterfly feet and wings in row 5) and suppressing background (such as row 4). The results above demonstrate that the method of this invention has significant advantages in target boundary localization accuracy and structural integrity. The model of this invention can robustly integrate RGB appearance information with Depth geometric information, ensuring the integrity of salient regions and avoiding the problems of missing or blurred salient regions that often occur in some baseline models. Furthermore, the model of this invention also exhibits excellent performance in edge refinement. Qualitative analysis results show that the model of this invention can provide reliable salientity prediction results in complex scenes.

[0055] To further verify the effectiveness of the proposed model, comprehensive ablation experiments were also conducted. These experiments aimed to evaluate the effectiveness of the individual modules proposed in this invention, and how they work together to improve overall performance.

[0056] The ablation experiment results are shown in Table 3, systematically verifying the contributions of each core component of the model—the Swin Transformer backbone network (Swin), the Feature Pyramid Network (FPN), the Edge Guided Refinement Module (EAR), and the Cross-Modal Bidirectional Fusion Module (CMBF)—to the final performance. This invention sequentially adds these components to a baseline configuration for evaluation.

[0057] (1) Baseline Configuration: The first row of results serves as the baseline configuration for this invention, using only the Swing Transformer as the encoder. Good initial performance has been achieved using only the Swing encoder, for example, on NJU2K. This provides a clear starting point for validating the effectiveness of subsequent modules.

[0058] (2) Effectiveness of FPN: The results after adding the Feature Pyramid Network (FPN) are shown in the second row. Performance across all datasets showed a consistent and significant improvement. For example, on the NLPR dataset, It decreased from 0.025 to 0.023. The accuracy was improved to 0.929. This demonstrates that FPN can effectively aggregate multi-scale features, which is crucial for improving the accuracy of salient object detection.

[0059] (3) Effectiveness of EAR: An edge-aware attention module (EAR) was introduced on top of Swin+FPN, and the results are shown in the third row. The model performance was further enhanced, especially in detail capture. For example, on the NJU2K dataset... The accuracy was improved to 0.957. This indicates that the EAR module successfully guided the model to focus on target boundary information, effectively improving the accuracy of the prediction results.

[0060] (4) Effectiveness of CMBF: Next, this invention introduces a cross-modal fusion module (CMBF), the results of which are shown in the fourth row. The purpose of this module is to efficiently fuse RGB and D-modal information. The results show that performance continues to improve after adding CMBF, especially in... In terms of metrics. For example, on the DUT dataset, Reaching 0.942, It decreased to 0.022. This strongly demonstrates the effectiveness of CMBF for cross-modal information exchange and complementary utilization.

[0061] (5) Final Full Model: The last row shows the performance of the full CENet model. The full model of this invention achieves best performance on all datasets and across all metrics, highlighted in red. For example, on the NJU2K dataset, It reached 0.927, then decreased to 0.028; on the DUT, Reaching 0.944, The score reached 0.974. This result clearly confirms that all proposed components are effective and mutually beneficial, and their synergistic effect significantly improves the overall performance of cross-modal saliency target detection.

[0062] Table 3 Ablation Experiment Results To verify the cross-modal adaptability of the proposed method, the model was retrained and tested on the RGB-T (RGB-Thermal) saliency object detection dataset. This experiment applies the network architecture and fusion strategy originally designed for the RGB-D task to the RGB-T scene, verifying whether the proposed method possesses the versatility to handle different types of auxiliary modal information, i.e., extending from processing geometric structure information (depth) to processing temperature contrast information (thermal infrared), thus demonstrating the modality independence of the network design and the universality of the fusion mechanism. Experiments were conducted on three datasets: VT821, VT1000, and VT5000. 2500 pairs of data were selected from VT5000 as the training set, and the remaining data were used as the test set. Ten state-of-the-art methods were compared, and the results are shown in Table 4. As can be seen from the table, the proposed method achieved best or near-best performance on all three datasets. Specifically, on the VT821 dataset, the proposed method... , and The values ​​reached 0.908, 0.891, and 0.946 respectively, with the lowest value being 0.026; on the VT1000 dataset, , and The results achieved Equities of 0.937, 0.941, and 0.976 respectively, with an MAE of only 0.016. Even on the larger VT5000 dataset, the proposed model maintained stable performance, outperforming suboptimal methods. , and Both increased by 0.3%. It decreased by 0.2%.

[0063] Table 4 Generalization Experiment Results Qualitative comparison results of different methods on the RGB-T dataset are as follows: Figure 7As shown, some methods (such as BASNet, ADFNet, and MIDD) tend to produce blurred or incomplete target boundaries under conditions of large illumination variations, thermal noise interference, or complex backgrounds. Other methods (such as DCNet, MGAI, and TCINet) can improve the integrity of salient regions to some extent, but they still have shortcomings in terms of detail preservation and interference suppression. In contrast, the proposed method can accurately highlight target regions in various complex scenes, and the generated saliency maps are closer to the real annotations in terms of target edges, shape consistency, and background suppression, fully demonstrating the robustness and superiority of the proposed model in cross-modal feature modeling and salient region extraction.

[0064] In summary, this paper proposes a salient object detection system and method based on the Swin transformer. Through two core modules—cross-modal bidirectional fusion and edge-guided refinement—it effectively addresses the shortcomings of existing methods in feature fusion and edge blurring. Experimental results show that the proposed method achieves excellent performance on multiple benchmark datasets, especially demonstrating stronger robustness in complex scenes.

[0065] It should be understood that, inspired by the inventive concept, those skilled in the art can make various improvements or modifications based on the above description without departing from the scope of the invention, but these still fall within the protection scope of the invention.

Claims

1. A salient target detection method based on the Swing transformer, characterized in that, include: Step 1: Construct a two-stream encoder: Use a two-stream Swing Transformer encoder to extract features from the input RGB image and Depth image respectively, to obtain a multi-scale RGB feature set. and Depth feature set ; Step 2: Feature Fusion: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] and Input is used to perform cross-modal bidirectional fusion, generating a multi-scale cross-modal fusion feature set. ; Step 3: Construct a multi-level decoder: As input, the resolution of the feature map is restored step by step. In at least one intermediate layer of the decoder, edge-guided thinning is introduced to enhance and refine the fused features at that level, and the final saliency prediction map is output. Step 4: Loss function calculation: The binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function are used to supervise the network. The binary cross-entropy (BCE) loss function measures the difference between the saliency map predicted by the model and the true saliency map, while the intersection-over-union (IoU) loss function is used to better optimize the prediction results from the perspective of overlap and spatial structure relationship. Step 5: Iterative training: Iteratively train the model built in Steps 1-3. In each training iteration, perform forward propagation to calculate the multi-task loss, then update the model parameters through backpropagation, select the optimal model parameters through validation set performance evaluation, and then input the test image into the trained model to generate a prediction map to achieve salient object detection. Step 6: Validation and Evaluation: Five evaluation metrics are used to evaluate the performance of the model method, including PR curve, S-measure, mean absolute error, F-measure, and E-measure. Step 7: Generalization performance verification: Retrain and test the model built in Step 1-3 on the RGB-Thermal saliency object detection dataset to verify the generalization performance of the model; In step 2, the following will be done: and As input, RGB features and depth features are used to act as query, key, and value, respectively. A fine-grained feature correspondence is established through cross-attention operations. After generating bidirectional attention features, attention mechanisms are applied to each branch, and the features are then concatenated and fused to generate a multi-scale cross-modal fusion feature set. ; In step 3, the two feature maps from different levels are... and As input, and For low-level features containing rich boundary details, firstly, bilinear interpolation is used to... Upsampling to Using the same spatial dimensions to achieve feature alignment, the two features are then respectively... Convolution performs dimensionality adjustment and feature extraction, providing preliminary feature enhancement and dimensionality unification to the original features, as shown below: in Indicates an upsampling operation; Max pooling can highlight the strongest response in local regions and capture the significant geometric features of edges and corners. Then, based on average pooling, the overall distribution characteristics of features are maintained through global averaging, preserving semantic coherence. For each feature, both pooling operations are applied simultaneously to capture different types of spatial information, as shown below: By fusing these two complementary spatial information through element-wise multiplication, we can simultaneously model local saliency and global consistency. Subsequently, a preliminary fusion of information from two levels is achieved through feature addition. An edge-guided gating mechanism is employed, which adaptively adjusts the gating based on the varying contributions of different feature components to the final edge prediction, highlighting the most critical feature information for edge discrimination. This is represented as follows: Where a and b represent the spatial information obtained from low-level features, respectively. Represents the edge information weight graph. This represents the feature information obtained through coordinate attention calculation; Using coordinate attention, spatial structural features of the target are captured by encoding positional information in the horizontal and vertical directions respectively, as follows: in, Indicates the input feature map, This represents the pooling feature along the height direction. This represents the pooling feature along the width direction. This indicates the features after splicing.

2. The salient target detection method based on the Swin transformer according to claim 1, characterized in that, In step 1, a Swin transformer is used as the backbone to extract features from the input RGB and Depth images layer by layer. The low-level features of the RGB and Depth images are first input into the Feature Pyramid Network (FPN) to obtain four features. , , and The remaining levels obtain features , , and .

3. The salient target detection method based on the Swin transformer according to claim 1, characterized in that, In step 2, the following will be done: and As input, RGB features and depth features are used as query, key, and value to each other. A fine-grained feature correspondence is established through cross-attention operations, including: For RGB features and depth features First, attention is obtained from RGB to depth. In this path, depth features are used as the query vector. The RGB features are used as key vectors. Sum value vector ; through cross-attention operations Each element of the RGB feature is assigned an attention weight, which reflects its correlation with each element in the deep feature; ultimately, an attention-weighted RGB feature is generated. In another approach, attention is obtained down to the RGB level, with RGB features serving as the query vector. Deep features serve as key vectors. Sum value vector ;pass Generate an attention-weighted deep feature. ; indicates as: in, Indicates the source mode, Indicates the target mode, , , Represents the learnable parameter matrix, This indicates a cross-attention operation. Indicates the scaling factor. This represents the attention feature calculated using RGB modal information as the query and depth information as the key and value. This represents the attention features calculated using deep modality information as the query and RGB information as the key and value.

4. The salient target detection method based on the Swin transformer according to claim 3, characterized in that, In step 2, an attention mechanism is applied to each branch, and the branches are then concatenated and fused to generate a multi-scale cross-modal fusion feature set. include: After generating bidirectional attention features, an attention mechanism is applied to each branch; for the two branches... and First, a 1×1 convolutional layer is used to process and generate... and Then, global average pooling is used to compress the spatial dimension to obtain the channel-level feature vector for each branch. Subsequently, the two resulting vectors are fed into a sigmoid activation function to generate channel attention weight vectors. and The weight vector and Features after 1×1 convolution and Cross-multiplication is performed separately to achieve adaptive weighting of the feature channels. After attention weighting, the features from the two branches are concatenated along the channel dimension, as follows: in This represents a two-dimensional convolution operation with a kernel size of 1×1. and These represent the global average pooling operation and the sigmoid activation function, respectively. This indicates a feature-by-feature multiplication operation. Indicates a splicing operation; The concatenated features are fed into a multi-layer convolutional block consisting of two 3×3 convolutional layers and one 1×1 convolutional layer. The 3×3 convolutional layers perform local feature fusion and information integration in the spatial dimension, while the final 1×1 convolutional layer further adjusts the number of channels and completes the final feature fusion, generating the final fused features. , is represented as: in, This represents a two-dimensional convolution operation with a kernel size of 3×3. This represents a two-dimensional convolution operation with a kernel size of 1×1.

5. The salient target detection method based on the Swin transformer according to claim 1, characterized in that, In step 4, the BCE loss function is used to measure the difference between the saliency map predicted by the model and the true saliency map, and the IoU loss function is used to better optimize the prediction results from the perspective of overlap and spatial structure relationship. The final loss function is defined as follows: in The model represents the first time. The predicted saliency map is generated at each scale, and its value is typically between 0 and 1, representing the probability that each pixel is a salient object. This represents a true saliency graph.

6. The salient target detection method based on the Swin transformer according to claim 1, characterized in that, In step 5, during the training and testing phases, the input RGB image and depth image are adjusted to a resolution of 256×256. During training, random flipping, rotation, and boundary cropping are used for data augmentation to prevent overfitting of the training images. The parameters of the backbone network are initialized using two Swin-B pre-trained models, and the remaining parameters use the default settings of PyTorch. The Adam optimizer is used during training, the batch size is 16, the initial learning rate is set to 5e-5, and the learning rate is decayed to 1 / 10 of the original every 50 epochs. The model converges within 100 epochs.

7. A detection system employing the salient target detection method based on the Swin transformer as described in claim 1, characterized in that, The detection system includes a Feature Pyramid (FPN), a Cross-Modal Bidirectional Fusion Module (CMBF), and an Edge Guided Refinement Module (EAR). Feature Pyramid Network (FPN) captures low-level feature information, enabling low-level features to express high-level semantic information while being rich in texture and edge information, thus making the feature expression more complete. The cross-modal bidirectional fusion module CMBF is connected to the output of the feature pyramid FPN. A symmetrical bidirectional cross-attention mechanism is designed so that the two modal features can enhance each other. The deep fusion of RGB and depth features is achieved through the design of the bidirectional attention interaction mechanism. Fine feature correspondence is established through cross-attention operation to achieve cross-modal information integration. The Edge-Guided Refinement (EAR) module is connected to the output of the Cross-Modal Bidirectional Fusion (CMBF) module. By deeply fusing low-level features with high-level semantic features and introducing them into the intermediate layer of the decoder, it effectively preserves and utilizes the boundary details in low-level features while maintaining the high-level semantic understanding capability. In addition, considering that the target may exhibit different feature performances at different spatial scales, multiple pooling strategies are adopted to capture multi-scale information and enhance the model's adaptability to complex scenes.

Images