Significance target detection method and device

Abstract

The application discloses a salient object detection method and device. After obtaining a to-be-detected image, the application extracts RGB features and depth features of the to-be-detected image through an encoder; the application respectively performs feature enhancement on high-level features of the RGB and depth through an attention feature enhancement module; the application inputs the extracted RGB features and depth features into a cross-modal fusion module to strengthen and fuse the RGB features and depth features; the application generates edge information of salient objects in the depth features by using an edge extraction module; the application inputs the strengthened and fused RGB features and depth features into a multi-scale feature aggregation module to perform multi-scale feature aggregation and obtain multi-level fusion features; the application inputs the multi-level fusion features into a cascaded correction decoder to refine and correct the multi-level fusion features, predict salient features, and then enhance the salient features by using the edge information to generate a saliency map; and the application breaks the limitations of the prior art in global context modeling and multi-scale feature aggregation.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and specifically to a method and apparatus for detecting salient targets. Background Technology

[0002] The purpose of salient object detection (SOD) is to locate the most visually prominent object in a given scene. SOD plays a crucial role in a range of real-world applications, such as image retrieval, object tracking, image / video segmentation, face recognition, and image understanding—many computer vision tasks.

[0003] RGB-D-based salient object detection is a crucial preprocessing operation for various visual tasks and has become a focus of attention in recent years. While significant breakthroughs have been achieved in RGB-D-based salient object detection, limitations remain in global context modeling and multi-scale feature aggregation. Recent methods based on RGB-D have gradually broken through the performance bottlenecks of traditional and RGB-based methods, particularly demonstrating excellent performance in complex scenes with similar foreground and background. This is mainly due to global context modeling, multi-scale feature aggregation, and edge-guided learning mechanisms. Previous research has focused on these three key factors. However, despite these advancements, the following limitations still exist.

[0004] First, most RGB-D-based salient object detection tasks use convolutional neural networks (CNNs) as the backbone for feature extraction. While achieving good results, the inherent limitations of sliding window feature extraction methods prevent them from effectively learning global semantic information. Some researchers have introduced dilated convolutions to extract global semantic information, expanding the receptive field of the convolutional kernel without introducing additional parameters, thus enhancing the network's global feature extraction capabilities. However, dilated convolutions cause a raster effect, losing the continuity of information. This can lead to detection failures in complex scenes and when the target is small. Other researchers have taken a feature extraction approach, using Transformers as the backbone to extract global features from the image and learn global semantic information. While this method can achieve global context modeling, the computational complexity introduced by Transformers is immeasurable.

[0005] Secondly, the differences between features at different scales are ignored during multi-scale feature aggregation. High-level features contain rich semantic information but lack accurate location information, while low-level features contain rich details and clear edge contours but have more background noise. Direct aggregation easily introduces noise and other redundant information. In addition, the information loss and boundary blurring caused by upsampling during the aggregation process are ignored. Some researchers use low-level features to generate edge information to solve the boundary blurring problem, but the problem of information loss has not been well explored. Therefore, it is necessary to propose a salient object detection method to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a salient object detection method that combines global modeling capabilities with the local modeling capabilities of CNNs; enhances high-level features and acquires more detailed semantic information; reduces the differences between features at different scales, thereby improving the representational power of features; and solves the problem of partial information loss caused by upsampling operations in multi-scale feature aggregation.

[0007] This invention provides a salient target detection method, comprising:

[0008] Acquire the image to be detected;

[0009] The encoder extracts RGB and depth features from the image to be detected.

[0010] The attention feature enhancement module is used to enhance the high-level features of RGB features and deep features respectively;

[0011] The extracted RGB and depth features are input into the cross-modal fusion module to enhance and fuse the RGB and depth features;

[0012] The edge extraction module is used to generate edge information of salient objects in the depth features;

[0013] The enhanced and fused RGB features and depth features are input into the multi-scale feature aggregation module to perform multi-scale feature aggregation and obtain multi-level fused features.

[0014] The multi-level fused features are input into the cascaded correction decoder to refine and correct the multi-level fused features, generate salient features, and then the salient features are enhanced using edge information to generate the final salient map.

[0015] Further, the encoder extracts RGB features and depth features from the image to be detected, including:

[0016] The encoder includes an RGB stream encoder and a depth stream encoder. The RGB stream encoder and the depth stream encoder use Swin-B as the backbone network to extract multi-level features of the RGB image and the depth image, respectively.

[0017] Furthermore, the attention feature enhancement module performs feature enhancement on the high-level features of RGB features and deep features respectively, including:

[0018] Attention feature enhancement modules are added to the tail of the RGB stream encoder and the depth stream encoder respectively. The attention feature enhancement modules are used to perform three types of dilated convolution with kernel size of 3 and dilation rate of 2, 4 and 6 on high-level features, as well as 1x1 convolution and global average pooling operations to obtain multi-scale features.

[0019] Perform self-attention mechanism operations on multi-scale features to obtain global context information;

[0020] The feature maps that have undergone self-attention, 1x1 convolution, and global average pooling are concatenated to obtain the enhanced features.

[0021] Furthermore, the enhanced RGB features and depth features are input into the cross-modal fusion module to enhance and fuse the RGB features and depth features, including:

[0022] The cross-modal feature fusion module includes a deep feature enhancement module, which uses a spatial attention mechanism to compute a shared spatial attention map of RGB features and deep features;

[0023] The spatial attention map is used as the weight of the depth features, and a residual connection is made between the original depth features and the spatial attention map.

[0024] Calculate the channel attention map using the channel attention mechanism;

[0025] The channel attention map is multiplied with the original depth features to obtain the enhanced depth features.

[0026] Furthermore, the enhanced RGB features and depth features are input into the cross-modal fusion module to enhance and fuse the RGB features and depth features, including:

[0027] The cross-modal feature fusion module includes an RGB feature enhancement module, which uses a spatial attention mechanism to compute a shared spatial attention map of RGB features and depth features.

[0028] The spatial attention map is used as the weight of the RGB features, and a residual connection is made between the original RGB features and the spatial attention map.

[0029] Calculate the channel attention map using the channel attention mechanism;

[0030] The channel attention map is multiplied with the original RGB features to obtain the enhanced RGB features.

[0031] Furthermore, the enhanced RGB features and depth features are input into the cross-modal fusion module to enhance and fuse the RGB features and depth features, including:

[0032] The cross-modal feature fusion module includes a fusion block, which fuses the enhanced depth features and RGB features.

[0033] Furthermore, the edge extraction module is used to generate edge information of salient objects in the depth features, including:

[0034] Perform 1x1 convolution and upsampling operations on the depth features to generate two features of the same scale;

[0035] The two features of the same scale are concatenated to generate edge features;

[0036] The edge features are processed using a channel attention mechanism and residual connections to generate edge information.

[0037] Furthermore, the enhanced and fused RGB features and depth features are input into the multi-scale feature aggregation module for multi-scale feature aggregation, resulting in multi-level fused features, including:

[0038] Operations include upsampling of high-level features, 3x3 convolution, and adjustment of the number of channels;

[0039] By multiplying elements, high-level features are transformed and fused with low-level features to obtain fused features, which possess the characteristics of both high-level and low-level features.

[0040] A channel attention mechanism is used to refine the fused features, resulting in multi-level fused features.

[0041] Furthermore, the multi-level fused features are input into a cascaded correction decoder to refine and correct the multi-level fused features, generating salient features, including:

[0042] The cascaded correction decoder includes a first sub-decoder and a second sub-decoder. The first and second sub-decoders each contain three multi-scale feature aggregation modules. The cascaded correction decoder includes two processes: bottom-up and top-down.

[0043] For the bottom-up process, the multi-scale feature aggregation module in the sub-decoder gradually aggregates from high-level features to low-level features. The aggregated features are used to generate a coarse saliency map, which is then supervised.

[0044] For the top-down process, the features output by the top multi-scale feature aggregation module in the first sub-decoder are directly downsampled into features and added to the multi-level features exported by the multi-scale feature aggregation module for refinement and correction.

[0045] The corrected features are sent to the next sub-decoder to generate features with complete information through the same process;

[0046] Generate a saliency map based on features with complete information.

[0047] This invention provides a salient target detection device, comprising:

[0048] The image acquisition module is used to acquire the image to be detected;

[0049] An encoder is used to extract RGB features and depth features from the image to be detected;

[0050] The attention feature enhancement module is used to enhance the high-level features of RGB features and deep features respectively;

[0051] A cross-modal fusion module is used to enhance and fuse the RGB features and depth features;

[0052] An edge extraction module is used to generate edge information of salient objects in the depth features;

[0053] The multi-scale feature aggregation module is used to perform multi-scale feature aggregation on the enhanced and fused RGB features and depth features to obtain multi-level fused features.

[0054] A cascaded correction decoder is used to refine and correct the multi-level fused features, generate salient features, and then use edge information to enhance the salient features to generate the final salient map.

[0055] This invention offers the following advantages: The salient object detection method and apparatus provided by this invention, after acquiring the image to be detected, extracts RGB and depth features from the image through an encoder; enhances the RGB and depth features respectively through an attention feature enhancement module; inputs the extracted RGB and depth features into a cross-modal fusion module to strengthen and fuse the RGB and depth features; utilizes an edge extraction module to generate edge information of salient objects in the depth features; inputs the strengthened and fused RGB and depth features into a multi-scale feature aggregation module for multi-scale feature aggregation to obtain multi-level fused features; inputs the multi-level fused features into a cascaded correction decoder to refine and correct the multi-level fused features, generating salient features; then, the edge information is used to enhance the salient features, generating the final salient map; this invention proposes a cascaded aggregation network based on the Swin Transformer, which to some extent alleviates the limitations of most RGB-D salient object detection methods in global context modeling and multi-scale feature aggregation. This invention proposes an attention feature enhancement module, which can capture multi-scale contextual information of objects, alleviate the phenomenon of local information loss after dilated convolution, and enhance the correlation between information. This invention proposes a multi-scale feature aggregation module, allowing the model to selectively aggregate features at different scales, preventing the introduction of redundant information during decoding, and maximizing the aggregation of useful semantic information during decoding. This invention also proposes a cascaded correction decoder, which further refines and improves features by cascading two decoder blocks with multi-scale feature aggregation modules to generate a complete saliency map. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of the salient target detection method provided by the present invention;

[0058] Figure 2 This is a diagram of the overall CATNet structure of the salient target detection method provided by the present invention;

[0059] Figure 3 This is a structural diagram of the attention feature enhancement module of the salient target detection method provided by the present invention;

[0060] Figure 4 This is a structural diagram of the edge extraction module of the salient target detection method provided by the present invention;

[0061] Figure 5 This is a visual comparison of the generation effects of the salient target detection method provided by this invention and other models;

[0062] Figure 6 This is a visualization of the feature map of the cascaded correction decoder of the salient target detection method provided by the present invention. Detailed Implementation

[0063] Please see Figure 1 This invention provides a salient target detection method, comprising:

[0064] S101, acquire the image to be detected.

[0065] S102, the encoder extracts RGB features and depth features from the image to be detected.

[0066] In this embodiment, please refer to Figure 2 This invention proposes a CATNet, which consists of an encoder, an attention feature enhancement module (AFEM), an edge generator, a cross-modal feature fusion module (CMFM), a multi-scale feature aggregation (MSFA) module, and a cascade correction decoder.

[0067] The encoder includes an RGB stream encoder and a depth stream encoder. The RGB stream encoder and depth stream encoder use Swin-B as the backbone network to extract multi-level features from the RGB image and depth image, respectively. The Swin Transformer is a hierarchical Transformer with a window attention mechanism. It introduces a window attention mechanism between each image patch, instead of a global attention mechanism, significantly reducing computational cost. Furthermore, it provides a patch merging mechanism, enabling gradual dimensionality reduction during encoding, simulating the behavior of convolutional coding layers. The Swin Transformer combines the advantages of Transformer and CNN, and is therefore used as the backbone network for feature extraction. Features of the RGB image and depth image are extracted from the backbone network, such as... Figure 2 As shown in Figure a, where the RGB features are represented as Deep features are represented as

[0068] S103 uses the attention feature enhancement module to enhance the high-level features of RGB features and deep features respectively.

[0069] For salient object detection, fully utilizing the semantic and multi-scale information in high-level features can improve model performance. Therefore, this invention designs an Attention Feature Enhancement Module (AFEM) to further extract semantic and multi-scale information from high-level features. A self-attention mechanism branch is added after the dilated convolution branch to extract global semantic information. To enable the model to extract more detailed semantic information from high-level features, the Attention Feature Enhancement Module is added to the tail of both the RGB stream encoder and the deep stream encoder.

[0070] The attention feature enhancement module has five branches. The first branch is a standard 1x1 convolution, designed to maintain the original receptive field and adjust the number of channels in the input feature set for easier subsequent operations. The second to fourth branches use dilated convolutions with dilation rates of 2, 4, and 6 and a kernel size of 3, respectively, to extract multi-scale information from high-level features. The branch following the dilated convolutions is the self-attention mechanism branch, which aims to obtain global contextual information from high-level features. The fifth branch is global average pooling, used to reduce the number of network parameters. Finally, the feature maps from the five branches are concatenated along the channel dimension to obtain information at different scales. See details... Figure 3 The network uses dilated convolution to extract multi-scale features, MHSA (Multi-Head Attention) to enhance global modeling capabilities, 1x1 convolution to adjust the number of channels, and GAP (Global Average Pooling) to reduce the number of network parameters.

[0071] Specifically, firstly, regarding high-level features Three types of dilated convolutions (kernel size 3, dilation rates 2, 4, and 6), along with 1x1 convolutions and global average pooling, are performed. After obtaining multi-scale features through the dilated convolution branch, a self-attention mechanism is then applied to acquire global contextual information. Finally, the feature maps processed by self-attention, 1x1 convolutions, and global average pooling are concatenated to obtain the enhanced features. This process can be defined as:

[0072]

[0073] Here, AFEM(·) represents the attention feature enhancement module operation.

[0074] S104, the extracted RGB features and depth features are input into the cross-modal fusion module to enhance and fuse the RGB features and depth features.

[0075] The cross-modal fusion module is used to enhance and fuse RGB features and depth features. It includes a depth feature enhancement module (DFEM), an RGB feature enhancement module (RFEM), and a fusion block (FB).

[0076] Deep Feature Enhancement Module: Due to the poor quality of depth maps, which can suppress effective information in RGB images during fusion and thus affect the model's final prediction performance, a deep feature enhancement module is proposed. Specifically, it incorporates the rich semantic information from RGB features into the deep features to enhance them.

[0077] The deep feature enhancement module uses a spatial attention mechanism to compute RGB features. and depth features A shared spatial attention map is generated; this spatial attention map is used as the weights for the depth features, and a residual connection is made between the original depth features and the spatial attention map; a channel attention map is calculated using a channel attention mechanism; the channel attention map is multiplied by the original depth features to obtain the enhanced depth features. The process is represented as follows:

[0078]

[0079] Where "×" represents element-wise multiplication, "+" represents element-wise addition, CA(·) represents channel attention, and SA(·) represents spatial attention. The computation process of spatial attention and channel attention can be defined as follows:

[0080] SA(x)=Sigmoid(Conv3(CGMP(x))) (3)

[0081] CA(x)=Sigmoid(Conv1(GMP(x))) (4)

[0082] Wherein, CGMP(·) represents global max pooling operation in the channel direction, GMP(·) represents global max pooling operation, Conv3(·) represents convolution operation with a kernel size of 3×3, Conv1(·) represents convolution operation with a kernel size of 1×1, and Sigmoid(·) represents sigmoid activation function.

[0083] RGB Feature Enhancement Module: Although RGB images contain rich semantic information, they suffer from background noise, and salient objects in the image are easily affected by the background. Therefore, an RGB feature enhancement module is proposed to introduce spatial features from the depth map into the RGB features, thereby enhancing the RGB features. Specifically, spatial attention and channel attention mechanisms are used to enhance the RGB features.

[0084] The RGB feature enhancement module uses a spatial attention mechanism to calculate RGB features. and depth features A shared spatial attention map is generated; this spatial attention map is used as the weights for the RGB features, and a residual connection is made between the original RGB features and the spatial attention map; a channel attention map is calculated using a channel attention mechanism; the channel attention map is multiplied by the original RGB features to obtain the enhanced RGB features. The process is represented as follows:

[0085]

[0086] Fusion Block: After obtaining the enhanced features from the two modalities, the fusion block fuses the enhanced depth features and RGB features. Assume... The features after fusion can be represented by the following process:

[0087] Here, FB(·) represents the operation of the merge block.

[0088] S105, use the edge extraction module to generate edge information of salient objects in the depth features.

[0089] To enable the model to predict salient maps with clear edge contours, this invention introduces an edge extraction module to generate edge information for salient objects. It is well known that low-level features contain more details and edge contour information. Meanwhile, salient objects in the depth map have more prominent structures, and their contours can be easily depicted through depth contrast. Therefore, shallow features generated by the depth encoder are used to produce edge information. Figure 4 The structure of the edge extraction module is shown.

[0090] Specifically, for deep features Perform a 1x1 convolution and upsampling operation to generate two features of the same scale; concatenate the two features of the same scale to generate the edge feature f′. e The calculation process is as follows:

[0091]

[0092] Here, Up2(x) represents upsampling x by a factor of 2, and Concat represents the concatenation operation.

[0093] Next, channel attention and residual connections are applied to the edge features to generate edge information f. e The process is defined as follows:

[0094] f e =f′e ×CA(BConv(f′ e ))+f′ e (8)

[0095] Where BConv(·) represents a 3x3 kernel convolution, batch normalization, and ReLU activation function operation.

[0096] S106 inputs the enhanced and fused RGB features and depth features into the multi-scale feature aggregation module to perform multi-scale feature aggregation and obtain multi-level fused features.

[0097] This invention proposes a Multiscale Feature Aggregation (MSFA) module to refine high-level and low-level features. These two types of features differ; low-level features retain rich details and background noise, and these features have clear boundaries, which are crucial for generating accurate saliency maps. In contrast, high-level features are coarse at the boundaries due to multiple upsampling. Although they lose much detailed information, they still contain rich semantic information. To suppress noise in low-level features and reduce redundant information, this invention designs a multiscale feature aggregation module. MSFA mitigates the differences between different features through feature multiplication and channel attention mechanisms. Specifically, firstly, the high-level feature f... h Upsampling, 3x3 convolution, and channel number adjustment are performed; element-wise multiplication is used to combine high-level features with low-level features f. l Feature transformation and fusion are performed to obtain fused features f′. These fused features possess the characteristics of both high-level and low-level features, namely clear boundaries and rich semantic information. However, since feature f′ contains redundant information and noise, feature multiplication alone cannot suppress noise and reduce redundant information. Therefore, a channel attention mechanism is used to refine the fused features f′, resulting in multi-level fused features with rich semantics and low noise. Finally, to enable the network to automatically select the weights of features at different scales, a gating mechanism is introduced. The entire process is as follows:

[0098] f′=f l ×BConv3(Up2(f h (9)

[0099] CA_G=CA(f′) (10)

[0100] f u_end = fl × CA_G + (1 - CA_G) × f h (11)

[0101] Where CA_G represents the gate signal with channel attention mechanism, f u_endThis represents the features after passing through the MSFA module. Although the multi-scale feature aggregation module can effectively suppress noise in low-level features, the problem of information loss caused by the upsampling process is not well resolved. Therefore, this invention further proposes a cascaded correction decoder to solve this problem.

[0102] S107, the multi-level fusion features are input into the cascaded correction decoder to refine and correct the multi-level fusion features, generate salient features, and then the salient features are enhanced using edge information to generate the final salient map.

[0103] The Cascaded Corrected Decoder (CCD) is built upon a multi-scale feature aggregation module to refine multi-level features and iteratively generate saliency maps. Previous methods directly aggregated multi-level features and decoded them to generate the final saliency map. However, features at different levels exhibit differences, and direct aggregation can lead to information loss or redundancy. Even with MSFA, this missing information is difficult to recover, potentially impacting the model's final performance. Therefore, this invention designs a Cascaded Corrected Decoder that supervises the features predicted by the top-level MSFA module and feeds these features back to the features of the previous layer for refinement and correction, thereby addressing the information loss problem.

[0104] Figure 2 Figure b illustrates the architecture of the cascaded correction decoder. It can be seen that the cascaded correction decoder consists of two sub-decoders: Decoder1 and Decoder2. Each sub-decoder contains three multi-scale feature aggregation modules. The cascaded correction decoder includes both bottom-up and top-down processes.

[0105] For the bottom-up process, the multi-scale feature aggregation module in the sub-decoder gradually aggregates high-level features to low-level features, and the aggregated features... It is used to generate a coarse saliency map while supervising this saliency map;

[0106] For a top-down process, the features output by the top multi-scale feature aggregation module in the first sub-decoder... Directly downsampled to feature {p i |i=1,2,3,4}, and add it to the multi-level features exported by the multi-scale feature aggregation module. In the middle, the features are refined and corrected; the corrected features are sent to the next sub-decoder to generate features sal that are sufficiently complete through the same process; a saliency map is generated based on sal.

[0107] The entire process is represented as:

[0108]

[0109]

[0110]

[0111] Among them, Down m (x) indicates that x is downsampled by a factor of m.

[0112] The loss function L is defined as:

[0113] L = L e +L s (15)

[0114] Where L e L represents the edge loss. s This indicates a significant loss.

[0115] Edge loss: The output f of the edge extraction module is obtained. e Then, convolution and upsampling operations are performed on it to generate the final edge map E. The process is as follows:

[0116] E=U p4 (Conv3(f e (16)

[0117] Here, Up4(x) represents a 4x upsampling operation on x, and Conv3(·) represents an operation with a 3x3 kernel size.

[0118] The binary cross-entropy loss function is used to supervise E, and its formula is as follows:

[0119] L e =L bce (E, G) e (17)

[0120] Where L bce (·) represents the binary cross-entropy loss function, G e This represents the true value of the edge map.

[0121] Significance loss: Significance loss L s It is divided into two parts: one part supervises the saliency map S1 predicted by the sub-decoder, and the other part supervises the final saliency map S obtained after the cascaded correction decoder. The formula is as follows:

[0122] L s =L s1 +L s2 (18)

[0123] Among them, L s For significant loss, G sL represents the true value of the saliency map. s1 L represents the saliency loss of the sub-decoder. s2 This represents the final saliency loss of the cascaded decoder.

[0124] The saliency loss of the sub-decoder yields the saliency features predicted by the sub-decoder. Then, convolution and upsampling operations are performed on it to obtain the final saliency map S1 predicted by the sub-decoder. The process is as follows:

[0125]

[0126] This invention uses a binary cross-entropy loss function to supervise S1, the formula of which is:

[0127] L s1 =L bce (S1, G) s (20)

[0128] Among them, L bce Let G represent the binary cross-entropy loss function. s This represents the true value of the saliency map.

[0129] After obtaining the final salient feature sal for the saliency loss part of the cascaded correction decoder, let it be compared with the edge information f. e The features S′ are concatenated, and then convolution and upsampling operations are performed on features S′ to obtain the final saliency map S. The process is as follows:

[0130] S′=Concat(sal,f e ) (twenty one)

[0131] S=Up4(Conv3(S′)) (22)

[0132] Similarly, this invention uses a binary cross-entropy loss function to supervise S, the formula of which is:

[0133] L s2 =L bce (S, G) s ) (twenty three)

[0134] Among them, G s This represents the true value of the saliency map.

[0135] Data sets and evaluation metrics

[0136] 1) Datasets: To validate the effectiveness of the model, experiments were conducted on seven public RGB-D datasets, including NLPR, SIP, LFSD, NJU2K, STERE, RGBD135, and DUT-RGBD. The training datasets consist of 1485 images from the NJU2K dataset, 700 images from the NLPR dataset, and 800 images from the DUT-RGBD dataset.

[0137] 2) Evaluation indicators: The F-measure (F...) is used. β MAE, S-measure (S) m ) and E-measure(E ξ Four commonly used evaluation metrics are used to evaluate the performance of the saliency detection model. E-measure (E ξ S-measure is used to measure local pixel-level error and global image-level error; m ) Evaluate the spatial structural similarity of region perception and object perception in saliency maps; F-measure (F β Precision and recall are the weighted harmonic mean of precision and recall, used to evaluate the overall performance of the system; MAE measures the average absolute difference per pixel between the saliency map and the ground truth map.

[0138] This model is implemented in PyTorch and trained on a GTX A6000 GPU with 48GB of memory. During training and testing, the input image size was resized to 384×384, and single-channel depth images were duplicated into three-channel images to meet requirements. During training, random horizontal flipping and normalized data augmentation methods were used to reduce overfitting. The backbone network of the feature encoder was initialized using a pre-trained Swin-B algorithm. The remaining parameters were initialized to PyTorch's default settings. The Adam optimizer was used to train the network, with a batch size of 8 and an initial learning rate of 5e-5, which was divided by 10 every 100 epochs. The network converged within 290 epochs, with a training time of approximately 19 hours.

[0139] To verify the effectiveness of the proposed model, it was compared with several state-of-the-art RGB-D saliency-based object detection methods, including CoNet, CDNet, JLDCF, HAINe, VS, TriTransNet, SwinNet, CIRNet, C2DFNet, SPSN, BPGNet, MTFormer, CAVER, and DCF. To ensure the fairness of the comparison results, the saliency maps used for evaluation were provided by the authors or generated by running the source code. The proposed method was used to evaluate these methods.

[0140] Quantitative Evaluation. This invention uses the same evaluation metrics to assess the performance of the proposed method against 14 other state-of-the-art methods. Table 1 compares the proposed method with CNN-based methods, showing that the model outperforms the latest CNN-based methods. Table 2 compares the proposed method with Transformer-based methods, again demonstrating that the model outperforms other Transformer-based methods. Experimental results also visually validate the effectiveness and robustness of the proposed model across different datasets and evaluation metrics.

[0141] Qualitative evaluation. Figure 5 A visual comparison between the model of this invention and other state-of-the-art models is presented. It can be observed that in many challenging scenarios, CATNet generates more accurate saliency maps and is more competitive than other models, such as: edges and details (rows 1-2), small objects (rows 3-4), low-quality depth maps (rows 5-6), complex backgrounds (rows 7-8), and large objects (rows 9-10). These results demonstrate the robustness and effectiveness of the proposed method. Furthermore, the model proposed in this invention can completely detect salient objects. The above results demonstrate the accuracy and robustness of the proposed method.

[0142] Table 1. Quantitative comparison of F-measure, MAE, S-measure, and E-measure between the CNN-based RGB-D SOD method and seven widely used RGB-D datasets.

[0143]

[0144]

[0145] Table 2. Quantitative comparison of F-measure, MAE, S-measure, and E-measure between the Transformer-based RGB-D SOD method and seven widely used RGB-D datasets.

[0146]

[0147]

[0148]

[0149] Qualitative comparison of the model of this invention with other state-of-the-art models. Compared with other models, CATNet can not only accurately locate salient objects, but also generate clear edges and complete salient objects in complex scenes. (Edges and details (rows 1-2), small objects (rows 3-4), low-quality depth maps (rows 5-6), complex backgrounds (rows 7-8), and large objects (rows 9-10))

[0150] ablation experiment

[0151] In this section, ablation experiments were conducted using the NJU2K and LFSD datasets to evaluate the contribution of key modules in the proposed method.

[0152] 1) Module Effectiveness Experiments: Starting with the encoder-decoder baseline (including a two-stream encoder composed of Swing Transformers, a cross-modal fusion module, an edge extraction module, and a decoder composed of ordinary convolutional layers), it was progressively extended using different models, including AFEM, MSFA, and CCD. As shown in Table 3, the baseline (No:1) and the addition of the AFEM module (No:2) were tested first, and the model showed a significant performance improvement. This is reasonable because AFEM can extract more detailed semantic information from high-level features. Then, adding the cascaded decoder (No:3) again showed a significant performance improvement. The reason for this is that the cascaded correction decoder can alleviate the information loss phenomenon in the upsampling operation. Finally, as expected, adding all the modules proposed in this invention (No:4) to the baseline model achieved the best performance.

[0153] Table 3 shows the ablation experiments of the designed modules, with the best performance indicated in bold. (The same applies below.)

[0154]

[0155]

[0156] 2) Verifying the effectiveness of the attention feature enhancement module: ASPP is also a feature enhancement method. Similar to ASPP, the AFEM of this invention also uses dilated convolution to obtain multi-scale information. However, this invention adds a self-attention mechanism branch after the dilated convolution to compensate for the grid effect caused by the dilated convolution. To verify the effectiveness of the attention feature enhancement module proposed in this invention, as shown in Table 4, this invention adds ASPP and the attention enhancement feature module proposed in this invention to the baseline model (a two-stream encoder composed of Swing Transformers, a cross-modal feature enhancement module, an edge extraction module, and a cascaded decoder composed of MSFA, No:1) for comparison. Experimental results show that the baseline model using the attention feature enhancement module (No:4) performs better than the baseline model using ASPP (No:2). The reason for this is that the added attention module can obtain long-distance contextual information, resulting in a high correlation of semantic information in high-level features. In addition, this invention replaces the dilated convolution in AFEM with three ordinary convolutions (No:3) to verify the impact of attention feature enhancement modules composed of different convolutions on model performance. As can be seen, compared with the AFEM of this invention, the attention feature enhancement module composed of ordinary convolutions did not bring significant performance improvement to the model. One explainable reason is that ordinary convolutions cannot capture multi-scale information in high-level features, nor can they capture more detailed semantic information in high-level features, thus affecting the model's performance. This also further confirms the effectiveness of the AFEM proposed in this invention.

[0157] Table 4. Effectiveness Experiment of Attention Feature Enhancement Module

[0158]

[0159]

[0160] 3) Verifying the effectiveness of the multi-scale feature aggregation module: To verify the effectiveness of the multi-scale feature aggregation module, this invention conducted different comparative experiments based on the baseline model (with the multi-scale feature aggregation module removed). As shown in Table 5, it can be seen that compared with the baseline model (No:1), the model performance is significantly improved after using the multi-scale feature aggregation module (No:2) proposed in this invention. To increase the interpretability of the multi-scale feature aggregation module, this invention visualizes the results of the multi-scale feature aggregation module. For example... Figure 6 As shown, it can be seen that low-level features (such as...) Figure 6 f in l It contains rich detail and background noise, and advanced features (such as...) Figure 6 f hThe features contain rich semantic information. After passing through the multi-scale feature aggregation module, the feature noise is reduced, and the semantic information is rich and the boundaries are clear. One possible explanation is that the channel attention mechanism in the multi-scale aggregation module can suppress noise in low-level features and make feature channels containing complete semantic information more prominent.

[0161] Table 5. Experimental results on the effectiveness of the multi-scale feature aggregation module.

[0162]

[0163] 4) Verifying the effectiveness of the cascaded correction decoder: To verify the effectiveness of the cascaded correction decoder, this invention conducted two sets of comparative experiments based on the baseline model (a two-stream encoder composed of Swing Transformers, a cross-modal feature enhancement module, an edge extraction module, an attention feature enhancement module, and a single-layer decoder with an MSFA module). As shown in Table 6, it can be seen that compared to the baseline model (No:1), the cascaded correction decoder (No:2) can bring a significant performance improvement to the model. This is mainly because the cascaded correction decoder can repair the information loss caused by the upsampling operation. It supervises the salient features predicted by the sub-decoder and adds these features to the features of the previous layer step by step, achieving the purpose of repairing and refining the features of the previous layer. Figure 6 It can be observed that the features (Up) after the upsampling operation have blurred boundaries and information loss; the features after the cascaded correction decoder (as shown in f in the figure) ccd The boundaries become clearer and the information more complete because of the effect of repairing "incomplete" features using supervised features. Furthermore, the saliency maps predicted by the model of this invention are highly similar to the ground truth, further validating the superior performance of the model of this invention.

[0164] Figure 6 Visualization of the cascaded calibration decoder feature map. From left to right, the image consists of: RGB image, depth map, low-level features, high-level features, aggregated features after multi-scale feature aggregation module, features after upsampling operation, output features of the cascaded decoder, saliency map predicted by the model, and ground truth saliency map.

[0165] Table 6. Effectiveness Experiment of Cascaded Correction Decoder

[0166]

[0167] This invention proposes a cascaded aggregation network based on the Swin Transformer. Specifically, it uses the Swin Transformer as the backbone network of the encoder to extract features from RGB and depth images. An attention-based feature enhancement module is proposed, which utilizes dilated convolution to extract multi-scale information of high-level features. Simultaneously, a self-attention branch is added after the dilated convolution to compensate for the loss of local information after dilated convolution. Furthermore, to maximize the aggregation of useful information during decoding, a multi-scale feature aggregation module and a cascaded correction decoder are proposed to refine and correct the fused features. Comprehensive experiments were conducted on seven benchmark datasets, and the experimental results demonstrate the effectiveness of each proposed module and the superior performance of the model.

[0168] The present invention also provides a salient target detection device, comprising:

[0169] The image acquisition module is used to acquire the image to be detected;

[0170] An encoder is used to extract RGB features and depth features from the image to be detected;

[0171] The attention feature enhancement module is used to enhance the high-level features of RGB features and deep features respectively;

[0172] A cross-modal fusion module is used to enhance and fuse the RGB features and depth features;

[0173] An edge extraction module is used to generate edge information of salient objects in the depth features;

[0174] The multi-scale feature aggregation module is used to perform multi-scale feature aggregation on the enhanced and fused RGB features and depth features to obtain multi-level fused features.

[0175] A cascaded correction decoder is used to refine and correct the multi-level fused features, generate salient features, and then use edge information to enhance the salient features to generate the final salient map.

[0176] This invention also provides a storage medium storing a computer program. When executed by a processor, the computer program implements some or all of the steps in the various embodiments of the salient target detection method provided by this invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0177] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0178] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the embodiments for the salient target detection device are basically similar to the method embodiments, so they are described more simply, and the relevant parts can be referred to the description in the method embodiments.

[0179] The embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A salient target detection method, characterized in that, include: Acquire the image to be detected; The encoder extracts RGB and depth features from the image to be detected. The attention feature enhancement module is used to enhance the high-level features of RGB features and deep features respectively; The extracted RGB and depth features are input into the cross-modal fusion module to enhance and fuse the RGB and depth features; The edge extraction module is used to generate edge information of salient objects in the depth features; The enhanced and fused RGB features and depth features are input into the multi-scale feature aggregation module to perform multi-scale feature aggregation and obtain multi-level fused features. The multi-level fused features are input into the cascaded correction decoder to refine and correct the multi-level fused features, generate salient features, and then the salient features are enhanced using edge information to generate the final salient map. The attention feature enhancement module performs feature enhancement on the high-level features of both RGB and deep features, including: Attention feature enhancement modules are added to the tail of the RGB stream encoder and the depth stream encoder respectively. The attention feature enhancement modules are used to perform three types of dilated convolution with kernel size of 3 and dilation rate of 2, 4 and 6 on high-level features, as well as 1x1 convolution and global average pooling operations to obtain multi-scale features. Perform self-attention mechanism operations on multi-scale features to obtain global context information; The feature maps that have undergone self-attention, 1x1 convolution, and global average pooling are concatenated to obtain the enhanced features. The extracted RGB and depth features are input into the cross-modal fusion module to enhance and fuse the RGB and depth features, including: The cross-modal feature fusion module includes a deep feature enhancement module, which uses a spatial attention mechanism to compute a shared spatial attention map of RGB features and deep features; The spatial attention map is used as the weight of the depth features, and a residual connection is made between the original depth features and the spatial attention map. Calculate the channel attention map using the channel attention mechanism; The channel attention map is multiplied with the original depth features to obtain the enhanced depth features; The enhanced RGB and depth features are input into the cross-modal fusion module to enhance and fuse the RGB and depth features, including: The cross-modal feature fusion module includes an RGB feature enhancement module, which uses a spatial attention mechanism to compute a shared spatial attention map of RGB features and depth features. The spatial attention map is used as the weight of the RGB features, and a residual connection is made between the original RGB features and the spatial attention map. Calculate the channel attention map using the channel attention mechanism; The channel attention map is multiplied with the original RGB features to obtain the enhanced RGB features; The enhanced RGB and depth features are input into the cross-modal fusion module to enhance and fuse the RGB and depth features, including: The cross-modal feature fusion module includes a fusion block, which fuses the enhanced depth features and RGB features. The edge extraction module is used to generate edge information of salient objects in the depth features, including: Perform 1x1 convolution and upsampling operations on the depth features to generate two features of the same scale; The two features of the same scale are concatenated to generate edge features; The edge features are processed using a channel attention mechanism and residual connections to generate edge information. The multi-level fused features are input into a cascaded correction decoder to refine and correct the multi-level fused features, generating salient features, including: The cascaded correction decoder includes a first sub-decoder and a second sub-decoder. The first and second sub-decoders each contain three multi-scale feature aggregation modules. The cascaded correction decoder includes two processes: bottom-up and top-down. For the bottom-up process, the multi-scale feature aggregation module in the sub-decoder gradually aggregates from high-level features to low-level features. The aggregated features are used to generate a coarse saliency map, which is then supervised. For the top-down process, the features output by the top multi-scale feature aggregation module in the first sub-decoder are directly downsampled into features and added to the multi-level features exported by the multi-scale feature aggregation module for refinement and correction. The corrected features are sent to the next sub-decoder to generate features with complete information through the same process; Generate a saliency map based on features with complete information.

2. The salient target detection method as described in claim 1, characterized in that, The encoder extracts RGB and depth features from the image to be detected, including: The encoder includes an RGB stream encoder and a depth stream encoder. The RGB stream encoder and the depth stream encoder use Swin-B as the backbone network to extract multi-level features of the RGB image and the depth image, respectively.

3. The salient target detection method as described in claim 1, characterized in that, The enhanced and fused RGB features and depth features are input into the multi-scale feature aggregation module for multi-scale feature aggregation, resulting in multi-level fused features, including: Operations include upsampling of high-level features, 3x3 convolution, and adjustment of the number of channels; By multiplying elements, high-level features are transformed and fused with low-level features to obtain fused features, which possess the characteristics of both high-level and low-level features. A channel attention mechanism is used to refine the fused features, resulting in multi-level fused features.

4. A salient target detection device, used to implement the salient target detection method as described in claim 1, characterized in that, include: The image acquisition module is used to acquire the image to be detected; An encoder is used to extract RGB features and depth features from the image to be detected; The attention feature enhancement module is used to enhance the high-level features of RGB features and deep features respectively; A cross-modal fusion module is used to enhance and fuse the RGB features and depth features; An edge extraction module is used to generate edge information of salient objects in the depth features; The multi-scale feature aggregation module is used to perform multi-scale feature aggregation on the enhanced and fused RGB features and depth features to obtain multi-level fused features. A cascaded correction decoder is used to refine and correct the multi-level fused features, generate salient features, and then use edge information to enhance the salient features to generate the final salient map.

Images