An RGB-D salient object detection method based on unwrapping representation learning

Abstract

The application discloses an RGB-D salient object detection method based on disentangled representation learning, which respectively learns the disentangled representation of salient objects, namely the main part and the detail part, by using RGB and depth branches. Specifically, in order to reduce the modal difference, the application proposes a focus-alignment-refinement module, guides fine-grained alignment through the semantic information of the salient object prototype, and refines the features by using supplementary modalities. Since some poor-quality depth maps inevitably introduce noise, damage the fine-grained representation, and thus lead to excessive or insufficient detection, the application designs an uncertainty feature perception module, which explores high-uncertainty areas by establishing a pixel-level probability distribution. In order to handle object scale changes, the application proposes a progressive scale perception module, which expands the range of feature perception by cascading convolutions of different sizes and inflation rates. Finally, the application introduces an adjacent shrinkage decoder, which generates high-quality masks by aggregating consecutive and adjacent features. Extensive experiments on five RGB-D SOD datasets prove that the application outperforms the state-of-the-art methods, and the results on underwater and mirror scene datasets further demonstrate impressive generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of salient object detection (SOD) technology, and more specifically, relates to an RGB-D salient object detection method based on unentangled representation learning. Background Technology

[0002] Salient object detection can guide downstream tasks in computer vision. Based on its development, salient object detection can be divided into two categories: the first involves hand-designed features or incorporating prior features, such as background priors, center priors, context priors, and shape priors; however, these methods heavily rely on expert knowledge and lack generalization ability. The second category utilizes convolutional neural networks, the Transformer family, or a combination of both to extract high-dimensional features from images, employing various enhancement strategies such as multi-scale perception, edge supervision, and hybrid losses to explore complete salient objects. While these methods have achieved promising results, there is still significant room for improvement in challenging scenarios such as low contrast, multiple objects, and complex backgrounds. Therefore, some recent work has focused on two main aspects: 1) increasing image resolution to enrich visual information; and 2) attempting to introduce additional visual or linguistic cues as guidance or supplementation to improve detection performance and robustness. For example, utilizing depth differences in depth maps can effectively locate approximate edges or contours of people or objects, laying the foundation for further salient object detection.

[0003] In terms of cross-modal feature fusion, current RGB-D salient object detection methods mainly focus on leveraging the complementary advantages of RGB and depth maps (D) and have designed various cross-modal fusion strategies. Sun et al. improved cross-modal feature integration by utilizing depth-sensitive attention and automatic architecture search, and refined it using geometric priors. See Peng Sun, Wenhu Zhang, Huanyu Wang, Songyuan Li, and Xi Li. 2021. Deep RGB-D saliency detection with depth-sensitive attention and automatic multimodal fusion. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 1407–1417. Liao et al. enhanced feature fusion by using a cross-collaborative encoder and a cross-modal decoder, achieving the aggregation of multi-scale complementary information and improving the robustness of detection. See Guibiao Liao, Wei Gao, Ge Li, Junle Wang, and Sam Kwong. 2022. Cross collaborative fusion-encoder network for robust RGB-thermal salient object detection. IEEE Transactions on Circuits and Systems for Video Technology 32, 11(2022), 7646–7661. Wan et al. proposed an architecture with a trimodal depth fusion encoder and a progressive feature enhancement decoder, aiming to integrate visible light, depth, and thermal infrared information. See Bin Wan, Xiaofei Zhou, Yaoqi Sun, Tingyu Wang, Chengtao Lv, Shuai Wang, Haibing Yin, and Chenggang Yan. 2023. MFFNet: Multi-modal Feature Fusion Network for VDTSalient Object Detection. IEEE Transactions on Multimedia (2023).Ma et al.

[47] proposed an innovative feature synthesis plug-in module in a dual-branch RGB-X network, which has an energy exchange module that facilitates interaction between different scales and modalities. See Jin Ma, Jinlong Li, Qing Guo, Tianyun Zhang, Yuewei Lin, and Hongkai Yu. 2023. RXFOOD: Plug-in RGB-X Fusion for Object of Interest Detection. arXiv preprint arXiv:2306.12621 (2023). Xiao et al. proposed a depth-guided fusion method that combines depth map enhancement techniques to improve the quality of depth information used in feature fusion. See Fen Xiao, Zhengdong Pu, Jiaqi Chen, and Xieping Gao. 2023. DGFNet: Depth guided cross-modality fusion network for RGB-D salient object detection. IEEE Transactions on Multimedia (2023). Wu et al. used Transformer fusion and pixel-level contrastive learning to capture long-range interactions and improve the ability to distinguish between pixels. See Jiesheng Wu, Fangwei Hao, Weiyun Liang, and Jing Xu. 2023. Transformer fusion and pixel-level contrastive learning for RGB-D salient object detection. IEEE Transactions on Multimedia (2023). Hu et al.

[17] combined dual-stream Swin Transformer with cross-modal attention mechanism for the fusion of complex features. See Xihang Hu, Fuming Sun, Jing Sun, Fasheng Wang, and Haojie Li. 2024. Cross ModalFusion and Progressive Decoding Network for RGB-D Salient Object Detection. International Journal of Computer Vision (2024), 1–19. These methods attempt to design various fusion strategies to integrate multimodal information, but lack exploration of modal characteristics.

[0004] In uncertainty estimation, fine-grained features often weaken or disappear due to the introduction of mismatched depth information. Therefore, it is necessary to estimate the uncertainty of feature maps. Uncertainty can be divided into aleatoric uncertainty and epistemic uncertainty, which describe the confidence of the input data and the prediction result, respectively. The former can be learned implicitly by the network, while the latter usually requires additional methods for measurement, such as Gaussian mixtures and deep ensembles. Unlike the methods mentioned above, these methods cannot use gradient optimization.

[0005] Specifically, the challenges in cross-modal feature fusion mainly come from three aspects: a) complex visual relationships in RGB images, b) imaging quality of depth maps, and c) alignment of corresponding features. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an RGB-D salient target detection method based on unentangled representation learning. This method mines common features to reduce interference from complex visual representations and low-quality depth information, and achieves pixel-by-pixel modal alignment, coarse-to-fine level complementary feature calibration. At the same time, it enables fine-grained differentiation and captures objects of different scales, generating more refined masks.

[0007] To achieve the above-mentioned objectives, the present invention provides an RGB-D salient target detection method based on unentangled representation learning, characterized by comprising the following steps:

[0008] (1) Obtaining multi-scale feature maps X i ,F i

[0009] Using a Pyramid Vision Transformer (PVT) network as the backbone network, for input images I∈R 3×H×W Extracting multi-scale feature maps Among them, C i H and W represent the number of channels, height, and width, respectively;

[0010] Depth map D of input image I o ∈R 1×H×W Adjusted to D via convolution c ∈R 3×H×W Then it is input into another pyramid visual transformer to extract multi-scale feature maps.

[0011] (2) Focus-Alignment-Refinement

[0012] The Focus-Alignment-Refinement (FAR) module is used to refine multi-scale feature maps X. i ,F i Align and fuse features of the same scale to generate feature maps.

[0013] 2.1) Alignment from thick to thin

[0014] First, for the multi-scale feature map X i ,F i Regularization and 3x3 convolution are performed, followed by the use of a softmax function and reshaping to generate attention. and Where m represents the number of prototypes, the process is expressed as:

[0015]

[0016] Where Att represents a combination of regularization, convolution, softmax, and reshaping;

[0017] Then, the cluster centers are randomly initialized. A significant prototype is obtained through summation and regularization.

[0018]

[0019] Where Reshape(·) represents the reshaping operation, All prototypes accumulated, Represents the L2 norm;

[0020] Then, the significant prototype With multi-scale feature map X i Significant prototype With multi-scale feature map F i The feature maps are fused to generate prototype-based feature maps. and

[0021]

[0022] Where Γ(,) represents combining by channel and performing a 1×1 convolution, followed by the ReLU activation function;

[0023] Then, a 1×1 convolution followed by a 3×3 depthwise separable convolution is used to process the multi-scale feature map X. i ,F i and feature maps Encode and generate feature maps and feature maps

[0024] Then, the feature map As a query, feature map As the key, calculate the correlation matrix in the channel dimension:

[0025]

[0026] Among them, the correlation matrix Where h is the number of attention heads, T represents transpose, σ is the softmax function, τ is the learnable scaling factor, and Reshape(·) represents the reshaping operation;

[0027] Finally, the enhanced feature map is obtained.

[0028]

[0029] 2.2) Refine

[0030] For feature maps They are spatially compressed separately and then fused to obtain a channel map. for:

[0031]

[0032] Where Avg represents average pooling, Concat represents channel concatenation, and superscript 1→C i This indicates taking channels 1 to C. i The element, superscript C i +1→2C i Indicates the acquisition of channel C i +1 to 2C i Element;

[0033] Finally, a refined feature map is generated.

[0034]

[0035] Where ⊙ represents element-wise multiplication;

[0036] (3) Perception of uncertainty characteristics

[0037] Uncertainty feature perception (UFP) module is used to obtain uncertainty perception feature M. u :

[0038] 3.1) Establish a Laplace distribution for each pixel of the feature map and construct an uncertainty map:

[0039] First, by refining the feature map The mean was obtained by using two different 1×1 convolutions:

[0040]

[0041] Where Γ′ and Γ′′ represent two different 1×1 convolutions,

[0042] Then, random sampling is performed multiple times to generate the variable ξ from the standard Laplace distribution, so as to obtain a new uncertainty distribution of the pixel, i.e., L = μ + ξb;

[0043] Then, the variance is calculated and normalized to obtain the uncertainty diagram.

[0044] U=Norm(Var(φ(sample(L)))) (9)

[0045] Where Norm(·), VAR(·), φ(·) and sample(·) represent normalization, sample variance, Sigmoid function and sampling, respectively;

[0046] Refine the feature map The inputs are fed into an adjacency shrinkage (AS) decoder to generate feature maps. Then, the feature maps are concatenated by channel to obtain the feature maps.

[0047]

[0048] in,

[0049] Perform a 1×1 convolution on the feature map M, then perform a 3×3 convolution three times to obtain the query M. q , key M k Sum M v The convolution kernels used in the three convolutions are different;

[0050] Apply the uncertainty graph U to query M q , key M k Generate the correlation matrix corr u :

[0051]

[0052] in, This represents matrix multiplication, where τ1 is a learnable scaling factor;

[0053] Obtain the uncertainty-aware feature map

[0054] M u =M q ⊙corr u (12)

[0055] (4) Progressive scale perception

[0056] By using a Progressive Scale-Aware Module (PSAM), dilated convolutions at different scales are introduced to gradually expand the receptive field, thereby capturing objects of different scales:

[0057] 4.1) Generate feature maps

[0058]

[0059] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 3×3,delation＝1 This represents a 3x3 convolution with a dilation rate of 1. 3×3 This represents a 3×3 convolution;

[0060] Using the same method but with different convolution kernels, feature maps are generated.

[0061]

[0062] 4.2) Generate feature maps

[0063]

[0064] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×3 Represents a 1×3 convolution, Conv 3×1 Represents a 3×1 convolution, Conv 3×3,delation＝ This represents a 3x3 convolution with a dilation rate of 3;

[0065] Using the same method but with different convolution kernels, feature maps are generated.

[0066]

[0067] 4.3) Generate feature maps

[0068]

[0069] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×5Represents a 1×5 convolution, Conv 5×1 Represents a 5×1 convolution, Conv 3×3,delation＝ This represents a 3x3 convolution with a dilation rate of 5;

[0070] Using the same method but with different convolution kernels, feature maps are generated.

[0071]

[0072] 4.4) Generating feature maps

[0073]

[0074] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×7 Represents a 1×7 convolution, Conv 7×1 Represents a 7×1 convolution, Conv 3×3,delation＝7 This represents a 3x3 convolution with a dilation rate of 7;

[0075] Using the same method but with different convolution kernels, feature maps are generated.

[0076]

[0077] (5) Refined Adjacent Shrink Decoding

[0078] By using a refined adjacent shrinkage (RAS) decoder to progressively aggregate consecutive and adjacent layers, the gap between high and low layer information is reduced. Specifically, the RGB branch features are aggregated as follows:

[0079]

[0080] Among them, T i T i ′、T i "The feature maps obtained by hierarchical aggregation, T is the RGB branch feature map, and CBR represents the combination of convolution, BN layer and ReLU function;

[0081] Similarly, deep branch features are aggregated as follows:

[0082]

[0083] Among them, D i 、D′ i D″ i The feature map is obtained by hierarchical aggregation, where D is the deep branch feature map;

[0084] Finally, the predicted saliency map S is obtained:

[0085] First, the RGB branch feature map T and the depth branch feature map D are concatenated by channel, and then the concatenated feature map is used as the refined feature map in step (3). Uncertainty feature perception is performed to obtain uncertainty-perceived feature map M. u The prediction saliency map F serves as the fusion branch;

[0086] (6) Training

[0087] Based on steps (1) to (5), a progressive decoupling network based on uncertainty awareness is obtained. The labeled images in the training dataset are fed into the progressive decoupling network based on uncertainty awareness and processed according to steps (1) to (5) to obtain the RGB branch feature map T, the depth branch feature map D, and the predicted saliency map F of the fusion branch. Then, the loss is compared with the labeled RGB feature map, the depth feature map, and the saliency map S, respectively. as well as Total loss

[0088]

[0089] Where α, β, and γ are hyperparameters, and three losses as well as Each loss in the algorithm consists of a weighted binary cross-entropy loss, a weighted IoU loss, and an SSIM loss.

[0090] Based on total losses Train the uncertainty-aware progressive decoupling network until it converges;

[0091] (7) Target Detection

[0092] The image to be detected is fed into a progressive decoupling network based on uncertainty perception and processed according to steps (1) to (5) to obtain a predicted saliency map F, thus completing the saliency target detection.

[0093] The objective of this invention is achieved as follows.

[0094] RGB-D Satisfactory Object Detection (SOD) aims to enhance spatial awareness and guide the localization of salient regions using depth maps. Currently, many methods attempt to design various fusion strategies to integrate multimodal information, but lack exploration of modal characteristics. To address this issue, this invention utilizes RGB and depth branches to learn decoupled representations of salient objects, namely the backbone and details. Specifically, to reduce modal discrepancies, this invention proposes a Focus-Alignment-Refinement (FAR) module, which guides fine-grained alignment using semantic information of the salient object prototype and refines features using complementary modalities. Since some low-quality depth maps inevitably introduce noise, impairing fine-grained representations (e.g., irregular regions) and leading to over- or under-detection, this invention designs an Uncertainty Feature Awareness (UFP) module to explore high-uncertainty regions by establishing pixel-level probability distributions. To handle object scale variations, this invention proposes a Progressive Scale Awareness (PSA) module, which expands the range of feature awareness by cascading convolutions of different sizes and dilation rates. Finally, we introduce a Neighbor-Received Shrink (AS) decoder to generate high-quality masks by aggregating continuous and adjacent features. Extensive experiments on five RGB-D SOD datasets demonstrate that the present invention surpasses state-of-the-art methods, and results on underwater and mirror scene datasets further showcase impressive generalization capabilities. Attached Figure Description

[0095] Figure 1 This is a comparison of the architectures of RGB-D SOD models, where (a) is a single-stream model based on image-level fusion, (b) is a two-stream model based on feature interaction, (c) is a three-stream model based on intermediate modalities or features, and (d) is the PDUNet proposed in this invention, exploring backbone and detail features respectively.

[0096] Figure 2 This is a general structural diagram of a specific implementation of the RGB-D salient target detection method based on unentangled representation learning of the present invention;

[0097] Figure 3 This is a flowchart of a specific implementation of the RGB-D salient target detection method based on unentangled representation learning of the present invention;

[0098] Figure 4 This is a comparison chart of the efficiency and performance balance between the present invention and existing salient object detection methods, where (a) is an RGB image, (b) is a depth map, (c) is a ground truth map, (d) is the predicted salient map output by the present invention, (e) is the salient map output by PICRNet, and (f) is the salient map output by CoNet.

[0099] Figure 5 This is a qualitative comparison chart of the present invention and existing salient target detection methods in natural scenes;

[0100] Figure 6 This is a qualitative comparison chart of the present invention and existing salient target detection methods in underwater scenarios;

[0101] Figure 7 This is a qualitative comparison chart of the present invention and existing salient target detection methods in a mirror scene;

[0102] Figure 8 This is a visualization of the ablation analysis results of this invention. Detailed Implementation

[0103] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0104] SOD (Signal Decoupling) aims to locate the most attractive regions or objects in an image and can facilitate other tasks such as segmentation, visual language navigation, and image generation. However, it is often susceptible to interference from complex backgrounds. To address this issue, some methods attempt to introduce additional visual cues to improve the robustness of detection. Inspired by the fact that RGB images are rich in visual appearance (e.g., texture, color) and that depth maps can indicate the spatial geometry of objects, thus highlighting edges or shapes, some methods have explored various image or feature fusion approaches to facilitate the extraction of complementary information. However, performance still needs improvement when dealing with some challenging scenarios. Inspired by the paper Jun Wei, Shuhui Wang, Zhe Wu, Chi Su, Qingming Huang, and Qi Tian. 2020. Label decoupling framework for salient object detection. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 13025–13034], this invention decomposes salient objects into two parts: the trunk and the details, where the trunk emphasizes the central region of the object and the details represent the edges and their surrounding areas. Edge regions in RGB images are often blurry, while depth maps can highlight object contours based on depth differences. Therefore, the backbone and details of salient objects can be derived from feature decoding of the RGB image and depth map, respectively. It's important to note that this invention does not directly use the edge map as supervision for the depth branch, primarily for two reasons: 1) Depth maps are susceptible to estimation errors and struggle to accurately reflect the location and contours of salient objects, especially in extreme cases. 2) The ratio of edge pixels to nearby background pixels is severely imbalanced; therefore, directly using edge supervision would hinder the optimization process during training.

[0105] This invention further rethinks existing detection frameworks and proposes a decoupled representation learning architecture. For example... Figure 1As shown, existing RGB-D SOD methods can be broadly categorized into single-stream, two-stream, and three-stream structures based on their fusion strategies. Single-stream networks typically employ early fusion, fusing information at the image level. However, they lack the ability to explore modal differences, leading to the introduction of numerous disturbances. Two-stream networks consist of RGB images and depth maps forming encoding and decoding branches, respectively, and perform feature interaction. Three-stream networks construct branches sharing a common modality at the image or feature level, jointly learning with branches of two specific modalities. However, this significantly increases computational complexity. These frameworks are developed based on the advantages of fusing different modalities without exploring appropriate supervision information for each modality. This invention combines the advantages of two-stream and three-stream networks, promoting feature interaction in the encoding stage and integrating modal-specific features in the decoding stage. In terms of training, this invention uses the backbone, details, and binarized ground truth map as supervision for the RGB, depth, and fusion branches, respectively.

[0106] Figure 2 This is a general structural diagram of a specific implementation of the RGB-D salient target detection method based on unentangled representation learning of the present invention.

[0107] Based on the above discussion, this invention proposes a progressively decoupled network based on uncertainty awareness, namely PDUNet. The motivation for this invention is to decompose salient objects into a backbone and detail components, which are predicted by RGB and depth branches respectively, and then fused. By employing a decomposition-fusion process, this invention aims to explore the features of different modalities and leverage their respective advantages. Figure 2As shown, this invention follows the general paradigm of an encoder-decoder architecture, comprising four key elements. Specifically, the challenges in feature fusion mainly stem from three aspects: a) complex visual relationships in RGB images; b) imaging quality of depth maps; and c) alignment of corresponding features. To address these challenges, this invention introduces a FAR module to achieve calibration from a local-global-local perspective. This invention focuses on the RGB and depth information of salient objects through prototype clustering, mining common features to reduce interference from complex visual representations and low-quality depth information. These features are then fused with initial RGB or depth features to achieve pixel-by-pixel modality alignment, calibrating complementary features from coarse to fine levels. Subsequently, this invention refines the features through the FAR module, generating cross-modal representations with different emphases. Through the FAR module, this invention can effectively learn modality-sharing and modality-specific knowledge. However, the introduction of depth errors is particularly detrimental to weakly salient feature representations or fine-grained regions (such as small objects and irregular areas). To overcome this problem, this invention designs a UFP module that utilizes Bayesian learning to model the probability distribution of objects and fuses it into the feature map. By exploring regions of high uncertainty, we enhance the RGB representation and suppress depth errors, achieving fine-grained distinction. Furthermore, we observed significant scale variations in salient objects, ranging from occupying one-tenth to more than half of the image. Therefore, this invention proposes a PSA module, which gradually expands the feature perception range by using cascaded convolutional kernels with progressively increasing size and dilation rate to capture objects at different scales. Finally, similar to the U-shaped decoder that propagates high-level semantic information to low-level detail information layer by layer, this invention introduces an AS decoder, which aggregates features from consecutive and adjacent layers with similar semantics and detail representations to generate a more refined mask. This invention further proposes a refined AS (RAS) decoder, which combines the output of the AS decoder with the feature map generated by the encoder to refine the initial features. It is worth noting that this invention uses a UFP module to further enhance the output of the RAS decoder and obtain the final predicted mask.

[0108] In summary, the contributions of this invention are as follows:

[0109] This invention proposes a novel decoupled representation learning framework that explores the features of RGB and depth images and their correspondence with the backbone and details.

[0110] This invention proposes PDUNet by designing FAR and UFP modules, which narrows the modality gap, emphasizes beneficial feature representations, and introduces a PSA module to handle variations in target scale. Furthermore, we have developed an AS decoder to obtain high-quality masks.

[0111] This invention outperforms state-of-the-art methods on five widely used natural scene RGB-D datasets, as well as underwater and mirror detection datasets (RGB-D type), achieving a better balance between performance and efficiency.

[0112] Figure 3 This is a flowchart of a specific implementation of the RGB-D salient target detection method based on unentangled representation learning according to the present invention.

[0113] In this embodiment, as Figure 3 As shown, the RGB-D salient target detection method based on unentangled representation learning of the present invention includes the following steps:

[0114] Step S1: Obtain multi-scale feature map X i ,F i

[0115] Using a Pyramid Vision Transformer (PVT) network as the backbone network, for input images I∈R 3×H×W Extracting multi-scale feature maps Among them, C i H and W represent the number of channels, height, and width, respectively.

[0116] Depth map D of input image I o ∈R 1×H×W Adjusted to D via convolution c ∈R 3×H×W Then it is input into another pyramid visual transformer to extract multi-scale feature maps.

[0117] In this embodiment, C1 = 64, C2 = 128, C3 = 320, and C4 = 640.

[0118] Step S2: Focus - Align - Refine

[0119] Based on the complex visual relationships in RGB images and the imaging quality of depth images, different regions of salient objects may have weakened features, while the most prominent parts still retain good representation. This invention generates salient prototypes through clustering to obtain common feature representations focused on core regions. Furthermore, this invention further utilizes the generated prototypes as a supplement to the initial features, achieving feature alignment from coarse to fine levels. It is noteworthy that, unlike previous works that align prototypes or pixels, leading to over- or under-alignment, this invention combines the advantages of both by guiding key features to align details. Moreover, since the representations learned by the RGB and depth branches differ, this invention refines them to obtain features with different emphases. Specifically, this invention utilizes a Focus-Alignment-Refinement (FAR) module to refine multi-scale feature maps X. i ,F i Align and fuse features of the same scale to generate feature maps.

[0120] Step S2.1: Alignment from coarse to fine

[0121] First, for the multi-scale feature map X i ,F i Regularization and 3x3 convolution are performed, followed by the use of a softmax function and reshaping to generate attention. and Where m represents the number of prototypes, the process is expressed as:

[0122]

[0123] Here, Att represents a combination of regularization, convolution, softmax, and reshaping.

[0124] Then, the cluster centers are randomly initialized. A significant prototype is obtained through summation and regularization.

[0125]

[0126] Where Reshape(·) represents the reshaping operation, All prototypes accumulated, This represents the L2 norm.

[0127] Then, the significant prototype With multi-scale feature map X i Significant prototype With multi-scale feature map F i The feature maps are fused to generate prototype-based feature maps. and

[0128]

[0129] Here, Γ(,) represents combining by channel and performing a 1×1 convolution, followed by the ReLU activation function.

[0130] Then, a 1×1 convolution followed by a 3×3 depthwise separable convolution is used to process the multi-scale feature map X. i ,F i and feature maps Encode and generate feature maps and feature maps

[0131] Then, the feature map As a query, feature map As the key, calculate the correlation matrix in the channel dimension:

[0132]

[0133] Among them, the correlation matrix Where h is the number of attention heads, T represents transpose, σ is the softmax function, τ is the learnable scaling factor, and Reshape(·) represents the reshaping operation.

[0134] Finally, the enhanced feature map is obtained.

[0135]

[0136] Step S2.2: Refine

[0137] For feature maps They are spatially compressed separately and then fused to obtain a channel map. for:

[0138]

[0139] Where Avg represents average pooling, Concat represents channel concatenation, and superscript 1→C i This indicates taking channels 1 to C. i The element, superscript C i +1→2C i Indicates the acquisition of channel C i +1 to 2C i Element.

[0140] Finally, a refined feature map is generated.

[0141]

[0142] Here, ⊙ represents element-wise multiplication.

[0143] Step S3: Uncertainty Feature Perception

[0144] By aligning layers from coarse to fine, this invention reduces the overall interference of depth errors on salient objects. However, for regions with weak feature representations or fine-grained details, introducing erroneous information can degrade or even blur the original representation, thereby increasing uncertainty. To address this issue, this invention proposes an Uncertainty Feature Perception (UFP) module, which introduces probabilistic modeling to locate regions with high uncertainty and further reduce depth noise. Specifically, the uncertainty-perceived feature M is obtained using the uncertainty feature perception module. u :

[0145] Step S3.1: Establish a Laplace distribution for each pixel of the feature map and construct an uncertainty map:

[0146] First, by refining the feature map The mean was obtained by using two different 1×1 convolutions:

[0147]

[0148] Where Γ′ and Γ′′ represent two different 1×1 convolutions,

[0149] Then, random sampling is performed multiple times to generate the variable ξ from the standard Laplace distribution, so as to obtain a new uncertainty distribution of the pixel, namely L=μ+ξb.

[0150] Then, the variance is calculated and normalized to obtain the uncertainty diagram.

[0151] U=Norm(Var(φ(sample(L)))) (9)

[0152] Where Norm(·), VAR(·), φ(·) and sample(·) represent normalization, sample variance, Sigmoid function and sampling, respectively.

[0153] Refine the feature map The inputs are fed into an adjacency shrinkage (AS) decoder to generate feature maps. Then, the feature maps are concatenated by channel to obtain the feature maps.

[0154]

[0155] in,

[0156] Perform a 1×1 convolution on the feature map M, then perform a 3×3 convolution three times to obtain the query M. q , key M k Sum M v The convolution kernels used in the three convolutions are different.

[0157] Apply the uncertainty graph U to query M q , key M k Generate the correlation matrix corr u :

[0158]

[0159] in, This represents matrix multiplication, where τ1 is a learnable scaling factor.

[0160] Obtain the uncertainty-aware feature map

[0161] M u =M q ⊙corr u (12)

[0162] Step S4: Progressive Scale Awareness

[0163] The significant differences in the size of salient regions make it difficult to locate small objects and detect large, intact objects. Therefore, this invention proposes a Progressive Scale-Aware Module (PSAM), which gradually expands the receptive field by introducing dilated convolutions of different scales to capture objects of varying sizes. Specifically, the Progressive Scale-Aware Module introduces dilated convolutions of different scales to gradually expand the receptive field, thereby capturing objects of different scales:

[0164] Step S4.1: Generate feature maps

[0165]

[0166] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 3×3,delation＝ This represents a 3x3 convolution with a dilation rate of 1. 3×3 This represents a 3×3 convolution.

[0167] Using the same method but with different convolution kernels, feature maps are generated.

[0168]

[0169] Step S4.2: Generate feature maps

[0170]

[0171] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×3 Represents a 1×3 convolution, Conv 3×1 Represents a 3×1 convolution, Conv 3×3,delation＝3 This represents a 3x3 convolution with a dilation rate of 3.

[0172] Using the same method but with different convolution kernels, feature maps are generated.

[0173]

[0174] Step S4.3: Generate feature maps

[0175]

[0176] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×5 Represents a 1×5 convolution, Conv 5×1 Represents a 5×1 convolution, Conv 3×3,delation＝5 This represents a 3x3 convolution with a dilation rate of 5;

[0177] Using the same method but with different convolution kernels, feature maps are generated.

[0178]

[0179] Step S4.4: Generate feature maps

[0180]

[0181] Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×7 Represents a 1×7 convolution, Conv 7×1 Represents a 7×1 convolution, Conv 3×3,delation＝7 This represents a 3x3 convolution with a dilation rate of 7.

[0182] Using the same method but with different convolution kernels, feature maps are generated.

[0183]

[0184] Step S5: Refined Adjacent Shrink Decoding

[0185] Existing decoding paradigms mainly include: 1) top-down approach, which may lead to the loss of semantic information; 2) dense interaction, resulting in high computational cost and feature redundancy; 3) separation of high and low layers, leading to conflicts between details and semantic features. Therefore, this invention proposes a Refined Adjacent Shrinkage (RAS) decoder that progressively aggregates continuous and adjacent layers to reduce the gap between high and low layer information, thereby capturing focused features at different levels to decode high-quality masks. Specifically, the RGB branch feature aggregation is as follows:

[0186]

[0187] Among them, T i 、T′ i 、T″ i The feature maps obtained by hierarchical aggregation are T, which is the RGB branch feature map, and CBR represents the combination of convolution, BN layer and ReLU function.

[0188] Similarly, deep branch features are aggregated as follows:

[0189]

[0190] Among them, D i 、D′ i D″ i The feature map is obtained by hierarchical aggregation, where D is the deep branch feature map.

[0191] Finally, the predicted saliency map S is obtained:

[0192] First, the RGB branch feature map T and the depth branch feature map D are concatenated by channel, and then the concatenated feature map is used as the refined feature map in step (3). Uncertainty feature perception is performed to obtain uncertainty-perceived feature map M. u The prediction saliency map F is the fusion branch.

[0193] Step S6: Training

[0194] Based on steps (1) to (5), a progressive decoupling network based on uncertainty awareness is obtained. The labeled images in the training dataset are fed into the progressive decoupling network based on uncertainty awareness and processed according to steps (1) to (5) to obtain the RGB branch feature map T, the depth branch feature map D, and the predicted saliency map F of the fusion branch. Then, the loss is compared with the labeled RGB feature map, the depth feature map, and the saliency map S, respectively. as well as Total loss

[0195]

[0196] Where α, β, and γ are hyperparameters, and three losses as well as Each loss in the algorithm consists of weighted binary cross-entropy loss, weighted IoU loss, and SSIM loss.

[0197] Based on total losses Train the uncertainty-aware progressive decoupling network until it converges;

[0198] Step S7: Detection of salient targets

[0199] The image to be detected is fed into a progressive decoupling network based on uncertainty awareness and processed according to steps S1 to S5 to obtain a predicted saliency map F, thus completing the salient target detection.

[0200] Figure 4 This is a comparison chart of the efficiency and performance balance between the present invention and existing salient object detection methods, where (a) is an RGB image, (b) is a depth map, (c) is a ground truth map, (d) is the predicted salient map output by the present invention, (e) is the salient map output by PICRNet, and (f) is the salient map output by CoNet.

[0201] from Figure 4 It can be seen that the present invention achieves the best balance between efficiency and performance.

[0202] experiment

[0203] 1. Experimental setup

[0204] Benchmark datasets: Experiments were conducted on seven datasets. RGB-D datasets for natural scenes: STERE1000

[53] , SIP

[12] , NJU2K

[27] , NLPR

[57] and DUT

[58] . Underwater dataset: USOD10K

[16] . Mirror dataset: RGBD-Mirror

[51] .

[0205] Implementation details: The invention was implemented using PyTorch and all experiments were conducted on an NVIDIA RTX A100. The training set consisted of 1485 images from the NJU2K dataset and 700 images from the NLPR dataset. Evaluation was performed on the corresponding test sets as well as the STERE1000 and SIP datasets. For the DUT dataset, the original training set and the training sets from NJU2K and NLPR were used, and the original test set was used for evaluation. For the USOD10K and RGBD-Mirror datasets, a PVT network

[69] pre-trained on ImageNet was used as the encoder for RGB images and depth maps. All input images were scaled to 384×384. During training, Adam was used as the optimizer for 200 epochs with an initial learning rate of 1e-4 and a batch size of 16. During testing, no tricks (e.g., test-time data augmentation) or post-processing (e.g., CRF) were used

[66] .

[0206] Evaluation metrics: Five evaluation metrics were used: S-measure (Sα)

[10] , mean E-measure (Eξ)

[11] , weighted F-measure (Fw), adaptive F-measure (Fβ)

[49] , and Mean Absolute Error (M). It should be noted that the higher the first four metrics, the better.

[0207] 2. Comparison with state-of-the-art methods

[0208] This invention is compared with 13 state-of-the-art RGB-D SOD models, including DANet

[86] , BBSNet

[13] , DCFNet

[21] , DSA2F

[64] , VST

[40] , C2DFNet

[83] , DCMF

[68] , SPSN

[30] , HiDANet

[73] , PICRNet

[74] , PopNet

[74] , DCTNet[2], DFormer

[78] , two RGB-T SOD models, namely CCFENet

[36] and LSNet

[88] , and two bimodal methods, namely HRTransNet

[65] and CAVER

[56] , on five natural scene datasets. To verify the generalization ability of this invention, it was compared with eight state-of-the-art methods on the USOD10K dataset, namely DANet, UCNet

[82] , TriTransNet

[44] , SPNet

[87] , HAINet

[34] , SVAMNet

[20] , DMRA

[23] , and TC-USOD

[16] (specifically for underwater SOD). In addition, it was compared with six state-of-the-art RGB-D SOD methods on the RGBD-Mirror dataset, namely BBSNet, HDFNet

[54] , DANet, S2MA

[39] , JL-DCF

[12] , VST, and two mirror detection methods, namely PDNet and SATNet. To ensure fairness, the evaluation results were obtained by retraining using the saliency maps provided by the authors or using the provided official source code.

[0209] 2.1 Quantitative Assessment

[0210]

[0211] Table 1

[0212] Table 1 shows the S values ​​on five benchmark datasets. α F β E ξ A quantitative comparison with M(MAE) is presented. As shown in Table 1, our invention achieves state-of-the-art performance on all metrics across five widely used datasets. Particularly noteworthy is our invention's performance on the large-scale and challenging SIP dataset. α F β E ξThe performance of the method is improved by 0.9%, 2.0%, 1.1%, and 0.6% compared to the second-best RGB-D SOD method, DFormer, on M and M, respectively. Furthermore, the invention surpasses the state-of-the-art bimodal SOD method, CAVER, with improvements of 1.3%, 3.0%, 1.6%, and 0.8%, respectively, and the RGB-T SOD method, LSNet, with improvements of 3.1%, 4.7%, 3.3%, and 2.1%, respectively.

[0213]

[0214] Table 2

[0215]

[0216] Table 3

[0217] Tables 2 and 3 show the quantitative comparisons on the USOD10K and RGBD-Mirror datasets, respectively. As shown in Tables 2 and 3, this invention also demonstrates excellent generalization ability, outperforming the state-of-the-art methods TC-USOD and SATNet on the USOD10K and RGBD-Mirror datasets, respectively, with improvements of 1.0%, 2.1%, 0.6%, 0.3% and 1.0%, 1.1%, 1.7%, -0.2%. These results verify that this invention can effectively handle various complex scenarios.

[0218] 2.2 Qualitative Assessment

[0219] like Figure 5 As shown, this experiment provides comparative examples of challenging scenarios. Our invention effectively establishes foreground-background differences in low-contrast (rows a and b) and low-quality depth maps (rows b, c, and d), utilizing beneficial depth information while reducing the interference of depth errors. Despite significant variations in object scale (rows c and d), our invention captures long-range and short-range feature dependencies, achieving accurate localization and detection. In scenes with composite objects (rows f and g), the depth difference between target and non-target regions is not significant, and most methods incorrectly detect trapezoidal and round rocks due to misleading depth maps. In contrast, our invention utilizes RGB information for separation. When dealing with irregularly shaped targets (rows e, h, and k), our invention leverages uncertainty modeling to utilize fine-grained feature cues and enhance detail information. Furthermore, our invention achieves complete detection without omission when handling multiple targets (rows i and j). Figure 6 and Figure 7 As shown, this experiment also provides a visual comparison of underwater and mirrored scenes. From Figure 6 and Figure 7 As can be seen, the present invention can address the challenges of specific tasks, such as camouflage and reflection.

[0220] 2.3 Ablation Experiment

[0221] To verify the effectiveness of each key module and hyperparameter, pruning experiments were conducted on the NJU2K and DUT datasets, including quantitative analysis (Tables 4, 5, and 6) and qualitative analysis (…). Figure 8 ).

[0222] 2.3.1 Validity of the FAR module.

[0223]

[0224] Table 4

[0225] Table 4 shows the ablation analysis of the modules proposed in this invention, where B, I1, I2, and I3 represent the baseline, FAR module, UFP module, and PSA module, respectively. As shown in Table 4, integrating the FAR module into the baseline model achieved average improvements of 1.6%, 1.1%, and 1.5% in Sα, Fβ, and Eξ, respectively, while simultaneously reducing M by an average of 0.5%. Figure 8 In (a, b, c), the baseline model exhibits significant deficiencies in low-quality depth maps. After integrating the FAR module, the detection of the horse tail (a), the left mirror (b), and the small fish (c) is significantly improved, demonstrating the effective feature alignment mechanism of the FAR module, which retains useful information while discarding harmful information.

[0226] 2.3.2 Validity of the UFP Module

[0227] As shown in Table 4, the average improvement after adding the UFP module was 0.8%, 0.7% (0.65%), and 0.8%, while the average reduction was 0.4% (0.35%) for M. Figure 8 In our experiments, adding the UFP module expanded the detection area (the horse's tail in a and the left mirror in b) and also improved the object's outline (the fish's tail in c). This demonstrates that the UFP module can enhance the perception of fine-grained features and edge regions, highlighting detailed information.

[0228] 2.3.3 Validity of the PSA Module

[0229] As shown in Table 4, adding the PSA module resulted in average improvements of 1% (0.95%), 1.3%, and 1.1% (1.05%), while the average reduction in performance was 0.5%. Furthermore, the combination with the FAR module further enhanced overall performance, demonstrating their complementarity. Figure 8 In (b), by incorporating the PSA module to improve the distance dependence of features, we can simultaneously locate three mirrors, although the localization of the right mirror is still inaccurate.

[0230] 2.3.4 Decoder Type

[0231]

[0232] Table 5

[0233] Table 5 presents the ablation analysis results for different decoders, where U, D, F, H, and S represent top-down, densely connected, feature feedback, high-low layer heterogeneity (separable), and feature contraction, respectively. To verify the superiority of the AS and RAS decoders, as shown in Table 5, we applied different types of decoders, including top-down, densely connected, feature feedback, high-low layer heterogeneity (separable), and feature contraction. Among these methods, we found that the AS and RAS decoders achieved the best performance, demonstrating the effectiveness of progressive aggregation of neighboring and continuous features.

[0234] 2.3.5 Weights of each loss function

[0235]

[0236] Table 6

[0237] To find the optimal weight combination for the three-branch loss function, we set the weight of the fusion branch to 1, and the weights β,γ ∈ [0.25, 0.50, 0.75, 1] ​​for the trunk and detail branches. Through multiple iterations of different weight combinations, as shown in Table 6, we found that the optimal combination is α = β = γ = 1.

[0238] This invention rethinks existing RGB-D salient object detection frameworks and proposes PDUNet. It focuses on exploring modal properties by unpacking salient objects and learning heterogeneous representations through different branches. To overcome the shortcomings of current modality fusion methods, it introduces a FAR module to align and fuse features in a local-global-local manner. A UFP module is proposed to locate regions with high uncertainty and extract fine-grained information. Furthermore, a PSA module is introduced to handle scale variations of salient objects. Finally, AS and RAS decoders are proposed to reduce semantic and detail gaps during the fusion process by progressively aggregating adjacent and continuous features. Extensive experiments on five RGB-D natural scene datasets and two downstream scene datasets demonstrate the superiority and generalization ability of the proposed method.

[0239] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A salient target detection method for RGB-D based on unentangled representation learning, characterized in that, Includes the following steps: (1) Obtaining multi-scale feature maps X i ,F i Using a Pyramid Vision Transformer (PVT) network as the backbone network, for input images I∈R 3×H×W Extracting multi-scale feature maps i = 1, 2, 3, 4, where C i H and W represent the number of channels, height, and width, respectively; Depth map D of input image I o ∈R 1×H×W Adjusted to D via convolution c ∈R 3×H×W Then it is input into another pyramid visual transformer to extract multi-scale feature maps. i = 1, 2, 3, 4; (2) Focus-Alignment-Refinement The Focus-Alignment-Refinement (FAR) module is used to refine multi-scale feature maps X. i ,F i Align and fuse features of the same scale to generate feature maps. 2.1) Alignment from thick to thin First, for the multi-scale feature map X i ,F i Regularization and 3x3 convolution are performed, followed by the use of a softmax function and reshaping to generate attention. and in, m represents the number of prototypes, and the process is expressed as: Where Att represents a combination of regularization, convolution, softmax, and reshaping; Then, the cluster centers are randomly initialized. A significant prototype is obtained through summation and regularization. Where Reshape(·) represents the reshaping operation, All prototypes accumulated, Represents the L2 norm; Then, the significant prototype With multi-scale feature map X i Significant prototype P i f With multi-scale feature map F i The feature maps are fused to generate prototype-based feature maps. and Where Γ(,) represents combining by channel and performing a 1×1 convolution, followed by the ReLU activation function; Then, a 1×1 convolution followed by a 3×3 depthwise separable convolution is used to process the multi-scale feature map X. i ,F i and feature maps Encode and generate feature maps and feature maps Then, the feature map As a query, feature map As the key, calculate the correlation matrix in the channel dimension: Among them, the correlation matrix Where h is the number of attention heads, T represents transpose, σ is the softmax function, τ is the learnable scaling factor, and Reshape(·) represents the reshaping operation; Finally, the enhanced feature map is obtained. 2.2) Refine For feature maps They are spatially compressed separately and then fused to obtain a channel map. for: Where Avg represents average pooling, Concat represents channel concatenation, and superscript 1→C i This indicates taking channels 1 to C. i The element, superscript C i +1→2C i Indicates the acquisition of channel C i +1 to 2C i Element; Finally, a refined feature map is generated. Where ⊙ represents element-wise multiplication; (3) Perception of uncertainty characteristics Uncertainty feature perception (UFP) module is used to obtain uncertainty perception feature M. u : 3.1) Establish a Laplace distribution for each pixel of the feature map and construct an uncertainty map: First, by refining the feature map The mean was obtained by using two different 1×1 convolutions: Where Γ′ and Γ″ represent two distinct 1×1 convolutions, Then, random sampling is performed multiple times to generate the variable ξ from the standard Laplace distribution, so as to obtain a new uncertainty distribution of the pixel, i.e., L = μ + ξb; Then, the variance is calculated and normalized to obtain the uncertainty diagram. U=Norm(Var(φ(Sample(L)))) (9) Where Norm(·), VAR(·), φ(·) and sample(·) represent normalization, sample variance, Sigmoid function and sampling, respectively; Refine the feature map The inputs are fed into an adjacency shrinkage (AS) decoder to generate feature maps. Then, the feature maps are concatenated by channel to obtain the feature maps. in, Perform a 1×1 convolution on the feature map M, then perform a 3×3 convolution three times to obtain the query M. q , key M k Sum M v The convolution kernels used in the three convolutions are different; Apply the uncertainty graph U to query M q , key M k Generate the correlation matrix corr u : in, This represents matrix multiplication, where τ1 is a learnable scaling factor; Obtain the uncertainty-aware feature map M u =M q ⊙corr u (12) (4) Progressive scale perception By using a Progressive Scale-Aware Module (PSAM), dilated convolutions at different scales are introduced to gradually expand the receptive field, thereby capturing objects of different scales: 4.1) Generate feature maps Among them, Conv 1×1 Represents a 1×1 convolution, Conv 3×3,delation＝1 This represents a 3x3 convolution with a dilation rate of 1. 3×3 This represents a 3×3 convolution; Using the same method but with different convolution kernels, feature maps are generated. 4.2) Generate feature maps Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×3 Represents a 1×3 convolution, Conv 3×1 Represents a 3×1 convolution, Conv 3×3,delation＝3 This represents a 3x3 convolution with a dilation rate of 3; Using the same method but with different convolution kernels, feature maps are generated. 4.3) Generate feature maps Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×5 Represents a 1×5 convolution, Conv 5×1 Represents a 5×1 convolution, Conv 3×3,delation＝5 This represents a 3x3 convolution with a dilation rate of 5; Using the same method but with different convolution kernels, feature maps are generated. 4.4) Generating Feature Maps Among them, Conv 1×1 Represents a 1×1 convolution, Conv 1×7 Represents a 1×7 convolution, Conv 7×1 Represents a 7×1 convolution, Conv 3×3,delation＝ This represents a 3x3 convolution with a dilation rate of 7; Using the same method but with different convolution kernels, feature maps are generated. (5) Refined Adjacent Shrink Decoding By using a refined adjacent shrinkage (RAS) decoder to progressively aggregate consecutive and adjacent layers, the gap between high and low layer information is reduced. Specifically, the RGB branch features are aggregated as follows: Among them, T i 、T′ i 、T″ i The feature maps obtained by hierarchical aggregation are T, which is the RGB branch feature map, and CBR represents the combination of convolution, BN layer and ReLU function; Similarly, deep branch features are aggregated as follows: Among them, D i 、D′ i D″ i The feature map is obtained by hierarchical aggregation, where D is the deep branch feature map; Finally, the predicted saliency map S is obtained: First, the RGB branch feature map T and the depth branch feature map D are concatenated by channel, and then the concatenated feature map is used as the refined feature map in step (3). Uncertainty feature perception is performed to obtain uncertainty-perceived feature map M. u The prediction saliency map F serves as the fusion branch; (6) Training Based on steps (1) to (5), a progressive decoupling network based on uncertainty awareness is obtained. The labeled images in the training dataset are fed into the progressive decoupling network based on uncertainty awareness and processed according to steps (1) to (5) to obtain the RGB branch feature map T, the depth branch feature map D, and the predicted saliency map F of the fusion branch. Then, the loss is compared with the labeled RGB feature map, the depth feature map, and the saliency map S, respectively. as well as Total loss Where α, β, and γ are hyperparameters, and three losses as well as Each loss in the algorithm consists of weighted binary cross-entropy loss, weighted IoU loss, and SSIM loss. Based on total losses Train the uncertainty-aware progressive decoupling network until it converges; (7) Target Detection The image to be detected is fed into a progressive decoupling network based on uncertainty perception and processed according to steps (1) to (5) to obtain a predicted saliency map F, thus completing the saliency target detection.

Images