RGB-D saliency target detection method based on separation and fusion of common and specific information

Abstract

The invention discloses an RGB-D saliency target detection method based on common and specific information separation and fusion, and relates to the technical field of artificial intelligence image processing and computer vision. According to the method, by separating shared features and specific features of RGB and depth modals layer by layer and combining shared feature fusion and specific feature selection strategies, multi-modal information fusion with finer granularity is achieved, modal specific information hidden in the background can be effectively mined, and meanwhile complementarity of shared information is reserved; besides, a cross-scale attention module based on an asymmetric attention matrix is designed for the aliasing effect problem existing in the up-sampling process in the decoding stage, and the definition of the edge of a saliency target is remarkably improved by fusing multi-scale features. Experimental results show that the method has the most advanced performance on nine RGB-D SOD data sets, is remarkably superior to other ten comparison methods, and particularly shows higher robustness and higher detection precision in a complex scene.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image processing and computer vision technology, and specifically to an RGB-D salient object detection method based on the separation and fusion of commonality and specificity information. Background Art

[0002] Most current RGB-D SOD and COD models use pre-trained, independent backbone networks to extract RGB and depth features. Due to the heterogeneity of RGB images and depth maps, the semantics of the RGB and depth features extracted by the backbone networks differ significantly. Therefore, numerous studies are exploring how to better fuse RGB and depth information. A cross-modal spatial attention mechanism is used to filter out the areas of interest to both modalities, and channel-wise attention is applied to these cross-modal features to capture the global information of each modality. Finally, the fused cross-modal features are aggregated. Spatial attention is used to guide each modality to focus on the areas of interest to the other modality, while channel-wise attention is used to filter and enhance the global information of each modality. RGB and depth features are then merged, and the areas of shared interest are then combined through max pooling. These methods all focus on maximizing the extraction of shared information between the two modalities. Their design relies heavily on the performance of the fusion module, which can be ineffective in some complex situations. Existing fusion modules tend to capture shared information between RGB and depth modalities while treating modality-specific information as noise and eliminating it. However, modality-specific information plays a key role in adjusting modal dependencies, helping models understand scene content, and integrating modality advantages.

[0003] Currently, there are a small number of salient object detection methods that separate modality-shared information from modality-specific information. A three-stream complementary network (TCN) consists of three sub-networks: an RGB sub-network extracts an RGB modality saliency map from an RGB image, an RGB-D sub-network extracts modality-common information from an RGB-D image, and a fusion sub-network combines the RGB image, depth image, and saliency maps of the first two sub-networks to generate more refined results. Current methods for separating shared information from specific information use three feature extraction networks to extract RGB-specific features, depth-specific features, and shared features, respectively. Because the shared features of each layer are derived from the shared features of the previous layer, there is a strong coupling relationship between the shared features of each layer, which cannot fully capture the subtle differences between the features of each layer and can lead to error accumulation.

[0004] In summary, the current RGB-D salient object detection (SOD) and collaborative saliency detection (COD) models have the following three major deficiencies in modality fusion:

[0005] 1. Imbalance between modality-specific and shared information: Existing methods mostly focus on maximizing the shared information between RGB and depth modalities, while neglecting the importance of modality-specific information. This specific information plays a key role in adjusting modality dependencies, enhancing scene understanding, and integrating modality advantages. However, existing fusion modules often treat this information as noise and eliminate it, limiting the model's performance in complex scenarios.

[0006] 2. Limitations of the Fusion Module: While existing fusion methods (such as cross-modal spatial attention and channel attention) have achieved some success in fusing modal information, their design is highly dependent on the performance of the fusion module. In complex scenarios, these modules may not effectively capture subtle differences between modalities, resulting in suboptimal fusion results and even the introduction of erroneous information.

[0007] 3. Coupling Issues in Shared Feature Extraction: In methods that separate shared information from specific information (such as three-stream complementary networks), the extraction of shared features often relies on the shared features of the previous layer, resulting in strong coupling between shared features at each layer. This coupling not only limits the model's ability to capture subtle differences in features across layers but can also lead to the accumulation of erroneous information, thus affecting the accuracy of the final detection results.

[0008] Therefore, a new solution to the above problems needs to be proposed. Summary of the Invention

[0009] The purpose of the present invention is to provide an RGB-D salient object detection method based on the separation and fusion of common and specific information, and to design and train a model through deep learning methods to achieve fast and accurate salient object detection, so as to solve the technical problems raised in the background technology.

[0010] To achieve the above objectives, the present invention provides the following technical solution: an RGB-D salient object detection method based on the separation and fusion of commonality and specificity information, comprising at least the following steps:

[0011] S1: Batch preprocessing of images in the public dataset;

[0012] S2: Merge the original dataset with the synthetic dataset to perform non-overlapping dataset division;

[0013] S3: Design a network model for object detection, using a layered shared feature and specific feature separation method. First, separate the features of each layer, then fuse the multimodal features using shared feature fusion and specific feature selection. This method can capture the commonalities and differences of each layer of multimodal information in a more fine-grained manner. Especially in complex scenarios, this layer-by-layer feature separation method can more accurately capture and process information from different modalities, improving the robustness of the model.

[0014] S4: Design a loss function for detecting the target, and use the loss function to supervise all branch outputs in the decoding process. The loss function includes weighted binary cross entropy loss and intersection-over-union loss.

[0015] S5: Use the trained model for measurement verification, and the model training is completed.

[0016] Furthermore, the input image in S1 has a size of 384×384, and the input image includes an RGB image and a depth image;

[0017] The single-channel depth image is repeated into three channels to match the input format;

[0018] To avoid overfitting, data preprocessing adopts a data augmentation strategy, which includes but is not limited to random flipping, rotation and cropping.

[0019] Furthermore, the target detection network model is a significant target detection model based on the separation and fusion of common information and specific information. The encoder of the target detection model consists of two P2T basic backbone networks, and the extracted features are divided into four levels B i , i = 1, 2, 3, 4, the different modal features of each level are expressed as and Moreover, features in the same layer have the same resolution;

[0020] Then, the two modal outputs and Cross-modal fusion features are obtained through the separation and fusion module of modal shared information and specific information

[0021] After cross-modal fusion, and Cross-scale feature fusion is performed with deep low-resolution features and obtained through the cross-scale attention module and

[0022] Finally, the obtained four-layer features will be screened and refined through the channel weight selection module, and the generated four-layer results will be supervised.

[0023] Furthermore, the modality shared information and specific information separation and fusion module is used to enhance the consistency and complementarity between modalities. Specifically, in the encoding feature fusion stage, the modality shared information and specific information separation and fusion module (Separation and Fusion of Sahred and Specific Module, SFSSM) includes shared attention fusion and feature separation selection. In this module, the shared attention matrix (Shared Matrix) extracts and retains the consistency information between the two modalities through the multi-head attention mechanism. This information is simultaneously passed to the shared attention operation units of the two branches;

[0024] At the same time, the Specific Matrix extracts modality-specific information that contributes significantly to the target task from the background features implicit in the shared information through a deep feature mining mechanism, thereby achieving effective screening of specific information.

[0025] Finally, the information of the two branches is adaptively combined by introducing the Inverted Residual Block Feed-Forward Network (IRB-FFN) to ensure the consistency and completeness of the fused information in the feature space.

[0026] Furthermore, the application of the modality shared information and specific information separation and fusion module includes at least the following steps:

[0027] The features from the two modalities and are multiplied to obtain a shared attention matrix, which retains the consistency information of the two modalities and is passed to the shared attention operation of the two branches to achieve complementarity between the modalities. Specifically:

[0028] Given features and First, perform layer normalization and linear projection, expressed as:

[0029]

[0030]

[0031] in, Represents the RGB feature map; represents a depth or thermal imaging feature map; LN(·) represents a LayerNorm operation; Ψ(·) represents a linear projection operation;

[0032] Q and V r Represents the query and value based on the RGB feature map, K and V respectivelyd The keys and values obtained based on the deep feature map are represented separately. These representations will be operated in the attention mechanism, and the similarity between the query and the key is calculated, and the different values are finally weighted to obtain the final feature representation;

[0033] Then perform matrix multiplication on Q and K to obtain the shared attention matrix M of the RGB modality branch s , because there are AB in matrix operations T =(BA T ) T The property of , so the shared attention matrix of the deep modality Able to pass M s And transpose it to get, no Softmax operation is applied at this stage, expressed as follows:

[0034]

[0035] Where D represents the length of the feature dimension; N represents the number of heads in the multi-head operation; K T is the transpose of the key vector, so the matrix multiplication QK T A matrix of similarity between the query and the key will be generated; after obtaining the shared matrix M s After that, the remaining attention operation of the two branches is expressed as follows:

[0036]

[0037] Among them, δ(·) represents the Softmax activation function, and Represent the results of each branch in shared attention respectively;

[0038] The features of the two branches are concatenated to obtain the modal shared features:

[0039]

[0040] After obtaining the shared features, since the modality-specific features are often implicit in the filtered background information, element-wise subtraction is used to separate the modality-specific information from the original RGB features and depth features. The separated RGB and depth modality-specific features are shown in the following formula:

[0041]

[0042]

[0043] Among them, - represents element subtraction. By subtracting the shared features from the original modal features, the unique features of each modality can be effectively extracted. This method not only retains the difference information between the modalities, but also avoids feature redundancy, thereby ensuring the adequacy and effectiveness of feature expression.

[0044] Based on the separated RGB and depth modality-specific features, we further designed a specific information screening module. The core of the specific information screening module is to filter and enhance the specific information of RGB and depth modalities respectively through two independent multi-head self-attention mechanisms. Specifically:

[0045] For each modality-specific information, its query matrix, key matrix, and value matrix are independently generated from the features of each modality, thereby ensuring the independence and integrity of the modality-specific information. Finally, the filtered modality-specific features are obtained through channel splicing. The process is expressed as:

[0046]

[0047] Subsequently, the modality-shared features and the filtered modality-specific features are concatenated through the channels and fed into the feedforward network, as shown in the following formula:

[0048]

[0049] In the feedforward network, the features of the two modalities are concatenated in the channel dimension and fused through an IRB-FFN with an expansion ratio of 2. A 3×3 convolutional layer is used to halve the number of feature channels, as shown in the following formula:

[0050] fuse'=IRB(fuse)+fuse (14)

[0051] Fi=Conv 3×3 (fuse') (15)

[0052] Among them, Conv 3×3 (·) represents a convolutional layer with a kernel size of n×n; F i Represents the output of the specific information screening module;

[0053] In addition, the IRB expression in the equation is as follows:

[0054] X1=Conv 1×1 (x) (16)

[0055] X2=DWConv 3×3 (x1) (17)

[0056] X3=Conv 1×1 (x2) (18)

[0057] Among them, DWConv 3×3 (·) indicates using depthwise separable convolution of size 3×3.

[0058] Furthermore, the Cross-scale Feature Interaction Module (CFIM) aims to achieve effective fusion of multi-scale features;

[0059] The cross-scale attention module integrates feature information of adjacent scales through upsampling operations. To suppress the spectrum aliasing phenomenon that may be introduced during the upsampling process, the cross-scale attention module adopts an asymmetric feature mapping strategy, that is, high-resolution features are used as query vectors, and low-resolution features are used as key and value inputs respectively.

[0060] This design enables low-resolution features to be adaptively reconstructed into high-resolution feature representations through a learnable attention weight matrix;

[0061] In the feature fusion stage, the cross-scale attention module realizes the deep fusion of multi-scale features through multi-head attention mechanism and dot product operation.

[0062] Furthermore, the application of the cross-scale attention module includes at least the following steps:

[0063] First, through layer normalization and linear projection operations, the high-resolution features are mapped to the query matrix Q, and the low-resolution features F are mapped to the query matrix Q. i+1 They are mapped to the key matrix K and the value matrix V respectively. At the same time, in order to ensure the performance lower limit of the module and enhance the robustness of the features, bilinear interpolation is used to upsample the low-resolution features to obtain residual features. These operations are expressed as:

[0064] Q=Ψ(LN(F i+1 )) (19)

[0065] K,V=Ψ(LN(F i )) (20)

[0066] R=Bi(LN(F i+1 )) (twenty one)

[0067] Among them, F i Represents the high-resolution feature map of the current layer; F i+1 represents the low-resolution feature map output from the previous layer; Bi(·) represents bilinear interpolation;

[0068] Next, Q and K are matrix multiplied to obtain an asymmetric matrix M, which represents the patch correspondence between the two resolution features. In addition, the low-resolution features will be upsampled to the high-resolution feature scale after two matrix multiplications. The feedforward network here also uses IRB-FFN. The process is expressed as follows:

[0069]

[0070] Then, a multi-head self-attention module is used to enhance the multi-scale and multi-modal information in the reconstructed features. The specific process is as follows:

[0071]

[0072] in, It is the low-resolution feature map after upsampling;

[0073] Finally, the Hadamard product With F i Fusion is performed and 3×3 convolution, batch normalization, and ReLU activation function are used, expressed as:

[0074]

[0075] Among them, CBP(·) represents the feature fusion operation, which combines convolution, batch normalization and activation function.

[0076] Furthermore, the loss function is defined as:

[0077] L i =L bce (GT i ,P i )+L iou (GT i ,P i ) (25)

[0078] Among them, L i Represents the loss between the saliency map of each layer and the label, GT i Yes and P i are the labels and predicted saliency maps of the corresponding layers, respectively.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] The present invention realizes finer-grained multimodal information fusion by separating the shared features and specific features of RGB and depth modalities layer by layer, and combines shared feature fusion with specific feature selection strategy, which can effectively mine modality-specific information hidden in the background while retaining the complementarity of shared information; in addition, to address the aliasing effect problem existing in the upsampling process in the decoding stage, a cross-scale attention module based on an asymmetric attention matrix is designed, which significantly improves the clarity of the edges of salient targets by fusing multi-scale features; experimental results show that the present method has achieved state-of-the-art performance on 9 RGB-D SOD datasets, significantly outperforming the other 10 comparison methods, especially showing stronger robustness and higher detection accuracy in complex scenes, effectively solving the key problems in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 The flowchart of the salient object detection provided by the present invention;

[0083] Figure 2 Schematic diagram of the target detection model for separation and fusion of common and specific information provided by the present invention;

[0084] Figure 3 Schematic diagram of the module for separating and fusing modal shared information and specific information provided by the present invention;

[0085] Figure 4 Schematic diagram of the cross-scale attention module provided by the present invention;

[0086] Figure 5 A visual comparison chart of different models provided by the present invention on the data set;

[0087] Figure 6 This is a visualization diagram of the module ablation provided by the present invention. DETAILED DESCRIPTION

[0088] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0089] See Figure 1 The RGB-D salient object detection method based on the separation and fusion of commonality and specificity information includes at least the following steps:

[0090] S1: Batch preprocessing of images in the public dataset;

[0091] The input image size in S1 is 384×384, and the input image includes RGB image and depth image;

[0092] The single-channel depth image is repeated into three channels to match the input format;

[0093] To avoid overfitting, data preprocessing adopts data enhancement strategies, which include but are not limited to random flipping, rotation and cropping.

[0094] S2: Merge the original dataset with the synthetic dataset to perform non-overlapping dataset division;

[0095] The specific method for dataset partitioning in S2 is as follows: The NJU2K dataset, constructed by Nanjing University, contains 500 pairs of stereo images collected from the internet, 3D movies, and a Fujifilm W3 stereo camera. Using a stereo matching algorithm, 2000 RGB images, along with their corresponding depth maps and pixel-level saliency annotations, are generated. This dataset covers a variety of complex scenes, including indoor and outdoor environments, varying lighting conditions, and complex backgrounds. It provides high-quality depth information and serves as a reliable experimental benchmark for RGB-D salient object detection.

[0096] The NLPR dataset, created by the Institute of Automation at the Chinese Academy of Sciences, contains 1,000 images with both RGB and depth information. The depth data is captured using a Microsoft Kinect sensor and post-processed for quality assurance. The dataset covers natural environments, indoor objects, varying lighting conditions, and multi-target scenes, and provides precise pixel-level saliency annotations.

[0097] The STERE dataset contains 1,000 pairs of stereo images, mainly from the Internet and 3D movies, and generates corresponding depth maps through a stereo matching algorithm, while providing pixel-level salient object annotations.

[0098] The DES dataset was created by Universidad Carlos de Madrid (UCL), Spain. It contains 135 RGB images and their corresponding depth maps. The depth information is captured by a Kinect sensor, and pixel-level salient object annotations are provided. The DES dataset primarily consists of single-object scenes, where the object is typically prominently positioned against a relatively simple background. Therefore, it is well-suited for developing depth-based salient object detection methods.

[0099] The SIP dataset, constructed by Xiamen University, contains 929 pairs of images from various scenes, including images of various challenging scenes captured by smartphones. Each image provides high-quality depth information and pixel-level salient object annotations. The images are from a variety of complex scenes, covering indoor and outdoor environments, varying lighting conditions, and complex backgrounds.

[0100] The LFSD dataset contains 100 pairs of images of indoor and outdoor scenes captured by a Lytro light field camera. LFSD is unique in that it is challenging to use low-resolution images, which are often affected by noise, blur, and compression. Therefore, the LFSD dataset provides a good benchmark for evaluating the performance of salient object detection models when working with low-resolution images.

[0101] The DUT-RGBD dataset, created by Dalian University of Technology, contains 1,200 pairs of RGB images and corresponding depth maps acquired by Kinect sensors, along with pixel-level saliency annotations. The DUT-RGBD dataset includes images from indoor and outdoor environments, natural environments, and urban landscapes, encompassing challenging conditions such as varying lighting, occlusion, and background clutter.

[0102] The COME15K dataset contains 15,625 pairs of indoor and outdoor images. The images in the COME15K dataset come from a wide range of sources, including various indoor and outdoor scenes, covering different lighting conditions, complex backgrounds, and occlusion situations. The depth map information comes from different types of depth sensors, ensuring the diversity and representativeness of the data.

[0103] For the aforementioned datasets, this paper sets the input network's RGB and depth images to 352×352 pixels and preprocesses them using data augmentation techniques such as random flipping, rotation, and boundary cropping. It's important to note that the RGB images and depth maps are fed into the network with three channels and one channel, respectively. To ensure a fair comparison, this experiment follows a previously developed evaluation strategy and objectively evaluates the saliency maps generated by each method.

[0104] S3: Design a network model for target detection, using a hierarchical shared feature and specific feature separation method. First, separate the features of each layer, and then fuse the multimodal features using a shared feature fusion and specific feature selection method. This can capture the commonalities and differences of each layer of multimodal information in a more fine-grained manner. Especially in complex scenarios, this layer-by-layer feature separation method can more accurately capture and process information from different modalities and improve the robustness of the model. See Figure 2 ;

[0105] The network model of target detection is a salient target detection model based on the separation and fusion of common information and specific information. The encoder of the target detection model consists of two P2T basic backbone networks, and the extracted features are divided into four levels B i , i = 1, 2, 3, 4, the different modal features of each level are expressed as and Moreover, features in the same layer have the same resolution;

[0106] Then, the two modal outputs and Cross-modal fusion features are obtained through the separation and fusion module of modal shared information and specific information

[0107] After cross-modal fusion, and Cross-scale feature fusion is performed with deep low-resolution features and obtained through the cross-scale attention module and

[0108] Finally, the obtained four-layer features will be screened and refined by the channel weight selection module, and the generated four-layer results will be supervised;

[0109] Finally, the obtained four-layer features will be screened and refined through the channel weight selection module, and the generated four-layer results will be supervised.

[0110] The Separation and Fusion of Shred and Specific Module (SFSSM) is used to enhance the consistency and complementarity between modalities. Specifically, in the encoding feature fusion stage, the Separation and Fusion of Shred and Specific Module (SFSSM) includes shared attention fusion and feature separation selection. In this module, the shared attention matrix extracts and retains the consistency information between the two modalities through a multi-head attention mechanism. This information is simultaneously passed to the shared attention operation units of the two branches.

[0111] At the same time, the Specific Matrix extracts modality-specific information that contributes significantly to the target task from the background features implicit in the shared information through a deep feature mining mechanism, thereby achieving effective screening of specific information.

[0112] Finally, the information of the two branches is adaptively combined by introducing the Inverted Residual Block Feed-Forward Network (IRB-FFN) to ensure the consistency and completeness of the fused information in the feature space. Figure 3 The detailed structure of the proposed modality-shared feature separation selection module is presented.

[0113] The application of the modality shared information and specific information separation and fusion module includes at least the following steps:

[0114] The features from the two modalities and are multiplied to obtain a shared attention matrix, which retains the consistency information of the two modalities and is passed to the shared attention operation of the two branches to achieve complementarity between the modalities. Specifically:

[0115] Given features and First, perform layer normalization and linear projection, expressed as:

[0116]

[0117] in, Represents the RGB feature map; represents a depth or thermal imaging feature map; LN(·) represents a LayerNorm operation; Ψ(·) represents a linear projection operation;

[0118] Q and V r Represents the query and value based on the RGB feature map, K and V respectively d The keys and values obtained based on the deep feature map are represented separately. These representations will be operated in the attention mechanism, and the similarity between the query and the key is calculated, and the different values are finally weighted to obtain the final feature representation;

[0119] Then perform matrix multiplication on Q and K to obtain the shared attention matrix M of the RGB modality branch s , because there are AB in matrix operations T =(BA T ) T The property of , so the shared attention matrix of the deep modality Able to pass M s And transpose it to get, no Softmax operation is applied at this stage, expressed as follows:

[0120]

[0121] Where D represents the length of the feature dimension; N represents the number of heads in the multi-head operation; K T is the transpose of the key vector, so the matrix multiplication QK T A matrix of similarity between the query and the key will be generated; after obtaining the shared matrix M s After that, the remaining attention operation of the two branches is expressed as follows:

[0122]

[0123] Among them, δ(·) represents the Softmax activation function, and Represent the results of each branch in shared attention respectively;

[0124] The features of the two branches are concatenated to obtain the modal shared features:

[0125]

[0126] After obtaining the shared features, since the modality-specific features are often implicit in the filtered background information, element-wise subtraction is used to separate the modality-specific information from the original RGB features and depth features. The separated RGB and depth modality-specific features are shown in the following formula:

[0127]

[0128] Among them, - represents element subtraction. By subtracting the shared features from the original modal features, the unique features of each modality can be effectively extracted. This method not only retains the difference information between the modalities, but also avoids feature redundancy, thereby ensuring the adequacy and effectiveness of feature expression.

[0129] Based on the separated RGB and depth modality-specific features, we further designed a specific information screening module. The core of the specific information screening module is to filter and enhance the specific information of RGB and depth modalities respectively through two independent multi-head self-attention mechanisms. Specifically:

[0130] For each modality-specific information, its query matrix, key matrix, and value matrix are independently generated from the features of each modality, thereby ensuring the independence and integrity of the modality-specific information. Finally, the filtered modality-specific features are obtained through channel splicing. The process is expressed as:

[0131]

[0132] Subsequently, the modality-shared features and the filtered modality-specific features are concatenated through the channels and fed into the feedforward network, as shown in the following formula:

[0133]

[0134] In the feedforward network, the features of the two modalities are concatenated in the channel dimension and fused through an IRB-FFN with an expansion ratio of 2. A 3×3 convolutional layer is used to halve the number of feature channels, as shown in the following formula:

[0135] fuse'=IRB(fuse)+fuse (14)

[0136] F i =Conv 3×3 (fuse') (15)

[0137] Among them, Conv 3×3 (·) represents a convolutional layer with a kernel size of n×n; F i Represents the output of the specific information screening module;

[0138] In addition, the IRB expression in the equation is as follows:

[0139] X1=Conv 1×1 (x) (16)

[0140] X2=DWConv 3×3 (x1) (17)

[0141] X3=Conv 1×1 (x2) (18)

[0142] Among them, DWConv 3×3 (·) indicates using depthwise separable convolution of size 3×3.

[0143] The Cross-scale Feature Interaction Module (CFIM) aims to achieve effective fusion of multi-scale features;

[0144] The cross-scale attention module integrates feature information from adjacent scales through upsampling. To suppress the spectral aliasing that may be introduced during the upsampling process, the cross-scale attention module adopts an asymmetric feature mapping strategy, where high-resolution features are used as query vectors and low-resolution features are used as key and value inputs, respectively.

[0145] This design enables low-resolution features to be adaptively reconstructed into high-resolution feature representations through a learnable attention weight matrix;

[0146] In the feature fusion stage, the cross-scale attention module realizes the deep fusion of multi-scale features through the multi-head attention mechanism and dot product operation. Compared with the traditional cross-scale feature fusion method, the cross-scale attention module has the following significant advantages: each newly generated patch in the reconstructed high-resolution feature integrates the contextual information of all input patches, ensuring the global consistency of the features; it abandons the bilinear interpolation operation commonly used in traditional methods, fundamentally avoiding the generation of invalid feature channels; by reducing the number of patches, it significantly reduces the computational complexity of matrix operations and improves the operation efficiency of the module. Figure 4 The detailed architectural design of the cross-scale feature interaction module is presented.

[0147] The application of the cross-scale attention module includes at least the following steps:

[0148] First, through layer normalization and linear projection operations, the high-resolution features are mapped to the query matrix Q, and the low-resolution features F are mapped to the query matrix Q. i+1 They are mapped to the key matrix K and the value matrix V respectively. At the same time, in order to ensure the performance lower limit of the module and enhance the robustness of the features, bilinear interpolation is used to upsample the low-resolution features to obtain residual features. These operations are expressed as:

[0149] Q=ψ(LN(F i+1 )) (19)

[0150] K,V=Ψ(LN(F i )) (20)

[0151] R=Bi(LN(F i+1 )) (twenty one)

[0152] Among them, F i Represents the high-resolution feature map of the current layer; F i+1 represents the low-resolution feature map output from the previous layer; Bi(·) represents bilinear interpolation;

[0153] Next, Q and K are matrix multiplied to obtain an asymmetric matrix M, which represents the patch correspondence between the two resolution features. In addition, the low-resolution features will be upsampled to the high-resolution feature scale after two matrix multiplications. The feedforward network here also uses IRB-FFN. The process is expressed as follows:

[0154]

[0155] Then, a multi-head self-attention module is used to enhance the multi-scale and multi-modal information in the reconstructed features. The specific process is as follows:

[0156]

[0157] in, It is the low-resolution feature map after upsampling;

[0158] Finally, the Hadamard product With F i Fusion is performed and 3×3 convolution, batch normalization, and ReLU activation function are used, expressed as:

[0159]

[0160] Among them, CBP(·) represents the feature fusion operation, which combines convolution, batch normalization and activation function.

[0161] S4: Design a loss function for detecting the target. Use the loss function to supervise all branch outputs in the decoding process. The loss function includes weighted binary cross entropy loss and intersection-over-union loss.

[0162] The loss function is defined as:

[0163] L i =L bce (GT i ,P i )+L iou (GT i ,P i ) (25)

[0164] Among them, L i Represents the loss between the saliency map of each layer and the label, GT i Yes and P i are the labels and predicted saliency maps of the corresponding layers, respectively.

[0165] S5: Use the trained model for measurement verification, and the model training is completed.

[0166] The specific method for verifying the effectiveness of the model in S5 is as follows: The present invention uses four common indicators to evaluate the performance of the salient object detection model:

[0167] 1) Regional similarity measure (F-measure, F m ) takes into account both precision (P) and recall (R), where the β used in this invention is 2 The value of is set to 0.3 as in other methods. The mathematical expression of F-measure is:

[0168]

[0169] 2) Mean Absolute Error (M) refers to the degree of pixel-level approximation between the saliency map (S) and the ground truth label (G). Its mathematical expression is:

[0170]

[0171] 3) Structured measurement (S-measure, S m ) is used to evaluate the object perception (S0) and region perception (S r ) structure and the similarity between the label. As with other methods, the value of α in the present invention is set to 0.5. Its mathematical formula is expressed as:

[0172] S m =α·S0+(1-α)·S r (28)

[0173] 4) Enhance its mapping (E-measure, E m ) is used to evaluate image-level statistics and local pixel matching information. It is an evaluation metric based on the enhanced alignment mapping between saliency maps and labels. Its mathematical expression is:

[0174]

[0175] where ф FM To enhance the alignment matrix.

[0176] The following comparative proof is proposed:

[0177] The proposed model is compared with 10 state-of-the-art models for RGB-D saliency detection, including SPNet, SPSN, CIRNet, HINet, AFNet, EGANet, and HIDANet in the first invention. The methods ASINet, HFMDNet, and MAGNet in the first invention are also added. To ensure a fair comparison, the official code is used and each model is trained on an RTX 4090 GPU to generate the corresponding saliency maps. Figure 5 and Figure 6 .

[0178] Table 1 shows that the proposed method significantly improves MAE and other indicators on 9 test datasets compared with the most advanced methods. In addition, the proposed method has fewer parameters and FLOPs than other methods with similar performance. This advantage is attributed to the use of fewer feature channels than other methods. At the same time, SFSSM and CFIM adopt a multi-head mechanism to enhance the utilization of feature channel information without further expanding the number of feature channels. Quantitative comparison results of the proposed method with other SOTA models. In the indicators, ↓ indicates that the smaller the value, the better, and ↑ indicates that the larger the value, the better. The computational cost of the model (FLOPs, F) is in G, and the parameter amount (Parameter, P). In the results, the bold black is the best and the underlined is the suboptimal.

[0179] Table 1

[0180]

[0181]

[0182] In summary:

[0183] Compared with existing technologies, the present invention has the following advantages and positive effects: First, by separating the shared and specific features of RGB and depth modalities layer by layer, the present invention can more fully utilize modality-specific information and avoid misclassifying it as noise, thereby significantly improving the robustness and detection accuracy of the model in complex scenarios. Second, the adoption of a layered decoupled shared feature extraction method reduces the strong coupling of inter-layer features, avoids the accumulation of erroneous information, and enables the model to more accurately capture the subtle differences in features at each layer. Finally, by introducing a cross-scale attention module based on an asymmetric attention matrix, the aliasing effect problem during the upsampling process in the decoding stage is effectively resolved, further enhancing the clarity of salient object edges. Experimental results show that the present invention achieves state-of-the-art performance on multiple public datasets, significantly outperforming existing methods. In particular, it exhibits stronger adaptability and higher detection accuracy in complex scenarios, providing an efficient and reliable technical solution for the field of RGB-D salient object detection.

[0184] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. An RGB-D salient object detection method based on the separation and fusion of commonality and specificity information, characterized by: At least the following steps are included: S1: Batch preprocessing of images in the public dataset; S2: Merge the original dataset with the synthetic dataset to perform non-overlapping dataset division; S3: Design a network model for object detection, using a layered shared feature and specific feature separation method. First, separate the features of each layer, then fuse the multimodal features using shared feature fusion and specific feature selection. This method can capture the commonalities and differences of each layer of multimodal information in a more fine-grained manner. Especially in complex scenarios, this layer-by-layer feature separation method can more accurately capture and process information from different modalities, improving the robustness of the model. S4: Design a loss function for detecting the target, and use the loss function to supervise all branch outputs in the decoding process. The loss function includes weighted binary cross entropy loss and intersection-over-union loss. S5: Use the trained model for measurement verification, and the model training is completed.

2. The RGB-D salient object detection method based on separation and fusion of commonality and specificity information according to claim 1 is characterized by: The input image in S1 has a size of 384×384, and the input image includes an RGB image and a depth image; The single-channel depth image is repeated into three channels to match the input format; To avoid overfitting, data preprocessing adopts a data augmentation strategy, which includes but is not limited to random flipping, rotation and cropping.

3. The RGB-D salient object detection method based on separation and fusion of commonality and specificity information according to claim 1 is characterized by: The target detection network model is a salient target detection model based on the separation and fusion of common information and specific information. The encoder of the target detection model consists of two P2T basic backbone networks, and the extracted features are divided into four levels B i , i = 1, 2, 3, 4, the different modal features of each level are expressed as and Moreover, features in the same layer have the same resolution; Then, the two modal outputs and Cross-modal fusion features are obtained through the separation and fusion module of modal shared information and specific information After cross-modal fusion, and Cross-scale feature fusion is performed with deep low-resolution features and obtained through the cross-scale attention module and Finally, the obtained four-layer features will be screened and refined through the channel weight selection module, and the generated four-layer results will be supervised.

4. The RGB-D salient object detection method based on separation and fusion of commonality and specificity information according to claim 3 is characterized by: The modality shared information and specific information separation and fusion module is used to enhance the consistency and complementarity between modalities. Specifically, in the encoding feature fusion stage, the modality shared information and specific information separation and fusion module includes shared attention fusion and feature separation selection. In this module, the shared attention matrix extracts and retains the consistency information between the two modalities through the multi-head attention mechanism. This information is simultaneously transmitted to the shared attention operation units of the two branches; At the same time, the specificity matrix extracts modality-specific information that contributes significantly to the target task from the background features implicit in the shared information through a deep feature mining mechanism, thereby achieving effective screening of specific information. Finally, an inverse residual block feed-forward network is introduced to adaptively combine the information of the two branches to ensure the consistency and completeness of the fused information in the feature space.

5. The RGB-D salient object detection method based on separation and fusion of commonality and specificity information according to claim 4 is characterized by: The application of the modality shared information and specific information separation and fusion module includes at least the following steps: The features from the two modalities and are multiplied to obtain a shared attention matrix, which retains the consistency information of the two modalities and is passed to the shared attention operation of the two branches to achieve complementarity between the modalities. Specifically: Given features and First, perform layer normalization and linear projection, expressed as: in, Represents the RGB feature map; represents a depth or thermal imaging feature map; LN(·) represents a LayerNorm operation; Ψ(·) represents a linear projection operation; Q and V r Represents the query and value based on the RGB feature map, K and V respectively d The keys and values obtained based on the deep feature map are represented separately. These representations will be operated in the attention mechanism, and the similarity between the query and the key is calculated, and the different values are finally weighted to obtain the final feature representation; Then perform matrix multiplication on Q and K to obtain the shared attention matrix M of the RGB modality branch s , because there are AB in matrix operations T =(BA T ) T The property of , so the shared attention matrix of the deep modality Able to pass M s And transpose it to get, no Softmax operation is applied at this stage, expressed as follows: Where D represents the length of the feature dimension; N represents the number of heads in the multi-head operation; K T is the transpose of the key vector, so the matrix multiplication QK T A matrix of similarity between the query and the key will be generated; after obtaining the shared matrix M s After that, the remaining attention operation of the two branches is expressed as follows: Among them, δ(·) represents the Softmax activation function, and Represent the results of each branch in shared attention respectively; The features of the two branches are concatenated to obtain the modal shared features: After obtaining the shared features, since the modality-specific features are often implicit in the filtered background information, element-wise subtraction is used to separate the modality-specific information from the original RGB features and depth features. The separated RGB and depth modality-specific features are shown in the following formula: Among them, - represents element subtraction. By subtracting the shared features from the original modal features, the unique features of each modality can be effectively extracted. This method not only retains the difference information between the modalities, but also avoids feature redundancy, thereby ensuring the adequacy and effectiveness of feature expression. Based on the separated RGB and depth modality-specific features, we further designed a specific information screening module. The core of the specific information screening module is to filter and enhance the specific information of RGB and depth modalities respectively through two independent multi-head self-attention mechanisms. Specifically: For each modality-specific information, its query matrix, key matrix, and value matrix are independently generated from the features of each modality, thereby ensuring the independence and integrity of the modality-specific information. Finally, the filtered modality-specific features are obtained through channel splicing. The process is expressed as: Subsequently, the modality-shared features and the filtered modality-specific features are concatenated through the channels and fed into the feedforward network, as shown in the following formula: In the feedforward network, the features of the two modalities are concatenated in the channel dimension and fused through an IRB-FFN with an expansion ratio of 2. A 3×3 convolutional layer is used to halve the number of feature channels, as shown in the following formula: fuse'=IRB(fuse)+fuse (14) F i =Conv 3×3 (fuse') (15) Among them, Conv 3×3 (·) represents a convolutional layer with a kernel size of n×n; F i Represents the output of the specific information screening module; In addition, the IRB expression in the equation is as follows: X1=Conv 1×1 (x) (16) X2=DWConv 3×3 (x1) (17) X3=Conv 1×1 (x2) (18) Among them, DWConv 3×3 (·) indicates using depthwise separable convolution of size 3×3.

6. The RGB-D salient object detection method based on separation and fusion of commonality and specificity information according to claim 5 is characterized by: The cross-scale attention module is designed to achieve effective fusion of multi-scale features; The cross-scale attention module integrates feature information of adjacent scales through upsampling operations. To suppress the spectral aliasing phenomenon that is likely introduced during the upsampling process, the cross-scale attention module adopts an asymmetric feature mapping strategy, that is, high-resolution features are used as query vectors, and low-resolution features are used as key and value inputs respectively. This design enables low-resolution features to be adaptively reconstructed into high-resolution feature representations through a learnable attention weight matrix; In the feature fusion stage, the cross-scale attention module realizes the deep fusion of multi-scale features through multi-head attention mechanism and dot product operation.

7. The RGB-D salient object detection method based on separation and fusion of commonality and specificity information according to claim 6 is characterized by: The application of the cross-scale attention module includes at least the following steps: First, through layer normalization and linear projection operations, the high-resolution features are mapped to the query matrix Q, and the low-resolution features F are mapped to the query matrix Q. i+1 They are mapped to the key matrix K and the value matrix V respectively. At the same time, in order to ensure the performance lower limit of the module and enhance the robustness of the features, bilinear interpolation is used to upsample the low-resolution features to obtain residual features. These operations are expressed as: Q=Ψ(LN(F i+1 )) (19) K,VΨ(LN(F i )) (20) R=Bi(LN(F i+1 )) (21) Among them, F i Represents the high-resolution feature map of the current layer; F i+1 represents the low-resolution feature map output from the previous layer; Bi(·) represents bilinear interpolation; Next, Q and K are matrix multiplied to obtain an asymmetric matrix M, which represents the patch correspondence between the two resolution features. In addition, the low-resolution features will be upsampled to the high-resolution feature scale after two matrix multiplications. The feedforward network here also uses IRB-FFN. The process is expressed as follows: Then, a multi-head self-attention module is used to enhance the multi-scale and multi-modal information in the reconstructed features. The specific process is as follows: in, It is the low-resolution feature map after upsampling; Finally, the Hadamard product With F i Fusion is performed and 3×3 convolution, batch normalization, and ReLU activation function are used, expressed as: Among them, CBP(·) represents the feature fusion operation, which combines convolution, batch normalization and activation function.

8. The RGB-D salient object detection method based on separation and fusion of commonality and specificity information according to claim 1 is characterized by: The loss function is defined as: IT i =L bce (GT i ,P i )+L iou (GT i ,P i ) (25) Among them, L i Represents the loss between the saliency map of each layer and the label, GT i Yes and P i are the labels and predicted saliency maps of the corresponding layers, respectively.

Images