A Multimodal Attention Perception and Adjacent Scale Modeling Method for RGB-D Salient Object Detection
The RGB-D salient object detection method, which combines multimodal attention perception and adjacent scale modeling, solves the problems of incomplete salient objects and loss of edge information. The generated salient map ensures both object integrity and enhanced edge details.
Patent Information
- Application Number
- CN202410892941.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-07-04
AI Technical Summary
Existing salient object detection methods suffer from incomplete salient objects and loss of edge information in complex scenes.
An RGB-D salient target detection method employing multimodal attention perception and adjacent scale modeling is proposed. By constructing a multimodal attention fusion module and an adjacent scale modeling module, and combining spatial attention, channel attention and self-attention operations, complementary information is mined from RGB and depth features. Furthermore, feature consistency is enhanced through inverse residual blocks and multi-head self-attention operations.
It effectively ensures the integrity of salient targets while enhancing the edge details of the targets. The generated salient map can contain semantic and global contextual information of high-level features as well as edge and texture details of low-level features.
Smart Images

Figure CN118823536B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and artificial intelligence technology, specifically involving an RGB-D salient object detection method based on multimodal attention perception and adjacent scale modeling. Background Technology
[0002] Salient object detection (SOD) is a popular research topic in computer vision, aiming to detect and segment the most visually attractive regions or objects in an image. As an effective image preprocessing technique, SOD has been widely applied in object tracking, object recognition, video analysis, image captioning, image segmentation, and image reconstruction.
[0003] In recent years, due to the rapid progress and excellent performance of deep learning in the field of saliency detection images, many deep learning-based SOD models have been proposed. With the continuous development of deep learning, Transformer-based SOD methods have now surpassed convolution-based methods, reaching a significant milestone. Existing SOD methods have achieved good results, but when faced with challenging and complex scenes (such as changes in illumination and low contrast), these SOD methods still suffer from problems such as incomplete salient objects and loss of edge information. Summary of the Invention
[0004] The purpose of this invention is to provide an RGB-D salient object detection method based on multimodal attention perception and adjacent scale modeling, which solves the problems of incomplete salient objects and loss of edge information in existing detection methods.
[0005] The technical solution adopted in this invention is an RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling, which is implemented according to the following steps:
[0006] Step 1, Data Preprocessing;
[0007] Step 2: Construct a multimodal attention fusion module;
[0008] Step 3: Construct the adjacent scale modeling module.
[0009] The invention is further characterized by:
[0010] The data pre-includes NLPR, NJUD, STERE, SIP, and DUT-RGBD in Step 1. The input size of the RGB and depth maps is adjusted to 384×384, and various data augmentation techniques are used to avoid overfitting.
[0011] In step 1, the P2T backbone is pre-trained on ImageNet, the remaining parameters of the model are initialized using PyTorch, the model is trained using the Adam optimizer with a batch size of 2, an initial learning rate of 5e-5, and the learning rate is reduced by a factor of 10 every 100 epochs.
[0012] Step 2 is implemented in the following steps:
[0013] Step 2.1: Use spatial attention operation to mine complementary information clues between RGB features and depth features, and pass the complementary information to each other. At the same time, apply spatial attention operation to itself to improve feature representation ability and reduce computational complexity. Then, use channel attention operation to filter important channels.
[0014] Step 2.2: Multiply the features of the two modalities to obtain a shared attention matrix, and pass it to the shared attention operation of the two branches to enhance the consistency of multimodality;
[0015] Step 2.3: Use the inverse residual block feedforward network to combine the two branch information to make the multimodal transport information consistent.
[0016] Step 2.1 is implemented as follows:
[0017] Step 2.1.1, for a given feature and Perform spatial attention operations as shown in the following equation:
[0018]
[0019] SA(X i )=σ(Conv 7×7 (gmp c (X i (2)
[0020] In the formula, Represents the RGB feature map. Represents the deep feature map, SA(·) is the spatial attention operation, X i Let σ(·) be the input feature map, and gmp be the sigmoid function. c (·) represents the global max pooling operation along the channel dimension, Conv n×n (·) represents a convolutional layer with a kernel size of n×n;
[0021] Step 2.1.2, for the generated respectively with and The combination transfers information-rich RGB / depth cues into RGB / depth features and reduces computational complexity, as shown in the following equation:
[0022]
[0023] In the formula, This represents element-wise multiplication;
[0024] Step 2.1.3: Apply channel attention to highlight important channels for further optimization, as shown in the following formula:
[0025]
[0026] CA(X s )=σ(Conv 1×1 (Conv 1×1 (gmp s (X s (6)
[0027] Where CA(·) represents the channel attention operation, gmp s (·) denotes a global max pooling operation along the spatial dimension, X rs and X ds X represents the RGB features and depth features after enhancing complementarity, respectively. s Refers to X rs or X ds .
[0028] Step 2.2 is implemented as follows:
[0029] Step 2.2.1, perform layer normalization and linear projection operations, as shown in equation (7):
[0030]
[0031] In the formula, LN(·) represents the layer normalization operation, ψ(·) represents the linear projection operation, and Q,V r K and V d To share the attention variables, matrix multiplication is then performed between Q and K to obtain the shared attention matrix M for the RGB modality branches. s ;
[0032] Step 2.2.2, the shared attention matrix M for the RGB modality branch. s Performing a transpose operation yields the attention matrix for the deep modality. As shown in equation (8):
[0033]
[0034] Where D represents the length of the feature dimension, N represents the number of heads in a multi-head operation; K T For the transpose of the shared interest variable K;
[0035] Step 2.2.3, obtain the shared attention matrix M s Afterwards, the remaining attention operations for the two branches are as shown in equation (9):
[0036]
[0037] Where δ(·) is the Softmax activation function, and V represents the result of each branch after the shared concern operation. r and V d To share variables of interest, and These represent the RGB features and depth features after spatial and channel attention enhancement, respectively.
[0038] Step 2.3 is implemented as follows:
[0039] Step 2.3.1: In the feedforward network, the two modal features are first connected along the channel dimension and then fused using an IRB with a scaling ratio of 2, as shown in the following equation:
[0040]
[0041] fuse′=IRB(fuse)+fuse (11)
[0042] Where Cat(·) is the cascade operation, fuse represents the features after initial fusion through the cascade operation, and fuse′ represents the enhanced fused features;
[0043] Step 2.3.2, use a 3×3 convolutional layer to halve the number of channels in the feature, as shown in Equation (12):
[0044] F i =Conv 3×3 (fuse′) (12).
[0045] Step 3 is implemented in the following steps:
[0046] Step 3.1: First, the low-resolution features are upsampled using bilinear interpolation. Then, two features at adjacent scales are concatenated, and a 3×3 convolution is applied to halve the number of feature channels, as shown in Equation (13):
[0047]
[0048] Among them, F i F represents the high-resolution features of the current layer. i+1 This represents the low-resolution features of the previous layer, and Bi(·) represents bilinear interpolation. Indicates preliminary fusion characteristics;
[0049] Step 3.2 explores the interdependencies between features at different scales using a G-ECA and IRB submodule, as shown in Equation (14):
[0050]
[0051] Step 3.3: Enhance the multi-scale and multi-modal information in the fused features using multi-head self-attention operations, as shown in the following equation:
[0052]
[0053] in, It is an enhanced fusion feature;
[0054] Step 3.4, to prevent the loss of F i Important information is obtained through element-wise multiplication. With F i Multiply, as shown in equation (18):
[0055]
[0056] Among them, F i ′ is the final output of the adjacent scale modeling module.
[0057] The beneficial effects of this invention are:
[0058] This invention presents an RGB-D salient object detection method based on multimodal attention perception and adjacent scale modeling. This method ensures the integrity of salient objects while enhancing their edge details. It employs two pyramid pool transformers (P2T) with multi-scale pooling as multimodal encoders to extract multi-scale features from the two modalities. A multimodal attention fusion module combines spatial attention, channel attention, and self-attention operations to mine complementary information between the two modalities, and uses the same feedforward block to explore modal consistency. By progressively fusing adjacent scale features from high to low, the diversity and complementarity of features at different scales are fully utilized. The final fused features contain both semantic and global contextual information from high-level features and detailed information such as edges and textures from low-level features. Therefore, the generated salient map not only ensures the integrity of salient objects but also enhances their edge details. Attached Figure Description
[0059] Figure 1 This is a diagram illustrating the overall model framework of the RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling of the present invention.
[0060] Figure 2This is a diagram of the architecture of the multimodal attention fusion module in step 2 of the RGB-D salient target detection method with multimodal attention perception and adjacent scale modeling of the present invention.
[0061] Figure 3 This is a diagram of the architecture of the adjacent scale modeling module in step 3 of the RGB-D salient target detection method with multimodal attention perception and adjacent scale modeling of the present invention.
[0062] Figure 4 This is a detection result image of the dataset obtained in step 3 of the RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling of the present invention. Detailed Implementation
[0063] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0064] Example 1
[0065] The RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling of this invention is implemented according to the following steps:
[0066] Step 1, Data Preprocessing;
[0067] Step 1 uses five of the most representative datasets, including NLPR, NJUD, STERE, SIP and DUT-RGBD, and adjusts the input size of the RGB and depth maps to 384×384. Various data augmentation techniques are used to avoid overfitting, including random flipping, rotation and cropping.
[0068] In step 1, the P2T backbone is pre-trained on ImageNet, and the remaining parameters of the model are initialized using PyTorch. The model is trained using the Adam optimizer with a batch size of 2 and an initial learning rate of 5e-5, which decreases by a factor of 10 every 100 epochs. The overall model framework of this invention is as follows: Figure 1 As shown;
[0069] Step 2, construct the multimodal attention fusion module, the architecture of which is as follows: Figure 2 As shown;
[0070] Step 3, construct the adjacent scale modeling module, the architecture of which is as follows: Figure 3 As shown.
[0071] Example 2
[0072] The present invention provides a multimodal attention perception and adjacent scale modeling method for RGB-D salient target detection, wherein step 2 is specifically implemented according to the following steps:
[0073] Step 2.1: Use spatial attention operation to mine complementary information clues between RGB features and depth features, and pass the complementary information to each other. At the same time, apply spatial attention operation to itself to improve feature representation ability and reduce computational complexity. Then, use channel attention operation to filter important channels.
[0074] Step 2.1 is implemented as follows:
[0075] Step 2.1.1, for a given feature and Perform spatial attention operations as shown in the following equation:
[0076]
[0077] SA(X i )=σ(Conv 7×7 (gmp c (X i (2)
[0078] In the formula, Represents the RGB feature map. Represents the deep feature map, SA(·) is the spatial attention operation, X i For input feature mapping (i.e. or ), where σ(·) is the sigmoid function, gmp c (·) represents the global max pooling operation along the channel dimension, Conv n×n (·) represents a convolutional layer with a kernel size of n×n;
[0079] Step 2.1.2, for the generated respectively with and The combination transfers information-rich RGB / depth cues into RGB / depth features and reduces computational complexity, as shown in the following equation:
[0080]
[0081] In the formula, This represents element-wise multiplication. and These represent the RGB features and depth features enhanced by spatial attention operations, respectively.
[0082] Step 2.1.3: Apply channel attention to highlight important channels for further optimization, as shown in the following formula:
[0083]
[0084] CA(xs )=σ(Conv 1×1 (Conv 1×1 (gmp s (X s (6)
[0085] Where CA(·) represents the channel attention operation, gmp s (·) denotes a global max pooling operation along the spatial dimension, X rs and X ds X represents the RGB features and depth features after enhancing complementarity, respectively. s Refers to X rs or X ds .
[0086] Step 2.2: Multiply the features of the two modalities to obtain a shared attention matrix, and pass it to the shared attention operation of the two branches to enhance the consistency of multimodality;
[0087] Step 2.2 is implemented as follows:
[0088] Step 2.2.1, perform layer normalization and linear projection operations, as shown in equation (7):
[0089]
[0090] In the formula, LN(·) represents the layer normalization operation, ψ(·) represents the linear projection operation, and Q,V r K and V d To share the attention variables, matrix multiplication is then performed between Q and K to obtain the shared attention matrix M for the RGB modality branches. s ;
[0091] Step 2.2.2, the shared attention matrix M for the RGB modality branch. s Performing a transpose operation yields the attention matrix for the deep modality. As shown in equation (8):
[0092]
[0093] Where D represents the length of the feature dimension, N represents the number of heads in a multi-head operation, and K T For the transpose of the shared interest variable K;
[0094] Step 2.2.3, obtain the shared attention matrix M s Afterwards, the remaining attention operations for the two branches are as shown in equation (9):
[0095]
[0096] Where δ(·) is the Softmax activation function, and V represents the result of each branch after the shared concern operation. r and V d To share variables of interest, and These represent the RGB features and depth features after spatial and channel attention enhancement, respectively.
[0097] Step 2.3: Use the inverse residual block feedforward network to combine the two branch information to make the multimodal transport information consistent;
[0098] Step 2.3 is implemented as follows:
[0099] Step 2.3.1: In the feedforward network, the two modal features are first connected along the channel dimension and then fused using an IRB with a scaling ratio of 2, as shown in the following equation:
[0100]
[0101] fuse′=IRB(fuse)+fuse (11)
[0102] Where Cat(·) is the cascade operation, fuse represents the features after initial fusion through the cascade operation, and fuse′ represents the enhanced fused features;
[0103] Step 2.3.2, use a 3×3 convolutional layer to halve the number of channels in the feature, as shown in Equation (12):
[0104] F i =Conv 3×3 (fuse′) (12)
[0105] Step 3 is implemented in the following steps:
[0106] Step 3.1: First, the low-resolution features are upsampled using bilinear interpolation. Then, two features at adjacent scales are concatenated. Since the number of feature channels doubles after concatenation, a 3×3 convolution is applied to halve the number of feature channels, as shown in Equation (13).
[0107]
[0108] Among them, F i F represents the high-resolution features of the current layer. i+1 This represents the low-resolution features of the previous layer, and Bi(·) represents bilinear interpolation. Indicates preliminary fusion characteristics;
[0109] Step 3.2 explores the interdependencies between features at different scales using a G-ECA and IRB submodule, as shown in Equation (14):
[0110]
[0111] Step 3.3: Enhance the multi-scale and multi-modal information in the fused features using multi-head self-attention operations, as shown in the following equation:
[0112]
[0113] in, It is an enhanced fusion feature;
[0114] Step 3.4, to prevent the loss of F i Important information is obtained through element-wise multiplication. With F i Multiply, as shown in equation (18):
[0115]
[0116] Among them, F i ′ This is the final output of the adjacent scale modeling module. Its detection results are as follows: Figure 4 As shown.
[0117] Example 3
[0118] The experimental results of the RGB-D salient object detection method based on multimodal attention perception and adjacent scale modeling proposed in this invention are shown in Table 1 below. The results indicate that the network model trained using the proposed method achieves superior performance on all five challenging datasets across the four most representative evaluation metrics (i.e., S-measure (S...). α ), F-measure(F β ), E-measure(E ξ Both () and MAE(M)) are superior to the other 15 methods.
[0119] Table 1. Comparison of results between the method of the present invention and the traditional method, where the results in bold represent the best results.
[0120]
[0121]
[0122]
[0123] This invention presents an RGB-D salient object detection method based on multimodal attention perception and adjacent scale modeling. It employs two pyramid pool transformers (P2T) with multi-scale pooling as multimodal encoders to extract multi-scale features from two modalities. Secondly, this invention proposes a novel multimodal attention fusion module that combines spatial attention, channel attention, and self-attention operations to mine complementary information between the two modalities, and uses the same feedforward block to explore modal consistency. Furthermore, this invention designs a novel adjacent scale modeling module to fuse multi-scale features. By progressively fusing adjacent scale features from high to low, it fully leverages the diversity and complementarity of features between different scales. This results in the final fused features containing both semantic and global contextual information from high-level features, as well as detailed information such as edges and textures from low-level features. Therefore, the final generated salient map not only ensures the integrity of salient objects but also enhances the edge details of the targets.
Claims
1. A multimodal attention-based RGB-D salient object detection method with adjacent scale modeling employs two pyramid pooling transformers with multi-scale pooling as multimodal encoders to extract multi-scale features from two modalities. It combines spatial attention, channel attention, and self-attention operations to mine complementary information between the two modalities and uses the same feedforward block to explore modal consistency. By progressively fusing adjacent scale features from high to low, it fully leverages the diversity and complementarity of features between different scales, ensuring that the final fused features contain both semantic and global contextual information from high-level features and edge and texture detail information from low-level features. Its key feature is... The specific steps are as follows: Step 1, Data Preprocessing; Step 2: Construct a multimodal attention fusion module; Step 3: Construct the adjacent scale modeling module; Step 2 is implemented in the following steps: Step 2.1: Use spatial attention operation to mine complementary information clues between RGB features and depth features, and pass the complementary information to each other. At the same time, apply spatial attention operation to itself to improve feature representation ability and reduce computational complexity. Then, use channel attention operation to filter important channels. Step 2.2: Multiply the features of the two modalities to obtain a shared attention matrix, and pass it to the shared attention operation of the two branches to enhance the consistency of multimodality; Step 2.3: Use the inverse residual block feedforward network to combine the two branch information to make the multimodal transport information consistent; Step 2.1 is implemented in the following steps: Step 2.1.1, for a given feature and Perform spatial attention operations as shown in the following equation: In the formula, Represents the RGB feature map. Represents a depth feature map. For spatial attention operations, For input feature mapping, For the sigmoid function, This is a global max pooling operation along the channel dimension. This represents a convolutional layer with a kernel size of n×n; Step 2.1.2, for the generated , respectively with and The combination transfers information-rich RGB / depth cues into RGB / depth features and reduces computational complexity, as shown in the following equation: In the formula, This represents element-wise multiplication. These represent the RGB features and depth features enhanced by spatial attention operations, respectively. Step 2.1.3: Apply channel attention to highlight important channels for further optimization, as shown in the following formula: in, This indicates channel attention operations. This represents the global max-pooling operation along the spatial dimension. and These represent the RGB features and depth features after enhancing complementarity, respectively. Reference or .
2. The RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling according to claim 1, characterized in that, The data in step 1 pre-includes NLPR, NJUD, STERE, SIP, and DUT-RGBD, and the input size of the RGB and depth maps is adjusted to 384. 384, employing various data augmentation techniques to avoid overfitting.
3. The RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling according to claim 1, characterized in that, In step 1, the P2T backbone is pre-trained on ImageNet, the remaining parameters of the model are initialized using PyTorch, the model is trained using the Adam optimizer with a batch size of 2 and an initial learning rate of 5e.
5. The learning rate decreases by a factor of 10 every 100 epochs.
4. The RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling according to claim 1, characterized in that, Step 2.2 is implemented in the following steps: Step 2.2.1, perform layer normalization and linear projection operations, as shown in equation (7): In the formula, Presentation layer normalization operation, This represents a linear projection operation. , and To share the variables of interest, then and Perform matrix multiplication between them to obtain the shared attention matrix for the RGB modality branches. ; Step 2.2.2, Shared attention matrix for RGB modality branches Performing a transpose operation yields the attention matrix for the deep modality. As shown in equation (8): in, N represents the length of the feature dimension, and N represents the number of heads in a multi-head operation. To share variables of interest Transpose of; Step 2.2.3, obtain the shared attention matrix Afterwards, the remaining attention operations for the two branches are shown in equation (9): in, The Softmax activation function is used. and These represent the results of each branch after the shared concern operation. and To share variables of interest, and These represent the RGB features and depth features after spatial and channel attention enhancement, respectively.
5. The RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling according to claim 1, characterized in that, Step 2.3 is implemented in the following steps: Step 2.3.1: In the feedforward network, the two modal features are first connected along the channel dimension and then fused using an IRB with a scaling ratio of 2, as shown in the following equation: in, For cascading operations, This indicates the features after initial fusion through cascading operations. This indicates the enhanced fusion features; Step 2.3.2, use a 3×3 convolutional layer to halve the number of channels in the feature, as shown in Equation (12):
6. The RGB-D salient target detection method based on multimodal attention perception and adjacent scale modeling according to claim 1, characterized in that, Step 3 is implemented in the following steps: Step 3.1: First, the low-resolution features are upsampled using bilinear interpolation. Then, two features at adjacent scales are concatenated, and a 3×3 convolution is applied to halve the number of feature channels, as shown in Equation (13): in, This represents the high-resolution features of the current layer. This represents the low-resolution features of the previous layer. Indicates bilinear interpolation. Indicates preliminary fusion characteristics; Step 3.2 explores the interdependencies between features at different scales using a G-ECA and IRB submodule, as shown in Equation (14): Step 3.3: Enhance the multi-scale and multi-modal information in the fused features using multi-head self-attention operations, as shown in the following equation: in, It is an enhanced fusion feature; Step 3.4, to prevent loss Important information is obtained through element-wise multiplication. and Multiply, as shown in equation (18): in, It is the final output of the adjacent scale modeling module.
Citation Information
Patent Citations
RGB-D saliency target detection method based on boundary deformable convolution guidance
CN115830420A
Multi-modal saliency target detection method based on fine-grained decision-making mechanism
CN117765236A
