RGB-D saliency target detection method and system based on mixed feature cross fusion

Abstract

According to the RGB-D saliency target detection method based on mixed feature cross fusion, through a mixed feature cross fusion module, a model can fully utilize complementary information between an RGB image and a depth image, and efficient fusion of features is achieved. The module enhances the interaction between RGB and depth features through a self-attention mechanism, so that the connection between different modal features is better modeled and captured, and the problem of feature distortion under the condition of poor depth image quality is solved. Besides, the invention provides a specific training strategy and an optimization scheme, and the model can rapidly converge on a large-scale data set by adopting the combination of an MAE pre-training model and a ViT model, and obtains excellent performance in a saliency target detection task.

Description

A method and system for RGB-D saliency target detection based on hybrid feature cross-fusion Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, specifically to an RGB-D salient object detection method and system based on the cross-fusion of hybrid features. Background Technology

[0002] Salience object detection, a crucial task in computer vision, aims to identify salient regions of interest to the human visual system from images or videos. Traditional salient object detection algorithms primarily rely on RGB images, but due to the greater geometric and structural information provided by depth images, RGB-D salient object detection has become a research hotspot in recent years. RGB-D salient object detection improves accuracy and robustness by fusing complementary information from RGB and depth images. However, the quality of depth images varies significantly across different datasets, making the effective fusion of information from RGB and depth images a pressing issue that needs to be addressed in this field.

[0003] Existing methods have limitations in RGB-D feature fusion. For example, early and mid-stage fusion methods fail to fully capture the interdependence between the two modalities, resulting in poor detection performance when the depth image quality is low. Therefore, designing more efficient fusion mechanisms to improve the robustness of models under different depth image conditions has become an important research direction in the field of RGB-D salient object detection. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes an RGB-D saliency target detection method based on MAE and ViT. By introducing a hybrid feature cross-fusion module, it effectively solves the problem of feature fusion between RGB images and depth images, significantly improving detection performance, especially achieving good detection results when the depth image quality is poor.

[0005] This invention proposes a novel RGB-D salient object detection method based on MAE-SOD, aiming to address the problem of insufficient feature fusion between RGB and depth images in existing technologies, especially the poor robustness of models when the depth image quality is low. This invention introduces a MAE (Masked AutoEncoder) pre-trained model to extract RGB image features and combines it with a ViT (Vision Transformer) model to process depth image features. A hybrid feature cross-fusion module is then used to achieve efficient fusion of the two modalities.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] A method for RGB-D salient target detection based on hybrid feature cross-fusion includes the following steps:

[0008] 1) Use the MAE model to extract features from RGB images;

[0009] 2) Extract features from the depth image corresponding to the RGB image;

[0010] 3) Perform feature cross-fusion on the two extracted image features;

[0011] 4) Decode the saliency image of the fused features to obtain the predicted saliency image.

[0012] Step 1) includes the following steps:

[0013] 1.1) Convert the RGB image I with dimensions H×W×C rgb Divide into blocks of size P×P, the number of blocks Where H, W, and C are the length, width, and number of channels of the image, respectively, and P is a set parameter;

[0014] 1.2) Map the segmented image and add positional encoding to generate RGB image feature vector T. rgb The dimensions are [N, P×P×3];

[0015] 1.3) Using a random masking method, the mask vector M∈[0,1} is defined. N M i =0 indicates that the i-th image patch is masked, M i =1 indicates that the i-th image patch is not masked, and the masked feature vector T is obtained by masking 75% of the tokens. rgb-masked ;

[0016] 1.4) T rgb-masked Feature extraction is performed within the Transformer architecture.

[0017] The T rgb and T rgb-masked Specifically:

[0018] T rgb =Patchify(I rgb )+E pos

[0019] T rgb-masked =F Encoder (T rgb ⊙M)

[0020] Where ⊙ represents the element-wise product, F Encoder E represents the coding layer in the Transformer architecture. pos This indicates location encoding; Patchify indicates the image segmentation process.

[0021] Step 2) specifically refers to:

[0022] Depth image I depth The image is divided into P×P blocks, and positional encoding is added to generate a depth image feature vector T. depth :

[0023] T depth =F Encoder Patchify depth )+E pos ).

[0024] Step 3) includes the following steps:

[0025] 3.1) Perform linear mapping on the feature vectors of the RGB image and the depth image respectively to obtain the query matrix Q, key matrix K and value matrix V of the two images;

[0026] 3.2) Calculate the self-attention Attn of the RGB images separately using a multi-head attention mechanism. r Self-attention Attn of depth images d RGB image-guided depth image attention Attn r-guide Attn for RGB images guided by depth images d-guide Their dimensions are all [B,N,P×P×3], where B is the batch size;

[0027] 3.3) Concatenate the four attention methods into a fusion feature tensor matrix Attn with dimensions [4, B, N, P × P × 3]. fused The matrix is ​​then transposed into a tensor matrix of dimensions [B, N, P × P × 3, 4], and weighted using a multilayer perceptron to obtain the fused RGB-D saliency attention feature tensor matrix T. fused .

[0028] The Attn fused and T fused Specifically:

[0029] Attn fused =Concat(Attn) r Attn d Attn r-guide Attn d-guide )

[0030] T fused=MLP(Transpose(Attn) fused )).

[0031] Step 4) includes the following steps:

[0032] 4.1) Use the Transformer decoder to decode the RGB image features and generate the restored RGB image Pred. rgb And use the image to train the MAE model;

[0033] 4.2) The fused features are decoded using a Transformer decoder to generate a saliency image Pred. sal .

[0034] The Pred rgb Specifically:

[0035] Pred rgb =Unpatchify(F MAE-Decoder (T rgb_masked ))

[0036] Where Unpatchify is the image restoration process, F MAE-Decoder This is the decoding layer for the MAE architecture.

[0037] The Pred sal Specifically:

[0038] Pred sal =Unpatchify(sigmoid(Linear(F Decoder (T fused ))))

[0039] Where Linear is a fully connected linear layer, F Decoder This is the decoding layer for the Transformer architecture.

[0040] An RGB-D salient target detection system based on hybrid feature cross-fusion includes:

[0041] The RGB image feature extraction module is used to extract features from RGB images using the MAE model.

[0042] The depth image feature extraction module is used to extract features from the depth image corresponding to the RGB image;

[0043] The hybrid feature cross-fusion module is used to perform hybrid feature cross-fusion on the two extracted image features;

[0044] The saliency image decoding module is used to decode the saliency image of the fused features to obtain the predicted saliency image.

[0045] The present invention has the following beneficial effects and advantages:

[0046] 1. Efficient Feature Fusion: This invention proposes a hybrid feature cross-fusion module that can effectively fuse features from RGB and depth images, overcoming the difficulty of synergistic effects between RGB and depth features in traditional methods. Compared to fusion methods that rely on only a single feature, this invention achieves more accurate cross-modal information exchange and fusion through a multi-head attention mechanism.

[0047] 2. Adaptability to Different Depth Image Quality: Traditional RGB-D salient object detection methods often exhibit unstable performance when dealing with depth images of varying quality. This invention demonstrates, through training and testing on multiple publicly available datasets, that the proposed hybrid feature cross-fusion module maintains high detection performance and exhibits stronger robustness when processing datasets with inconsistent depth image quality.

[0048] 3. Strong Model Generalization Ability: Comparative experiments with various feature fusion schemes verified the superior performance of this invention on multiple public datasets, especially on test sets with poor depth image quality. This model possesses strong generalization ability and can adapt to salient object detection tasks in various complex scenarios.

[0049] 4. Efficient Training and Optimization: This invention employs the MAE model as the backbone network for RGB feature extraction, combined with the ViT model for depth image feature extraction. The AdamW optimizer and cosine annealing learning rate are used for optimization, significantly improving training efficiency and shortening model convergence time. Simultaneously, the use of the MSE loss function ensures that the generated salient images have high similarity to real images.

[0050] 5. Strong image reconstruction capability: This invention utilizes the autoencoder-decoder architecture of MAE and trains the model with 75% masking, enabling the model to have a strong image reconstruction capability, effectively improving the accuracy of RGB image feature extraction, and ensuring the successful implementation of the RGB-D salient object detection task. Attached Figure Description

[0051] Figure 1 is a network structure diagram of the algorithm model of the present invention;

[0052] Figure 2 shows the fusion attention generation diagram guided by depth image;

[0053] Figure 3 shows the structure of the hybrid cross-feature fusion module. Detailed Implementation

[0054] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0055] The main innovation of this invention lies in the design of a hybrid feature cross-fusion module. This module calculates four different attention matrices for RGB and depth images (RGB image attention, depth image attention, RGB image-guided fusion attention, and depth image-guided fusion attention), and then cross-fused these attention matrices to generate the final fused feature. This module can better capture the complementary information between RGB and depth images, overcoming the performance limitations of existing technologies when the depth image quality is uneven.

[0056] Furthermore, this invention employs a multi-head attention mechanism and a multilayer perceptron (MLP) structure to further improve the feature fusion effect. Experimental verification on multiple publicly available RGB-D datasets (such as COME15K, NJU2K, NLPR, DUTLF-Depth, etc.) shows that the proposed model achieves significant performance improvements on multiple datasets, especially on datasets with poor depth image quality.

[0057] The RGB-D saliency target detection model proposed in this invention adopts the MAE-SOD structure, and the algorithm mainly includes the following steps:

[0058] 1. RGB Image Feature Extraction

[0059] The salient features of the RGB image are extracted using a MAE (Masked AutoEncoder) model. The specific steps are as follows:

[0060] Given an input RGB image I rgb Its dimensions are H×W×C. The image is divided into blocks of size P×P, and the number of blocks is... Map it and add position encoding to generate an RGB image token, denoted as T. rgb The dimensions are [N, P×P×3]. Then, a random masking method is used, defining a mask matrix M∈{0,1}. N M i =0 indicates that the i-th image patch is masked, M i =1 indicates that the i-th image block is not masked. Masking 75% of the tokens yields the masked T. rgb-masked The processed token is then fed into the Transformer architecture for feature extraction. This process can be represented as:

[0061] T rgb =Patchify(I rgb )+E pos

[0062] T rgb-masked=F Encoder (T rgb ⊙M)

[0063] Where ⊙ represents the element-wise product, F Encoder This refers to the encoder portion of the Transformer architecture.

[0064] 2. Depth Image Feature Extraction

[0065] Feature extraction is performed on the input depth image using the ViT (Vision Transformer) model. The specific steps are as follows: the depth image is also divided into blocks to generate depth image tokens (feature vectors). These tokens are then processed by the ViT encoder to extract the feature vectors of the depth image.

[0066] Depth image I depth Divide the image into P×P blocks, add positional encoding to generate a depth image token, denoted as T. depth This process can be represented as:

[0067] T depth =F Encoder Patchify depth )+E pos )

[0068] Where E pos This indicates location encoding; Patchify indicates the image segmentation process.

[0069] 3. Hybrid Feature Cross-fusion Module

[0070] To effectively fuse features from RGB and depth images, a hybrid feature cross-fusion module is proposed. The specific steps are as follows: Linear mapping is performed on the feature vectors of the RGB and depth images respectively to obtain the query matrix (Q), key matrix (K), and value matrix (V). A multi-head attention mechanism is then used to calculate the self-attention of the RGB image, the self-attention of the depth image, the RGB image-guided depth image attention, and the depth image-guided RGB image attention. The attention mechanism formula can be expressed as:

[0071]

[0072] Where d k Let Q and K be the dimensions of the matrices. The formulas for the four attention methods can be expressed as follows:

[0073]

[0074] The four attention matrices are concatenated to form a fused feature matrix. The fused feature matrix is ​​then transposed and weighted using a multilayer perceptron (MLP) to obtain the fused RGB-D salient features.

[0075] Furthermore, a multi-head attention mechanism is used to compute multiple different attention heads in parallel, and each head is denoted as Attn. r Attn d Attn r-guide and Attn d-guide Its dimensions are [B, N, P × P × 3], where B is the batch size. The four attention parameters are concatenated into a fusion feature tensor matrix Attn with dimensions [4, B, N, P × P × 3]. fused The matrix is ​​then transposed into a tensor matrix of dimensions [B, N, P × P × 3, 4], and then weighted using a multilayer perceptron to obtain the final fused RGB-D saliency attention feature tensor matrix T. fused The process can be represented as:

[0076] Attn fused =Concat(Attn) r Attn d Attn r-guide Attn d-guide )

[0077] T fused =MLP(Transpose(Attn) fused ))

[0078] 4. Saliency Image Decoding

[0079] The fused features are decoded using a Transformer decoder to generate the predicted saliency image. The specific steps are as follows: the fused features are input into the decoder, and features of the saliency image are extracted through multiple decoding operations. A fully connected layer is added at the end of the decoder, and a sigmoid activation function is used to map the output to the range of the saliency map. Finally, the decoded features are reassembled to the original image size using an unpatchify operation to generate the final saliency image.

[0080] Simultaneously, during training, by optimizing the reconstruction loss for RGB image restoration, the model can learn to infer and reconstruct the masked portion based on known image patches. Finally, the image blocks of dimension [N, P×P×C] obtained from the decoder are restored to an image of dimension H×W×C through an unpatchify operation, and the restored RGB image Pred is predicted. rgb The process can be represented as follows:

[0081] Pred rgb =Unpatchify(F MAE-Decoder (T rgb_masked ))

[0082] The saliency image feature decoding part has a similar decoder architecture to the image reconstruction decoding part, also using a Transformer decoding layer. However, the saliency image feature decoding adds a linear fully connected layer and a Sigmoid activation function after decoding the fused features obtained from the feature fusion module through the decoding layer. Finally, unpatchify generates a saliency image Pred with dimensions H×W×1. sal The salient feature decoding process can be represented as:

[0083] Pred sal =Unpatchify(sigmoid(Linear(F Decoder (T fused ))))

[0084] 5. Training and Optimization

[0085] The model training uses mean squared error (MSE) as the loss function, and the AdamW optimizer is used during optimization, with a cosine annealing learning rate strategy for fine-tuning. The MSE loss function captures the pixel-level differences between the original and reconstructed images, promoting the model to generate reconstructed images that are more similar to the input image. Given a predicted saliency map S and a ground truth map G, where S(i,j) and G(i,j) represent the values ​​of the pixels in the i-th row and j-th column of S and G respectively, the MSE loss function can be expressed as:

[0086]

[0087] Where W and H represent the width and height of the image, respectively.

[0088] The training process includes: comparing the predicted saliency images with the true saliency images, calculating the MSE loss, and optimizing the model parameters. Through multiple rounds of training, the model weights are continuously updated, enabling the model to better adapt to RGB-D saliency detection tasks on different datasets.

[0089] Example

[0090] Figure 1 shows the overall structure of the RGB-D salient object detection network proposed in this invention. The structure consists of two main parts: RGB image processing and depth image processing.

[0091] RGB Image Processing: Input RGB Image I rgbFirst, the image is processed by the MAE (Masked Autoencoder) encoder module, which divides it into several unmasked and masked image blocks to form T. rgb-masked These image patches are then fed into the MAE decoder module for reconstruction to obtain the reconstructed RGB image I. pred .

[0092] Depth Image Processing: Input depth image I depth After passing through the ViT (Vision Transformer) encoder, the depth image features are extracted to form a T... depth RGB and depth image features are fused through a hybrid cross-feature fusion module, enhancing the collaborative information between the two.

[0093] Hybrid Feature Cross-Fusion Module: This module enables the model to effectively fuse features from RGB and depth images. The fused features are further processed by a decoder to generate a saliency image (Sal). pred The final output saliency image after the fusion module and decoder is compared with the ground truth image. The figure shows the comparison between the final output saliency image and the ground truth annotation, demonstrating the model's effectiveness.

[0094] Figure 2 shows the generative network structure for fusion attention guided by the depth image obtained by cross-fusion of RGB images and depth images.

[0095] RGB features of the image T rgb-masked After passing through the projection module, a query vector Q is generated. rgb Deep features T depth After passing through the projection module, a key-value vector K is generated. depth and V depth The query vector Q for RGB image features. rgb Key vector K of depth image features depth Matrix multiplication (matmul) is performed to calculate the relevance score. The score is then scaled and normalized using a sigmoid activation function to obtain the depth-guided weights. These normalized weights are then compared with the depth image feature value vector V. depth Matrix multiplication is performed to generate a deep-guided attention graph Attn. depth-guide .

[0096] Figure 3 shows the structure of the multi-attention weighted cross-fusion module used for feature fusion.

[0097] Attention Module: This module contains multiple attention channels, such as Attn. r Attn d Attn r-guide and Attn d-guideThese different attention channels process and enhance the features of the RGB and depth images respectively.

[0098] Multilayer Perceptron (MLP) Block: Features output from different attention channels are fused by the MLP module to generate a weighted fused attention feature Attn. fused Its dimensions are [B, N, P × P × 3]. This fused feature represents the final feature after weighted attention fusion, which can capture the multimodal correlation between RGB and depth image features, further improving the detection accuracy of the model.

[0099] The above description is only the best specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.

[0100] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A method for RGB-D salient target detection based on hybrid feature cross-fusion, characterized in that, Includes the following steps: 1) Use the MAE model to extract features from the RGB image; 2) Extract features from the depth image corresponding to the RGB image; 3) Perform cross-fusion of the extracted two types of image features; 4) Decode the saliency image of the fused features to obtain the predicted saliency image.

2. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 1, characterized in that, Step 1) includes the following steps: 1.1) Converting the RGB image I with dimensions H×W×C... rgb Divide into blocks of size P×P, the number of blocks Where H, W, and C are the length, width, and number of channels of the image, respectively, and P is a set parameter; 1.2) Map the segmented image and add position encoding to generate an RGB image feature vector T. rgb The dimensions are [N, P×P×3]; 1.3) Using a random masking method, the mask vector M∈{0,1} is defined. N M i =0 indicates that the i-th image patch is masked, M i =1 indicates that the i-th image patch is not masked, and the masked feature vector T is obtained by masking 75% of the tokens. rgb-masked ;1.4) T rgb-masked Feature extraction is performed within the Transformer architecture.

3. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 2, characterized in that, The T rgb and T rgb-masked Specifically: T rgb =Patchify(I rgb )+E pos T rgb-masked =F Encoder (T rgb ⊙M) where ⊙ represents the element-wise product, F Encoder E represents the coding layer in the Transformer architecture. pos This indicates location encoding; Patchify indicates the image segmentation process.

4. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 1, characterized in that, Step 2) specifically involves: transferring the depth image I... depth The image is divided into P×P blocks, and positional encoding is added to generate a depth image feature vector T. depth :T depth =F Encoder Patchify depth )+E pos ).

5. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 1, characterized in that, Step 3) includes the following steps: 3.1) Perform linear mapping on the feature vectors of the RGB image and the depth image respectively to obtain the query matrix Q, key matrix K, and value matrix V of the two images; 3.2) Calculate the self-attention Attn of the RGB image respectively through a multi-head attention mechanism. r Self-attention Attn of depth images d RGB image-guided depth image attention Attn r-guide Attn for RGB images guided by depth images d-guide Their dimensions are all [B, N, P×P×3], where B is the batch size; 3.3) The four attentions are concatenated into a fusion feature tensor matrix Attn with dimensions [4, B, N, P×P×3]. fused The matrix is ​​then transposed into a tensor matrix of dimensions [B, N, P×P×3, 4], and weighted using a multilayer perceptron to obtain the fused RGB-D saliency attention feature tensor matrix T. fused .

6. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 5, characterized in that, The Attn fused and T fused Specifically: Attn fused =Concat(Attn) r Attn d Attn r-guide Attn d-guide )T fused =MLP(Transpose(Attn) fused )).

7. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 1, characterized in that, Step 4) includes the following steps: 4.1) Decode the RGB image features using a Transformer decoder to generate the restored RGB image Pred. rgb 4.2) The fused features are then used to train the MAE model; the Transformer decoder is used to decode the fused features and generate a saliency image Pred. sal .

8. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 7, characterized in that, The Pred rgb Specifically: Pred rgb =Unpatchify(F MAE-Decoder (T rgb_masked Among them, Un patchify is the image restoration process, F MAE-Decoder This is the decoding layer for the MAE architecture.

9. The RGB-D saliency target detection method based on hybrid feature cross-fusion according to claim 7, characterized in that, The Pred sal Specifically: Pred sal =Unpatchify(sigmoid(Linear(F Decoder (T fused Where Linear is a fully connected linear layer, F Decoder This is the decoding layer for the Transformer architecture.

10. An RGB-D salient target detection system based on hybrid feature cross-fusion, characterized in that, include: The module is divided into four parts: RGB image feature extraction module, which is used to extract features from RGB images using the MAE model; depth image feature extraction module, which is used to extract features from the depth image corresponding to the RGB image; hybrid feature cross-fusion module, which is used to perform hybrid feature cross-fusion on the two extracted image features; and saliency image decoding module, which is used to perform saliency image decoding on the fused features to obtain the predicted saliency image.

Images