Attention refining method for RGB-D saliency target detection
By using multiple attention and cross-attention integration in the RGB-D significant target detection, the problem of information redundancy and significant picture quality of information when cross-modular fusion is solved, and the high quality is significantly significant. Graph generation.
Patent Information
- Application Number
- CN202411466157.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-08-15
- Filing Date
- 2024-10-21
- Publication Date
- 2025-01-24
AI Technical Summary
The existing RGB-D significance object detection algorithms are problematic due to low quality of depth information during cross-modal fusion, resulting in low information quality and low significant graph quality.
Using an end-to-end structure, the fusion of cross-modal features is achieved through the fusion of multiple attention refinements and cross-attention. The attention interaction module and multi-scale compensation module are designed to reduce information redundancy, reduce the interference of background noise, and achieve the fusion of local and global context of multi-scale features.
It effectively avoids the influence of background noise caused by poor depth map quality during the cross-modal fusion process, maximizes the use of model performance, and generates high-quality saliency maps.
Smart Images
Figure CN119360170A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer image processing, and specifically relates to an attention refinement method for RGB-D salient target detection, which adopts an end-to-end structure, extracts features from RGB and depth images and performs inter-modal complementation, and generates high-quality salient maps by decoding through a multi-scale fusion method. The present invention relates to a method for extracting and processing RGB and depth images, and particularly to a method for interacting the two modalities by combining a spatial and channel attention mechanism with a cross attention mechanism. Background Art
[0002] Most existing RDB-D (Depth) salient object detection algorithms use depth information for cross-modal fusion, but due to the low quality of depth information, blur and difficulty in distinguishing complex scenes, a large amount of information redundancy will result, resulting in low quality of the generated salient maps. Summary of the invention
[0003] In response to the problems of insufficient cross-modal fusion and poor quality of depth maps in RDB-D salient target detection, the present invention proposes an attention refinement method for RGB-D salient target detection, which adopts an end-to-end structure and realizes the fusion of cross-modal features by means of multiple attention refinement and cross-attention fusion. Specifically, the present invention designs an attention interaction module, which uses multiple attention and cross-attention to refine and fuse the two modalities, reduces the information redundancy generated by cross-modal interaction, and reduces the interference of background noise. On this basis, the present invention designs a multi-scale compensation module to guide the fusion of multi-scale features step by step, and realizes the fusion of local and global contexts of multi-scale features.
[0004] The attention refinement method for RGB-D salient object detection includes the following steps:
[0005] A. Extract features from RGB and depth images to generate high-quality saliency maps.
[0006] A1. Using the Scale-Aware Modulation (SMT) framework, a two-stream symmetric network is designed to form an encoder to extract features from RGB and depth images respectively.
[0007] A2. The two features are fused through multiple attention and cross attention mechanisms and fed into the decoder to generate a saliency map.
[0008] B. Feature fusion is performed through an attention interaction module consisting of multiple attentions and cross attentions.
[0009] B1. The extracted RGB and depth features are input into a triple extraction network consisting of spatial attention and channel attention for feature refinement.
[0010] B2. The features refined by multiple attentions are sent to the cross-attention module for fusion to obtain cross-modal features.
[0011] C. The fusion of local and global contexts of multi-scale features is achieved through gradual guidance of the multi-scale compensation module.
[0012] C1. Increase the feature receptive field by dilating the convolution.
[0013] C2. Capture global context information through global average pooling and use ordinary convolution to obtain local context information.
[0014] C3. The extracted global information and local information are concatenated, the spatial attention module is used to guide the extracted feature context information, and the skip connection is used to compensate for the information loss caused by convolution and pooling operations to generate a high-quality saliency map.
[0015] Its advantages are:
[0016] Firstly, considering the details and spatial differences of RGB and depth features, this paper designs an attention interaction module, which establishes attention connections between features of different modalities and uses multiple attention and cross attention to refine and fuse the two modalities, thereby enhancing the fusion effect. Secondly, a multi-scale compensation module is designed to guide the fusion of multi-scale features step by step through a refinement aggregation method.
[0017] This method of interacting different modalities through multiple attention and cross-attention effectively avoids the influence of background noise caused by poor quality of depth maps during cross-modal fusion, thereby maximizing the performance of the model and synthesizing high-quality saliency maps. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Attached Figure 1 It is the overall flow chart of the multi-attention refinement network in the present invention.
[0019] Attached Figure 2 It is a specific flow chart of the attention interaction module in the present invention.
[0020] Attached Figure 3 It is a specific flow chart of the cross-attention module in the present invention.
[0021] Attached Figure 4 It is a specific flow chart of the multi-scale compensation module in the present invention.
[0022] Attached Figure 5It is a PR (precision; recall) curve diagram of the present invention on five public data sets.
[0023] Attached Figure 6 It is an example of a significant pattern in the present invention.
[0024] Attached Figure 7 yes Figure 5 A partial enlargement of the first part.
[0025] Attached Figure 8 yes Figure 5 A partial enlargement of the second part.
[0026] Attached Fig. 9 yes Figure 5 A partial enlarged view of the third part.
[0027] Attached Fig.10 yes Figure 5 A partial enlarged view of the fourth part.
[0028] Attached Fig.11 yes Figure 5 A partial enlarged view of the fifth part. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention is clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. The overall algorithm flow chart is as follows: Figure 1 As shown:
[0030] A. Extracting RGB and depth map features to generate high-quality saliency maps.
[0031] The step A specifically comprises the following steps:
[0032] A1. First, the features of RGB and depth images are extracted through the SMT two-stream symmetric network. The five public RGB-D datasets used are: NJU2K (containing 2003 images), NLPR (containing 1000 images), STERE (containing 1000 images), LFSD (containing 100 images) and DUT-RGBD (containing 1200 images). The training dataset includes 1485 images from NJU2K, 700 images from NLPR and 800 images from DUT, and the remaining image pairs are used for testing. The RGB and depth image datasets are input to the SMT two-stream symmetric network encoder.
[0033] A2. The two features are fused through the multi-attention and cross-attention mechanisms and fed into the decoder to generate a saliency map. The relevant parameters are batch_size=8, epoch=200, learning_rate=0.0001, decay_rate=0.1.
[0034] B. The features are fused through the attention interaction module composed of multiple attention and cross attention. Figure 2 shown.
[0035] The step B specifically comprises the following steps:
[0036] B1. The extracted RGB and depth features are input into a triple extraction network consisting of spatial attention and channel attention for feature refinement.
[0037] B2. The features refined by multiple attentions are sent to the cross-attention module for fusion to obtain cross-modal features. Figure 3 shown.
[0038] C. The fusion of local and global context of multi-scale features is achieved by gradually guiding the multi-scale compensation module. Figure 4 shown.
[0039] C1. Increase the feature receptive field by dilating convolution.
[0040] C2. Capture global context information through global average pooling and use ordinary convolution to obtain local context information.
[0041] C3. The extracted global information and local information are concatenated, the spatial attention module is used to guide the extracted feature context information, and the skip connection is used to compensate for the information loss caused by convolution and pooling operations to generate a high-quality saliency map.
Claims
1. An attention refinement method for RGB-D salient object detection, characterized by The following steps are involved: A. Extract RGB and depth map features to generate high-quality saliency maps; B. Feature fusion is performed through an attention interaction module consisting of multiple attentions and cross attentions; C. The fusion of local and global contexts of multi-scale features is achieved through gradual guidance of the multi-scale compensation module.
2. The attention refinement method for RGB-D salient object detection as claimed in claim 1, characterized in that The method comprises the following steps: Step A specifically comprises the following steps: A1. Use scale-aware modulation units to design a two-stream symmetric network to form an encoder to extract features from RGB and depth images respectively; A2. The two features are fused through multiple attention and cross attention mechanisms and fed into the decoder to generate a saliency map.
3. The attention refinement method for RGB-D salient object detection as claimed in claim 1, characterized in that The method comprises the following steps: Step B specifically comprises the following steps: B1. Input the extracted RGB and depth features into a triple extraction network consisting of spatial attention and channel attention for feature refinement; B2. The features refined by multiple attentions are sent to the cross-attention module for fusion to obtain cross-modal features.
4. The attention network method for RGB-D salient object detection as claimed in claim 1, characterized in that The method comprises the following steps: Step C specifically comprises the following steps: C1. Increase the feature receptive field by dilating the convolution; C2. Capture global context information through global average pooling and use ordinary convolution to obtain local context information; C3. The extracted global information and local information are concatenated, the spatial attention module is used to guide the extracted feature context information, and the skip connection is used to compensate for the information loss caused by convolution and pooling operations to generate a high-quality saliency map.