An RGB-D salient object detection method based on cross-modal edge guidance
By employing a cross-modal edge-guided RGB-D salient object detection method, which utilizes the Swing Transformer and attention fusion network, the problems of cross-modal information fusion and edge segmentation are solved, achieving more accurate salient object detection.
Patent Information
- Application Number
- CN202310708088.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-14
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2043-06-14
AI Technical Summary
Existing RGB-DSOD methods suffer from errors and information loss in cross-modal information fusion and salient target edge segmentation, which affects detection accuracy.
We employ a cross-modal edge-guided RGB-D salient object detection method. We extract features using the Swing Transformer, combine it with a channel and spatial attention fusion network to construct an edge-aware module and a decoder, and perform multi-scale feature fusion to generate a salient map.
It improves the accuracy of cross-modal feature fusion and the ability to preserve edge details, thereby enhancing the precision of salient target detection.
Smart Images
Figure CN116958736B_ABST
Abstract
Description
Technical fields:
[0001] This invention relates to the fields of computer vision and image processing, and in particular to an RGB-D salient target detection method based on cross-modal edge guidance. Background technology:
[0002] RGB-D salient object detection is a technique that uses RGB images and depth images (i.e., depth represents depth) for object detection. Compared to using only RGB images, RGB-D salient object detection can obtain more accurate scene information by using depth information.
[0003] Although many methods have emerged and made some progress in the field of RGB-DSOD, there are currently two main challenges hindering its development: (1) Fusion of cross-modal information. RGB images and Depth images are based on optical imaging and structured light principles, respectively, and therefore have different imaging principles and noise characteristics. This difference may lead to certain errors in the fusion of cross-modal information, and directly using some traditional multimodal fusion algorithms may result in the loss of feature information or the omission of key information. Therefore, it is necessary to explore more effective fusion algorithms and make full use of their connections and complementarities. (2) Segmentation of salient target edges. In RGB-DSOD, edge quality is of concern because edges are usually the boundaries between different objects or backgrounds in an image, and therefore are crucial for the accurate localization and segmentation of targets. The quality of the edges directly affects the accuracy of salient target detection. Some existing algorithms use edge information to determine salient targets in an image and distinguish them from the background. Therefore, when performing salient target detection, it is necessary to consider the quality of the extracted edges to ensure more accurate localization and segmentation of salient targets.
[0004] Therefore, there is an urgent need to design an RGB-D salient object detection method based on cross-modal edge guidance to better integrate cross-modal features and preserve edge details, so as to solve the above problems. Summary of the Invention:
[0005] The technical problem to be solved by the present invention is to provide an RGB-D saliency target detection method based on cross-modal edge guidance, which can better integrate cross-modal features and preserve edge details.
[0006] The specific technical solution adopted in this invention is as follows:
[0007] A method for RGB-D salient object detection based on cross-modal edge guidance, comprising the following steps: 1. Obtaining the RGB-D dataset for training and testing the task;
[0008] 2. Establish salient object detection model networks for extracting RGB image features and Depth image features respectively; 3. Establish a network structure for cross-modal attention fusion;
[0009] 4. Establish a network structure for edge sensing;
[0010] 5. Construct an edge-guided fusion decoder to decode multi-scale fusion features from high to low and concatenate and fuse them with edge features to obtain a saliency map.
[0011] Furthermore, in step 1, the NJU2K dataset and the NLPR dataset are used as training sets, and the remaining parts of the NJU2K dataset, the remaining parts of the NLPR dataset, the RGBD135 dataset, the SSD dataset, and the STERE dataset are used as test sets.
[0012] Furthermore, step 2 includes the following sub-steps:
[0013] 2.1) The Swing Transformer is used as the backbone network of the model in this invention to extract RGB image features and corresponding Depth image features, which are represented as follows: and Where i represents the layer number, corresponding to the output of each layer of the Swing Transformer.
[0014] 2.2) The SwinTransformer weights used in this invention to construct the backbone network are initialized using pre-trained parameter weights.
[0015] Furthermore, the network structure used for cross-modal attention fusion in step 3 is as follows:
[0016] 3.1) By using channel attention to filter out the most expressive and relevant feature channels, interference from non-relevant information during the process is reduced. This is done by... and Channel attention processing is performed on the channel features to obtain RGB channel attention vectors and depth channel attention vectors.
[0017]
[0018] in, and Let these represent the parameters and biases of the fully connected layer for the i-th level RGB features, respectively. and Let Avg(·) represent the parameters and biases of the i-th level deep feature fully connected layer, respectively. Let Avg(·) represent the global average pooling operation, σ(·) represent the softmax activation function, and + represent the element-aware matrix addition operation.
[0019] 3.2) Aggregate the results obtained in step 3.1) through addition operations. and We obtain more discriminative and representative feature representations of the RGB-D image in the channel dimension. Then, we normalize the aggregated attention vector to obtain the cross-modal channel attention vector. To enhance important feature channels and reduce channels that do not contribute to the prediction results, based on The weights of the RGB features and depth features are respectively related to... and Perform channel multiplication to obtain RGB weighted enhanced features. and depth-weighted augmented features
[0020]
[0021] in, This indicates an element-wise multiplication operation, and N(·) is used to normalize the result to between 0 and 1.
[0022] 3.3) Further refine the results obtained in step 3.2). and Spatial attention processing is performed on the spatial features to obtain the RGB spatial attention vector. and depth space attention vector Then to and The purpose of performing the multiplication operation again is to preserve the information of the original feature map and avoid the attention mechanism from overemphasizing local features. Then, the processed features are concatenated to obtain RGB space-weighted features. and depth space weighted features
[0023]
[0024] Where Max(·) represents the global max pooling operation, and C represents the cascading operation.
[0025] 3.4) For the results obtained in step 3.1) and RGB channel weighted features are generated through multiplication operations. and depth channel weighted features
[0026]
[0027]
[0028] 3.5) For the results obtained in step 3.4) and With the results obtained in step 3.3 and Perform element-wise addition operations to obtain the RGB channel spatial enhancement features. and depth channel spatial enhancement features
[0029] 3.6) For the results obtained in step 3.5) and Element-based multiplication and addition are applied, and the results are concatenated and then fed into a 3×3 convolutional layer to obtain a smooth representation, ultimately yielding cross-attention fusion features.
[0030]
[0031] Here, Bconv(·) is a sequence operation that combines batch normalization after 3×3 convolution with the ReLU function. Furthermore, the network structure used for edge detection in step 4 is specifically configured as follows:
[0032] 4.1) For the features extracted from step 2, and RGB edge features are obtained by addition respectively. and depth edge features
[0033]
[0034] Up2(·) represents the 2×upsampling operation.
[0035] 4.2) The result obtained in step 4.1) and Channel attention and spatial attention are performed separately, and then the results are residually connected to generate clearer edge information. and
[0036]
[0037] Here, SA represents spatial attention operation, and SE is a channel-type attention mechanism.
[0038] 4.3) The result obtained in step 4.2) and Aggregate the data to generate edge-aware features F. e :
[0039]
[0040] Furthermore, step 5 includes the following sub-steps:
[0041] 5.1) Using the information obtained in step 3 Using the input as input, the multi-scale feature fusion result is obtained by adding elements one by one.
[0042]
[0043] 5.2) The edge features F obtained in step 4.3) e The multi-scale fusion features obtained in step 5.1) Combined, an edge-guided salient graph S is generated:
[0044]
[0045] Where Sig(·) represents the Sigmoid activation function.
[0046] Compared with previous RGB-D saliency target detection methods, the advantages of this invention are as follows:
[0047] First, deep learning techniques are used to construct the relationship between RGB-D image pairs and salient objects in the images through encoder and decoder structures. Saliency prediction is obtained through cross-modal feature extraction and fusion. Second, edge awareness is used to better preserve edge details. Finally, an edge-guided decoder is constructed to perform multi-scale, multi-modal feature fusion to predict the final saliency map. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of the model structure of the present invention;
[0049] Figure 2 This is a schematic diagram of the cross-modal attention fusion module;
[0050] Figure 3 This is a schematic diagram of the edge sensing module;
[0051] Figure 4 This is a comparison chart of the results of this invention with other RGB-D saliency target detection methods. Detailed Implementation
[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Furthermore, the described examples are merely a part of the examples of the present invention, and not all of them. All other examples obtained by those skilled in the art based on the examples in this invention without inventive effort are within the scope of protection of this invention.
[0053] Reference Appendix Figure 1A method for RGB-D saliency target detection based on cross-modal edge guidance mainly includes the following steps:
[0054] 1. Obtain the RGB-D dataset for training and testing this task, and define the algorithm objective of this invention. Use the NJU2K dataset and NLPR dataset as the training set, and use the remaining parts of the NJU2K dataset, the remaining parts of the NLPR dataset, the RGBD135 dataset, the SSD dataset, and the STERE dataset as the test set.
[0055] 2. Establish salient object detection model networks for extracting RGB image features and corresponding Depth image features respectively: 2.1) Utilize the Swin Transformer as the backbone network of the model in this invention to extract RGB image features and corresponding Depth image features, denoted as follows: and Where i represents the layer number, corresponding to the output of each layer of the SwinTransformer.
[0056] 2.2) The SwinTransformer weights used in this invention to construct the backbone network are initialized using pre-trained parameter weights.
[0057] 3. Reference Appendix Figure 2 As shown, the four levels of RGB image features generated in step 2 are combined through a cross-modal attention fusion module. and Depth image features By performing fusion, four levels of cross-attention fusion features are obtained. The main steps are as follows:
[0058] 3.1) By using channel attention to filter out the most expressive and relevant feature channels, interference from non-relevant information during the process is reduced. This is done by... and Channel attention processing is performed on the channel features to obtain RGB channel attention vectors. and deep channel attention vector
[0059]
[0060] in, and Let these represent the parameters and biases of the fully connected layer for the i-th level RGB features, respectively. and Let Avg(·) represent the parameters and biases of the i-th level deep feature fully connected layer, respectively. Let Avg(·) represent the global average pooling operation, σ(·) represent the softmax activation function, and + represent the element-aware matrix addition operation.
[0061] 3.2) Aggregate the results obtained in step 3.1) through addition operations. and We obtain more discriminative and representative feature representations of the RGB-D image in the channel dimension. Then, we normalize the aggregated attention vector to obtain the cross-modal channel attention vector. To enhance important feature channels and reduce channels that do not contribute to the prediction results, based on The weights of the RGB features and depth features are respectively related to... and Perform channel multiplication to obtain RGB weighted enhanced features. and depth-weighted augmented features
[0062]
[0063] in, This indicates an element-wise multiplication operation, and N(·) is used to normalize the result to a range of 0 to 1.
[0064] 3.3) Further refine the results obtained in step 3.2). and Spatial attention processing is performed on the spatial features to obtain the RGB spatial attention vector. and depth space attention vector Then to and The purpose of performing the multiplication operation again is to preserve the information of the original feature map and avoid the attention mechanism from overemphasizing local features. Then, the processed features are concatenated to obtain RGB space-weighted features. and depth space weighted features
[0065]
[0066]
[0067] Where Max(·) represents the global max pooling operation, and C represents the cascading operation.
[0068] 3.4) For the results obtained in step 3.1) and RGB channel weighted features are generated through multiplication operations. and depth channel weighted features
[0069]
[0070] 3.5) For the results obtained in step 3.4) and Acquired with S33 and Perform element-wise addition operations to obtain the RGB channel spatial enhancement features. and depth channel spatial enhancement features
[0071] 3.6) For the results obtained in step 3.5) and Element-based multiplication and addition are applied, and the results are concatenated and then fed into a 3×3 convolutional layer to obtain a smooth representation, ultimately yielding cross-attention fusion features.
[0072]
[0073] Bconv(·) is a sequence operation that combines batch normalization after 3×3 convolution with the ReLU function.
[0074] 4. Reference Appendix Figure 3 Edge features F are extracted using an edge sensing module. e The main steps are as follows:
[0075] 4.1) For the features extracted from step 2, and RGB edge features are obtained by addition respectively. and depth edge features
[0076]
[0077] Up2(·) represents the 2×upsampling operation.
[0078] 4.2) The result obtained in step 4.1) and Channel attention and spatial attention are performed separately, and then the results are residually connected to generate clearer edge information. and
[0079]
[0080] Here, SA represents spatial attention operation, and SE is a channel-type attention mechanism.
[0081] 4.3) The result obtained in step 4.2) and Aggregate the data to generate edge-aware features F. e :
[0082]
[0083] 5. Reference Appendix Figure 4 A decoder for edge-guided fusion is constructed to decode multi-scale fusion features from high to low and concatenate them with edge features to obtain a saliency map.
[0084] 5.1) Using the information obtained in step 3 Using the input as input, the multi-scale feature fusion result is obtained by adding elements one by one.
[0085]
[0086] 5.2) The edge features F obtained in step 4.3) e The multi-scale fusion features obtained in step 5.1) Combined, an edge-guided salient graph S is generated:
[0087]
[0088] Where Sig(·) represents the Sigmoid activation function.
[0089] This invention presents a cross-modal edge-guided RGB-D salient object detection method. Utilizing deep learning technology, it constructs the relationship between RGB-D image pairs and salient objects in the images through an encoder and decoder structure. Saliency prediction is obtained through cross-modal feature extraction and fusion. Secondly, edge awareness is used to better preserve edge detail information. Finally, an edge-guided decoder is constructed to perform multi-scale, multi-modal feature fusion, predicting the final salient map.
[0090] Example
[0091] The RGB-D image salient object detection method described in this embodiment uses 1485 images from the NJU2K dataset and 700 images from the NLPR dataset as the training set. The remaining images from the NJU2K and NLPR datasets, as well as the entire RGBD135, SSD, and STERE datasets, are used as the test set for testing. During the training and testing phases, the input RGB-D images are resized to 384*384 pixels, and the training set undergoes data augmentation operations such as random rotation, flipping, and border cropping. The model training uses the Adam optimizer with an initial learning rate of 1e-5, a batch size of 3, and employs Swin-B pre-training parameters and default PyTorch settings. The graphics card used is an NVIDIA RTX A4000 GPU. The model converges after 150 generations of training, taking approximately 18 hours. The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for RGB-D salient target detection based on cross-modal edge guidance, characterized in that, The method includes the following steps: 1) Obtain the RGB-D dataset for training and testing this task; 2) Establish salient object detection model networks for extracting RGB image features and Depth image features respectively; 3) Establish a network structure for cross-modal attention fusion; 4) Establish a network structure for edge sensing; 5) Construct an edge-guided fusion decoder to decode multi-scale fusion features from high to low and concatenate and fuse them with edge features to obtain a saliency map; The network structure used for cross-modal attention fusion in step 3 is as follows: 3.1) By using channel attention to filter out the most expressive and relevant feature channels, interference from non-relevant information during the process is reduced. This is done by... and Channel attention processing is performed on the channel features to obtain RGB channel attention vectors. and deep channel attention vector 3.2) Aggregate the results obtained in step 3.1) through addition operations. and We obtain more discriminative and representative feature representations of the RGB-D image in the channel dimension. Then, we normalize the aggregated attention vector to obtain the cross-modal channel attention vector. To enhance important feature channels and reduce channels that do not contribute to the prediction results, based on The weights of the RGB features and depth features are respectively related to... and Perform channel multiplication to obtain RGB weighted enhanced features. and depth-weighted augmented features 3.3) Further refine the results obtained in step 3.2). and Spatial attention processing is performed on the spatial features to obtain the RGB spatial attention vector. and depth space attention vector Then to and The purpose of performing the multiplication operation again is to preserve the information of the original feature map and avoid the attention mechanism from overemphasizing local features. Then, the processed features are concatenated to obtain RGB space-weighted features. and depth space weighted features 3.4) For the results obtained in step 3.1) and RGB channel weighted features are generated through multiplication operations. and depth channel weighted features 3.5) For the results obtained in step 3.4) and With the results obtained in step 3.3 and Perform element-wise addition operations to obtain the RGB channel spatial enhancement features. and depth channel spatial enhancement features 3.6) For the results obtained in step 3.5) and Element-based multiplication and addition are applied, and the results are concatenated and then fed into a 3×3 convolutional layer to obtain a smooth representation, ultimately yielding cross-attention fusion features. The specific structure of the network used for edge sensing in step 4 is as follows: 4.1) For the features extracted from step 2, and RGB edge features are obtained by addition respectively. and depth edge features Where Up2(·) represents a 2×upsampling operation; 4.2) The result obtained in step 4.1) and Channel attention and spatial attention are performed separately, and then the results are residually connected to generate clearer edge information. and Here, SA represents spatial attention operation, and SE is a channel-type attention mechanism; 4.3) The result obtained in step 4.2) and Aggregate the data to generate edge-aware features F. e : Step 5 includes the following sub-steps: 5.1) Using the information obtained in step 3 Using the input as input, the multi-scale feature fusion result is obtained by adding elements one by one. 5.2) The edge features F obtained in step 4.3) e The multi-scale fusion features obtained in step 5.1) Combined, an edge-guided salient graph S is generated: Where Sig(·) represents the Sigmoid activation function.
2. The RGB-D saliency target detection method based on cross-modal edge guidance according to claim 1, characterized in that: The specific method for step 1) is as follows: The NJU2K and NLPR datasets were used as training sets, while the remaining parts of the NJU2K dataset, the remaining parts of the NLPR dataset, the RGBD135 dataset, the SSD dataset, and the STERE dataset were used as test sets.
3. The RGB-D saliency target detection method based on cross-modal edge guidance according to claim 1, characterized in that: The specific method for step 2) is as follows: 2.1) The Swin Transformer is used as the backbone network of the model to extract RGB image features and corresponding depth image features, which are represented as follows: and Where i represents the layer number, corresponding to the output of each layer of the SwinTransformer; 2.2) Initialize the Swin Transformer weights used to build the backbone network using pre-trained parameter weights.