RGB-d salient object detection method based on multi-scale adaptive fusion

Abstract

The present application belongs to the field of image detection, and particularly relates to an RGB-D salient object detection method based on multi-scale adaptive fusion. Based on the research on early and middle fusion strategies and cross-modal multi-scale interaction, the present application proposes a double-flow network to encode RGB-D features in shallow, middle and deep multi-scale adaptive integration through a cross-modal fusion encoder. For the first problem, different from the traditional early fusion of RGB-D images in 2D space, the present application projects the RGB-D images into a three-dimensional space and represents them as point cloud data, learns the point cloud representation in the three-dimensional space, explicitly models the pixel-level relationship and further explores the appearance and geometric information. For the second problem, the present application applies each scale feature to a point cloud-based fusion module to fully utilize the complementary information of the two modalities.

Description

Technical Field

[0001] This invention belongs to the field of image detection, specifically relating to an RGB-D salient target detection method based on multi-scale adaptive fusion. Background Technology

[0002] In recent years, salient object detection (SOD) technology, aiming to locate and identify the most attractive regions in a scene, has received widespread attention. As an effective preprocessing method, it has been applied to various computer vision tasks, such as scene classification, visual tracking, and image editing. With the increasing prevalence of depth cameras such as Kinect and RealSense, salient object detection from RGB-D ("D" stands for depth) input is becoming an attractive research direction. Researchers can learn geometric and positional information from depth images that are complementary to RGB images, which helps to identify salient objects from clutter, even generating discriminative SOD models in very cluttered environments. This task aims to detect salient objects from RGB and depth information.

[0003] Given the significant differences in the distributions of RGB and depth data, existing RGB-D SOD algorithms typically focus on exploring effective fusion strategies to model complementary information patterns between the two. These fusion strategies can be categorized into early fusion, mid-fusion, and late fusion. While these strategies improve the performance of saliency models, several issues remain. First, early fusion strategies assemble RGB-D images and then input them into a feature extractor. However, RGB and depth images contain asynchronous information. A simple concatenation operation will eliminate the unique features offered by the two modes. Furthermore, feature extractors (such as VGG, ResNet, etc.) are often pre-trained on RGB-based benchmarks, which are insufficient to learn appearance and geometric features from the merged RGB-D data. Second, mid-level fusion strategies are the most important operation for integrating cross-modal features from RGB and depth images. However, most existing algorithms are designed and use the same fusion operation at different network scales, neglecting the unique properties of features across multiple scales, such as appearance and geometric information at shallow scales, structural cues at mid-scales, and high-level semantic features at deeper scales. Although some researchers have proposed algorithmic models that consider the diversity of multi-scale features, the representational power of the fusion module remains poor. Therefore, these methods exhibit limited capabilities, failing to explore differentiated cross-modal features from different layers of the network, resulting in suboptimal performance in the final prediction. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention proposes an RGB-D salient object detection network based on multi-scale adaptive fusion. Based on research into early and mid-stage fusion strategies and cross-modal multi-scale interactions, this invention proposes a two-stream network that adaptively integrates RGB-D features at shallow, mid, and deep scales through a cross-modal fusion encoder. Regarding the first problem, unlike traditional methods that fuse RGB-D images in early 2D space, this invention projects them into 3D space, representing them as point cloud data. By learning the point cloud representation in 3D space, pixel-level relationships are explicitly modeled, further exploring appearance and geometric information. Regarding the second problem, this invention applies features at each scale to the point cloud-based fusion module to fully utilize the complementary information of the two modalities.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] The specific implementation steps of the RGB-D salient target detection method based on multi-scale adaptive fusion are as follows:

[0007] Step S1: Data preprocessing. Based on the input image set (RGB image and depth image), a two-stream network is used to extract multi-scale features of the two modalities respectively;

[0008] Furthermore, step S1 specifically includes:

[0009] Step S11: Combine the input RGB image and the corresponding depth image with VGG-16 as the backbone network and a partial decoder to independently extract multi-level features.

[0010] Step S12: Use the overall attention module to process the feature map and initial saliency map from the optimization layer, accurately identify edge information, and uniformly segment the entire object;

[0011] Step S13: Output a preliminary prediction map of the RGB image and the corresponding depth image;

[0012] Step S2: Employ a scale-adaptive fusion encoder to integrate its hierarchical features based on cross-modal multi-scale attributes;

[0013] Furthermore, step S2 specifically includes:

[0014] Step S21: For shallow-scale features, feature extraction is performed based on the PointNet++ method, and feature representations are learned in the 3D point cloud space;

[0015] Step S22: For mesoscale features, design a spatial contrast refinement module to accurately integrate multimodal features by mining spatial contrast information;

[0016] Step S23: For deep-scale features, utilize depth-aware channel attention to adaptively enhance the semantic cue representation capability in the RGB-D modality;

[0017] Step S3: Fuse multi-scale features to generate a predicted salient target map;

[0018] Furthermore, step S3 specifically includes:

[0019] Step S31: Fuse multi-scale features;

[0020] Step S32: Integrate the fused features with the RGB saliency prediction map and the depth saliency prediction map;

[0021] Step S33: Output the final predicted salient target feature map, using the ground truth image as supervision;

[0022] Step S4: Train the entire algorithm based on the loss function to make the algorithm's prediction results fit the correct results;

[0023] Step S5: Train the RGB-D salient object detection model using the training set, and validate the model being trained using the validation set. The resulting algorithm model is the tool for RGB-D salient object detection.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) This invention studies effective early and mid-term fusion strategies and focuses on designing multi-scale adaptive fusion modules to fully explore complementary information from multiple modalities.

[0026] (2) To fuse multimodal features, this invention inputs features at each scale into a point cloud-based fusion module to fully utilize the complementary information of the two modalities. For shallow-scale features, the PointNet++ method is used to learn point-by-point representations in 3D point cloud space to achieve detailed neighborhood clue mining; for mesoscale features from which structural information is extracted, spatial contrast refinement is adopted to integrate RGB-D features, thereby learning more discriminative representations from RGB-D features and fully distinguishing between the target and the background; to enhance the representational ability of semantic features at the depth scale, a depth-aware channel attention mechanism is selected to associate synchronous features with cross-modal features. Attached Figure Description

[0027] Figure 1 This is an overall structural diagram of the design of this invention;

[0028] Figure 2 This is the specific process of image preprocessing in this invention;

[0029] Figure 3This is the specific process of feature fusion in this invention;

[0030] Figure 4 This is a visual comparison result between the present invention and advanced algorithms in the same field. Specific implementation methods

[0031] The technical solution of the present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0032] The RGB-D salient target detection method based on multi-scale adaptive fusion has the following steps:

[0033] Step S1: As Figure 2 As shown, perform data preprocessing operations. Based on the input image group (RGB image I... c and depth image I d Multi-scale features of the two modalities are extracted using a two-stream network.

[0034] Step S1 specifically involves:

[0035] Step S11: Using VGG-16 as the backbone network combined with a partial decoder, process the input RGB image I. c With the corresponding depth image I d Multi-level features are extracted independently;

[0036] Step S12: Select the overall attention module to process the feature map and initial saliency map from the optimization layer, accurately identify edge information, and uniformly segment the entire object. After image processing, the three nearest neighboring features of the last part of the decoder are taken as shallow-scale, medium-scale, and deep-scale features, respectively denoted as RGB image multi-level features. and multi-level features of depth images

[0037] Step S13: Output the preliminary prediction map S of the RGB image and the corresponding depth image. RGB and S D ;

[0038] Step S2: Employ a scale-adaptive fusion encoder to integrate its hierarchical features based on cross-modal multi-scale attributes;

[0039] Step S2 specifically involves:

[0040] Step S21: Shallow-scale features of RGB and depth images and The input 2D RGB-D image (x,y) is preprocessed to be converted into a 3D point cloud representation (x,y). ′ ,y ′ ,z ′ ):

[0041]

[0042] Among them, I d (x, y) is the depth value at the (x, y) position, f x and f y are focal length parameters, and s is the scale factor of the camera. The above can be initially expressed as the shallow-scale features of the RGB image and the depth image in the channel dimension, denoted as To learn the point-wise representation, the "encoder-decoder" architecture of PointNet++ is used to capture the fine-grained information around the neighborhood at multiple levels. The encoder learns the representation hierarchically through a set of abstraction levels, which in this invention include three parts: a sampling layer, a grouping layer, and a PointNet layer. At each set abstraction level, the sampling layer selects N′ (N′ < N) points as centroids to cover the entire point cloud in the best form within a suitable receptive field. Then, the grouping layer uses the ball query strategy to aggregate the adjacent points of each centroid into a group. Finally, the PointNet layer encodes the features of each group to learn the local pattern. To understand the features of each original point in the point cloud, the decoder uses a set of feature propagation levels, skip connections, and multi-layer perceptrons (MLPs) to propagate the point features output by the encoder to all the original points in a hierarchical manner, and outputs the feature Since the shallow-scale fusion features are in 3D space, in order to seamlessly integrate them with the subsequent features, in this invention, a normalization operation is used to project them into 2D space to alleviate the distribution difference. Therefore, there is the shallow-scale feature f shallow :

[0043]

[0044] Among them, InstanceNorm(·) represents the instantiation layer, and RFB(·) is the receptive field module (Receptive Field Block, RFB);

[0045] Step S22: The medium-scale feature f mid is obtained from the shallow-scale feature f shallow through an encoding operation. This scale processing method introduces a graph neural network (Graph Neural Network, GNN) to extract the structural information of the scene. This network refines the fused multi-modal features based on the spatial contrast relationship with other regions. First, the RGB image features of the corresponding level and the depth image features are connected in the channel dimension through a convolutional layer to obtain f mid :

[0046]

[0047] Here, Concat(·,·) is the join operation. and Represented by θ c and θ d This is a convolution operation with parameters. Then, based on f... mid Construct a fully connected graph G = (V, E), where V represents the set of vertices, E represents the set of edges, and f mid Each pixel in the array is considered as point n i ∈V(i=1,2,…,K), K represents f mid The number of pixels, node n i With n j The edge between can be considered as e ij The weights are defined as the distances between nodes in 3D space. Based on graph G, and combined with the refinement operation of GNN, related pixels with the same salient label tend to cluster together, outputting refined mesoscale features f. middle This scale feature, by mining spatial contrast information and accurately integrating multimodal features, can obtain more accurate prediction results;

[0048] Step S23: For deep-scale features, utilize depth-aware channel attention to adaptively enhance the semantic cue representation capability in the RGB-D modality. First, deep-scale features... After convolution and activation layer operations, a pixel-level attention vector f is generated. d_att Represents spatially important information about depth features; reshapes deep-scale RGB features. The correlation between learning depth and RGB mode is used to extract the attention vector a. c :

[0049]

[0050] Where R(·) represents the reshaping operation. Channel attention a c Highlight important dimensional information in the RGB feature map and output feature map f. deep Representing high-level semantic information:

[0051]

[0052] The tile(·) function tile the channel attention vector into a matrix. For the same size, ⊙ is an element-wise multiplication operation.

[0053] Step S3: As Figure 3 As shown, multi-scale features are fused to generate a predicted salient target map;

[0054] Furthermore, step S3 specifically includes:

[0055] Step S31: Use attention-guided gated fusion units to mine global contextual information, and process the multi-scale features obtained in step S2 through the decoder. Integrate, and obtain f respectively sm f md Further operations on the above results yield f smd :

[0056]

[0057]

[0058]

[0059] Here, Gate(·) is the gate function, represented as a 3×3 convolutional sequence, a batch normalization layer, and an activation function layer. Based on this, the prediction result S can be obtained using a 3×3 convolutional layer. fusion ;

[0060] Step S32: Combine the fused features S fusion Compared with the RGB saliency prediction map S in step S1 RGB , depth saliency prediction map S D The integrated output is S final :

[0061]

[0062] Step S33: Output the final predicted salient target feature map, using the ground truth image as supervision;

[0063] Step S4: Train the entire algorithm based on the loss function to make the algorithm's prediction results fit the correct results;

[0064] Furthermore, step S4 specifically includes:

[0065] Calculate the binary cross-entropy loss function between the predicted image and the ground truth image.

[0066] L K (S K ,G)=GlogS k +(1-G)log(1-S k ),k∈{RGB,D,fusion}

[0067] In the above formula, S K Let G represent the saliency map and G represent the ground truth image. Based on this, the RGB streaming loss L is calculated separately. RGB Depth flow loss L D and the loss of fused data stream L fusion The total loss function L can be calculated. total :

[0068] L total =L RGB +L D +L fusion

[0069] Step S5: Train the RGB-D salient object detection model using the training set. The resulting algorithm model is the tool for RGB-D salient object detection.

[0070] The algorithm described above is the step of an RGB-D salient target detection method based on multi-scale adaptive fusion. Multiple sets of control experiments were conducted on the same dataset to compare efficiency. Figure 4 This document presents a visual comparison of the present invention with other methods in the field. Furthermore, in terms of quantitative performance evaluation in terms of F-measure, S-measure, E-measure, and MAE, the proposed invention achieves satisfactory results on six datasets (DUT-D, NLPR, NJUD, STERE, SIP, and LFSD), outperforming state-of-the-art algorithms, demonstrating the effectiveness of the present invention.

Claims

1. A salient target detection method based on multi-scale adaptive fusion of RGB-D, characterized in that, The steps are as follows: Step S1: Data preprocessing; Based on the input image set, extract multi-scale features of the two modalities using a two-stream network; The image set includes RGB images and depth images; Step S2: Employ a scale-adaptive fusion encoder to integrate its hierarchical features based on cross-modal multi-scale attributes; Step S21: For shallow-scale features, feature extraction is performed based on the PointNet++ method, and feature representations are learned in the 3D point cloud space; Step S22: For mesoscale features, design a spatial contrast refinement module to accurately integrate multimodal features by mining spatial contrast information; Mesoscale features From shallow scale features After encoding, this scaling method introduces a graph neural network to extract the structural information of the scene. This network refines the fused multimodal features based on their spatial contrast with other regions. First, the RGB image features at the corresponding levels are... With depth image features The channel dimension is obtained by connecting through convolutional layers. : ; in, It's a connection operation. and They represent respectively with and Convolution operation for parameters; then based on Construct a fully connected graph ,in, Represents a set of points. Describe the set of edges. Each pixel in the image is considered a point. , express Pixel count, nodes and The edges between can be considered as The weight is defined as the distance between nodes in 3D space; Step S23: For deep-scale features, utilize depth-aware channel attention to adaptively enhance the semantic cue representation capability in the RGB-D modality; Step S3: Fuse multi-scale features to generate a predicted salient target map; Step S4: Train the entire algorithm based on the loss function to make the algorithm's prediction results fit the correct results; Step S5: Train the RGB-D salient object detection model using the training set, and validate the model being trained using the validation set. The resulting algorithm model is the tool for RGB-D salient object detection.

2. The RGB-D salient target detection method based on multi-scale adaptive fusion as described in claim 1, characterized in that, Step S1 specifically includes: Step S11: Combine the input RGB image and the corresponding depth image with VGG-16 as the backbone network and a partial decoder to independently extract multi-level features. Step S12: Use the overall attention module to process the feature map and initial saliency map from the optimization layer, accurately identify edge information, and uniformly segment the entire object; Step S13: Output a preliminary prediction map of the RGB image and the corresponding depth image.

3. The RGB-D salient target detection method based on multi-scale adaptive fusion as described in claim 1 or 2, characterized in that, Step S3 specifically includes: Step S31: Fuse multi-scale features; Step S32: Integrate the fused features with the RGB saliency prediction map and the depth saliency prediction map; Step S33: Output the final predicted salient target feature map, using ground truth images as supervision.

4. The RGB-D salient target detection method based on multi-scale adaptive fusion as described in claim 1, characterized in that, Step S3 specifically includes: Step S31: Fuse multi-scale features; Step S32: Integrate the fused features with the RGB saliency prediction map and the depth saliency prediction map; Step S33: Output the final predicted salient target feature map, using ground truth images as supervision.

Images