RGB-D SOD Method Based on Local Feature Extraction and Semantic Segmentation

Abstract

The RGB-D SOD method based on local feature extraction and semantic segmentation belongs to the technical field of image saliency detection, in order to solve the problem of low precision of the existing saliency extraction method, the method comprises the following steps: step 1, constructing a network model: the whole network is based on the framework design of the encoder-decoder structure model; step 2, preparing a data set: selecting a commonly used RGB-D picture to construct a data set; step 3, training the network model: inputting the data set prepared in step 2 into the network model constructed in step 1 for training; step 4, selecting the minimum loss function value and the optimal evaluation index; step 5, saving the model: solidifying the finally determined model parameters; if the RGB-D image saliency detection operation is performed, the color image and the depth image are directly input into the network to obtain the final saliency map. The present application has higher precision and can be well applied to robot technology, and has higher universality.

Description

Technical Field

[0001] This invention relates to an RGB-D SOD (Salient Object Detection) method based on local feature extraction and semantic segmentation, belonging to the field of image saliency detection technology. Background Technology

[0002] Salient object detection (SOD) aims to mimic biological vision and capture the most salient parts of an image. It is widely used in many downstream tasks, such as object detection, image classification, semantic segmentation, cosine saliency detection, and object tracking. SOD models typically perform two tasks: 1. Detecting the most salient object. 2. Segmenting the accurate region of that object. To improve the effectiveness of these two tasks, intrinsic and extrinsic cues are commonly used in salient detection. Depth maps are one type of extrinsic cue that can improve the effectiveness of salientity models. The advent of depth cameras makes it easier for users to access depth maps of the current scene. Appropriate feature extractors and fusion schemes can be selected to allow depth information to play a better role in salient extraction. Salientity models can utilize edge information, multi-scale aggregation, integrity, and context to extract salient objects. These methods all improve the effectiveness of salientity modeling to varying degrees. However, existing models' utilization of extrinsic cues, especially fusion schemes for color and depth maps and methods for image semantic segmentation, is too coarse, and the results remain unsatisfactory.

[0003] BBS-Net is an internationally recognized excellent saliency detection model. See the journal "Cham: Springer International Publishing, 2020: 275-292", where Deng-Ping Fan et al. published a paper titled "BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network". This method first uses channel / spatial attention to fuse color and depth maps of images in the dataset; then, it utilizes a depth enhancement module (DEM) to extract depth cue information from the channel and spatial views; next, it adjusts the model parameters through a loss function to obtain the optimal model; finally, it saves the model and evaluates its performance. Although this method is fast, the accuracy of the salient objects it obtains is relatively low, which cannot meet the needs of current robotic applications. Summary of the Invention

[0004] To address the low accuracy of existing saliency extraction methods, this invention provides an RGB-D SOD method based on local feature extraction and semantic segmentation. By applying local feature extraction, the method improves performance in multimodal and multi-scale fusion of color and depth images. Furthermore, by applying image semantic segmentation, it enhances saliency feature enhancement, resulting in higher accuracy of the final salient object identification, which is more consistent with biological visual perception methods. Moreover, the proposed method offers higher accuracy than existing methods, making it well-suited for robotics applications and more versatile.

[0005] The solution to the technical problem of this invention is:

[0006] The RGB-D SOD method based on local feature extraction and semantic segmentation includes the following steps:

[0007] Step 1, Building the Network Model: The entire network is designed based on an encoder-decoder architecture. The image preprocessing module of the encoder part loads a pre-trained weighted backbone network of dual PVTv2 (Pyramid Vision Transformer v2) to extract basic features from the color and depth maps of the input multimodal image, respectively. The entire fusion is performed on four scales (96x96, 48x48, 24x24, 12x12). For the depth map, a ConvNeXt (Convolutional Next Xtreme) module is added for local feature processing, while the color map is not processed by the ConvNeXt module. Then, the depth map features processed by PVTv2+ConvNeXt are fed into the CSA (Channel and SpatialAttention) module, and the result is added to the color map features processed by PVTv2. Finally, the added result is fed into the ConvNeXt module for local feature extraction, and then convolved by the convolution module to complete the entire RGB-D fusion.

[0008] Step 2, Prepare the dataset: Select commonly used RGB-D images to construct the dataset;

[0009] Step 3, train the network model: input the dataset prepared in step 2 into the network model built in step 1 for training;

[0010] Step 4, Select the minimum loss function value and the optimal evaluation metric: Minimize the loss function between the network output image and the label until the number of training iterations reaches a set threshold and the loss value tends to converge. This indicates that the model parameters have been pre-trained and the model parameters are saved. At the same time, select the optimal evaluation metric to measure the accuracy of the algorithm and evaluate the performance of the system.

[0011] Step 5, Save the model: Solidify the finalized model parameters; if performing RGB-D image saliency detection, simply input the color image and depth image into the network to obtain the final saliency map.

[0012] The decoder section includes: an MFE (Multi-feature Extraction) module, an SSE (Semantic Segmentation Enhancement) module, and an LGE (Local / Global Agreement and Edge Detection) module;

[0013] The MFE module employs three different types and sizes of convolution kernels: ordinary convolution, cross convolution, and dynamic convolution.

[0014] The SSE module enhances the saliency of RGB-D features through semantic segmentation using DeepLabv3+, which employs spatial pyramid pooling optimized by dilated convolution and an Encoder-Decoder architecture. Specifically, it uses an Up-Down-Sampling (UDS) module to integrate the relationship between upper and lower scales; and BN is added before ReLU. Then, it uses DeepLabv3+ modules with three different dilation rates (3, 6, and 9) and sums them to enhance the saliency features through image segmentation.

[0015] The LGE module performs saliency detection through interactions across four scales, guided by local / global consistency checks and edge detection. A capsule network is used to handle the local and global consistency relationships of salient objects. Additionally, a Double Fusion module based on edge detection is introduced to improve the performance of the LGE module. The pose vector output from the capsule network is then concatenated with the edge information processed by Double Fusion to obtain the final output pose vector. In the Double Fusion module, in addition to convolution and pixel shuffle operations, a Smish activation function is designed, which effectively improves the optimization effect during training without Batch Normalization (BN) by using convolution, sigmoid, ln, and tanh operations. Finally, the output pose vector of the capsule network and the output of Double Fusion are concatenated to obtain the output of the LGE module; the output of the LGE module is the final output of the entire network.

[0016] Step 2 involves collecting the same RGB-D saliency dataset based on the comparison network BBS-Net, using NJU2K, NLPR, STERE, LFSD, SSD, and SIP; conducting supervised training on the images in the dataset; performing image augmentation on the images in the dataset, normalizing, randomly cropping, and randomly flipping the same set of images as input to the entire network, and using the labeled images in the dataset as labels.

[0017] In step 3, the RGB-D saliency dataset used in the pre-training process is NJU2K, NLPR, STERE, LFSD, SSD, and SIP. Supervised training is performed on the images in the dataset. Image augmentation is performed on the images in the dataset by normalizing, randomly cropping, and randomly flipping the same set of images, including color images, depth images, and label images, and adjusting them to a uniform size of 384×384, which are then used as the input to the entire network. The labeled images in the dataset are used as saliency labels.

[0018] In step 4, the optimal evaluation index is selected as: S α Focus on capturing the structural similarity of images; E ξ This indicates that it integrates local pixel matching and image-level statistics, and obtains the maximum value by changing the threshold; F β Performance is evaluated using precision and recall, and the maximum value can be obtained by changing the threshold. Finally, MAE calculates the average pixel-by-pixel difference between the predicted image and the ground truth. These four metrics can evaluate the accuracy and quality of the algorithm and measure the effectiveness of the segmentation network.

[0019] The beneficial effects of this invention are as follows:

[0020] 1. An image preprocessing module is used as the encoder. With the support of the PVTv2 backbone, the channel and spatial attention (CSA) module and the local feature extraction module are used to process RGB and Depth in different ways to achieve better cross-modal fusion. Absolute position encoding (APE) is also added to embed more information about the relationship between image patches.

[0021] 2. In the decoder section, a Semantic Segmentation Enhancement (SSE) module is introduced. The SSE module combines spatial pyramid pooling and an encoder-decoder structure to integrate semantic information at different scales, enhancing complex semantic segmentation tasks. Furthermore, the SSE module utilizes natural convolution to optimize the boundary recognition capabilities of the spatial pyramid pooling.

[0022] 3. Comparisons with BBS-Net were performed on six challenging datasets, and saliency maps generated using these datasets are presented. Experimental results show that the proposed method achieves higher accuracy than other state-of-the-art techniques. The method of this invention demonstrates superior performance in multi-scale, multi-modal fusion RGB-D saliency detection tasks. Attached Figure Description

[0023] Figure 1 This is a flowchart of the RGB-D SOD method based on local feature extraction and semantic segmentation of the present invention.

[0024] Figure 2 This is a network structure diagram of the RGB-D SOD method based on local feature extraction and semantic segmentation of this invention.

[0025] Figure 3 This is a network structure diagram of the Multiple Feature Extraction (MFE) module described in this invention.

[0026] Figure 4 This is a network structure diagram of the semantic segmentation enhancement (SSE) module based on dilated convolution optimization and spatial pyramid pooling as described in this invention.

[0027] Figure 5 This is a network structure diagram of the Local / Global Consistency Check and Edge Detection (LGE) module described in this invention. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the RGB-D SOD method based on local feature extraction and semantic segmentation specifically includes the following steps:

[0030] Step 1, construct the network model. For example... Figure 2 As shown, the entire network is designed based on an encoder-decoder architecture. The image preprocessing module of the encoder part loads a pre-trained weighted backbone network with dual PVTv2 architectures to extract basic features from the color and depth maps of the input multimodal image. The entire fusion process is performed at four scales. Adding a ConvNeXt module for local feature processing of the depth map improves the RGB-D fusion accuracy. The color map, however, is not processed by the ConvNeXt module. Then, the depth map features processed by PVTv2+ConvNeXt are fed into the CSA module, and the result is added to the color map features processed by PVTv2. Finally, the added result is fed into the ConvNeXt module for local feature extraction, and then a convolutional module is used for convolution to complete the RGB-D fusion.

[0031] The decoder section includes: MFE module, SSE module, and LGE module.

[0032] The decoder section includes, in order: 1. First, such as... Figure 3 As shown, the MFE module employs three different types and sizes of convolutional kernels to extract diverse features and enrich feature information at multiple levels. The three different convolutions are: ordinary convolution, cross convolution, and dynamic convolution. 2. Then, as... Figure 4 As shown, the SSE module, based on spatial pyramid pooling optimized by dilated convolution and the DeepLabv3+ Encoder-Decoder architecture, enhances the saliency of RGB-D features through semantic segmentation. Specifically, an upsampling / downsampling (UDS) module is used to integrate the relationship between upper and lower scales. Furthermore, BN is added before ReLU to alleviate the optimization difficulty caused by channel transformation. Next, DeepLabv3+ modules with three different dilation rates (3, 6, and 9) are used and summed to enhance saliency features through image segmentation. Finally, as... Figure 5 As shown, LGE completes saliency detection through interactions across four scales, guided by local / global consistency checks and edge detection. A capsule network is used to handle the local and global consistency relationships of salient objects. Additionally, a Double Fusion module based on edge detection is introduced to improve the performance of the LGE module. Then, the pose vector output by the capsule network is concatenated with the edge information processed by Double Fusion to obtain the final output pose vector. In the Double Fusion module, in addition to convolution and pixel shuffle operations, a Smish activation function is designed, which effectively improves the optimization performance during training without Batch Normalization (BN) by using convolution, sigmoid, ln, and tanh operations. Finally, the output pose of the capsule network and the output of Double Fusion are concatenated as the output of the LGE module. The output of the LGE module is the final output of the entire network.

[0033] Step 2, Prepare the dataset. Collect the same RGB-D saliency dataset as the comparison network BBS-Net, using NJU2K, NLPR, STERE, LFSD, SSD, and SIP. Supervised training is performed on the images in the dataset; image augmentation is performed on the images by normalizing, randomly cropping, and randomly flipping the same set of images, using these as input to the entire network, and using the labeled images from the dataset as the labels.

[0034] Step 3, train the network model. Input the dataset prepared in Step 2 into the network model built in Step 1 for training.

[0035] Step 4: Select the minimum loss function value and the optimal evaluation metric: Minimize the loss function between the network's output image and the label until the training iterations reach a set threshold and the loss value converges. This indicates the model parameters are pre-trained and saved. Simultaneously, select the optimal evaluation metric to measure the algorithm's accuracy and evaluate the system's performance. The choice of loss function significantly impacts the model's quality, accurately reflecting the difference between predicted and true values ​​and providing correct feedback on model quality. Optimal evaluation metric selection: S α Focus on capturing the structural similarity of images; E ξ This indicates that it integrates local pixel matching and image-level statistics, and we can obtain the maximum value by changing the threshold; F β Performance is evaluated using precision and recall, and we can also obtain the maximum value by changing the threshold; finally, MAE calculates the average pixel-by-pixel difference between the predicted image and the ground truth. These four metrics can evaluate the accuracy and quality of the algorithm and measure the effectiveness of the segmentation network.

[0036] Step 5, save the model. The finalized model parameters are then saved.

[0037] When performing saliency detection on RGB-D images, the final saliency objects can be obtained by directly inputting the color image and depth image into the network.

[0038] Example:

[0039] In step 1, within the decoder section: 1. In the MFE module, to dynamically adjust the weight of each pixel when extracting local features through convolution, this invention proposes a dynamic convolution method. Dynamic convolution enhances the perception of local features by introducing an attention mechanism, enabling more flexible determination of the importance of different pixels in the image during the convolution process. Dynamic convolution makes basic feature extraction more robust, and experiments have shown that it is helpful for enriching aggregated features in saliency detection. 2. In the SSE module, to mine information between multiple scales, an up-down-sampling (UDS) module is used to integrate the relationship between upper and lower scales. For the current level, upsampling is performed on the next level, and downsampling is performed on the previous level, so that the three adjacent levels can reach the same image size. If there is no previous / next level, the current level is used as the result of up / downsampling. Then, the features of the three levels are concatenated and output. A series of operations are used to enhance the output features of the SSE module. Through a convolution, the number of channels added to the output layer of the UDS module is restored, and BN is added before ReLU to alleviate the optimization difficulty caused by channel transformation. Next, we used DeepLabv3+ modules with three different dilation rates (3, 6, and 9) and summed them to enhance salient features through image segmentation. Then, we completed all module operations through convolution, batch normalization (BN), and another convolution. After the SSE enhancement module, the edges of the overall salient objects became clearer and more defined. 3. The LGE module utilizes a Double Fusion module to extract image edge information, thereby enhancing the pose of the capsule network output. In the Double Fusion module, in addition to convolution and pixel shuffle operations, a Smish activation function was designed. Through +1, sigmoid, ln, and tanh operations, it effectively improved the optimization effect during training when BN was not present. 4. Finally, the network output consists of 5 terms. in One of the outputs of the SSE module. f lge _ pose This is the output of the LGE module. The loss function is calculated separately for each of these five items, and then the five loss values ​​are summed to obtain a total loss value.

[0040] In step 2, to achieve a fully comparable comparison with BBS-Net, the RGB-D image saliency detection model was evaluated on six challenging datasets: NJU2K, NLPR, STERE, LFSD, SSD, and SIP. This invention selected 1485 images from NJU2K and 700 images from NLPR as the training dataset, using the remaining images from these two datasets as the test dataset during training. Evaluation was conducted using the full NJU2K, NLPR, STERE, LFSD, SSD, and SIP datasets as the test datasets. NJU2K contains 1985 images collected from the internet, movies, and a Fujifilm W3 camera; the remaining 500 images, besides those in the training set, were used for testing. NLPR consists of 1000 RGB images and depth maps; the remaining 300 images, besides those in the training set, were used for testing. STERE collected 1000 stereo images using 100 indoor and outdoor scenes collected by a Lytro light field camera. The SSD uses 80 samples constructed from three stereoscopic movies. The SIP contains 1,000 high-resolution, realistic photos in different poses.

[0041] In step 3, image enhancement is performed on the images in the dataset. Random diffraction transformation is applied to the same group of images, and the same group of images (color images, depth images, and label images) are normalized, randomly cropped, and randomly flipped, then adjusted to a uniform 384×384 size as input to the entire network. Normalization, random cropping, and random flipping can be implemented using dataset processing software algorithms. Then, using labeled images from the dataset as labels allows the network to learn better feature extraction capabilities, ultimately achieving better segmentation results.

[0042] In step 4, the network output and labels are used to calculate a loss function. Minimizing this loss function achieves better fusion results. Binary cross-entropy (BCE) and intersection-over-union (IoU) loss functions are used as supervision strategies. The final loss function is L. lfss L bce Let L be the BCE loss function. iou Let IoU be the loss function. The final loss function formula is:

[0043] L lfss =L bce +L iou

[0044] The BCE loss function is defined by H and W, where H represents the height and width of the image, and W represents the ground truth and predicted location values, respectively. The calculation formula is as follows:

[0045]

[0046] The formula for calculating the IoU loss function is:

[0047]

[0048] The entire network operation yields five features as the final output of the model. The final prediction under the constraint of the loss function serves as the supervision strategy. The sum of these five losses is the final loss.

[0049] This invention ran all experimental code on a PC with an Intel i7-12700K 3.6GHz CPU, an Nvidia GeForce RTX 3090 (24GB VRAM), and 32GB 3000MHz RAM. Python 3.7.2 for Windows 11 and torch 1.13.0 + cu117 were used as the deep learning platform. During training / testing, the PVTv2 backbone scaled the input images to 384×384. Images were normalized, randomly cropped, and randomly flipped. The SGD optimizer parameters were initialized to lr = 0.05, momentum = 0.9, weight_decay = 1e-4, and nesterov = True, while the batch size was set to batch_size = 4 and epoch = 200. Finally, APEX 0.1 was used to accelerate training.

[0050] After the network training is completed in step 5, all parameters of the network need to be saved. Then, the color image and depth image of the target to be extracted are input into the network to obtain the saliency map. This network has no requirements on the size of the input image; any size is acceptable.

[0051] Among them, the implementations of convolution, Transformer (PVTv2), activation functions, matrix concatenation, matrix multiplication, batch normalization (BN), residual networks, ConvNeXt, DeepLabv3+, etc. are algorithms well known to those skilled in the art, and the specific processes and methods can be found in the corresponding textbooks or technical documents.

[0052] This invention constructs an RGB-D salient object detection network based on local feature extraction and semantic segmentation, which can directly generate salient maps from images. This enables higher-precision salient object extraction in robot vision, providing excellent preprocessing for downstream tasks. Under the same conditions, the feasibility and superiority of this proposed method are further verified by calculating the image correlation metrics obtained by existing methods. A comparison of the correlation metrics between existing technologies and the proposed method is shown in Table 1.

[0053] Table 1 compares the prior art and the method proposed in this invention across 6 datasets using 4 metrics ("↑" indicates that the higher the metric, the better; "↓" indicates that the lower the metric, the better; bold indicates the best accuracy).

[0054]

[0055] As can be seen from the table, the method proposed in this invention has a higher accuracy rate than existing methods in the four basic indicators (S). α E ξ F β The method proposed in this invention surpasses the original method in terms of both saliency extraction (MAE) and other metrics, which further demonstrates that the method has better saliency extraction accuracy.

Claims

1. An RGB-D SOD method based on local feature extraction and semantic segmentation, characterized by: The method includes the following steps: Step 1, Building the Network Model: The entire network is designed based on an Encoder-Decoder structure model. The image preprocessing module of the encoder part loads a pre-trained weighted backbone network with dual PVTv2, which performs basic feature extraction on the color image and depth image of the input multimodal image, respectively. The entire fusion is performed on four scales. For the depth image, a ConvNeXt module is added for local feature processing, while the color image is not processed by the ConvNeXt module. Then, the depth image features processed by PVTv2+ConvNeXt are fed into the CSA module, and the result is added to the color image features processed by PVTv2. Finally, the added result is fed into the ConvNeXt module for one local feature extraction, and then passed through the convolution module for one convolution to complete the entire RGB-D fusion. Step 2, Prepare the dataset: Select commonly used RGB-D images to construct the dataset; Step 3, train the network model: input the dataset prepared in step 2 into the network model built in step 1 for training; Step 4, Select the minimum loss function value and the optimal evaluation metric: Minimize the loss function between the network output image and the label until the number of training iterations reaches a set threshold and the loss value tends to converge. This indicates that the model parameters have been pre-trained and the model parameters are saved. At the same time, select the optimal evaluation metric to measure the accuracy of the algorithm and evaluate the performance of the system. Step 5, Save the model: Solidify the finalized model parameters; if performing RGB-D image saliency detection, simply input the color image and depth image into the network to obtain the final saliency map; The decoder section includes: MFE module, SSE module, and LGE module; The MFE module employs three different types and sizes of convolution kernels: ordinary convolution, cross convolution, and dynamic convolution. The SSE module enhances the saliency of RGB-D features through semantic segmentation using DeepLabv3+ with spatial pyramid pooling optimized by dilated convolution and an Encoder-Decoder architecture. Specifically, it employs an upsampling / downsampling UDS module to integrate the relationship between upper and lower scales, and adds BN before ReLU. Then, it uses DeepLabv3+ modules with three different dilation rates (3, 6, and 9) and sums them to enhance the saliency features through image segmentation. The LGE module performs saliency detection through interactions across four scales, guided by local / global consistency checks and edge detection. A capsule network is used to handle the local and global consistency relationships of salient objects. Additionally, a Double Fusion module based on edge detection is introduced to improve the performance of the LGE module. The pose vector output from the capsule network is then concatenated with the edge information processed by Double Fusion to obtain the final output pose vector. In the Double Fusion module, in addition to convolution and pixel shuffle operations, a Smish activation function is designed, which effectively improves the optimization effect during training without Batch Normalization (BN) by using convolution, sigmoid, ln, and tanh operations. Finally, the output pose vector of the capsule network and the output of Double Fusion are concatenated to obtain the output of the LGE module; the output of the LGE module is the final output of the entire network.

2. The RGB-D SOD method based on local feature extraction and semantic segmentation according to claim 1, characterized in that, Step 2 involves collecting the same RGB-D saliency dataset based on the comparison network BBS-Net, using NJU2K, NLPR, STERE, LFSD, SSD, and SIP; conducting supervised training on the images in the dataset; performing image augmentation on the images in the dataset, normalizing, randomly cropping, and randomly flipping the same set of images as input to the entire network, and using the labeled images in the dataset as labels.

3. The RGB-D SOD method based on local feature extraction and semantic segmentation according to claim 1, characterized in that, In step 3, the RGB-D saliency dataset used in the pre-training process is NJU2K, NLPR, STERE, LFSD, SSD, and SIP. Supervised training is performed on the images in the dataset. Image augmentation is performed on the images in the dataset by normalizing, randomly cropping, and randomly flipping the same set of images, including color images, depth images, and label images, and adjusting them to a uniform size of 384×384, which are then used as the input to the entire network. The labeled images in the dataset are used as saliency labels.

4. The RGB-D SOD method based on local feature extraction and semantic segmentation according to claim 1, characterized in that, In step 4, the optimal evaluation index is selected as: S α Focus on capturing the structural similarity of images; E ξ This indicates that it integrates local pixel matching and image-level statistics, and obtains the maximum value by changing the threshold; F β Performance is evaluated using precision and recall, and the maximum value can be obtained by changing the threshold. Finally, MAE calculates the average pixel-by-pixel difference between the predicted image and the ground truth. These four metrics can evaluate the accuracy and quality of the algorithm and measure the effectiveness of the segmentation network.

Images