A saliency detection method based on RGB-D multi-channel information fusion

Abstract

The present invention discloses a saliency detection method based on RGB-D multi-channel information fusion, comprising the following steps: obtaining a training model data set, data preprocessing, network model training, encoder feature extraction, cross-modal feature fusion, decoder decoding, and generating a saliency detection result. Based on the general RGB-D saliency detection, the present invention adds an RGB-D fusion module to extract the connection features between RGB-D, optimizes the ordinary decoder, adopts a partial cascade method to decode and combine color image features, depth image features and saliency image features to generate a saliency prediction map, and improves the traditional SOD. The RGB-D fusion module can effectively link depth features with RGB features, enhance the separation features of blurred foreground and background boundaries, and the partial cascade decoder can effectively remove interference information in the features extracted by the last three layers of the encoder network, making the saliency map clearer.

Description

Technical Field

[0001] The present invention relates to the technical field of saliency detection, and in particular to a saliency detection method based on RGB-D multi-channel information fusion. Background Art

[0002] When the human eye sees things, it first captures and memorizes everything in front of it, and then transmits the information to the brain for understanding. Saliency detection studies human perception and processing of complex environmental information. The scene includes two important components: background and objects.

[0003] Saliency detection is based on the fact that scene images contain both background and objects. Saliency detection aims to separate the background from the objects, pay more attention to the objects, and ignore the background, so as to quickly locate the important areas in the image scene.

[0004] Since the 20th century, through research in fields such as eye movements, psychology, and neuroscience, the academic community's understanding of human vision has gradually matured. Recognizing scenes with the naked eye is an innate talent given to humans by nature, and there is no need for deliberate practice. However, using computers to extract important areas of a scene is an extremely difficult challenge. Therefore, based on people's research on human vision, the scientific community began to use computers to simulate algorithms and models of vision and brain mechanisms.

[0005] With the rapid development of deep learning, deep learning-based saliency detection methods are gradually replacing traditional methods based on biological models and becoming a major research direction. This paper summarizes the basic ideas of deep learning-based saliency detection methods and categorizes them into the following three categories: non-end-to-end deep saliency detection algorithms, end-to-end deep saliency detection algorithms, and GAN-based deep saliency detection algorithms.

[0006] Over the past few years, the increasing popularity of 3D sensors (such as lidar, depth-sensing cameras, and radar) has spurred the need for saliency detection capable of processing the data captured by these devices. The field of computer vision has recently made significant progress in 3D saliency detection, and its application scenarios have become more extensive. It can be used in computer vision tasks such as object detection and recognition, key point localization, visual tracking, semantic segmentation, image thumbnail generation, and image retargeting. Combining saliency detection with prior knowledge in related fields can help quickly detect target objects in practical application scenarios, thereby facilitating subsequent processing. For example, in autonomous driving perception systems, saliency detection algorithms can help quickly detect targets such as pedestrians, vehicles, and traffic signs in traffic scenes, providing autonomous vehicles with information about the external environment and road conditions.

[0007] However, existing saliency detection methods suffer from blurred edges and unclear segmentation. Furthermore, due to large variations in scene luminosity, cluttered objects, and irregular layouts, foreground-background separation is difficult, resulting in inefficient and inaccurate saliency detection results. Therefore, this paper proposes a saliency detection method based on RGB-D multi-channel information fusion to address these problems in the existing technology. Summary of the Invention

[0008] In response to the above problems, the purpose of the present invention is to propose a saliency detection method based on RGB-D multi-channel information fusion to solve the problem that the saliency detection results of existing saliency detection methods are not efficient and accurate enough.

[0009] To achieve the purpose of the present invention, the present invention is implemented by the following technical solution: a saliency detection method based on RGB-D multi-channel information fusion, comprising the following steps:

[0010] Step 1: Obtain a training model dataset and perform hierarchical processing on the dataset so that the dataset contains paired color images and depth images and their corresponding saliency images;

[0011] Step 2: Perform data augmentation on the dataset by preprocessing the images in the dataset by random horizontal movement, random rotation, and scaling.

[0012] Step 3: Input the color modal image and depth modal image in the data augmentation dataset into the network at the same time and perform end-to-end training on the network;

[0013] Step 4: Extract the features of the color modal image through the encoder network to obtain the color modal image features, and extract the features of the depth modal image through the encoder network to obtain the depth modal image features;

[0014] Step 5: Select the last two layers of output features extracted by the encoder network from the color modal image features and their corresponding depth modal image features, input them into the RGB-D fusion network and splice them together to achieve cross-modal feature fusion of the network and obtain the RGB-D fusion network features;

[0015] Step 6: The extracted color modal map features, depth modal map features, and RGB-D fusion network features are input into the decoder for decoding to obtain the saliency prediction data under the three features;

[0016] Step 7: Aggregate the saliency prediction data under the three features decoded by the decoder to generate the saliency detection result.

[0017] A further improvement is that in step 1, the dataset obtained is the public dataset COM15k, which contains 15,625 image pairs and provides binary annotations, instances and annotations.

[0018] A further improvement is that in step 1, the data set after hierarchical processing is divided into three levels: simple, easy, and difficult. When training the model, the simple data set is used first, then the easy data set, and finally the difficult data set.

[0019] A further improvement is that in step 2, the images in the data set are pre-processed by random horizontal movement, random rotation and scaling, and then unified into images with a length of 224*width of 224*channel number 3.

[0020] A further improvement is that in step 4, the two encoder network architectures are the same and do not share weights.

[0021] A further improvement is that in step 5, the specific steps of cross-modal feature fusion are:

[0022] S1, first extract the features of the i-th color modal graph feature network and the features of the i-th deep modality graph feature extraction network Input to the RGB-D fusion network, and then use the global average pooling layer to obtain the global statistical information of the color modal map features and the depth modal map features. Then the two features are sent to the fully connected layer and a softmax activation function is used to obtain the attention vector, where the attention vector extracted from the color modal map feature is recorded as The attention vector of deep modality graph feature extraction is recorded as Then, the attention vector is applied to the input features in a channel-by-channel multiplication manner to obtain

[0023] S2, will and Aggregate through a maximization function and save important information, then normalize the output to 0-1 through a standard operation N(.) to obtain the attention vector of the fusion of color modality map features and depth modality map features, recorded as Expressed as

[0024] S3, finally the fused attention vector Multiply the input feature extraction network channel by channel, and then and Sum them separately and get The two summed feature vectors are concatenated and input into a 1*1 convolution layer to generate the final RGB-D fusion feature.

[0025] A further improvement is that in step six, the decoder adopts a cascaded partial decoder and only focuses on the high-level semantic information of the last three layers after decoding, aggregates and decodes the features extracted by the last three layers of the network to obtain a primary feature map.

[0026] The beneficial effects of the present invention are as follows: on the basis of general RGB-D saliency detection, the present invention adds an RGB-D fusion module to extract the connection features between RGB-D, and optimizes the ordinary decoder. In a partial cascade manner, the color image features, depth image features and saliency image features are finally decoded and combined to generate a saliency prediction map, thereby improving the traditional SOD. The RGB-D fusion module of the present invention can effectively link the depth features with the RGB features, and enhance the separation features of the blurred foreground and background boundaries. The partial cascade decoder of the present invention can effectively remove the interference information in the features extracted by the last three layers of the encoder network, making the obtained saliency map clearer. For saliency detection, the present invention has good practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 It is a schematic flow chart of the method of the present invention;

[0029] Figure 2 are the RGB, depth, and GT images in the embodiment of the present invention;

[0030] Figure 3 This is a structural diagram of an encoder in an embodiment of the present invention;

[0031] Figure 4 4 is a structural diagram of a cascaded partial decoder in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] See also Figure 1、 Figure 2 、 Figure 3 、 Figure 4 This embodiment provides a saliency detection method based on RGB-D multi-channel information fusion, including the following steps:

[0034] Step 1: Obtain the training model dataset and perform hierarchical processing on the obtained dataset, such as Figure 2 As shown, the dataset contains paired color images (RGB images) and depth images (depth images) and their corresponding saliency (SOD) images (GT images). The obtained dataset is the public dataset COM15k. The public dataset COME15K contains 15,625 image pairs and provides binary annotations, instances, and annotations. The dataset after hierarchical processing is divided into three levels: simple, easy, and difficult. When training the model, the simple dataset is used first, then the easy dataset, and finally the difficult dataset.

[0035] Step 2: The images in the acquired dataset are preprocessed by random horizontal translation (t), random rotation (r), and scaling (s) to unify them into images with a length of 224*width of 224*channel number 3. This is used to enhance the dataset and improve the generalization ability of the model. The changes in the images caused by preprocessing can be summarized as T = {t, r, s}.

[0036] Step 3: During the network training process, the processed data set is used to operate the image through data preprocessing to obtain the data required for training and verification, which is expressed as R rgbd (R rgb +R depth ), the color mode map (R rgb ) and deep modality graph (R depth ) Input the network at the same time and train the network end-to-end;

[0037] Step 4: Extract the features of the color modal image through the encoder network to obtain the color modal image features, and extract the features of the depth modal image through the encoder network to obtain the depth modal image features. The two encoder network architectures are the same and do not share weights. The encoder network model in this embodiment adopts a similar model to vgg16. In order to adapt to the target extraction of the task in this embodiment, the encoder structure diagram is as follows: Figure 3 As shown;

[0038] Step 5: Select the last two layers of output features extracted by the encoder network from the color modal image features and their corresponding depth modal image features, input them into the RGB-D fusion network and splice them to achieve cross-modal feature fusion of the network and obtain the RGB-D fusion network features. The specific steps of the cross-modal feature fusion are as follows:

[0039] S1, first extract the features of the i-th color modal graph feature network and the features of the i-th deep modality graph feature extraction network Input to the RGB-D fusion network, and then use the global average pooling layer to obtain the global statistical information of the color modal map features and the depth modal map features. Then the two features are sent to the fully connected layer and a softmax activation function is used to obtain the attention vector, where the attention vector extracted from the color modal map feature is recorded as The attention vector of deep modality graph feature extraction is recorded as Then, the attention vector is applied to the input features in a channel-by-channel multiplication manner to obtain

[0040] S2, will and Aggregate through a maximization function and save important information, then normalize the output to 0-1 through a standard operation N(.) to obtain the attention vector of the fusion of color modality map features and depth modality map features, recorded as Expressed as

[0041] S3, finally the fused attention vector Multiply the input feature extraction network channel by channel, and then and Sum them separately and get The two summed feature vectors are concatenated and input into a 1*1 convolution layer to generate the final RGB-D fusion feature.

[0042] Expressed as:

[0043]

[0044] Step 6: The extracted color modal image features, depth modal image features and RGB-D fusion network features are input into the decoder for decoding. The decoder adopts a cascaded partial decoder with the following structure: Figure 4 As shown in the figure, since the amount of feature semantic information extracted by the first few layers of the Vgg16 network is not as large as that of the latter layers, the decoder of this embodiment only focuses on the high-level semantic information of the last three layers. It first aggregates and decodes the features extracted by the last three layers of the network to obtain a primary feature map. The last three layers of the Vgg16 network are represented as follows: The decoding process is expressed as: And get a primary attention map S iThen, a holistic attention module is used to process the obtained primary saliency map to obtain an enhanced attention map S h , and then use the enhanced attention map S h The features extracted by the last three layers (f3, f4, f5) of the vgg16 network are multiplied element by element and are recorded as: Then we get the refined features of the last three layers, expressed as The enhanced attention map can effectively remove the interference information in the features extracted by the last three layers of the vgg16 network, and integrate it by building another decoder. The saliency map we need is generated and recorded as S d : , expressed as: The two decoder g here a and g d We use the same structure;

[0045] Step 7: Aggregate the saliency prediction data under the three features decoded by the decoder to generate the saliency detection result.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A saliency detection method based on RGB-D multi-channel information fusion, characterized in that: The following steps are involved: Step 1: Obtain a training model dataset and perform hierarchical processing on the dataset so that the dataset contains paired color images and depth images and their corresponding saliency images; Step 2: Perform data augmentation on the dataset by preprocessing the images in the dataset by random horizontal movement, random rotation, and scaling. Step 3: Input the color modal image and depth modal image in the data augmentation dataset into the network at the same time and perform end-to-end training on the network; Step 4: Extract the features of the color modal image through the encoder network to obtain the color modal image features, and extract the features of the depth modal image through the encoder network to obtain the depth modal image features; Step 5: Select the last two layers of output features extracted by the encoder network from the color modal image features and their corresponding depth modal image features, input them into the RGB-D fusion network and splice them to achieve cross-modal feature fusion of the network and obtain the RGB-D fusion network features. The specific steps of the cross-modal feature fusion are as follows: S1, first extract the features of the i-th color modal graph feature network and the features of the i-th deep modality graph feature extraction network Input to the RGB-D fusion network, and then use the global average pooling layer to obtain the global statistical information of the color modal map features and the depth modal map features. Then the two features are sent to the fully connected layer and a softmax activation function is used to obtain the attention vector, where the attention vector extracted from the color modal map feature is recorded as The attention vector of deep modality graph feature extraction is recorded as Then, the attention vector is applied to the input features in a channel-by-channel multiplication manner to obtain S2, will and Aggregate through a maximization function and save important information, then normalize the output to 0-1 through a standard operation N(.) to obtain the attention vector of the fusion of color modality map features and depth modality map features, recorded as Expressed as S3, finally the fused attention vector Multiply the input feature extraction network channel by channel, and then and Sum them separately and get The two summed feature vectors are concatenated and input into a 1*1 convolution layer to generate the final RGB-D fusion feature. Step 6: The extracted color modal map features, depth modal map features, and RGB-D fusion network features are input into the decoder for decoding to obtain the saliency prediction data under the three features; Step 7: Aggregate the saliency prediction data under the three features decoded by the decoder to generate the saliency detection result.

2. The saliency detection method based on RGB-D multi-channel information fusion according to claim 1, characterized in that: In the step 1, the dataset obtained is the public dataset COM15k, which contains 15,625 image pairs and provides binary annotations, instances, and annotations.

3. The saliency detection method based on RGB-D multi-channel information fusion according to claim 1, characterized in that: In the step 1, the data set after hierarchical processing is divided into three levels: simple, easy, and difficult. When training the model, the simple data set is used first, then the easy data set is used, and finally the difficult data set is used.

4. The saliency detection method based on RGB-D multi-channel information fusion according to claim 1, characterized in that: In the step 2, the images in the data set are pre-processed by random horizontal movement, random rotation and scaling, and then unified into images with a length of 224*width of 224*channel number 3.

5. The saliency detection method based on RGB-D multi-channel information fusion according to claim 1, characterized in that: In step 4, the two encoder networks have the same architecture and do not share weights.

6. The saliency detection method based on RGB-D multi-channel information fusion according to claim 1, characterized in that: In step six, the decoder uses a cascaded partial decoder and only focuses on the high-level semantic information of the last three layers after decoding, aggregates and decodes the features extracted by the last three layers of the network to obtain a primary feature map.

Images