A cross-modal interactive RGB-D image salient region detection method

Abstract

The method of the present invention discloses a cross-modal interactive RGB-D image salient region detection method. The method of the present invention first extracts RGB features and depth features from the color image and depth image respectively using the VGG16 network; then, in the RGB-D interaction module, the RGB features and depth features are mutually corrected and edge details are enhanced; in the depth weight allocation module, the fusion weights of the processed multiple features are calculated based on the depth features; finally, the fused features are refined in the attention update module. The method of the present invention links RGB with depth, and enhances the target area by supplementing rich boundary information; the weights are calculated in the multi-scale depth weight module, and the degree of influence on the salient area is determined according to the quality of the depth map, thereby eliminating the negative impact of low-quality depth maps and improving the overall performance of the algorithm. The method of the present invention can be directly applied in engineering fields such as 3D image processing, 3D quality assessment, and object recognition.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, specifically the field of stereo image processing technology, and relates to a cross-modal interactive RGB-D image salient area detection method based on an attention mechanism. Background Art

[0002] In the human visual system, the brain cannot simultaneously process and analyze all visual information received from the eyes. Therefore, in daily life, the brain always prioritizes areas of interest based on experience. To enable computers to process visual information like the human eye, researchers must design algorithms similar to the visual attention mechanism to extract the areas of interest in images / videos.

[0003] Currently, researchers have proposed numerous RGB-based models for detecting salient regions in videos and images. Most unsupervised methods detect salient objects by calculating low-level features (such as brightness, position, motion, texture, color, and gradient), while deep learning algorithms predict salient objects by training on high-level features such as contextual semantics. RGB video and image saliency detection algorithms are relatively mature, but the human world is three-dimensional, and relying solely on two-dimensional RGB information to detect salient objects has its limitations. The addition of depth information makes visual saliency prediction more accurate. The depth information unique to stereo vision can supplement the spatial structural information lacking in two-dimensional visual saliency. However, depth maps often introduce some noise or holes during acquisition or calculation, and avoiding these issues in the computation is a worthy issue.

[0004] Existing RGB image saliency detection methods have made great progress, but when it comes to challenging complex scenes, due to the lack of precise spatial constraints, the prediction accuracy of saliency detection is low when the color contrast between the foreground and background is very low or the background is cluttered. Most RGB-D algorithms use depth information as a supplement to RGB information. Traditional fusion features and deep learning algorithms that use contextual semantics have difficulty solving the regional errors and boundary blurring problems caused by depth information. In order to reduce / eliminate the impact of low-quality depth maps on the accuracy of saliency detection. This invention focuses on effectively exploring and establishing the complementarity and collaboration of cross-modal features, while avoiding the negative impact of erroneous depth maps. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the existing technology, focus on effectively exploring and establishing the complementarity and collaboration of cross-modal features, while avoiding the negative impact of erroneous depth maps, and provide a cross-modal interactive RGB-D image salient area detection method based on the attention mechanism.

[0006] The proposed method first extracts RGB features and depth features from the color image and depth image using the VGG16 network. Secondly, in the RGB-D interaction module, RGB features and depth features are mutually corrected and edge details are enhanced. Then, in the depth weight allocation module, the fusion weights of the processed multiple features are calculated based on the depth features. Finally, in the attention update module, the fused features are refined to obtain the final saliency map. The specific steps are as follows:

[0007] Step (1) Extract RGB features and depth features from the color image and depth image using the VGG16 network: This step uses two VGG16-based networks (without the fully connected layers) to learn features from the RGB and depth images respectively. The VGG16 network structure contains 13 convolutional layers, 3 fully connected layers, and 5 pooling layers.

[0008] (1-1) The convolution kernel size of VGG16 is 3×3; the fully connected layer of VGG16 consumes huge resources and occupies a large amount of memory when used, so the training speed is improved by removing the fully connected layer.

[0009] The two datasets are divided into training sets and test sets: the number of training images in NJU-2000 is 1485, and the number of test images is 500; the number of training images in DUT-RGBD is 800, and the number of test images is 400.

[0010] After layer-by-layer processing by VGG16, the number of channels of the input image tensor will gradually double, and the width and height of the image tensor will be halved layer by layer.

[0011] (1-2) Comprehensive prediction error of edge, pixel, and region, total loss function L = L bce +L e +L r Among them, L bce is the binary cross loss, L e To enhance edge loss, L r Predict losses for a region.

[0012] N is the total number of pixels in the input image, and H and W are the height and width of the input image respectively.

[0013] Calculate the error between the true pixel value g and the predicted value p at each position, and then accumulate and average the errors of all positions to obtain the enhanced edge loss Intermediate parameters where ρ(·) represents the average pooling operation with a 5×5 sliding window, P = {p|0<p<1}∈R N×1×H×W , G={g|0<g<1}∈R N ×1×H×W; Get the local area close to the outline of the true value image through e, and calculate the difference L between the predicted value p and the true value g of the pixel in this area e . Among them, the forecast loss Background prediction loss

[0014] Step (2) In the RGB-D interaction module, RGB features and depth features are mutually corrected and edge details are enhanced. Specifically:

[0015] This step discards the first two layers of low-level features extracted from VGG16, and processes the lth layer of RGB features and depth features (l = 3, 4, 5) in three RGB-D interaction modules. The RGB-D interaction module is mainly composed of a cross-channel attention unit and a boundary enhancement unit. The former mainly functions to effectively interact with RGB features and depth features, thereby supplementing RGB information and enhancing salient objects. The latter addresses the edge blurring caused by the acquisition and processing of depth maps.

[0016] (2-1) Cross-channel attention unit processing: First, the depth map (0-1) is divided into m binary markers to find the depth region that is useful for saliency detection. For the i-th marker, if its pixel in the depth map has a corresponding pixel value greater than and less than The pixel value in the identification map is 1, otherwise it is 0; For the RGB module, a rough initial saliency map S supervised by the true value map is first generated pr ; It generates the initial saliency map from the 5th layer of RGB features through the convolution operation Conv(·), that is, in is the 5th layer RGB feature extracted from the VGG16 network.

[0017] The calculation process of deep features supplementing RGB features is as follows: the lth layer RGB features after interactive processing in, is the RGB feature of the lth layer, N is the total number of pixels in the image, d i represents the i-th binary identification map, d i,j represents the pixel value of the jth pixel in the i-th identification image, S j pr Represents the initial saliency map S pr The pixel value of the jth pixel in .

[0018] When encountering a depth map of low quality, the accuracy of saliency detection may be reduced, so we added Further processing was performed: Among them, Conv(·) represents the convolution operation; Cat(·) represents the concatenation operation; CA(·) is the channel attention operation, including an average pooling operation and a maximum pooling operation; S″ is the transformed form of the initial saliency map. When l = 5, S″ = S pr In other cases, S″=1-S pr By S pr The remaining details maintain high-level semantic information, and the network can keep learning more reliable information from RGB features.

[0019] For the processing of deep features, multiple binary identification maps are weighted based on the initial saliency map. is the lth layer deep feature learned from the VGG16 network.

[0020] (2-2) Boundary enhancement unit processing: Design a multi-scale boundary feature extractor, which contains four parallel dilated convolution blocks with different receptive fields (1, 3, 5, 7), and then combine them to obtain significant edge enhancement features and significant features of each level. represents element addition, F b represents the boundary features extracted from the RGB pattern after the multi-scale boundary feature extractor, F l is the depth / RGB feature extracted from the VGG16 network; for F b , the edge truth map generated by the saliency truth map is supervised to ensure the accuracy of boundary features.

[0021] Step (3) The depth weight distribution module calculates the fusion weights of multiple features after processing based on the depth features; specifically:

[0022] After the VGG16 network that extracts deep features, a global pooling layer and a convolution layer are added. The convolution layer has several parallel kernels with different sizes and expansion rates. Processing the fifth layer of deep features Get k multi-scale features with the same resolution but different contexts. In order to obtain the k-th multi-scale deep feature Corresponding depth weight right First perform global average pooling and then convolution, and then use the softmax function to calculate: Among them, AvePooling(·) represents the average pooling operation, and V represents the corresponding convolution parameter matrix (given by The size of the network is determined by *, * represents the convolution operation, and δ(·) represents the softmax function.

[0023] When objects of different sizes appear at different depths, processing multi-scale depth information helps to increase the accuracy of saliency detection and better fuse the enhanced RGB features and depth features. The three-layer enhanced features after the fusion of the two enhanced features in step (2) To enhance the depth boundary features, Enhance the features of RGB boundaries. To enhance the features of deep interaction, is the RGB interaction enhancement feature, w is the weight when calculating the depth weight distribution, Where K is the number of multi-scale deep features, K=6.

[0024] Step (4) refines the fused features in the attention update module. The specific calculation process of the attention update module is as follows:

[0025] The attention mechanism and ConvLSTM network are combined to form an attention update module. This module can infer the current prediction results from the saliency prediction results stored in the previous stage. Through multiple iterations, the spatiotemporal dependencies between salient features at different times are learned. By understanding the scene in the previously stored salient features, the detection details are optimized to improve the accuracy and precision of saliency prediction. t and F s Generated attention map Att(h t ,F s )=δ(AvgPooling(W0×h t +W1×F s )); where the subscript t represents the time step in ConvLSTM, W0 and W1 are h t and F s The weight parameter, h t represents the previously stored memory used to remember the scene, F s is the input feature; Represents element-wise multiplication.

[0026] The features processed by the attention block This is then fed into the ConvLSTM to further update the spatial correlation between different semantic features. The detailed computational process of the ConvLSTM is omitted here as it represents the existing network structure.

[0027] After updating and iteration, the final feature F is obtained M , the final saliency map S is obtained through a 1×1 convolution layer and upsampling operation.

[0028] This method proposes a novel stereo image saliency detection technique in the field of stereo image processing research, simulating the human visual attention mechanism as closely as possible. It improves existing stereo image salient object detection algorithms and fully utilizes the important role of depth information in stereo visual saliency detection.

[0029] The RGB-D interaction module of the present invention links RGB and depth, enhancing the target area by supplementing it with rich boundary information. A multi-scale depth weighting module calculates weights, determining their influence on salient areas based on the quality of the depth map, thereby mitigating the negative impact of low-quality depth maps. Finally, a recurrent attention module comprehensively understands scene information and learns the internal high-level semantic relationships of fused features, improving the overall performance of the algorithm. This stereo image saliency method can be directly applied to engineering fields such as 3D image processing, 3D quality assessment, and object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of the method of the present invention;

[0031] Figure 2 This is the VGG16 network framework diagram;

[0032] Figure 3 This is the architecture diagram of the RGB-D interaction module;

[0033] Figure 4 This is the architecture diagram of the cross-channel attention unit;

[0034] Figure 5 This is the structure diagram of the attention update module;

[0035] Figure 6 It is the result map of salient region detection. DETAILED DESCRIPTION

[0036] A cross-modal interactive RGB-D image salient region detection method, such as Figure 1 :

[0037] Step (1) Extract RGB features and depth features from color images and depth images using the existing VGG16 network: This step uses two VGG16-based networks (without the fully connected layers) to learn features from RGB and depth images respectively. The VGG16 structure contains 13 convolutional layers, 3 fully connected layers, and 5 pooling layers. The specific structure is as follows: Figure 2 .

[0038] (1-1) The convolution kernel size of VGG16 is 3*3, which is generally recognized as having good results. The fully connected layer of VGG16 consumes a lot of resources and occupies a large amount of memory when used, so this step improves the training speed by removing the fully connected layer.

[0039] The two datasets are divided into training sets and test sets: the number of training images in NJU-2000 is 1485, and the number of test images is 500; the number of training images in DUT-RGBD is 800, and the number of test images is 400.

[0040] After layer-by-layer processing by VGG16, the number of channels of the input image tensor will gradually double, and the width and height of the image tensor will be halved layer by layer. Figure 2 Taking a 224*224 image as an example, the specific changes are described.

[0041] (1-2) Comprehensive prediction error of edge, pixel, and region, total loss function L = L bce +L e +L r Among them, L bce = is the binary cross loss, L e To enhance the edge loss, L r Predict losses for a region.

[0042] N is the total number of pixels in the input image, and H and W are the height and width of the input image respectively.

[0043] Calculate the error between the true pixel value g and the predicted value p at each position, and then accumulate and average the errors of all positions to obtain the enhanced edge loss Intermediate parameters where ρ(·) represents the average pooling operation with a 5×5 sliding window, P = {p|0<p<1}∈R N×1×H×W , G={g|0<g<1}∈R N ×1×H×W ; Get the local area close to the outline of the true value image through e, and calculate the difference L between the predicted value p and the true value g of the pixel in this area e . Among them, the forecast loss Background prediction loss

[0044] Step (2) In the RGB-D interaction module, RGB features and depth features are mutually corrected and edge details are enhanced. The specific steps are:

[0045] This step discards the first two layers of low-level features extracted from VGG16, and processes the lth layer RGB features and depth features (l = 3, 4, 5) in three RGB-D interaction modules. The RGB-D interaction module is mainly divided into a cross-channel attention unit and a boundary enhancement unit. The specific structure is shown in Figure 3The main function of the former is the effective interaction between RGB features and depth features, which has achieved the effect of supplementing RGB information and enhancing salient targets; the latter solves the edge blur problem caused by the acquisition and processing of depth maps.

[0046] (2-1) Cross-channel attention unit processing ( Figure 4 ): First, the depth map (0-1) is divided into m binary markers to find the depth region that is useful for saliency detection. For the i-th marker, if its pixel in the depth map has a corresponding pixel value greater than and less than The pixel value in the identification map is 1, otherwise it is 0; For the RGB module, a rough initial saliency map S supervised by the true value map is first generated pr ; It generates the initial saliency map from the 5th layer of RGB features through the convolution operation Conv(·), that is, in is the 5th layer RGB feature extracted from the VGG16 network.

[0047] The calculation process of deep features supplementing RGB features is as follows: the lth layer RGB features after interactive processing in, is the RGB feature of the lth layer, N is the total number of pixels in the image, d i represents the i-th binary identification map, d i,j represents the pixel value of the jth pixel in the i-th identification image, S j pr Represents the initial saliency map S pr The pixel value of the jth pixel in .

[0048] When encountering a depth map of low quality, the accuracy of saliency detection may be reduced, so we added Further processing was performed: Among them, Conv(·) represents the convolution operation; Cat(·) represents the concatenation operation; CA(·) is the channel attention operation, which includes an average pooling operation and a maximum pooling operation; S″ is the transformed form of the initial saliency map. When l = 5, S″ = S pr In other cases, S″=1-S pr By S pr The remaining details maintain high-level semantic information, and the network can keep learning more reliable information from RGB features.

[0049] For the processing of deep features, multiple binary identification maps are weighted based on the initial saliency map. is the lth layer deep feature learned from the VGG16 network.

[0050] (2-2) Boundary enhancement unit processing: Design a multi-scale boundary feature extractor, which contains four parallel dilated convolution blocks with different receptive fields (1, 3, 5, 7), and then combine them to obtain significant edge enhancement features and significant features of each level. represents element addition, F b represents the boundary features extracted from the RGB pattern after the multi-scale boundary feature extractor, F l is the depth / RGB feature extracted from the VGG16 network, for F b , the edge truth map generated by the saliency truth map is supervised to ensure the accuracy of boundary features.

[0051] The specific steps of calculating the fusion weights of multiple features processed according to the depth feature in step (3) of the depth weight allocation module are:

[0052] After the VGG16 network that extracts deep features, a global pooling layer and a convolution layer are added. The convolution layer has several parallel kernels with different sizes and expansion rates. Processing the fifth layer of deep features Get k multi-scale features with the same resolution but different contexts. In order to obtain the k-th multi-scale deep feature Corresponding depth weight right First perform global average pooling and then convolution, and then use the softmax function to calculate: Among them, AvePooling(·) represents the average pooling operation, and V represents the corresponding convolution parameter matrix (given by The size of the network is determined by *, * represents the convolution operation, and δ(·) represents the softmax function.

[0053] When objects of different sizes appear at different depths, processing multi-scale depth information helps to increase the accuracy of saliency detection and better fuse the enhanced RGB features and depth features. The three-layer enhanced features after the fusion of the two enhanced features in step (2) To enhance the depth boundary features, Enhance the features of RGB boundaries. To enhance the features of deep interaction, is the RGB interaction enhancement feature, w is the weight calculated by depth weight distribution, Where K is the number of multi-scale deep features, K=6.

[0054] Step (4) refines the fused features in the attention update module. The specific structure is shown in Figure 5 , the specific calculation process of the attention update module is as follows:

[0055] The attention mechanism and ConvLSTM network are combined to form an attention update module. This module can infer the current prediction results from the saliency prediction results stored in the previous stage. Through multiple iterations, the spatiotemporal dependencies between salient features at different times are learned. By understanding the scene in the previously stored salient features, the detection details are optimized to improve the accuracy and precision of saliency prediction. t and F s Generated attention map Att(h t ,F s )=δ(AvgPooling(W0×h t +W1×F s )); where the subscript t represents the time step in ConvLSTM, W0 and W1 are h t and F s The weight parameters are W0=0.4 and W1=0.6 in this paper, h t represents the previously stored memory used to remember the scene, F s is the input feature; Represents element-wise multiplication.

[0056] The features processed by the attention block This is then fed into the ConvLSTM to further update the spatial correlation between different semantic features. The detailed computational process of the ConvLSTM is omitted here as it represents the existing network structure.

[0057] After updating and iterating, the final feature F is obtained M , the final saliency map S is obtained through a 1×1 convolution layer and upsampling operation.

[0058] Considering the detection results of the present invention in different scenes, three groups of RGB-D images in different scenes were selected for experiments, and the stereo video saliency detection method of the present invention was used for detection. The detection results are shown in the figure. Figure 6 The experimental results show that the proposed method can effectively detect the salient areas of RGB-D stereo images.

[0059] The protection content of the present invention is not limited to the above examples. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.

Claims

1. A cross-modal interactive RGB-D image salient region detection method, characterized by: First, RGB features and depth features are extracted from the color image and depth image using the VGG16 network respectively; Secondly, in the RGB-D interaction module, RGB features and depth features are mutually corrected and edge details are enhanced. Then, in the depth weight allocation module, the fusion weights of multiple processed features are calculated based on the depth features. Finally, in the attention update module, the fused features are refined to obtain the final saliency map. The steps are as follows: Step (1) extracting depth features and RGB features from two improved VGG16 networks respectively, and designing a loss function that comprehensively considers pixel-level, edge, and regional losses; the VGG16 network structure includes 13 convolutional layers, 3 fully connected layers, and 5 pooling layers; (1-1) Remove the fully connected layer; (1-2) Comprehensive prediction error of edge, pixel, and region, total loss function L = L bce +L e +L r Among them, L bce is the binary cross loss, L e To enhance edge loss, L r forecasting losses for the region; N is the total number of pixels in the input image, H and W are the height and width of the input image respectively; Calculate the error between the true pixel value g and the predicted value p at each position, and then accumulate and average the errors of all positions to obtain the enhanced edge loss Intermediate parameters where ρ(·) represents the average pooling operation with a 5×5 sliding window, P = {p|0<p<1}∈R N×1×H×W , G={g|0<g<1}∈R N×1×H×W ; Get the local area close to the outline of the true value image through e, and calculate the difference L between the predicted value p and the true value g of the pixel in this area e ; Forecast loss Background prediction loss Step (2) supplements the RGB features in the RGB-D interaction module, corrects the errors caused by the depth map, and enhances the edge features; discards the first two layers of low-level features extracted from VGG16, and processes the lth layer RGB features and depth features in three RGB-D interaction modules, l = 3, 4, 5, specifically: (2-1) Cross-channel attention unit processing: First, the depth map (0-1) is divided into m binary identification maps to find the depth region useful for saliency detection; for the i-th identification map, if its pixel in the depth map has a corresponding pixel value greater than and less than The pixel value in the identification map is 1, otherwise it is 0; For the RGB module, a rough initial saliency map S supervised by the true value map is first generated pr , is to generate the initial saliency map from the 5th layer of RGB features through the convolution operation Conv(·), that is, in is the 5th layer RGB feature extracted from the VGG16 network; The calculation process of deep features supplementing RGB features is as follows: the lth layer RGB features after interactive processing in, is the RGB feature of the lth layer, N is the total number of pixels in the image, d i represents the i-th binary identification map, d i,j represents the pixel value of the jth pixel in the i-th identification image, S j pr Represents the initial saliency map S pr The pixel value of the jth pixel in ; right Further processing: Among them, Conv(·) represents the convolution operation; Cat(·) represents the concatenation operation; CA(·) is the channel attention operation, including an average pooling operation and a maximum pooling operation; S″ is the transformed form of the initial saliency map, when l = 5, S″ = S pr In other cases, S″=1-S pr ; For the processing of deep features, multiple binary identification maps are weighted based on the initial saliency map. is the lth layer deep feature learned from the VGG16 network; (2-2) Boundary enhancement unit processing: Design a multi-scale boundary feature extractor, which contains four parallel dilated convolution blocks with different receptive fields (1, 3, 5, 7), and then combine them to obtain significant edge enhancement features and significant features of each level. represents element addition, F b represents the boundary features extracted from the RGB pattern after the multi-scale boundary feature extractor, F l is the depth / RGB feature extracted from the VGG16 network; for F b , marginal truth map supervision generated by saliency truth map; In step (3), the depth weight distribution module calculates the fusion weights of multiple features processed according to the depth features, and fuses the two features enhanced in step (2) according to the characteristics of the depth features, specifically: Add a global pooling layer and a convolution layer after the VGG16 network that extracts deep features; process the 5th layer of deep features Get k multi-scale features with the same resolution but different contexts; for the kth multi-scale depth feature Perform global average pooling and then convolution, using the softmax function to calculate Corresponding depth weight Where AvePooling(·) represents the average pooling operation, V represents the corresponding convolution parameter matrix, * represents the convolution operation, and δ(·) represents the softmax function; Three-layer enhanced features after fusion of two enhanced features To enhance the depth boundary features, Enhance the features of RGB boundaries. To enhance the features of deep interaction, is the RGB interaction enhancement feature, w is the weight when calculating the depth weight distribution, Where K is the number of multi-scale deep features, K = 6; Step (4) refines the fused features in the attention update module and combines the attention mechanism and the ConvLSTM network to update the salient features. Specifically: According to h t and F s Generated attention map Att c (h t ,F s )=δ(AvgPooling(W0*h t +W1*F s )); where the subscript t represents the time step in ConvLSTM, W0 and W1 are h t and F s The weight parameter, h t represents the previously stored memory used to remember the scene, F s is the input feature; represents element-wise multiplication; The features processed by the attention block Input ConvLSTM to further update the spatial correlation between different semantic features; After updating and iteration, the final feature F is obtained M ,The final saliency map S is obtained through a 1×1 convolution layer and upsampling operation.

Images