Salient object detection method based on absolute and relative depth induced network
By adopting an absolute and relative depth-induced network method in significance target detection, combining the adaptive fusion of depth and RGB features and spatial geometric information exploration, complex background and noise problems in the prior art are solved, and higher detection is achieved Accuracy and robustness.
Patent Information
- Application Number
- CN202010695446.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-07-17
AI Technical Summary
Existing significant object detection methods have challenges in dealing with complex backgrounds, low brightness environments, and transparent objects, lack of global context information and semantic clues, asynchronous fusion depth and RGB features lead to cluttered noise, and detailed significance information in local areas is not fully explored , limits the improvement of model performance.
Using a method based on absolute and relative depth induction networks, the residual network training is carried out, combined with absolute depth induction module and relative depth induction module to realize cross-modal feature fusion and spatial geometric information exploration, using gated recursive units and graph convolutional networks Adaptive fusion and refinement of depth and RGB features are performed to avoid noise interference and enhance local feature representation.
It improves the accuracy and robustness of significant object detection, accurately locates objects, enhances model performance, reduces messy noise interference, and has broad application prospects.
Smart Images

Figure CN111814895B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of salient target detection, and specifically relates to a salient target detection method based on an absolute and relative depth induction network. Background Art
[0002] Salient object detection is a basic operation in computer image processing. It aims to locate and segment the most visually distinctive objects in an image. In recent years, it has been widely used in various fields, such as relocalization, scene classification, visual tracking, and semantic segmentation. Before performing relevant image processing operations, computers can use saliency detection technology to filter out irrelevant information, thereby greatly reducing the image processing work and improving efficiency.
[0003] Early salient object detection methods mainly designed hand-crafted features (such as brightness, color, and texture) to detect salient objects in images. In recent years, various deep learning-based models have been proposed due to the development of CNN. In 2017, Hou et al. proposed a short connection mechanism between layers and used it to aggregate feature maps from multiple scales. In 2017, Zhang et al. explored multi-level features at each scale and generated saliency maps in a recursive manner. In 2019, Feng et al. proposed an attention feedback module to better explore the structure of salient objects. However, these recently proposed methods have certain challenges in extremely complex situations such as semantically complex backgrounds, low-light environments, and transparent objects. To address this problem, we propose to use depth information to supplement RGB images. In this way, we can explore the spatial structure and 3D geometric information of the scene, thereby improving the effectiveness and robustness of the network.
[0004] The features extracted by traditional RGB-D salient object detection methods lack global context information and semantic clues in the features. In recent years, effective integration methods of depth and RGB features have been a key issue for this task. In 2019, Zhao et al. designed a contrast loss to explore the prior contrast in depth images. Then, an attention map was generated by fusing the refined depth and RGB features. The final saliency map was output by a fluid pyramid integration strategy that fully utilized multi-scale cross-modal features. In 2019, Pial et al. hierarchically integrated depth and RGB images and refined the final saliency map through a recursive attention model. However, the current methods of fusing depth and RGB image feature spaces are asynchronous, which will introduce clutter noise in the network.
[0005] In summary, existing salient object detection techniques have the following defects: First, most existing methods only extract features from RGB images, which are not sufficient to distinguish salient objects from cluttered background areas; Second, most existing methods extract depth and RGB features through separate networks and directly fuse them using different strategies. However, the cross-modal feature space is inconsistent. Directly fusing them will lead to noisy responses in the prediction results; Third, although the absolute depth induction module can accurately locate salient objects, it still does not deeply explore the detailed saliency information of local areas, which also limits the further improvement of model performance. Summary of the invention
[0006] 1. Technical issues to be resolved
[0007] In view of the deficiencies in the prior art, the present invention provides a salient object detection method based on absolute and relative depth induced networks, which solves the problems mentioned in the background technology.
[0008] (II) Technical solution
[0009] To achieve the above object, the present invention provides the following technical solution: a salient object detection method based on absolute and relative depth induction network, comprising the following steps:
[0010] a. Deep induced network training with residual network as the backbone network: remove the last pooling layer and fully connected layer of ResNet-50, adjust the network input image to 256×256, normalize the data set, generate the corresponding side output map through the pyramid method, and then perform fusion operations from top to bottom in the network;
[0011] b. Absolute depth induction module cross-modal feature fusion and object positioning: The depth image of the input image is input into a set of convolutions to obtain a depth feature map with the same size as the Res2_x feature map. The absolute depth induction network is applied multiple times to recursively integrate the depth feature map and the RGB feature map to achieve cross-modal feature fusion, avoiding the noise interference caused by simply fusing two asynchronous modal features, strengthening the deep interaction between depth and color features, and adaptively fusing RGB and depth features at each scale;
[0012] c. The relative depth induction module establishes a spatial geometric model to supplement the detail information: First, the feature map from the last stage of the decoding network Res5_x is upsampled and integrated with the feature map obtained by cross-modal fusion of the absolute depth induction module to generate a new feature map, which is then input into the relative depth induction module together with the depth map generated by the absolute depth induction module to explore the spatial structure and detailed saliency information of the image, and the relative depth information is wrapped in the network to improve the performance of the saliency model;
[0013] Furthermore, when the input network images described in step a are of the same size, we use the bilinear interpolation method to operate on the data set.
[0014] Furthermore, when the side output map is generated in step a, the output feature maps of the four residual blocks are input into a 1*1 convolutional layer to reduce the channel dimension of the feature map, which is the side output map, so as to be used for the subsequent top-down integrated multi-level feature map.
[0015] Furthermore, the depth feature map and RGB2 feature map are integrated together in a recursive manner as described in step b. The absolute depth induction network is implemented by a gated recurrent unit (GRU), which is designed to handle sequence problems. We formulate the multi-scale feature integration process as a sequence problem and regard each scale as a time step.
[0016] Furthermore, at each time step, the depth feature map is first reduced in dimension, and then the depth and RGB feature maps are cascaded and transformed through global maximum pooling to generate a new feature vector, which is then passed through operations such as the fully connected layer to adaptively fuse RGB and depth features at each scale.
[0017] Furthermore, the relative depth induction module described in step c is used to explore the spatial structure and detailed saliency information of the image, and the module uses a graph convolutional network (GCN) to explore relative depth information.
[0018] Furthermore, the proposed graph convolutional network (GCN) projects image pixels into 3D space according to their spatial positions and depth values, making up for the disadvantage that adjacent pixels in 2D space are not strongly associated in 3D point cloud space, and performs information propagation in local areas based on short-range relative depth relationships. By exploring spatial structures and geometric information at multiple scales, the local feature representation capabilities are successively enhanced. In this way, detailed saliency information can be utilized in the relative induction network, which helps to accurately predict the final result.
[0019] (III) Beneficial effects
[0020] Compared with the prior art, the present invention provides a salient object detection method based on absolute and relative depth induction networks, which has the following beneficial effects:
[0021] The present invention not only extracts RGB image features from the residual network, but also proposes to use depth information to assist the salient target detection task. Most existing RGB-D models simply extract depth and RGB features and heuristically fuse them. The absolute depth induction module is used to fuse RGB image features and depth image information across modalities in a coarse-to-fine manner, avoiding the cluttered noise interference caused by the asynchronous characteristics of the two spaces, thereby accurately locating objects; the relative depth induction module is used to establish a spatial graph convolution model to explore spatial structure and geometric information to enhance the local feature representation capability, thereby improving the accuracy and robustness of detection, so that it can achieve excellent detection effects, help integrate with other fields, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is a flowchart of salient target measurement based on absolute and relative depth induction network proposed by the present invention. DETAILED DESCRIPTION
[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] See also Figure 1 The utility model provides a technical solution: a salient target detection method based on absolute and relative depth induction network, comprising the following steps:
[0025] Deep induced network training with residual network as the backbone network: the last pooling layer and fully connected layer of Resnet-50 are removed, the network input image is uniformly adjusted to 256×256, and the data set is normalized. The feature maps generated by the five convolution blocks are generated into the corresponding side output map through the pyramid method. Then the fusion operation is performed from top to bottom in the network.
[0026] Deep induced network training with residual network as the backbone network: The last pooling layer and the fully connected layer of ResNet-50 are removed. The backbone network includes five convolution blocks, Conv1, Res2_x, ... Res5_x. The input RGB image of size W*H is used to generate convolution blocks of size Feature map Shallow layers capture low-level information of the image, such as texture and spatial details, while deep feature maps contain high-level semantic information. We fuse feature maps in a pyramidal manner. Use a 1*1 convolution kernel to reduce the channel to C and get the side output map Then, the multi-level feature maps are integrated in a top-down manner.
[0027]
[0028] Among them, σ(.) is the ReLU activation function, CAT[.,.] is the cascade operation of connecting two feature maps with the same channel dimension, UP(.) is the upsampling operation with bilinear interpolation, and W l , b l are trainable parameters in the network.
[0029] Absolute depth induction module cross-modal feature fusion, localization of objects: First, the input depth image D of size W*H is input into a set of convolutional layers and generates a size The feature map f d Then, the Absolute Depth Induction Module (ADIM) is applied multiple times to recursively integrate the depth feature map with the RGB feature map are integrated to enhance the deep interaction between depth and color features.
[0030]
[0031] in, is the updated deep feature, It is the aggregation result of the depth and RGB information in the lth layer.
[0032] According to the above embodiment, ADIM is preferably implemented by a gated recurrent unit (GRU), which is designed to handle sequence problems. We formulate the multi-scale feature integration process as a sequence problem and regard each scale as a time step. In each time step, we transform the RGB feature is regarded as the input of GRU, and the deep features It can be regarded as the hidden state of the last step. The two feature maps are cascaded and transformed through the global maximum pooling (GMP) operation to generate a feature vector. A fully connected layer is then applied to the feature vector to generate the reset gate r and the update gate z. The values of these two gates are normalized by the S-type function. In fact, gate r controls the integration of depth and RGB features, and z controls In this way, RGB and depth features can be adaptively fused at each scale. The interaction between depth and RGB features is enhanced through the network. The generated multi-scale feature map is then Combined with the feature map in the decoding state, formula (1) is reformulated as:
[0033]
[0034] The relative depth induction module builds a spatial geometric model to supplement the detailed information: The relative depth induction module (RDIM) is used in the decoding stage. First, the feature map from the last stage of the decoding network is Upsampling and feature maps Integrated together, as described in formula (3), the generated feature map is expressed as Then RDIM is applied to the feature map and depth images to embed relative depth information in the network
[0035]
[0036] According to the above embodiment, preferably, RDIM is implemented by a graph convolutional network (GCN). In order to explore the relative depth relationship between pixels, we first convert the feature map generated by ADIM It is represented as G = (V, E), where the node set is V and the edge set is E. We regard each node ni in the graph as a point in the 3D coordinate system and represent the coordinates as (x i ,y i , d i ), where (x i ,y i ) is the spatial position map of the feature And di is the corresponding depth value. The node set is represented as V = {n 1 , n 2 ,…,n k}, k is the number of nodes. We define the edge set e of the 3D coordinate and its adjacent m elements i,j ∈E, calculate edge e i,j The weight w on i,j , as a relative depth value to measure the spatial correlation between nodes ni and nj,
[0037] w i,j =|(x i ,y i , d i )-(x j ,y j , d j )| (5)
[0038] To describe the semantic relationship between nodes ni and nj, we define edge e i,j , defines an attribute feature a i,j In order to further consider the global context information of the image, the feature map Apply GAP to extract high-level semantic information and output the feature vector f g .
[0039] The spatial GCN consists of a set of stacked graph convolutional layers (GCL). For each GCL, we first update the edge e i,j Attribute characteristics of a i,j ,
[0040]
[0041] in, and They are feature maps The position (x i ,y i ) and (x j ,y j ), and use MLP to update the function of each node.
[0042]
[0043] Where N(n i ) is the set of adjacent pixels w of node ni i,j Considered as edge i,j In this way, RDIM pays more attention to the area with a larger relative distance, and the message is transmitted through the edge of the adjacent nodes. Then, the updated features of all nodes are Feed it to the global maximum pooling layer and obtain the updated global feature vector f g Finally, we pass the last The feature map is obtained in is the overall output of RDIM of scale l. By using GCN to transmit messages between nodes, the features of each node will be updated and refined according to its relationship with all other neighboring nodes. In our network, we apply RDIM at level 3 and 4 of the decoding stage. The generated RDIM feature map is then input to the next decoding stage.
[0044] Select the feature map generated by the last decoding stage To predict the final saliency map, which combines absolute and relative depth information, we first use a bilinear interpolation operation to transform the feature map The image is upsampled to make it the same size as the input image and finally input into the convolutional layer of a single channel to obtain the final saliency map S. During the training process, the final saliency map is converted from the true value map to the real value map through the cross entropy loss function. Supervision,
[0045]
[0046] in and Si,j are the saliency values at position (i, j) in the ground truth map and saliency map, respectively.
[0047] The present invention not only extracts RGB image features from the residual network, but also proposes to use depth information to assist the salient target detection task. Most existing RGB-D models simply extract depth and RGB features and heuristically fuse them. The present invention designs an absolute depth induction module to fuse RGB image features and depth image information across modalities in a coarse-to-fine manner, avoiding the cluttered noise interference caused by the asynchronous characteristics of the two spaces, thereby accurately locating objects; at the same time, a relative depth induction module is designed to establish a spatial graph convolution model to explore spatial structure and geometric information to enhance the local feature representation capability, thereby improving the accuracy and robustness of detection, so that it can achieve excellent detection effects, help integrate with other fields, and have broad application prospects.
[0048] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0049] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.
[0050] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for salient object detection based on absolute and relative depth induction network, characterized in that: The steps include: a. Deep induced network training with residual network as the backbone network: remove the last pooling layer and fully connected layer of ResNet-50, adjust the network input image to 256×256, normalize the data set, generate the corresponding side output map through the pyramid method, and then perform fusion operations from top to bottom in the network; b. Absolute depth induction module cross-modal feature fusion and object positioning: The backbone network includes five convolution blocks, Conv1, Res2_x, ... Res5_x. The depth image of the network input image is input into a set of convolutions to obtain a depth feature map with the same size as the Res2_x feature map. The absolute depth induction network is applied multiple times to recursively integrate the depth feature map and the RGB feature map to achieve cross-modal feature fusion, avoiding the noise interference caused by simply fusing two asynchronous modal features, strengthening the deep interaction between depth and color features, and adaptively fusing RGB and depth features at each scale; First, the depth image D of size W×H is input into a set of convolutional layers and a size of The feature map f d Then, the absolute depth induction module ADIM is applied multiple times to recursively combine the depth feature map with the RGB feature map are integrated to enhance the deep interaction between depth and color features. in, is the updated deep feature, It is the aggregation result of the depth and RGB information in the lth layer; c. The relative depth induction module establishes a spatial geometric model to supplement the detail information: First, the feature map from the last stage of the decoding network Res5_x is upsampled and integrated with the feature map obtained by cross-modal fusion of the absolute depth induction module to generate a new feature map, which is then input into the relative depth induction module together with the depth map generated by the absolute depth induction module to explore the spatial structure and detailed saliency information of the image, and the relative depth information is wrapped in the network to improve the performance of the saliency model; When generating the side output map in step a, the output feature maps of the four residual blocks are input into a 1×1 convolutional layer to reduce the channel dimension of the feature map, which is the side output map, so as to be used for the subsequent top-down integrated multi-level feature map.
2. The method for salient object detection based on absolute and relative depth induced network according to claim 1, characterized in that: When the network input images in step a have the same size, the data set is operated using a bilinear interpolation method; Input RGB image of size W×H, and generate convolution blocks of size Feature map Shallow layers capture low-level information of the image, including texture and spatial details, while deep feature maps contain high-level semantic information; feature maps are fused in a pyramidal manner Use a 1×1 convolution kernel to reduce the channel to C and get the side output map Then, the multi-level feature maps are integrated in a top-down manner. Where σ is the ReLU activation function, CAT[.,.] is the concatenation operation of two feature maps with the same channel dimension, UP is the upsampling operation with bilinear interpolation, and W l , b l are trainable parameters in the network.
3. The method for salient object detection based on absolute and relative depth induced network according to claim 2, characterized in that: In step b, the depth feature map and the RGB2 feature map are integrated together in a recursive manner, and the absolute depth induction network is implemented by a gated recursive unit GRU; At each time step, the RGB feature map is regarded as the input of GRU, and the deep features It can be regarded as the hidden state of the last step. The two feature maps are cascaded and transformed through the global maximum pooling GMP operation to generate a feature vector. Then a fully connected layer is applied to the feature vector to generate a reset gate r and an update gate z. The values of these two gates are normalized by the S-type function. In fact, the reset gate r controls the integration of depth and RGB features, and z controls In this way, RGB and depth features can be adaptively fused at each scale; The interaction between depth and RGB features is enhanced through the network; then the generated multi-scale feature map Combined with the feature map in the decoding state, formula (1) is reformulated as: The relative depth induction module establishes a spatial geometric model to supplement the detailed information: The relative depth induction module RDIM is used in the decoding stage. First, the feature map from the last stage of the decoding network is Upsampling and feature maps Integrated together, as described in formula (3), the generated feature map is expressed as Then RDIM is applied to the feature map and depth images to embed relative depth information in the network.
4. The method for salient object detection based on absolute and relative depth induced network according to claim 1, characterized in that: At each time step, the depth feature map is first reduced in dimension, and then the depth and RGB feature maps are cascaded and transformed through global maximum pooling to generate a new feature vector, which is then operated through a fully connected layer to achieve adaptive fusion of RGB and depth features at each scale.
5. The method for salient object detection based on absolute and relative depth induced network according to claim 1, characterized in that: In step c, a relative depth induction module is used to explore the spatial structure and detailed saliency information of the image. The relative depth induction module uses a graph convolutional network (GCN) to explore relative depth information.
6. The method for salient object detection based on absolute and relative depth induced network according to claim 1, characterized in that: RDIM is implemented by graph convolutional network GCN. In order to explore the relative depth relationship between pixels, the feature map generated by ADIM is first It is expressed as G = (V, E), where the node set is V and the edge set is E; each node ni in the graph is regarded as a point in the 3D coordinate system, and the coordinates are expressed as (x i ,y i , d i ), where (x i ,y i ) is the spatial position of the feature, and di is the corresponding depth value; the node set is represented as V = {n1, n2, ..., n k }, k is the number of nodes; define the edge set e of the 3D coordinate and its adjacent m elements i,j ∈E, calculate edge e i,j The weight w on i,j , as a relative depth value to measure the spatial correlation between nodes ni and nj, w i,j =|(x i ,y i ,d i )-(x j ,y j ,d j )| (5) In order to describe the semantic relationship between nodes ni and nj, we can define an edge i,j , defines an attribute feature a i,j In order to further consider the global context information of the image, the feature map Apply GAP to extract high-level semantic information and output the feature vector f g ; The spatial GCN consists of a set of stacked graph convolutional layers GCL. For each GCL, we first update the edge e i,j Attribute characteristics of a i,j , in, and They are feature maps The position (x i ,y i ) and (x j ,y j ), and use MLP to update the function of each node. Where N(n i ) is the set of adjacent pixels w of node ni i,j Considered as edge i,j The attention value of the relative depth value on the node, in this way, RDIM pays more attention to the area with a larger relative distance, in this way, RDIM pays more attention to the area with a larger relative distance, the message is transmitted through the edge of the adjacent nodes; then, the updated feature map of all nodes Feed it to the global maximum pooling layer and obtain the updated global feature vector f g , and finally, through the last The feature map is obtained in is the overall output of the RDIM of scale l; by using GCN to transmit messages between nodes, the function of each node will be updated and improved according to its relationship with all other neighboring nodes; In the network, RDIM is applied at the 3rd and 4th levels of the decoding stage; the generated RDIM feature map is then input to the next decoding stage; Select the feature map generated by the last decoding stage To predict the final saliency map, which combines absolute and relative depth information, we first use a bilinear interpolation operation to transform the feature map The image is upsampled to make it the same size as the input, and finally input into the convolution layer of a single channel to obtain the final saliency map S. During the training process, the final saliency map is converted from the true value map to the real value map through the cross entropy loss function. Supervision, in and S i,j are the saliency values at position (i, j) in the ground truth map and saliency map, respectively.
