Dense depth map generation method and system

Abstract

The present invention proposes a method and system for generating a dense depth map. The method comprises the following steps: performing instance segmentation learning on an RGB color image in a deep learning network model to obtain edge features of an object, and performing semantic segmentation learning to obtain edge features between a scene and an object; re-predicting the relationship between the target object and the target object, and the positional relationship between the scene and the object; performing information fusion and complementary learning on the edge features of the object and the preliminary dense depth map to obtain an edge distance field; inputting the sparse depth map, image confidence, and edge distance field into a hierarchical multi-scale network structure to repair the edge features of the depth map; inputting the repaired depth map into a multi-scale optimization module to optimize the edge features of the depth map to obtain the final dense depth map. The dense depth map obtained by this method has a high precision.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and system for generating a dense depth map. Background Art

[0002] Depth map is a common way to express 3D scene information, also known as distance image. Each pixel value represents the distance from the image collector to each position in the scene. At present, there are two main ways to obtain depth information in the scene: passive and active. Using a binocular camera to obtain the depth information of the scene is the most common passive way to obtain depth information. However, due to the problems of poor system stability and complex calculation when using a binocular camera for ranging, these problems limit the practical application of binocular camera ranging technology. The principle of active depth information acquisition is to use the sensor's own light source to emit lasers or rays into space to collect the depth information of the scene. Compared with the passive depth information acquisition method, it can obtain depth information faster and more accurately, so it has been widely used.

[0003] Whether it is the acquisition of indoor (such as Kinect depth sensor) or outdoor (Li DAR laser radar), the number of points scanned per unit time is limited by the performance of the device itself. When the 3D point cloud captured by the laser radar is projected onto the image plane, the generated depth image is uneven and sparse, and the depth information is expressed in the form of discrete points, and the depth values ​​of many areas are missing. For such an image with missing depth information in most areas, it is called a sparse depth image.

[0004] Currently, there is no cheap solution to obtain dense and accurate depth maps. How to inpaint the sparse depth map from low-cost LiDAR to generate dense high-quality depth maps has become a hot topic worth studying.

[0005] The goal of sparse depth map inpainting is to generate dense and accurate depth maps. The difficulties and challenges of sparse depth map inpainting at present mainly include the inconsistency between depth edges and RGB image edges, the precise positioning of depth edges, and the large differences in depth features between different near and far areas. The existing technologies for extracting depth edges, RGB image edges, and depth map edges are all based on convolutional neural networks. The edge information obtained is incomplete, and it is difficult to guess what the edge is without human intervention. Summary of the invention

[0006] In order to overcome the defects in the above-mentioned prior art, an object of the present invention is to provide a method and system for generating a dense depth map.

[0007] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for generating a dense depth map, comprising the following steps:

[0008] Obtain an RGB color image and a sparse depth map of the RGB color image, calculate the image confidence of the RGB color image, and obtain a preliminary dense depth map based on the sparse depth map;

[0009] Extracting RGB image edge features from the RGB color image in a deep learning network model, including performing instance segmentation learning on the RGB color image to obtain object edge features, and performing semantic segmentation learning on the RGB color image to obtain edge features between scenes and objects;

[0010] Extract all possible relationships between target objects from the edge features of the objects, and re-predict the target objects and the relationships between them based on the graph neural network; extract all possible positional relationships between the scene and the objects from the edge features between the scene and the objects, and re-predict the positional relationship between the scene and the objects based on the graph neural network;

[0011] Performing information fusion and complementary learning on the edge features of the object and the preliminary dense depth map to obtain an edge distance field;

[0012] The sparse depth map, image confidence, and edge distance field are input into a hierarchical multi-scale network structure to repair the edge features of the depth map;

[0013] The repaired depth map is input into a multi-scale optimization module to optimize the edge features of the depth map to obtain the final dense depth map.

[0014] This method achieves accurate positioning of depth edges by extracting edge features from RGB images and using the extracted edge features of RGB images in a graph neural network to re-predict the positional relationships between objects and between scenes and objects. The depth edges are made consistent with the RGB image edges by patching the edge features of the depth map. The depth feature differences between different near and far regions are reduced by optimizing the edge features of the depth map, and the resulting dense depth map has high accuracy.

[0015] The preferred solution of the dense depth map generation method: the hierarchical multi-scale network structure is constructed based on the edge-guided convolution layer, and the depth information of different scales is extracted in the hierarchical multi-scale network structure. The confidence output of the last edge-guided convolution layer is max-pooled, while keeping the index of the confidence max-pooling the same as the depth information and edge distance, and selecting the feature with the highest confidence to enter the network of the next scale. In this way, the depth information of different scales can be extracted, and the inpainting accuracy can be further improved.

[0016] The present invention also proposes a dense depth map generation system based on the above-mentioned dense depth map generation method, comprising:

[0017] The RGB image edge feature extraction module is used to extract the RGB image edge features from the previously acquired RGB color pictures to obtain the edge features of the objects and the edge features between the scenes and the objects;

[0018] A scene understanding module based on a graph neural network is used to learn the edge features of the object and re-predict the target object and the relationship between the target objects; it is used to learn the edge features between the scene and the object and re-predict the positional relationship between the scene and the object;

[0019] A depth map edge extraction module performs information fusion and complementary learning on the edge features of the object and the preliminary dense depth map to obtain an edge distance field;

[0020] The depth edge inpainting module inpaints the depth map edge based on the sparse depth map, image confidence, and edge distance field;

[0021] The depth feature optimization module optimizes the edge features of the repaired depth map to obtain the final dense depth map.

[0022] In the preferred solution of the dense depth map generation system, the depth feature optimization module includes a multi-scale optimization module, which is composed of three cascaded hourglass networks. Each hourglass network inputs the depth map of different resolutions output by the depth edge repair module, and is trained at its own resolution. The object relationship graph neural network is combined to enhance the accuracy of the object edge, and the image semantic scene position graph neural network is combined to enhance the accuracy of the edge depth information. This enhances the accuracy of the edge depth information and further improves the repair effect.

[0023] The beneficial effects of the present invention are as follows: the present invention can better solve the three problems of accurate positioning of depth edges, inconsistency between depth edges and RGB image edges, and large differences in depth features between different near and far areas, so as to obtain a dense depth map with high accuracy.

[0024] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0026] Figure 1 It is a schematic diagram of the steps of a dense depth map generation method;

[0027] Figure 2 It is a dense deep graph generation network model diagram based on graph neural network;

[0028] Figure 3 It is the RGB image edge feature extraction module;

[0029] Figure 4 This is a schematic diagram of the scene understanding module learning the edge features of objects;

[0030] Figure 5 It is a schematic diagram of the scene understanding module learning the edge features between the scene and the object;

[0031] Figure 6 It is the depth map edge extraction module;

[0032] Figure 7 It is a schematic diagram of edge-guided convolutional layer;

[0033] Figure 8 It is a schematic diagram of the deep feature optimization module. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0035] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0036] like Figure 1 and Figure 2 As shown, the present invention provides a method for generating a dense depth map, comprising the following steps:

[0037] Obtain an RGB color image and a sparse depth map of the RGB color image, calculate the image confidence of the RGB color image, and obtain a preliminary dense depth map according to the sparse depth map. Here, the image confidence is calculated by an existing method, and a preliminary dense depth map is obtained by an existing method.

[0038] The RGB color picture is used in the deep learning network model to extract RGB image edge features, such as Figure 3As shown in the figure, it includes using a deep learning network model to perform instance segmentation learning on RGB color images to obtain object edge features, and performing semantic segmentation learning on RGB color images to obtain edge features between scenes and objects. Object edge features are accurate but redundant but sharp edge features of each object.

[0039] like Figure 4 As shown, all possible relationships between target objects are extracted from the edge features of the objects, and the target objects and the relationships between target objects are re-predicted based on the graph neural network. Specifically, all target objects learned from instance segmentation and all possible relationships between them are extracted, so that a dense graph with target objects in the image as nodes and relationships between any two nodes is obtained. This dense graph is pruned and sparsed to obtain a potential scene graph representation, but each node in this scene graph is directly generated from the image, so there are fewer mutual connections and conventions. Therefore, a graph neural network is used to model this scene graph, and the representation of the nodes is re-learned, so that the information of other entities around the nodes can also be included. Finally, the target objects represented by the nodes and the relationships between them are re-predicted based on the new representation learned by the graph neural network.

[0040] like Figure 5 As shown, all possible positional relationships between the scene and the object are extracted from the edge features between the scene and the object, and the positional relationship between the scene and the object is re-predicted based on the graph neural network. Specifically, all possible positional relationships between the scene and the object are extracted from the edge features learned by semantic segmentation. Similar to the processing of the instance segmentation part, a dense graph with the scene semantics parsed from the image as nodes and a positional relationship between any two nodes can be obtained. A graph neural network is used to model the positional relationship graph, and the node representation is re-learned. The scene semantics represented by the node and the relationship between the scene semantics are re-predicted based on the re-learned node representation, so that the information of other entities around the node can also be included.

[0041] In this embodiment, when re-predicting the positional relationship between a scene and an object, and the positional relationship between a scene and an object based on a graph neural network, an object position relationship graph neural network and a scene position graph neural network are generated by using a knowledge graph as prior knowledge.

[0042] The edge features of the object and the preliminary dense depth map are subjected to information fusion and complementary learning to obtain the edge distance field, such as Figure 6 As shown. In this embodiment, the edge distance field is a continuous edge distance field designed based on the obtained depth edge map. An edge distance threshold is set near the depth edge. For a given depth edge position (i, j), E i,jis the edge distance at edge position (i, j). The area where the edge distance is less than the edge distance threshold is the edge distance field. The minimum value in the edge distance field is located at the depth edge. Starting from the depth edge, the edge distance increases uniformly and is equal to the edge distance at the depth edge distance. Therefore, E i±1,j =E i,j+1 , E i±2,j =E i,j±2 , and so on. The maximum value in the edge distance field is E max , this position is the smooth area in the depth map.

[0043] Object edge features are accurate but redundant but sharp edge features of each object. Rough but consistent edge features are learned from the preliminary dense depth map. Figure 2 The complementary edge information can obtain more accurate depth edge.

[0044] The sparse depth map, image confidence, and edge distance field are input into a hierarchical multi-scale network structure to repair the edge features of the depth map. During the repair process, the depth information on one side of the depth edge is prevented from affecting the other side, thereby ensuring that the repaired depth map retains sharp edges and improves the accuracy of the depth image.

[0045] The hierarchical multi-scale network structure is built on the edge-guided convolution layer. The depth information of different scales is extracted in the hierarchical multi-scale network structure. The confidence of the output of the last edge-guided convolution layer is max-pooled. At the same time, the index of the confidence max-pooling is kept the same as the depth information and edge distance. The feature with the highest confidence is selected to enter the network of the next scale. Here, the edge distance field is used as the input of the edge-guided convolution layer to prevent the depth value on one side from crossing the depth edge and affecting the other side during the convolution process.

[0046] The validity mask of the input sparse depth map is regarded as a continuous confidence field, and the proposed confidence propagation convolution layer allows the confidence to be propagated between convolutional layers, thereby allowing a point-by-point continuous confidence map to be output for the network. Based on the edge distance field and confidence propagation convolution layer defined above, the edge-guided convolution layer proposed in this embodiment utilizes data confidence to process sparse input, while using the continuous edge distance field as a guide for sparse depth map patching. In the CNN framework, the standard convolution layer is replaced by the edge-guided convolution layer. The schematic diagram of the edge-guided convolution layer is shown in FIG. Figure 7 As shown, it accepts three input quantities: depth information, confidence, and edge distance field, and processes and outputs these three types of information at the same time, where convolution is represented by *, dot product is represented by ⊙, summation is represented by ∑, and the inverse operation of dot product is represented by 1 / x.

[0047] The formula for transferring depth information in the edge-guided convolutional layer between convolutional layers is:

[0048] in for l th The output depth value of the layer at position (i, j), convolution kernel C l-1 、E l-1 l-1 respectively th The depth value, confidence and edge distance field of the layer output, b l is the offset, ε is a non-zero constant, m and n represent the range of neighbors of point (i, j), Γ() is the activation function, if m ranges from (-1, 1) and n ranges from (-1, 1), then the convolution kernel The value obtained is a 3x3 matrix. If the value range of m is (-2, 2) and the value range of n is (-2, 2), then the convolution kernel The value obtained is a 5x5 matrix.

[0049] The confidence transfer formula is: Indicates l th The confidence of the output of the layer at position i,j, C l-1 is the confidence of the output of the previous layer.

[0050] Since the mask will be oversaturated in the early convolutional layers of the model, this will undoubtedly reduce the performance of the network. However, the confidence transfer formula effectively solves the above-mentioned "validity mask saturation" problem by propagating confidence between convolutional layers.

[0051] In order to allow the propagation of edge distance quantities between convolutional layers, the transfer formula of depth map edge features is: E l-1 is the edge distance of the previous layer output, is a filter kernel with the middle element being 1 and the rest being 0.

[0052] The depth edge inpainting module outputs depth maps of different resolutions, such as Figure 8 As shown, these repaired depth maps are input into a multi-scale optimization module for training, and combined with the object relationship graph neural network to enhance the accuracy of object edges, and combined with the image semantic scene position graph neural network to enhance the accuracy of edge depth information, further improve the repair effect, optimize the edge features of the depth map, and obtain the final dense depth map.

[0053] The present invention also proposes an embodiment of a dense depth map generation system based on the above-mentioned dense depth map generation method, such as Figure 1-2As shown, the dense depth map generation system specifically includes: an RGB image edge feature extraction module, a scene understanding module based on a graph neural network, a depth map edge extraction module, a depth edge repair module and a depth feature optimization module.

[0054] like Figure 3 As shown, the RGB image edge feature extraction module is used to extract RGB image edge features from the RGB color picture acquired in advance, and obtain the edge features of the object and the edge features between the scene and the object.

[0055] The RGB image edge feature extraction module includes an instance segmentation module and a semantic segmentation module. The instance segmentation module takes the RGB color image as input and uses a deep learning network model to learn the edge features of each object from the RGB color image; all possible relationships between target objects are extracted from the object edge features, and the target objects and the relationships between target objects are re-predicted based on the graph neural network to obtain a dense graph with the target objects in the image as nodes and any two nodes having a relationship.

[0056] The semantic segmentation module takes RGB color images as input and learns the edge features between scenes and objects. All possible positional relationships between scenes and objects are extracted from the edge features between scenes and objects learned through semantic segmentation. Similar to the instance segmentation part, a dense graph can be obtained in which nodes are the scene semantics parsed from the image and there is a positional relationship between any two nodes.

[0057] Scene understanding modules based on graph neural networks, such as Figure 4-5 As shown, it is used to learn the edge features of the object and re-predict the target object and the relationship between the target object; it is used to learn the edge features between the scene and the object and re-predict the position relationship between the scene and the object. The specific re-prediction method refers to the content described in the above-mentioned dense depth map generation method, which will not be repeated here.

[0058] The depth map edge extraction module accepts the object edge features obtained by instance segmentation of the RGB image edge feature extraction module and the dense depth map after patching the sparse depth map using the existing algorithm as input. The object edge features are accurate but redundant but sharp edge features of each object. The rough but relatively consistent edge features are learned from the dense depth map. By combining the complementary edge information of the two for information fusion and complementary learning, a more accurate depth edge is obtained, thereby obtaining an edge distance field.

[0059] The depth edge inpainting module inpaints the depth map edge based on the sparse depth map, image confidence, and edge distance field. The specific inpainting method refers to the content described in the above dense depth map generation method, which will not be repeated here.

[0060] The depth feature optimization module optimizes the edge features of the repaired depth map to obtain the final dense depth map. Specifically, the depth feature optimization module includes a multi-scale optimization module. In this embodiment, a standard convolution layer + RELU structure is used as the multi-scale optimization module. Figure 8 As shown, the multi-scale optimization module consists of three cascaded hourglass networks, each of which inputs the depth map of different resolutions output by the depth edge inpainting module and is trained at its own resolution. The object relationship graph neural network is then combined with the image semantic scene position graph neural network to enhance the accuracy of the edge depth information (e.g., the outputs of the object relationship graph neural network and the image semantic scene position graph neural network are used as the input of the optimization network, and the optimization network filters the useful information therein to enhance the accuracy), to obtain the final dense depth map.

[0061] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0062] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A dense depth map generation method, characterized in that: The following steps are involved: Obtain an RGB color image and a sparse depth map of the RGB color image, calculate the image confidence of the RGB color image, and obtain a preliminary dense depth map based on the sparse depth map; Extracting RGB image edge features from the RGB color image in a deep learning network model, including performing instance segmentation learning on the RGB color image to obtain object edge features, and performing semantic segmentation learning on the RGB color image to obtain edge features between scenes and objects; Extract all possible relationships between target objects from the edge features of the objects, and re-predict the target objects and the relationships between them based on the graph neural network; extract all possible positional relationships between the scene and the objects from the edge features between the scene and the objects, and re-predict the positional relationship between the scene and the objects based on the graph neural network; Performing information fusion and complementary learning on the edge features of the object and the preliminary dense depth map to obtain an edge distance field; The sparse depth map, image confidence, and edge distance field are input into a hierarchical multi-scale network structure to repair the edge features of the depth map; The repaired depth map is input into a multi-scale optimization module to optimize the edge features of the depth map to obtain the final dense depth map.

2. The method for generating a dense depth map according to claim 1, characterized in that: The specific steps of re-predicting the target object and the relationship between the target objects based on the graph neural network are: Extract all possible relationships between target objects from the edge features of the objects to obtain a dense graph with target objects in the image as nodes and any two nodes having a relationship, thin the dense graph to obtain a potential scene graph representation, use a graph neural network to model the scene graph, relearn the node representation, and re-predict the target objects represented by the nodes and the relationship between the target objects based on the re-learned node representation; The specific steps of re-predicting the positional relationship between the scene and the object based on the graph neural network are: All possible positional relationships between the scene and the object are extracted from the edge features between the scene and the object, and a dense graph with the scene semantics parsed from the image as nodes and a positional relationship between any two nodes is obtained. A graph neural network is used to model the positional relationship graph, and the node representation is relearned. The scene semantics represented by the node and the relationship between the scene semantics are re-predicted based on the re-learned node representation.

3. The dense depth map generation method according to claim 1, characterized in that: The edge distance field is a region in the depth map where the edge distance to a given edge position is less than a preset edge distance threshold.

4. The method for generating a dense depth map according to claim 1, characterized in that: The hierarchical multi-scale network structure is constructed based on the edge-guided convolution layer. Depth information of different scales is extracted in the hierarchical multi-scale network structure. The confidence output of the last edge-guided convolution layer is maximum pooled, while the index of the confidence maximum pooling is kept the same as the depth information and the edge distance, and the feature with the highest confidence is selected to enter the network of the next scale.

5. The method for generating a dense depth map according to claim 4, characterized in that: The formula for transferring the depth information in the edge-guided convolutional layer between convolutional layers is: in for l th The output depth value of the layer at position (i, j), convolution kernel C l-1 、E l-1 l-1 respectively th The depth value, confidence and edge distance field of the layer output, b l is the offset, ε is a non-zero constant, m and n represent the range of neighbors of point (i, j), and Γ() is the activation function; The confidence transfer formula is: Indicates l th The confidence of the output of the layer at position i,j, C l-1 is the confidence of the output of the previous layer; The transfer formula of the edge feature of the depth map is: E l-1 is the edge distance of the previous layer output, is a filter kernel with the middle element being 1 and the rest being 0.

6. A dense depth map generation system based on the dense depth map generation method according to any one of claims 1 to 5, characterized in that: include: The RGB image edge feature extraction module is used to extract the RGB image edge features from the previously acquired RGB color pictures to obtain the edge features of the objects and the edge features between the scenes and the objects; A scene understanding module based on a graph neural network is used to learn the edge features of the object, re-predict the target object and the relationship between the target objects, and obtain an object relationship graph neural network; it is used to learn the edge features between the scene and the object, re-predict the positional relationship between the scene and the object, and obtain an image semantic scene position graph neural network; A depth map edge extraction module performs information fusion and complementary learning on the edge features of the object and the preliminary dense depth map to obtain an edge distance field; The depth edge inpainting module inpaints the depth map edge based on the sparse depth map, image confidence, and edge distance field; The depth feature optimization module optimizes the edge features of the repaired depth map to obtain the final dense depth map.

7. The dense depth map generation system according to claim 6, characterized in that: The RGB image edge feature extraction module includes an instance segmentation module and a semantic segmentation module. The instance segmentation module takes the RGB color image as input and uses a deep learning network model to learn the edge features of each object from the RGB color image; The semantic segmentation module takes RGB color images as input and learns the edge features between scenes and objects.

8. The dense depth map generation system according to claim 6, characterized in that: The scene understanding module includes an object edge feature understanding module, which extracts all possible relationships between target objects from the object edge features and re-predicts the target objects and the relationships between the target objects based on a graph neural network; The scene understanding module also includes an edge feature understanding module between the scene and the object, which extracts all possible positional relationships between the scene and the object from the edge features between the scene and the object, and re-predicts the positional relationship between the scene and the object based on the graph neural network.

9. The dense depth map generation system according to claim 6, characterized in that: The deep feature optimization module includes a multi-scale optimization module, which is composed of three cascaded hourglass networks. Each hourglass network inputs a depth map of different resolutions output by the deep edge patching module and is trained at its own resolution. The object relationship graph neural network is combined to enhance the accuracy of object edges, and the image semantic scene position graph neural network is combined to enhance the accuracy of edge depth information.

Images