A depth estimation space division optimization method and system based on depth prior

Abstract

This paper discloses a method for optimizing spatial partitioning for depth estimation based on depth priors. The method first obtains an RGB image and a corresponding sparse depth map. The sparse depth map is processed using a lightweight feature extraction module to obtain a spatial partitioning result. A depth estimation network is then used to perform probabilistic predictions. Finally, the spatial partitioning result and the probabilities are combined to produce a final depth map. Compared to traditional classification-regression monocular depth estimation methods, this method adds guidance based on the sparse depth map, significantly improving the quality of the depth space partitioning and resulting in more accurate depth estimation results.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to a depth sensing system, and specifically relates to a depth estimation space division optimization method and system based on depth prior. Background Art

[0002] From smartphone cameras and augmented reality (AR) / virtual reality (VR) applications to autonomous driving and even more complex robotics tasks, many different computer vision algorithms are becoming increasingly common in our daily lives. To effectively solve these problems, obtaining accurate and reliable three-dimensional scene information is crucial. Considering that most recent mobile devices have more than one camera, or even a LiDAR sensor, it is generally believed that integrating multiple sensor inputs for 3D reconstruction is a more efficient and practical approach. Therefore, using a depth sensor to obtain more accurate initial depth information is also preferred. However, one of the main drawbacks of commercial depth sensors (such as 3D LiDAR and Kinect cameras) is the sparsity of the measured depth. To address this issue, various methods have emerged that attempt to "complete" sparse depth measurements into dense depth maps, namely "depth completion."

[0003] Many mainstream methods currently formulate the depth estimation task as a pixel-by-pixel regression. Pixel-by-pixel regression methods can accurately predict pixel depth and have become a common paradigm. Despite their demonstrated success, these methods still suffer from slow convergence and unsatisfactory results. Other methods discretize continuous depth values ​​and formulate them as a pixel-by-pixel classification problem through deep network learning. While this approach significantly improves model performance, the discretization of depth values ​​leads to poor visual quality and noticeable discontinuities. To address this issue, some methods reformulate the depth estimation task as a pixel-by-pixel classification-regression task, learning a probabilistic representation for each pixel and predicting the final depth value as a linear combination of the center of each discrete bin. The bin centers are predefined in uniform / log-uniform space (UD / SID) or a trained space (for each dataset). These methods combine the advantages of both tasks to achieve better performance. However, there remains significant room for development in effectively partitioning the discrete space to improve depth estimation. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the present invention provides a depth estimation space partition optimization method and system based on depth prior.

[0005] The present invention mainly targets a type of method that regards depth estimation as a combination of classification and regression tasks. This type of method mainly has two main subtasks, namely spatial partitioning (discrete depth interval partitioning) and pixel-by-pixel probability prediction. Since the final depth map is obtained by combining the results of these two tasks, improving the effect of any one of the subtasks will improve the overall depth estimation effect. Among them, the spatial partitioning subtask mainly requires discrete division of the depth interval of a given scene, and this task generally only has one RGB image as input each time, that is, we need to perform spatial division based on the limited information provided by this RGB image, which is a very challenging task.

[0006] To alleviate the spatial information loss caused by only having an RGB image, this paper first introduces a sparse depth map as depth prior knowledge to help divide the discrete depth intervals. Then, using the current advanced Transformer technology, a targeted lightweight feature extraction module is designed to extract features from this sparse depth map, fully utilizing the effective information in the introduced sparse depth map. This method improves the rationality of spatial partitioning, thereby improving the final depth estimation effect.

[0007] A depth estimation space partition optimization method based on depth prior includes the following steps:

[0008] Step 1: Get the RGB image and the corresponding sparse depth image;

[0009] The camera obtains the RGB image of a given scene, and the depth sensor obtains the corresponding sparse depth map.

[0010] Step 2: Process the sparse depth map through the lightweight feature extraction module to obtain the spatial partitioning result;

[0011] Step 3: Probability prediction through depth estimation network;

[0012] Step 4: Combine the obtained spatial division results and probabilities to obtain the final depth map;

[0013] Furthermore, the specific method of step 2 is as follows;

[0014] The lightweight feature extraction module first extracts the coordinates of valid depth points in the sparse depth map to form an initial spatial feature. This feature has an N×3 dimension, where N is the number of valid depth points and 3 refers to the x-coordinate, y-coordinate, and depth of each point. It then uses a group of fully connected neural network layers to transform the second dimension of this feature to dim-3. This feature is then concatenated with the initial N×3-dimensional feature, resulting in a feature dimension of N×dim (dim=256 in practice). A 1D convolutional layer then scales the first dimension of the spatial feature to M, resulting in an M×dim feature dimension (M=128 in practice). This results in an M×dim spatial feature that contains the spatial information from the sparse depth map. This M×dim-dimensional spatial feature serves as the initial spatial partitioning. Finally, it is further processed through a Transformer module to obtain the final spatial partitioning feature. The Transformer module does not change the feature size but enhances the correlation between its components.

[0015] Furthermore, the specific method of step 3 is as follows;

[0016] The RGB image and the corresponding sparse depth image obtained in step 1 are processed using a depth estimation network with a U-net structure that has been jointly trained with a lightweight feature extraction module, and a probability map P is output that represents each pixel in the RGB image and the sparse depth image corresponding to the spatial partitioning feature in step 2.

[0017] Furthermore, the specific method of step 4 is as follows;

[0018] First, the spatial division features obtained in step 2 are passed through a fully connected layer to obtain the final spatial division vector b. This vector consists of M values, which represent the width of each small depth interval divided out in turn. The center depth value of each depth interval is obtained through this vector b representing the interval width:

[0019]

[0020] where b i Indicates the width of the i-th depth interval; d min and d max Respectively represent the minimum depth value and maximum depth value of the entire depth range; b j Indicates the width of a depth interval before the i-th interval. Calculate the depth interval vector c(b): = {c(b1), c(b2), ..., c(b N )}. Combined with the probability map P obtained in step 3, the depth value of each pixel is calculated

[0021]

[0022] in is the depth value obtained, c(b k ) is the depth value of the center of the kth depth interval, p k is the probability that the pixel belongs to the kth depth interval. After calculating the depth values ​​of all pixels, the final depth map is obtained.

[0023] Furthermore, in the training phase, random sampling is performed on the depth labels in the dataset to simulate the sparse depth map obtained by the depth sensor in the real scene, and 500 depth points are fixedly sampled on the depth label as the sparse depth map; the dataset adopts the existing NYUv2 and KITTI datasets.

[0024] A depth estimation space partition optimization system based on depth prior includes a data acquisition module, a lightweight feature extraction module, a depth estimation module and a final depth map acquisition module.

[0025] The data acquisition module is used to obtain an RGB image and a corresponding sparse depth map; the RGB image of a given scene is obtained through a camera, and the corresponding sparse depth map is obtained using a depth sensor.

[0026] The lightweight feature extraction module is used to process the sparse depth map to obtain a spatial partitioning result;

[0027] The lightweight feature extraction module first extracts the coordinates of valid depth points in the sparse depth map to form an initial spatial feature. This feature has an N×3 dimension, where N is the number of valid depth points and 3 refers to the x-coordinate, y-coordinate, and depth of each point. It then uses a group of fully connected neural network layers to transform the second dimension of this feature to dim-3. This feature is then concatenated with the initial N×3-dimensional feature, resulting in a feature dimension of N×dim (dim=256 in practice). A 1D convolutional layer then scales the first dimension of the spatial feature to M, resulting in an M×dim feature dimension (M=128 in practice). This results in an M×dim spatial feature that contains the spatial information from the sparse depth map. This M×dim-dimensional spatial feature serves as the initial spatial partitioning. Finally, it is further processed through a Transformer module to obtain the final spatial partitioning feature. The Transformer module does not change the feature size but enhances the correlation between its components.

[0028] The depth estimation module is used to perform probability prediction;

[0029] The depth estimation module uses the existing U-net structure depth estimation network that has been jointly trained with the lightweight feature extraction module to process the RGB image and the corresponding sparse depth image obtained by the data acquisition module, and outputs a probability map P that can represent each pixel in the RGB image and the sparse depth image corresponding to the spatial division feature in the lightweight feature extraction module.

[0030] The final depth map acquisition module combines the obtained spatial division result and the probability to obtain the final depth map;

[0031] First, the spatial partitioning features obtained by the lightweight feature extraction module are passed through a fully connected layer to obtain the final spatial partitioning vector b. This vector consists of M values, which represent the width of each small depth interval in turn. The center depth value of each depth interval is obtained through this vector b representing the interval width:

[0032]

[0033] where b i Indicates the width of the i-th depth interval; d min and d max Respectively represent the minimum depth value and maximum depth value of the entire depth range; b j Indicates the width of a depth interval before the i-th interval. Calculate the depth interval vector c(b): = {c(b1), c(b2), ..., c(b N )}. Combined with the probability map P obtained by the depth estimation module, the depth value of each pixel is calculated

[0034]

[0035] in is the depth value obtained, c(b k ) is the depth value of the center of the kth depth interval, p k is the probability that the pixel belongs to the kth depth interval. After calculating the depth values ​​of all pixels, the final depth map is obtained.

[0036] The beneficial effects of the present invention are as follows:

[0037] Compared with the traditional classification-regression monocular depth estimation method, the present invention adds guidance based on sparse depth maps, which improves the quality of depth space division to a certain extent, achieving the effect of making the depth estimation result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a diagram of the overall network model of an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of a lightweight feature extraction module according to an embodiment of the present invention; DETAILED DESCRIPTION

[0040] The present invention will be described in detail below with reference to specific embodiments.

[0041] like Figure 1 As shown, the present invention proposes a depth estimation space division optimization method based on depth prior, which includes the following steps:

[0042] Step 1: Get the RGB image and the corresponding sparse depth image;

[0043] The camera obtains the RGB image of a given scene, and the depth sensor obtains the corresponding sparse depth map.

[0044] Step 2: Process the sparse depth map through the lightweight feature extraction module to obtain the spatial partitioning result;

[0045] Since there are only a limited number of valid depth points in the sparse depth map, such as 500 points in the training phase, there is no need to use a redundant and complex large neural network for feature extraction. Therefore, the present invention designs a lightweight feature extraction module for sparse depth maps. Figure 2 As shown, the lightweight feature extraction module first extracts the coordinates of valid depth points in the sparse depth map to form an initial spatial feature. This feature has an N×3 dimension, where N is the number of valid depth points and 3 refers to the x-coordinate, y-coordinate, and depth of each point. It then transforms the second dimension of this feature into dim-3 through a set of fully connected neural network layers. This feature is then concatenated with the initial N×3-dimensional feature, resulting in a feature dimension of N×dim (dim=256 in actual implementation). A 1D convolutional layer then scales the first dimension of the spatial feature to M, resulting in an M×dim feature dimension (M=128 in actual implementation). This results in an M×dim spatial feature that contains the spatial information in the sparse depth map. This M×dim-dimensional spatial feature serves as the initial spatial partitioning. Finally, it is further processed through a Transformer module to obtain the final spatial partitioning feature. The Transformer module does not change the feature size but enhances the correlation between its components.

[0046] Step 3: Probability prediction through depth estimation network;

[0047] Since the focus of the present invention is to improve the quality of spatial partitioning rather than the accuracy of probability prediction, the probability prediction part uses a depth estimation network with a U-net structure that has been jointly trained with a lightweight feature extraction module to process the RGB image and the corresponding sparse depth image obtained in step 1, and outputs a probability map P that can represent each pixel in the RGB image and the sparse depth map corresponding to the spatial partition feature in step 2.

[0048] Step 4: Combine the obtained spatial division results and probabilities to obtain the final depth map;

[0049] First, the spatial division features obtained in step 2 are passed through a fully connected layer to obtain the final spatial division vector b. This vector consists of M values, which represent the width of each small depth interval divided out in turn. The center depth value of each depth interval is obtained through this vector b representing the interval width:

[0050]

[0051] where b i Indicates the width of the i-th depth interval; d min and d max Respectively represent the minimum depth value and maximum depth value of the entire depth range; b j Indicates the width of a depth interval before the i-th interval. Calculate the depth interval vector c(b): = {c(b1), c(b2), ..., c(b N )}. Combined with the probability map P obtained in step 3, the depth value of each pixel is calculated

[0052]

[0053] in is the depth value obtained, c(b k ) is the depth value of the center of the kth depth interval, p k is the probability that the pixel belongs to the kth depth interval. After calculating the depth values ​​of all pixels, the final depth map is obtained.

[0054] During the training phase, random sampling is performed on the depth labels in the dataset to simulate the sparse depth map obtained by depth sensors such as LiDAR in real scenes. A fixed sampling of 500 depth points on the depth labels is used as the sparse depth map; the dataset uses the existing NYUv2 and KITTI datasets.

[0055] A depth estimation space partition optimization system based on depth prior includes a data acquisition module, a lightweight feature extraction module, a depth estimation module and a final depth map acquisition module.

[0056] The data acquisition module is used to obtain the RGB image and the corresponding sparse depth map; the RGB image of a given scene is obtained through the camera, and the corresponding sparse depth map is obtained using the depth sensor.

[0057] The lightweight feature extraction module is used to process the sparse depth map and obtain the spatial partitioning result;

[0058] The lightweight feature extraction module first extracts the coordinates of valid depth points in the sparse depth map to form an initial spatial feature. This feature has an N×3 dimension, where N is the number of valid depth points and 3 refers to the x-coordinate, y-coordinate, and depth of each point. It then uses a group of fully connected neural network layers to transform the second dimension of this feature to dim-3. This feature is then concatenated with the initial N×3-dimensional feature, resulting in a feature dimension of N×dim (dim=256 in practice). A 1D convolutional layer then scales the first dimension of the spatial feature to M, resulting in an M×dim feature dimension (M=128 in practice). This results in an M×dim spatial feature that contains the spatial information from the sparse depth map. This M×dim-dimensional spatial feature serves as the initial spatial partitioning. Finally, it is further processed through a Transformer module to obtain the final spatial partitioning feature. The Transformer module does not change the feature size but enhances the correlation between its components.

[0059] The depth estimation module is used to make probability predictions;

[0060] The depth estimation module uses the existing U-net structure depth estimation network that has been jointly trained with the lightweight feature extraction module to process the RGB image and the corresponding sparse depth image obtained by the data acquisition module, and outputs a probability map P that can represent the spatial partitioning features in the lightweight feature extraction module for each pixel in the RGB image and the sparse depth image.

[0061] The final depth map acquisition module combines the obtained spatial division results and probabilities to obtain the final depth map;

[0062] First, the spatial partitioning features obtained by the lightweight feature extraction module are passed through a fully connected layer to obtain the final spatial partitioning vector b. This vector consists of M values, which represent the width of each small depth interval in turn. The center depth value of each depth interval is obtained through this vector b representing the interval width:

[0063]

[0064] where b i Indicates the width of the i-th depth interval; d min and d maxRespectively represent the minimum depth value and maximum depth value of the entire depth range; b j Indicates the width of a depth interval before the i-th interval. Calculate the depth interval vector c(b): = {c(b1), c(b2), ..., c(b N )}. Combined with the probability map P obtained by the depth estimation module, the depth value of each pixel is calculated

[0065]

[0066] in is the depth value obtained, c(b k ) is the depth value of the center of the kth depth interval, p k is the probability that the pixel belongs to the kth depth interval. After calculating the depth values ​​of all pixels, the final depth map is obtained.

[0067] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0068] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A depth estimation space partition optimization method based on depth prior, characterized in that: The steps are as follows: Step 1: Get the RGB image and the corresponding sparse depth image; Obtain an RGB image of a given scene through a camera, and use a depth sensor to obtain a corresponding sparse depth map; Step 2: Process the sparse depth map through the lightweight feature extraction module to obtain the spatial partitioning result; Step 3: Probability prediction through depth estimation network; Step 4: Combine the obtained spatial division results and probabilities to obtain the final depth map; Step 2: The specific method is as follows; The lightweight feature extraction module first extracts the coordinates of the effective depth points in the sparse depth map to form the initial spatial features. The feature dimension is , where N is the number of valid depth points, and 3 refers to the horizontal coordinate, vertical coordinate and depth of each point; then the second dimension is transformed into dim-3 through a group of fully connected neural network layers, and then combined with the initial dimensional features are spliced ​​together, that is, the feature dimension becomes , in the actual implementation process, dim=256; then a one-dimensional convolution layer is used to change the first dimension of the spatial feature to M, that is, the feature dimension becomes , in the actual implementation process, M=128; so we get a dimension of The spatial features of the sparse depth map contain the spatial information. dimensional spatial features as the initial spatial division; finally, it is further processed through a Transformer module to obtain the final spatial division features; the Transformer module does not change the size of the features, but enhances the association between the various parts within the features.

2. The depth estimation space partition optimization method based on depth prior according to claim 1, characterized in that: Step 3: The specific method is as follows; The RGB image and the corresponding sparse depth image obtained in step 1 are processed using a depth estimation network with a U-net structure that has been jointly trained with a lightweight feature extraction module, and a probability map P is output that represents each pixel in the RGB image and the sparse depth image corresponding to the spatial partitioning feature in step 2.

3. The depth estimation space partition optimization method based on depth prior according to claim 2, characterized in that: Step 4: The specific method is as follows; First, the spatial division features obtained in step 2 are passed through a fully connected layer to obtain the final spatial division vector b. This vector consists of M values, which represent the width of each small depth interval divided out in turn. The center depth value of each depth interval is obtained through this vector b representing the interval width: ； in Indicates the width of the i-th depth interval; and Respectively represent the minimum depth value and maximum depth value of the entire depth range; Indicates the width of a depth interval before the i-th interval; calculate the depth interval vector ; Combined with the probability map P obtained in step 3, calculate the depth value of each pixel ; ； in is the depth value obtained, is the depth value of the center of the k-th depth interval, is the probability that the pixel belongs to the kth depth interval; After calculating the depth values ​​of all pixels, the final depth map is obtained.

4. A depth estimation space partition optimization method based on depth prior according to claim 2 or 3, characterized in that: During the training phase, random sampling is performed on the depth labels in the dataset to simulate the sparse depth map obtained by the depth sensor in the real scene, and 500 depth points are fixedly sampled on the depth labels as the sparse depth map; the dataset uses the existing NYUv2 and KITTI datasets.

5. A depth estimation space partitioning optimization system based on depth prior, characterized in that: It includes data acquisition module, lightweight feature extraction module, depth estimation module and final depth map acquisition module; The data acquisition module is used to obtain an RGB image and a corresponding sparse depth map; the RGB image of a given scene is obtained by a camera, and the corresponding sparse depth map is obtained using a depth sensor; The lightweight feature extraction module is used to process the sparse depth map to obtain a spatial partitioning result; The lightweight feature extraction module first extracts the coordinates of the effective depth points in the sparse depth map to form the initial spatial features. The feature dimension is , where N is the number of valid depth points, and 3 refers to the horizontal coordinate, vertical coordinate and depth of each point; then the second dimension is transformed into dim-3 through a group of fully connected neural network layers, and then combined with the initial dimensional features are spliced ​​together, that is, the feature dimension becomes , in the actual implementation process, dim=256; then a one-dimensional convolution layer is used to change the first dimension of the spatial feature to M, that is, the feature dimension becomes , in the actual implementation process, M=128; so we get a dimension of The spatial features of the sparse depth map contain the spatial information. The dimensional spatial features are used as the initial spatial partitioning; finally, they are further processed through a Transformer module to obtain the final spatial partitioning features; the Transformer module does not change the size of the features, but strengthens the association between the parts within the features; The depth estimation module is used to perform probability prediction; The depth estimation module uses an existing U-net structured depth estimation network that has been jointly trained with the lightweight feature extraction module to process the RGB image and the corresponding sparse depth image obtained by the data acquisition module, and outputs a probability map P that can represent each pixel in the RGB image and the sparse depth image corresponding to the spatial partitioning features in the lightweight feature extraction module; The final depth map acquisition module combines the obtained spatial division result and the probability to obtain the final depth map; First, the spatial partitioning features obtained by the lightweight feature extraction module are passed through a fully connected layer to obtain the final spatial partitioning vector b. This vector consists of M values, which represent the width of each small depth interval in turn. The center depth value of each depth interval is obtained through this vector b representing the interval width: ； in Indicates the width of the i-th depth interval; and Respectively represent the minimum depth value and maximum depth value of the entire depth range; Indicates the width of a depth interval before the i-th interval; calculate the depth interval vector ; Combined with the probability map P obtained by the depth estimation module, the depth value of each pixel is calculated ; ； in is the depth value obtained, is the depth value of the center of the k-th depth interval, is the probability that the pixel belongs to the kth depth interval; After calculating the depth values ​​of all pixels, the final depth map is obtained.

Images