A weakly aligned RGB-D image guided depth completion method and system

Abstract

The application discloses a kind of weak alignment RGB-D image guided depth map completion method and system, carry out inconsistent pixel detection to weak alignment RGB-D image, use neural network technology to divide weak alignment RGB-D image into flat area and depth structure area, select surface normal feature constraint flat area depth value smoothness;Select Gaussian weight feature constraint the accuracy of depth structure area structure information, adopt kernel function to adjust the size of Gaussian weight;Combining Markov Random Field optimization model, define depth completion as global optimization model, use data constraint term, normal constraint term and Gaussian weight constraint term to construct the cost function of global optimization model, output high-quality completion depth map by iterative solution.The application can not only complete original depth map with serious data loss under the guidance of weak alignment RGB-D image, but also can guarantee the structure accuracy and smoothness of completion depth map.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and system for depth map completion guided by weakly aligned RGB-D images. Background Technology

[0002] Depth maps effectively provide distance and geometric information about objects in a real-world scene. They are commonly used to supplement RGB images in 3D perception applications, such as robotic navigation, autonomous vehicles, virtual reality, and augmented reality. Compared to depth maps estimated from monocular or binocular RGB images, depth sensors can more accurately acquire object distance information. Therefore, depth sensors are widely used in 3D perception applications as the primary method for depth map acquisition. However, due to the immaturity of current technologies, depth maps acquired by depth sensors often contain a large amount of missing depth data. Existing depth sensor technologies mainly include time-of-flight (TOF) technology, structured light technology, and lidar technology. TOF sensors acquire depth maps by emitting infrared light into the scene and measuring the distance between the object and the receiving light. Structured light sensors acquire depth maps by projecting light signals onto the surface of a target object and calculating the distance based on the changes in the light signal caused by the object's surface. However, depth maps acquired by TOF and structured light technologies often suffer from depth data loss due to light refraction and object transparency. Compared to TOF and structured light technologies, lidar technology provides a more robust way to acquire depth maps corresponding to real-world scenes. It acquires point clouds of a scene by emitting laser pulses instead of infrared light, and then converts the point clouds into depth maps. However, due to the sparsity of laser pulses, the depth maps acquired by LiDAR technology are often very sparse. For example, depth maps acquired by low-precision LiDAR devices (e.g., 4 scan lines) typically contain a large amount of missing data. These depth maps with a large amount of missing data will negatively impact related 3D perception applications. For example, robot navigation systems cannot accurately locate targets due to the large number of missing depth values ​​in the depth map, and similar problems have been extensively studied in natural image processing fields such as image inpainting. However, due to the feature differences between natural images and depth maps, inpainting methods are generally not suitable for depth maps. Therefore, how to accurately predict and acquire missing depth values ​​in depth maps has become an important topic in the field of 3D vision in recent years.

[0003] In recent years, some studies have attempted to infer missing content from the original depth map from multiple depth maps. For example, stereo matching techniques are used to estimate missing depth data from two depth maps of the left and right viewpoints. However, in real-world scenes, it is often difficult to obtain multi-view depth maps of the same scene, thus limiting the applicability of these methods. Generally, the structure of a depth map is consistent with the structure of its corresponding RGB image. Therefore, some studies have proposed multimodal methods to address the problem of missing depth map data. These methods typically infer missing data from the original depth map guided by the corresponding RGB image. While multimodal methods can complete missing data in the original depth map, they have two significant drawbacks:

[0004] I. Depth Structure Distortion. Unlike RGB images, depth maps typically consist of multiple flat regions separated by discontinuous depth structures (i.e., object contours). Compared to flat regions, depth structures, often used to represent the geometric information of objects, constitute only a small portion of the entire depth map. Even slight errors near these depth structures can cause severe distortion in related applications (such as 3D reconstruction), making depth structures crucial in depth maps. Traditional multimodal methods, while effectively completing missing data in flat regions, suffer from severe distortion in depth structure completion. This is because these methods typically assume that once flat regions are accurately completed, the depth structure can be accurately predicted. However, this approach struggles to accurately complete missing depth structures. To address the structural distortion problem, some depth completion techniques use local geometric features of the image (e.g., surface normals, image gradients) as constraints to complete the original depth map. However, surface normals are often difficult to calculate accurately near the object's structure, resulting in depth maps generated by these methods still exhibiting significant distortion in depth structure.

[0005] II. Weakly Aligned RGB-D Images. Existing multimodal methods typically assume that the depth map and the RGB image acquired by the RGB camera are accurately aligned. However, RGB-D cameras often experience calibration errors or are affected by noise in real-world scenes. In such cases, the depth map acquired by the RGB-D camera and the RGB image are usually not aligned. Specifically, weakly aligned RGB-D images can be broadly categorized into two types: distorted boundaries and misaligned boundaries. For example, depth maps acquired by depth sensors are easily affected by noise, causing distortion of object boundaries in the original depth map. However, object boundaries in high-quality RGB images are almost unaffected by noise. This results in weak alignment of the same object structures in the depth map and RGB image. Furthermore, calibration errors of the RGB-D camera can cause a slight offset between the entire depth map and the RGB image, resulting in the inability to align all object boundaries in the depth map and RGB image. Existing multimodal completion methods generally rely on structural information provided by the RGB image to complete missing data in the original depth map. Once the RGB-D image is weakly aligned, these methods cannot work accurately. In recent years, some preprocessing (e.g., image registration) and postprocessing (e.g., depth map restoration) methods have been proposed to address the weak alignment problem of RGB-D images in depth completion. Although these methods can improve the accuracy of object boundaries in the completed depth map, they also make the system more complex and are often too time-consuming to be deployed in real-time systems.

[0006] Therefore, how to accurately complete the depth map under the guidance of a weakly aligned RGB-D image has become the key to the depth completion problem. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a depth map completion method and system guided by a weakly aligned RGB-D image, which can solve the technical problem that structural information in the depth map cannot be accurately completed under the guidance of a weakly aligned RGB-D image.

[0008] The present invention adopts the following technical solution:

[0009] A depth map completion method guided by a weakly aligned RGB-D image includes the following steps:

[0010] Input a pair of weakly aligned RGB-D images, and use a Gaussian kernel-based boundary inconsistency detection technique to detect and mark erroneous pixels in the depth map of the weakly aligned RGB-D images, generating depth map inconsistency detection weights;

[0011] Neural network technology is used to divide the depth map after the inconsistency detection algorithm into flat regions and deep structure regions.

[0012] The surface normal feature is used to constrain the smoothness of the depth values ​​in flat regions, resulting in the surface normal constraint term; the Gaussian weight feature is used to constrain the accuracy of the structural information of the depth structure region, and the Gaussian weight is adjusted according to the local texture complexity of the image using a kernel function, resulting in the Gaussian weight constraint term; the data constraint term for the depth map in the original weakly aligned RGB-D image is then constructed.

[0013] A global optimization model is constructed by combining the Markov random field model. Depth completion is defined as a global optimization model. The cost function of the global optimization model is constructed using data constraint terms, surface normal constraint terms, and Gaussian weight constraint terms. The global optimization model is then converted into an iterative optimization model.

[0014] The iterative optimization model is solved iteratively until the improvement rate is lower than the set value or the maximum number of iterations is reached, and then the completed depth map is output.

[0015] Specifically, boundary inconsistency detection involves:

[0016] Given a weakly aligned RGB-D image as input, use a Gaussian kernel to extract the depth Gaussian weights ω from the RGB-D image. D (p,q) and colored Gaussian weights ω c (p,q);

[0017] By measuring the similarity between the color Gaussian weights and the depth Gaussian weights, inconsistency detection weights with values ​​ranging from [0,1] are generated.

[0018] Specifically, using neural network technology to divide a weakly aligned RGB-D image into flat regions and deep structure regions involves:

[0019] Input a weakly aligned RGB-D image, use neural network technology to predict the object boundaries in the RGB-D image, and transform the boundary map into structure weights B, which are used to represent the probability that any pixel belongs to a structure region.

[0020] Set a threshold for dividing structural regions and flat regions. When the probability of a certain pixel in the structural weight B is less than the threshold, the corresponding pixel belongs to the flat region. When the probability of a certain pixel is greater than the threshold, the corresponding pixel belongs to the deep structural region.

[0021] Specifically, the surface normal constraint term E norm for:

[0022]

[0023] Where <·> represents the vector dot product operation, φ(p) denotes the 3×3 neighborhood centered at coordinate point p, (T x ,T y) are the positive x-axis and positive y-axis tangent vectors in the camera coordinate system, respectively. D is the input depth map, and N(p) is the surface normal at coordinate point p.

[0024] Specifically, the Gaussian weight constraint term E Gauss for:

[0025]

[0026] Where ω(p,q) is the colored Gaussian weight, φ(p) represents the 3×3 neighborhood centered at coordinate point p, and D(p) and D... o (p) represents the depth value at pixel p in the completed depth map and the initial depth map, respectively. This is the kernel function.

[0027] Furthermore, kernel functions for:

[0028]

[0029] Where x is the input value, and μ is the texture metric parameter, when the texture richness factor v(p) > threshold v thre When μ is 1; when texture richness factor v(p) ≤ threshold v thre When μ is 8.

[0030] Specifically, data constraint term E Data as follows:

[0031]

[0032] Where D(p) and D o (p) represents the depth value at pixel p in the completed depth map and the initial depth map, respectively.

[0033] Specifically, the iterative optimization model is as follows:

[0034]

[0035] The cost function of the global optimization model is:

[0036]

[0037] Among them, D (n+1) This is the completed depth map output in the (n+1)th iteration, where D is the original depth map, n represents the iteration number, and λ is the depth map output in the (n+1)th iteration. Data ,λ Norm , and λ Gauss These are the regularization parameters for the data constraint term, surface normal constraint term, and Gaussian weight constraint term, respectively, and B is the structural weight.

[0038] Specifically, the termination condition for the iterative solution is as follows:

[0039]

[0040] Where N is the number of pixels in the neighborhood, and D (n) (i) represents the i-th completed depth map output in the n-th iteration, D (n -1) (i) is the i-th completed depth map output in the (n-1)-th iteration.

[0041] Secondly, embodiments of the present invention provide a depth map completion system guided by a weakly aligned RGB-D image, characterized in that it includes:

[0042] The detection module is used to take as input pairs of weakly aligned RGB-D images, and uses a Gaussian kernel-based boundary inconsistency detection technique to detect and mark erroneous pixels in the depth map of the weakly aligned RGB-D images, and generate depth map inconsistency detection weights.

[0043] The partitioning module is used to divide the depth map obtained by the detection module after processing by the inconsistency detection algorithm into flat regions and deep structure regions using neural network technology.

[0044] The constraint module is used to determine the smoothness of the depth values ​​of the flat regions obtained by the surface normal feature constraint partitioning module, thus obtaining the surface normal constraint term; the accuracy of the structural information of the depth structure regions obtained by the Gaussian weight feature constraint partitioning module is determined by adjusting the magnitude of the Gaussian weights according to the local texture complexity of the image using a kernel function, thus obtaining the Gaussian weight constraint term; and the data constraint term for constructing the depth map in the original weakly aligned RGB-D image is also determined.

[0045] The optimization module is used to construct a global optimization model by combining the Markov random field model. It defines depth completion as a global optimization model and uses the data constraint terms, surface normal constraint terms, and Gaussian weight constraint terms constructed by the constraint module to construct the cost function of the global optimization model, and converts the global optimization model into an iterative optimization model.

[0046] The completion module is used to iteratively solve the iterative optimization model obtained by the optimization module until the improvement rate is lower than the set value or the maximum number of iterations is reached, and then outputs a completed depth map.

[0047] Compared with the prior art, the present invention has at least the following beneficial effects:

[0048] This invention presents a depth map completion method guided by weakly aligned RGB-D images. It combines boundary inconsistency detection technology and Markov random field optimization model technology. Based on the guidance of the RGB image, it detects and eliminates erroneous pixels in the depth map, and divides the depth map into regions according to the structural features of the depth map. Then, based on the structural characteristics of different regions, it uses different local geometric features to constrain the depth completion. Therefore, the object structure information in the completion result of this method is clear and accurate. It has a good correction ability for weakly aligned RGB-D images, and the model has high accuracy and strong robustness.

[0049] Furthermore, by using a Gaussian kernel to detect pixels with inconsistent boundaries in the depth map, the detected erroneous pixels are eliminated during the completion process, while the correct pixels are constrained by local geometric features during the completion process.

[0050] Furthermore, by utilizing neural network technology to divide the depth map into different structures, it is divided into flat regions and depth structure regions. Based on the structural characteristics of different regions, different local geometric features are selected for targeted processing to complete a clearer and more accurate object structure.

[0051] Furthermore, by using neural network technology to predict the surface normal features of RGB images and using the generated normal features to complete the constraints of flat regions in the depth map, this process can effectively complete the missing depth data in flat regions, eliminate outliers, and make the depth values ​​in flat regions sufficiently smooth.

[0052] Furthermore, by utilizing a Gaussian kernel to extract structure maps from RGB images and depth maps, and performing a similarity measurement between the depth structure map and the color structure map, a Gaussian structure weight ω is generated. p,q and using ω p,q The structural information represents the constraints on the completion of the deep structural region, making the object structure in the completion result more accurate and clear.

[0053] Furthermore, by utilizing kernel functions The local texture complexity of the RGB image is calculated and the size of the Gaussian weights is dynamically adjusted to weaken the constraint of the Gaussian weights in areas with overly rich textures, thereby alleviating the texture duplication artifact problem caused by the Gaussian kernel.

[0054] Furthermore, by utilizing constraint term E Data Limit the fidelity between the completed depth map and the original depth map to prevent the original depth value from being affected by global optimization.

[0055] Furthermore, by constructing a global optimization model for the depth completion problem using a Markov random field optimization model and iteratively solving the optimization model, a completed depth map without missing depth data is generated. Under the influence of multiple local geometric feature constraints, the final completed depth map has a clear and accurate depth structure and smooth flat regions, and completely eliminates erroneous pixels in the weakly aligned RGB-D image.

[0056] Furthermore, by setting termination conditions for iterative solutions, we can prevent the problem of insufficient iterations leading to incomplete filling of holes in the depth map or excessive iterations leading to wasted computational resources.

[0057] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0058] In summary, the method of the present invention can not only complete the original depth map with severely missing data under the guidance of weakly aligned RGB-D images, but also ensure the structural accuracy and smoothness of the completed depth map.

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0060] Figure 1 This is an overall flowchart of the present invention;

[0061] Figure 2 This is a diagram of the neural network architecture used in this invention;

[0062] Figure 3 The input image of this invention is shown in Figure 1, where (a) is an RGB image and (b) is a raw depth image with missing data acquired by a depth sensor.

[0063] Figure 4 The present invention uses Gaussian kernels to extract color Gaussian weights and depth Gaussian weights, where (a) is an RGB image, (b) is a color Gaussian weight, (c) is a depth map, and (d) is a depth Gaussian weight.

[0064] Figure 5 These are erroneous pixels detected by the inconsistency detection technology of the present invention, wherein (a) is a depth map, (b) is an RGB image, and (c) is the detected erroneous pixel (black area);

[0065] Figure 6 This is the completion result diagram output by the present invention;

[0066] Figure 7This paper presents a subjective comparison between the completed depth map of this invention and the completed depth map of existing completion methods. Among them, (a) is the Non-Local Spatial Propagation Network (NLSPN) completion algorithm, (b) is the Convolutional Spatial Propagation Network (CSPN) completion algorithm, (c) is the Deep Network-based Completion Algorithm (DDC), (d) is the Boundary Consistency and Attention-Based Completion Algorithm (BCSA), (e) is the Normal Vector-Guided Completion Algorithm (SG), and (f) is the completion result of the Spatial Invariant Convolutional Method (SACI). Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0069] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0070] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" relationship.

[0071] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0072] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0073] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0074] This invention provides a depth map completion method guided by weakly aligned RGB-D images. First, paired weakly aligned RGB-D images are input. Error pixels in the depth map are detected and marked using a Gaussian kernel-based boundary inconsistency detection technique. Then, neural network technology is used to divide the depth map of the weakly aligned RGB-D images into flat regions and depth structure regions. For flat regions, surface normal features are selected to constrain the smoothness of the depth values. For depth structure regions, Gaussian weight features are selected to constrain the accuracy of the structural information. To eliminate texture duplication artifacts caused by Gaussian weights, a kernel function is used to automatically adjust the magnitude of the Gaussian weights based on the local texture complexity of the image. Combining a Markov random field optimization model, depth completion is defined as a global optimization model. A cost function for the global optimization model is constructed using data constraints, normal constraints, and Gaussian weight constraints. The solution is iteratively solved until the improvement rate is lower than a set threshold or the maximum number of iterations is reached, outputting a high-quality completed depth map and ending the computation. This invention can not only complete the original depth map with severely missing data under the guidance of weakly aligned RGB-D images, but also ensure the structural accuracy and smoothness of the completed depth map.

[0075] Please see Figure 1 The present invention provides a depth map completion method guided by weakly aligned RGB-D images, comprising the following steps:

[0076] S1, Inconsistent pixel detection;

[0077] S101, Input the weakly aligned depth map as follows Figure 3 (b) and RGB images as shown Figure 3(a) Using a Gaussian kernel, the depth Gaussian weight ω is extracted from the depth map and the RGB image, respectively. D (p, q) and colored Gaussian weights ω c (p, q), the calculation formula is as follows:

[0078]

[0079]

[0080] Where, σ c Let φ(p) be the variance of the Gaussian kernel, φ(p) represent the neighborhood centered at pixel p, and q represent any pixel within that neighborhood.

[0081] S102. Generate inconsistency detection weights by measuring the similarity between the color Gaussian weights and the depth Gaussian weights. The weight ranges from [0,1], reflecting the consistency strength of pixel p. The closer the weight of a pixel is to 0, the more likely pixel p belongs to an inconsistent boundary region within the object. Conversely, when pixel p is not in an inconsistent boundary region within the object, the value approaches 1. The calculation formula is as follows:

[0082]

[0083]

[0084] Here, Thre is the binarization threshold, which is used to weight the inconsistency detection. Binarization simplifies subsequent calculations.

[0085] S2, Depth Map Region Division;

[0086] S201, Input a weakly aligned depth map as follows Figure 3 (b) and RGB images as shown Figure 3 (a) Using neural network technology (network architecture such as Figure 2 (As shown) Predict the object boundaries of the image and transform the boundary map into structural weights B, which are used to represent the probability that any pixel belongs to a structural region.

[0087] S202. Set the threshold for dividing structural regions and flat regions. When the probability of a certain pixel in the structural weight B is less than the threshold, the pixel is considered to belong to the flat region. Conversely, if the probability of a certain pixel is greater than the threshold, the pixel is considered to belong to the deep structural region.

[0088] S3. Extract surface normal features from RGB images to constrain flat regions:

[0089] S301. Based on the structure prediction weight B generated in step S2, surface normal features N are extracted from the flat region corresponding to the RGB image. According to the principle of structural similarity between the depth map and the RGB image, this feature is used to constrain the depth completion of the flat region of the depth map.

[0090] S302. Calculate the corresponding point P(p) of a pixel D(p) in the 3D coordinate system using the camera calibration parameters. The calculation formula is as follows:

[0091]

[0092] Among them, (f x f y (x0, y0) represents the camera focal length, and (x0, y0) represents the coordinates of the camera principal point, i.e., the camera distortion center.

[0093] S303. Take the partial derivative of point P(p) in the three-dimensional spatial coordinate system as the tangent vector in the positive x-axis direction and the tangent vector in the positive y-axis direction (T). x T y The calculation formula is as follows:

[0094]

[0095]

[0096] Among them, T x T is the tangent vector along the positive x-axis of the three-dimensional coordinate system. y It is the tangent vector along the positive y-axis of the three-dimensional coordinate system;

[0097] S304. Based on the orthogonality between the image surface normal and the image tangent vector, it can be known that the dot product of the normal and the tangent vector at any pixel is 0. Using this property, a surface normal constraint term E is established. norm The constraint term is defined as follows:

[0098]

[0099] Where <·> represents the vector dot product operation, φ(p) denotes the 3×3 neighborhood centered at coordinate point p, (T x ,T y These are the tangent vectors along the positive x-axis and positive y-axis in the camera coordinate system, respectively.

[0100] S305. Applying structural prediction weights B to the surface normal constraint term E norm By applying weights, the constraints of surface normals near deep structures are weakened, so that they only play a role in flat regions.

[0101] S4. Extract Gaussian weighted feature constrained deep structure regions from RGB images;

[0102] S401. For a given RGB image I, use a Gaussian kernel to extract its structural features and generate a colored Gaussian weight ω(p,q). Based on the principle of structural similarity between the RGB image and the depth map, the colored Gaussian weight can be used to constrain the depth structure region of the depth map.

[0103] S402. Based on the color Gaussian weights ω(p,q) obtained above, construct the Gaussian weight constraint term E using a weighted least squares model. Gauss The Gaussian weight constraint terms are shown below:

[0104]

[0105] Where ω(p,q) is the colored Gaussian weight, and φ(p) represents the 3×3 neighborhood centered at coordinate point p. This is achieved by minimizing the formula E. Gauss It can constrain the structural similarity between the completed depth map and the RGB image;

[0106] S403. To mitigate the texture replication artifact problem introduced by Gaussian weights, the Gaussian weight constraint term E... Gauss An improved Gaussian weight constraint term is obtained by incorporating a kernel least squares model.

[0107]

[0108] Kernel function The kernel function can adaptively adjust the Gaussian weights based on the local texture complexity of the RGB image. The definition is as follows:

[0109]

[0110] Here, μ is a texture metric parameter that can be adaptively adjusted according to the texture complexity in the RGB image.

[0111] The local texture complexity of an RGB image is represented by the texture richness factor v(p), and the calculation formula is shown below:

[0112]

[0113] Where μ(p) represents a 9×9 neighborhood centered at pixel p in the RGB image, N represents the number of pixels in the window μ(p), and x min (q) and x max (q) represents the minimum and maximum values ​​in a 3×3 neighborhood centered at pixel q, respectively.

[0114] The larger the texture richness factor v(p), the fewer the textures near pixel p, and vice versa.

[0115] Based on the characteristics of the texture richness factor v(p), the value of parameter μ at any pixel can be determined. First, a threshold v is set for v(p). thre Then, based on the texture richness factor v(p), the value of parameter μ can be determined in the following way:

[0116] (1) When v(p)>v thre At that time, the value of parameter μ is set to 1;

[0117] (2) When v(p)≤v thre At that time, the value of parameter μ was set to 8.

[0118] S5. Construct the data constraints for the original depth map;

[0119] For the missing depth map collected by the depth sensor, data constraints are constructed to protect the initial data from being altered and to limit the data fidelity between the completed depth map and the original depth map.

[0120] Data constraint term E Data The definition is as follows:

[0121]

[0122] Where D(p) and D o (p) represents the depth value at pixel p in the completed depth map and the initial depth map, respectively.

[0123] S6. Construct a global optimization model based on the Markov random field model;

[0124] This invention is based on a Markov random field optimization model, defining depth completion as a classic global optimization model. It constructs the cost function of the global optimization model using the proposed surface normal constraints, Gaussian weight constraints, and data constraints. Given the original depth map D... o The cost function of this global optimization model is defined as follows:

[0125]

[0126] Where, λ Data ,λ Norm , and λ Gauss These parameters represent the regularization parameters for the data constraint, surface normal constraint, and Gaussian weight constraint, respectively. These three parameters are mainly used to adjust the strength of the three constraint terms, and their initial values ​​are set to [800, 400, 1].

[0127] To complete depth maps with severely missing depth data and to output more accurate depth structure, the global optimization model is converted into an iterative optimization model, defined as follows:

[0128]

[0129]

[0130]

[0131]

[0132] Where n represents the number of iterations, D (n) This represents the completed depth map output in the nth iteration.

[0133] The update rules for the three regularization parameters are as follows:

[0134]

[0135]

[0136]

[0137] All parameters with superscript (n) represent parameters updated in the nth iteration.

[0138] S7. Solve the global optimization model.

[0139] For the iterative optimization model constructed in step S6, in order to ensure that the output depth map is clear and accurate, the method of iteratively executing the optimization model is used to refine the output results.

[0140] The objective function is solved using the Cholesky decomposition method in each iteration, and the termination condition of the iteration is as follows:

[0141]

[0142] Where N is the number of pixels in the neighborhood, and D (n) (i) represents the i-th completed depth map output in the n-th iteration, D (n-1) (i) is the i-th completed depth map output in the (n-1)-th iteration.

[0143] In another embodiment of the present invention, a depth map completion system guided by a weakly aligned RGB-D image is provided. This system can be used to implement the aforementioned depth map completion method guided by a weakly aligned RGB-D image. Specifically, the depth map completion system guided by a weakly aligned RGB-D image includes a detection module, a segmentation module, a constraint module, an optimization module, and a completion module.

[0144] The detection module is used to input paired weakly aligned RGB-D images, and to detect and mark erroneous pixels in the depth map of the weakly aligned RGB-D images using a Gaussian kernel-based boundary inconsistency detection technique, thereby generating depth map inconsistency detection weights.

[0145] The partitioning module is used to divide the depth map obtained by the detection module after processing by the inconsistency detection algorithm into flat regions and deep structure regions using neural network technology.

[0146] The constraint module is used to determine the smoothness of the depth values ​​of the flat regions obtained by the surface normal feature constraint partitioning module, thus obtaining the surface normal constraint term; the accuracy of the structural information of the depth structure regions obtained by the Gaussian weight feature constraint partitioning module is determined by adjusting the magnitude of the Gaussian weights according to the local texture complexity of the image using a kernel function, thus obtaining the Gaussian weight constraint term; and the data constraint term for constructing the depth map in the original weakly aligned RGB-D image is also determined.

[0147] The optimization module is used to construct a global optimization model by combining the Markov random field model. It defines depth completion as a global optimization model and uses the data constraint terms, surface normal constraint terms, and Gaussian weight constraint terms constructed by the constraint module to construct the cost function of the global optimization model, and converts the global optimization model into an iterative optimization model.

[0148] The completion module is used to iteratively solve the iterative optimization model obtained by the optimization module until the improvement rate is lower than the set value or the maximum number of iterations is reached, and then outputs a completed depth map.

[0149] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0150] Please see Figure 4 The color Gaussian weights and depth Gaussian weights extracted by the Gaussian kernel in this invention clearly show that for RGB images and depth maps in the same scene, their color Gaussian weights and depth Gaussian weights have strong structural consistency. By utilizing this structural consistency principle, the missing structural information in the depth map can be effectively constrained.

[0151] Please see Figure 5 These are the erroneous pixels detected by the inconsistency detection technology of this invention. The black areas in the third column of the image are the inconsistency areas detected by the inconsistency detection algorithm. It can be clearly seen that after the inconsistency areas are eliminated, the depth map structure can maintain structural consistency with the RGB image. This method can effectively eliminate erroneous pixels in the depth map.

[0152] Please see Figure 6 The image shown is the completion result output by this invention. It can be clearly seen that the output completed depth map has a clear and sharp object outline, and the missing depth values ​​are effectively completed, which verifies the good depth map structure completion capability of this invention.

[0153] Please see Figure 7 A subjective comparison between the completed depth map and the completed depth maps of existing completion methods clearly shows that all the comparison methods have serious structural distortion problems near the object outline in the depth map, while the completion result output by the present method has very clear and accurate object structural information. This subjective comparison experiment proves the superiority of the present method in structural completion.

[0154] Table 1. Objective comparison of the depth images after restoration with other methods.

[0155]

[0156] Please refer to Table 1. The method of this invention outperforms the other six comparative methods in all eight indicators. This objective comparative experiment proves the superiority of this method in depth map completion.

[0157] The main benefits of this invention are reflected in two aspects:

[0158] Its main advantage is its ability to correctly complete the structural information of the depth map under the guidance of weakly aligned RGB-D images, compared to... Figure 7 The depth completion results generated by different completion methods show that, under the guidance of weakly aligned RGB-D structures, the depth structure of most methods is severely deformed and distorted, while the completion result of the method described in this invention generates a clear and accurate depth structure that is basically consistent with the object outline in the RGB image.

[0159] Secondly, the method of the present invention can accurately complete depth images with severely missing depth data. For original depth maps with a depth data missing rate of more than 80%, the method of the present invention can still ensure that all missing depth values ​​in the original depth map are completed and generate a complete result with a clear and accurate depth structure.

[0160] In summary, the depth map completion method and system guided by weakly aligned RGB-D images of the present invention can effectively correct the depth map under weak RGB-D alignment, and can effectively complete and restore the missing depth values ​​in the original depth map, while ensuring high structural accuracy and computational efficiency.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A weakly aligned RGB-D image guided depth completion method, characterized in that, The method comprises the following steps: Input a pair of weakly aligned RGB-D images, detect and mark the error pixels of the depth map in the weakly aligned RGB-D images by using a boundary inconsistency detection technology based on a Gaussian kernel, and generate depth map inconsistency detection weights; Divide the depth map processed by the inconsistency detection algorithm into a flat region and a depth structure region by using a neural network technology; Constrain the smoothness of the depth value of the flat region by using a surface normal feature to obtain a surface normal constraint term; constrain the accuracy of the structure information of the depth structure region by using a Gaussian weight feature, adjust the size of the Gaussian weight according to the local texture complexity of the image by using a kernel function, and obtain a Gaussian weight constraint term; and construct a data constraint term of the depth map in the original weakly aligned RGB-D image; Construct a global optimization model in combination with a Markov random field model, define the depth completion as a global optimization model, construct a cost function of the global optimization model by using the data constraint term, the surface normal constraint term and the Gaussian weight constraint term constructed by the constraint module, and convert the global optimization model into an iterative optimization model; Solve the iterative optimization model by iteration until a set improvement rate is lower than or a maximum iteration number is reached, and output a completed depth map.

2. The weakly aligned RGB-D image guided depth completion method of claim 1, wherein, The boundary inconsistency detection specifically comprises: Input weakly aligned RGB-D images, extract depth Gaussian weights ω D (p,q) and color Gaussian weights ω c (p,q) from the RGB-D images using Gaussian kernels An inconsistency detection weight ranging from [0, 1] is generated by measuring the similarity between the color Gaussian weight and the depth Gaussian weight 3. The weakly aligned RGB-D image guided depth completion method of claim 1, wherein, Dividing the weakly aligned RGB-D image into a flat region and a depth structure region by using a neural network technology specifically comprises: Input the weakly aligned RGB-D image, predict the object boundary of the RGB-D image by using a neural network technology, and convert the boundary map into a structure weight B, which is used to represent the probability that any pixel point belongs to a structure region; Set a division threshold of the structure region and the flat region, when the probability of a certain pixel point of the structure weight B is less than the division threshold, the corresponding pixel point belongs to the flat region, and when the probability of a certain pixel point is greater than the division threshold, the corresponding pixel point belongs to the depth structure region.

4. The weakly aligned RGB-D image guided depth completion method of claim 1, wherein, Surface normal constraint term E norm is: where <·> is the vector dot product operation, φ(p) denotes a 3x3 neighborhood centered at coordinate point p, (T x ,T y ) are the tangent vectors in the positive x and y directions of the camera coordinate system, D is the input depth map, and N(p) is the surface normal at coordinate point p.

5. The weakly aligned RGB-D image guided depth completion method of claim 1, wherein, Gaussian weight constraint term E Gauss is: where ω(p, q) is a color Gaussian weight, φ(p) represents a 3x3 neighborhood centered at coordinate point p, D(p) and D o (p) represent the depth values of the completed depth map and the initial depth map at pixel point p, respectively, is a kernel function.

6. The weakly aligned RGB-D image guided depth completion method of claim 5, wherein, Kernel function is: where x is the input value, μ is a texture measure parameter, and when the texture richness factor v(p) > a threshold v thre μ is 1; and when the texture richness factor v(p) ≤ a threshold v thre μ is 8.

7. The weakly aligned RGB-D image guided depth completion method of claim 1, wherein, Data constraint item E Data As follows: where D(p) and D o (p) represent the depth values of the completed depth map and the initial depth map at pixel point p, respectively.

8. The weakly aligned RGB-D image guided depth completion method of claim 1, wherein, The iterative optimization model is: The cost function of the global optimization model is: where D (n+1) is the output of the n+1th iteration, D is the original depth map, n represents the iteration number, λ Data , λ Norm , and λ Gauss are regularization parameters of the data constraint term, the surface normal constraint term, and the Gaussian weight constraint term, respectively, and B is the structure weight.

9. The weakly aligned RGB-D image guided depth completion method of claim 1, wherein, The termination condition of the iterative solution is as follows: where N is the number of pixels in the neighborhood, D (n) (i) is the i-th completed depth map output in the n-th iteration, D (n-1) (i) is the i-th completed depth map output in the n-th iteration, D 10. A weakly aligned RGB-D image guided depth completion system, comprising: It comprises: A detection module configured to input a pair of weakly aligned RGB-D images, detect and mark the error pixels of the depth map in the weakly aligned RGB-D images by using a boundary inconsistency detection technology based on a Gaussian kernel, and generate depth map inconsistency detection weights; A division module configured to divide the depth map processed by the inconsistency detection algorithm obtained by the detection module into a flat region and a depth structure region by using a neural network technology; A constraint module configured to constrain the smoothness of the depth value of the flat region obtained by the division module by using a surface normal feature to obtain a surface normal constraint term; constrain the accuracy of the structure information of the depth structure region obtained by the division module by using a Gaussian weight feature, adjust the size of the Gaussian weight according to the local texture complexity of the image by using a kernel function, and obtain a Gaussian weight constraint term; and construct a data constraint term of the depth map in the original weakly aligned RGB-D image; An optimization module configured to construct a global optimization model in combination with a Markov random field model, define the depth completion as a global optimization model, construct a cost function of the global optimization model by using the data constraint term, the surface normal constraint term and the Gaussian weight constraint term constructed by the constraint module, and convert the global optimization model into an iterative optimization model; The completion module is configured to solve the iterative optimization model obtained by the optimization module by iteration until a set promotion rate is lower than or a maximum iteration number is reached, and output a completed depth map.

Images