A method and apparatus for estimating surface normal vectors of a three-dimensional model

Abstract

The application relates to a three-dimensional model surface normal vector estimation method and device based on multidirectional dynamic programming and iterative polynomial interpolation, wherein the method comprises the following steps: defining a smoothness energy function and a state transition variable of each pixel point based on a smooth energy transfer equation, and initializing the energy function and the state transition variable based on depth and gradient information; calculating the state transition variable based on a path smooth norm, and judging whether to introduce a neighborhood point to complete normal vector estimation of the current point; defining an iterative polynomial interpolation algorithm and an iterative gradient update equation; based on the iterative gradient update equation, the depth gradient is iteratively optimized and updated by using a dynamic programming method for a plurality of directional neighborhood points; and a three-dimensional normal vector is estimated by using a vector cross product method based on the depth gradient. Compared with the prior art, the application has the advantages of high estimation accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and apparatus for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation. Background Technology

[0002] Surface normal vector estimation techniques based on multi-directional dynamic programming and iterative polynomial interpolation aim to generate accurate estimates of 3D object surface normal vectors using structured depth data. This is a key issue in computer vision fields such as 3D scene understanding and semantic segmentation, and it has important significance for applications such as assisting 3D scene understanding, improving the accuracy of semantic segmentation networks, and inversely optimizing depth estimation.

[0003] In related technologies, existing surface normal vector estimation methods have several shortcomings: Existing optimization-based methods, such as PlaneSVD, PlanePCA, VectorSVD, and QuadSVD, use high-precision laser sensors to acquire high-precision point cloud data, and then apply various optimization methods to fit local planes to the 3D point cloud data. While the approach is relatively clear and simple, it suffers from poor real-time performance. Furthermore, the high-density point cloud processing incurs significant computational overhead, hindering large-scale deployment and application in production and daily life, and also limiting its application in fields such as real-time robot navigation.

[0004] Existing mean-based filtering methods, such as AreaWeighted and AngleWeighted, estimate the normal vector of the current point by assigning different weights to the neighboring points of the point to be estimated, using the normal vectors of the neighboring points. However, these methods also incur significant computational overhead. Furthermore, the large differences in normal vectors between different neighboring points at edges introduce substantial estimation noise, resulting in poor accuracy in estimating normal vectors at edges.

[0005] Existing end-to-end estimation methods, such as 3F2N, leverage the inherent geometric correlation between normal vectors and structured depth data, utilizing two gradient filters and a central tendency measurement filter to directly deduce 3D normal vector information end-to-end from depth maps or disparity maps. However, in discontinuous regions on the surface of 3D objects (occlusions, edges, ridges), the accuracy of these normal vector estimators is less than ideal because they often incorrectly introduce adjacent 3D points on different surfaces. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation, thereby improving the accuracy of surface normal vector estimation of a three-dimensional model under certain computational complexity conditions.

[0007] The objective of this invention can be achieved through the following technical solutions: A method for estimating the surface normal vector of a 3D model based on multi-directional dynamic programming and iterative polynomial interpolation includes the following steps: S1. Define the smoothness energy function and state transition variable for each pixel based on the smooth energy transfer equation, and initialize the energy function and state transition variable based on the depth and its gradient information; S2. Calculate the state transition variables based on the path smoothing norm, and determine whether to introduce neighboring points to complete the normal vector estimation of the current point; S3. Define the iterative polynomial interpolation algorithm and the iterative gradient update equation; S4. For neighborhood points in multiple directions, based on the iterative gradient update equation, use dynamic programming to complete the iterative optimization update of the depth gradient; S5. Based on the depth gradient, the three-dimensional normal vector is estimated using the vector cross product method.

[0008] The smoothness energy function is: in, To smooth the energy transfer equation, Represents the current iteration number. Represents the smoothness energy function. , The energy function representing the smoothness of a horizontal path. The energy function representing the smoothness of a vertical path. This is a state transition variable.

[0009] The state transition variable is: in, These are state transition variables used to describe the positions of the neighboring points introduced in the next iteration. , Represents the state transition variables of the horizontal path. Represents the state transition variables for the horizontal path.

[0010] The energy function and state transition variables initialized based on depth and its gradient information are as follows: Utilizing depth Figure 2 The first-order partial derivative defines the weights of the forward and backward difference operators. and initialize depth gradient : in, For the initial depth gradient, For Hadama accumulation, Z For depth, For the forward first-order difference with respect to depth, Backward first-order difference with respect to depth; Initialize smoothness energy function and In adjacent pixels and The minimum value; Initialize state transition variables for .

[0011] The path smoothness norm is: in, For path smoothness norm, functions respectively The second-order partial derivative, This is the integration path.

[0012] The method for calculating state transition variables based on path smoothness norm is as follows: in, Represents the state transition variables of the horizontal path. Represents the state transition variables for horizontal and vertical paths. For path smoothness norm, For depth value, Let the coordinates of the point to be determined be... To meet the conditions All pixels.

[0013] The iterative polynomial interpolation algorithm is as follows: To discrete points The polynomial obtained by interpolation Expand it to the following form: in, They are respectively from and The polynomial obtained by interpolation.

[0014] The iterative gradient update equation is: in,( , ) represents the current pixel, ( , ) represents the estimated depth gradient of that pixel. Let be the number of iteration rounds, ( , ) represents the depth value of that pixel.

[0015] The equation for iterative optimization and updating of the depth gradient using dynamic programming in S4 is as follows: in, These represent the neighboring points on the horizontal, vertical, and diagonal lines, respectively. , ) is an estimated value of the depth gradient of a certain pixel.

[0016] A device for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements the method described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) The surface normal vector estimation method of the present invention is a self-sensing surface normal vector estimation method for spatial discontinuous regions. It has high estimation accuracy and wide applicability.

[0018] (2) The surface normal estimation method of the present invention has a lower estimation error than all other normal vector estimation methods in areas where gradient information is discontinuous, such as the edge of the object surface and the intersection of different planes on the object surface.

[0019] (3) This invention utilizes a multi-directional dynamic programming algorithm to introduce neighborhood points in appropriate directions such as longitudinal, transverse, and diagonal directions to complete the high-precision fitting of the depth function of the object surface curve, ensuring that the normal vector estimation has good adaptability and robustness at the object edge and at the intersection of different planes.

[0020] (4) The present invention uses an iterative multi-directional interpolation algorithm to simplify the fitting calculation complexity, further obtain high-precision depth gradient information, ensure the efficiency of depth gradient calculation, and improve the calculation efficiency.

[0021] (5) This invention only requires depth maps acquired by consumer-grade depth cameras, which is inexpensive. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of point-by-point state transitions during deep gradient updates; Figure 3 A schematic diagram illustrating the path smoothing energy minimization process and gradient update; Figure 4 This is a schematic diagram of the three-dimensional normal vector estimation result in one embodiment. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments. This embodiment provides a method for estimating the surface normal vector of a 3D model based on multi-directional dynamic programming and iterative polynomial interpolation, such as... Figure 1 As shown, it includes the following steps: S1. Define the smoothness energy function and state transition variable for each pixel based on the smooth energy transfer equation, and initialize the energy function and state transition variable based on the depth and its gradient information.

[0024] In this embodiment, the smoothness energy function is: in, To smooth the energy transfer equation, Represents the current iteration number. Represents the smoothness energy function. , The energy function representing the smoothness of a horizontal path. The energy function representing the smoothness of a vertical path. This is a state transition variable.

[0025] In this embodiment, the state transition variable is: in, These are state transition variables used to describe the positions of the neighboring points introduced in the next iteration. , Represents the state transition variables of the horizontal path. Represents the state transition variables for the vertical path.

[0026] In the specific implementation of this invention, it is necessary to know the structured depth information of the target object. The depth information can be stored in PNG format, JPG format or other image formats. At the same time, it is necessary to know the intrinsic parameters of the consumer-grade depth camera used in the detection process.

[0027] After obtaining the depth information, the energy function and state transition variables can be initialized based on the depth and its gradient information. Specifically: Utilizing depth Figure 2 The first-order partial derivative defines the weights of the forward and backward difference operators. and initialize depth gradient : in, For the initial depth gradient, For Hadama accumulation, Z For depth, For the forward first-order difference with respect to depth, Backward first-order difference with respect to depth; Initialize smoothness energy function and In adjacent pixels and The minimum value; Initialize state transition variables for .

[0028] S2. Calculate the state transition variables based on the path smoothing norm, and determine whether to introduce neighboring points to complete the normal vector estimation of the current point.

[0029] The path smoothing norm is: in, For path smoothness norm, functions respectively The second-order partial derivative, This is the integration path.

[0030] In this embodiment, the method for calculating the state transition variable based on the path smoothness norm is as follows: in, Represents the state transition variables of the horizontal path. Represents the state transition variables for the vertical path. For path smoothness norm, For depth value, Let the coordinates of the point to be determined be... To meet the conditions All pixels.

[0031] , The values ​​belong to {-1, 0, 1}, representing three different transition directions: =-1 indicates a leftward shift. =0 indicates no transfer. =1 indicates a rightward transition; if =-1 indicates a downward transfer. =0 indicates no transfer. =1 indicates an upward transfer.

[0032] S3. Define the iterative polynomial interpolation algorithm and the iterative gradient update equation.

[0033] S31. Define the iterative polynomial interpolation algorithm: To discrete points The polynomial obtained by interpolation Expand it to the following form: in, They are respectively from and The polynomial obtained by interpolation.

[0034] S32. Define the iterative gradient update equation: in,( , ) represents the current pixel, ( , This is an estimate of the depth gradient of that pixel. Let be the number of iteration rounds, ( , ) represents the depth value of that pixel.

[0035] S4. For neighborhood points in multiple directions, based on the iterative gradient update equation, use dynamic programming to complete the iterative optimization update of the depth gradient.

[0036] The equation for iterative optimization and updating of the depth gradient using dynamic programming in S4 is as follows: in, These represent the neighboring points on the horizontal, vertical, and diagonal lines, respectively. , ) is an estimated value of the depth gradient of a certain pixel.

[0037] The pointwise state transition process during deep gradient update is as follows: Figure 2 As shown, the path smoothness energy minimization process and gradient update are as follows: Figure 3 As shown.

[0038] S5. Based on the depth gradient, the three-dimensional normal vector is estimated using the vector cross product method.

[0039] Figure 4 A comparison chart of estimation results based on the method of this invention and the SoTA normal vector estimation method is given. Figure 4 It can be found that the method described in this invention has higher estimation accuracy.

[0040] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation, characterized in that, Includes the following steps: S1. Define the smoothness energy function and state transition variable for each pixel based on the smooth energy transfer equation, and initialize the energy function and state transition variable based on depth and its gradient information. The state transition variable is: in, These are state transition variables used to describe the positions of the neighboring points introduced in the next iteration. , Represents the state transition variables of the horizontal path. For the state transition variables of the vertical path; S2. Calculate the state transition variable based on the path smoothing norm, and determine whether to introduce neighboring points to complete the normal vector estimation of the current point. The path smoothing norm is: in, For path smoothness norm, functions respectively The second-order partial derivative, For integration path; The method for calculating state transition variables based on path smoothness norm is as follows: in, For depth value, Let the coordinates of the point to be determined be... To meet the conditions All pixels; S3. Define the iterative polynomial interpolation algorithm and the iterative gradient update equation; S4. For neighborhood points in multiple directions, based on the iterative gradient update equation, use dynamic programming to complete the iterative optimization update of the depth gradient; S5. Based on the depth gradient, the three-dimensional normal vector is estimated using the vector cross product method.

2. The method for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation according to claim 1, characterized in that, The smoothness energy function is: in, To smooth the energy transfer equation, Represents the current iteration number. Represents the smoothness energy function. , The energy function representing the smoothness of a horizontal path. The energy function representing the smoothness of a vertical path. This is a state transition variable.

3. The method for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation according to claim 1, characterized in that, The energy function and state transition variables initialized based on depth and its gradient information are as follows: The weights of the forward and backward difference operators are defined using the second-order partial derivatives of the depth map. and initialize depth gradient : in, For the initial depth gradient, For Hadama accumulation, Z For depth, For the forward first-order difference with respect to depth, Backward first-order difference with respect to depth; Initialize smoothness energy function and In adjacent pixels and The minimum value; Initialize state transition variables for .

4. The method for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation according to claim 1, characterized in that, The iterative polynomial interpolation algorithm is as follows: To discrete points The polynomial obtained by interpolation Expand it to the following form: in, They are respectively from and The polynomial obtained by interpolation.

5. The method for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation according to claim 4, characterized in that, The iterative gradient update equation is: in,( , ) represents the current pixel, ( , This is an estimate of the depth gradient of that pixel. Let be the number of iteration rounds, ( , ) represents the depth value of that pixel.

6. The method for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation according to claim 5, characterized in that, The equation for iterative optimization and updating of the depth gradient using dynamic programming in S4 is as follows: in, These represent the neighboring points on the horizontal, vertical, and diagonal lines, respectively. , ) is an estimated value of the depth gradient of a certain pixel.

7. A device for estimating the surface normal vector of a three-dimensional model based on multi-directional dynamic programming and iterative polynomial interpolation, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-6.

Images