A non-continuous region adaptive three-dimensional object normal vector estimation method and device

Abstract

The application relates to a non-continuous region adaptive three-dimensional object normal vector estimation method and device, wherein the method comprises the following steps: preliminarily calculating the components of a three-dimensional space object normal vector in x and y directions based on image inverse depth information and a gradient filter; preliminarily estimating the component of the three-dimensional space object normal vector in the z direction based on a concentration trend filter for the neighborhood points of a target point; calculating a depth curvature based on depth information, wherein the depth curvature of the target point is the maximum value of the depth curvatures of the neighborhood points in each direction; comparing the depth curvature values of the target point and the neighborhood points, and replacing the three-dimensional space object normal vector estimation value of the target point with the three-dimensional space object normal vector estimation value of the point with the minimum depth curvature value by using the preliminarily obtained three-dimensional space object normal vector estimation value. Compared with the prior art, the application has the advantages of high precision and strong robustness to noise data.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and apparatus for adaptive estimation of normal vectors of three-dimensional objects in discontinuous regions. Background Technology

[0002] The end-to-end normal vector estimation technique for non-continuous regions aims to directly estimate the normal vectors of 3D objects based on depth information. As an important visual feature in 3D object recognition, the normal vector plays an auxiliary role in computer vision applications such as semantic segmentation, 3D reconstruction, and 6D object pose estimation.

[0003] Existing normal vector estimation methods and their limitations:

[0004] Existing geometry-based methods, such as PlaneSVD, PlanePCA, and VectorSVD, use statistical analysis or optimization methods to fit local planes or surfaces to 3D point cloud information. These methods have high computational complexity and are not suitable for practical applications. AreaWeighted and AngleWeighted methods calculate weights based on neighborhood triangles of the target point and then estimate the target point's normal vector using the normal vectors of these neighborhood triangles. These methods also have high computational costs, and the significant differences in neighborhood triangles in discontinuous regions can introduce substantial estimation noise, affecting the accuracy of the normal vector estimation results.

[0005] Existing machine / deep learning-based methods typically utilize data-driven classification / regression models, such as Convolutional Neural Networks (CNNs) and Deep Convolutional Neural Networks (DCNNs), to infer normal vector information from RGB or depth images. These methods require finding the optimal CNN parameters through a large amount of manually labeled training data. Furthermore, normal vector estimation in these methods is merely an auxiliary function for computer vision tasks, not a dedicated research area for normal vector estimation. Methods for estimating normal vectors from RGB images cannot clearly define the geometric relationship between the RGB image and the normal vector information. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive method and apparatus for estimating the normal vector of a 3D object in a discontinuous region. Based on depth information, the method uses two gradient filters and a central tendency filter to estimate the normal vector. At the same time, it uses an algorithm based on minimizing depth curvature to optimize the normal vector in the discontinuous region, thereby further improving the estimation accuracy of the normal vector, thus achieving low-cost and high-precision 3D object normal vector estimation.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] An adaptive method for estimating the normal vector of a 3D object in a discontinuous region includes the following steps:

[0009] Step S1: Based on the inverse depth information of the image and the gradient filter, initially calculate the normal vector of the 3D object in space. x , y Components in direction;

[0010] Step S2: For the neighborhood points of the target point, preliminarily estimate the normal vector of the 3D object in space based on the central tendency filter. z Components in direction;

[0011] Step S3: Calculate the depth curvature based on the depth information, where the depth curvature of the target point is the maximum value of the depth curvature of its neighboring points in each direction;

[0012] Step S4: Compare the depth curvature values ​​of the target point and its neighboring points. Using the preliminary 3D object normal vector estimates obtained in steps S1 and S2, replace the 3D object normal vector estimate of the target point with the 3D object normal vector estimate of the point with the smallest depth curvature value.

[0013] Steps S1 and S2 are the process of using three filters to initially estimate the normal vector of the three-dimensional object, while steps S3 and S4 are the process of using the minimum value of depth curvature to optimize the normal vector of the three-dimensional object.

[0014] In step S1, the normal vector of the three-dimensional object is... x , y The method for calculating the directional components is as follows:

[0015]

[0016] in, , The normal vectors are respectively in x , y Components in direction, For depth information, These are the pixel coordinates. and The camera is in , Focal length in direction, inverse depth information Partial derivatives in the horizontal and vertical directions , Approximate calculations are performed using horizontal and vertical image gradient filters, respectively.

[0017] The inverse depth information The partial derivatives in the horizontal and vertical directions are estimated by convolution of the inverse depth image with the image kernel:

[0018]

[0019] Where * denotes convolution, G x and G y These represent the image gradients of the convolution kernel in the horizontal and vertical directions, respectively. Different convolution kernels correspond to... G x and G y different.

[0020] The convolution kernel is a Sobel, Scharr, Prewitt, FD, or Roberts convolution kernel. Specifically, the Sobel, Scharr, Prewitt, and FD kernels convolve the image in both the horizontal and vertical directions, respectively using... The convolution kernel, represented by the Sobel kernel, shows the image gradients in the horizontal and vertical directions as follows:

[0021]

[0022] The image gradients of the Scharr kernel in the horizontal and vertical directions are as follows:

[0023]

[0024] The image gradients of the Prewitt kernel in the horizontal and vertical directions are as follows:

[0025]

[0026] The image gradients of the FD kernel in the horizontal and vertical directions are as follows:

[0027]

[0028] The Roberts convolution kernel performs convolution along the diagonal direction, and its image gradients along the main diagonal and secondary diagonal directions are as follows:

[0029]

[0030] When using the Roberts convolution kernel, because it performs convolution in the diagonal direction, but considering... , The calculations are performed in both the horizontal and vertical directions; therefore, after performing inverse depth image and image convolution kernel convolution, the... G u and G v Angle compensation is performed in the horizontal direction, where the compensation angle is:

[0031]

[0032] In step S2, for the target point neighborhood points Using a central tendency filter to apply the normal vector in The components in the direction are estimated, denoted as... :

[0033]

[0034] in, , The normal vectors are respectively in x , y Components in direction, , The number of neighboring points. This represents a central tendency filter, employing different central tendency measurement methods for neighborhood points. The value is operated on, corresponding to the eight neighboring points. The values ​​are sorted by numerical value and represented as follows: Central tendency measurement methods include eight methods: mean method, median method, trimean method, midhinge method, truncated mean method, winsorized mean method, midrange method, and harmonic mean method.

[0035] The mean method calculates the average of all neighboring points. Value estimation target point normal vector in Components in direction ;

[0036] The median method is used for neighboring points. The values ​​are sorted, and the target point is estimated by solving the median. ;

[0037] Both the three-mean method and the central pivot method calculate the quartiles. The calculation method for the three-mean method is as follows:

[0038]

[0039] The calculation method of the central pivot method is as follows:

[0040]

[0041] The truncated mean method and the extreme value adjusted mean method calculate the mean by removing and replacing extreme values, respectively. The calculation method for the truncated mean method is as follows:

[0042]

[0043] The calculation method for the extreme value adjusted average method is as follows:

[0044]

[0045] The midpoint range method calculates the target point using the maximum and minimum values. :

[0046]

[0047] The harmonic mean method is based on Calculate the target point by the reciprocal :

[0048]

[0049] In step S3, the depth curvature is calculated based on the depth information for the horizontal, vertical, main diagonal, and secondary diagonal directions. This specifically includes the following steps:

[0050] S31: Calculation depth at , First partial derivative in the direction:

[0051]

[0052] in, For depth information, and The camera is in , Focal length in direction, These are the pixel coordinates. Represents the origin of the image. , ;

[0053] S32: Calculate the main diagonal of the depth in the camera coordinate system and the second diagonal First partial derivative in the direction:

[0054]

[0055] S33: Calculation depth at , , , Second partial derivative in direction:

[0056]

[0057]

[0058] in, , ;

[0059] S34: Calculate pixel coordinates , Main diagonal and the second diagonal Depth curvature :

[0060]

[0061]

[0062]

[0063]

[0064] S35: Determine the depth curvature of the target point p:

[0065]

[0066] In step S4, the point with the minimum depth curvature value is:

[0067]

[0068] in, q i The target point represents the neighborhood point, and p represents the pixel coordinates. This represents the depth curvature at point a;

[0069] Then the estimated value of the target point normal vector ,in, n a This is the estimated normal vector value of the point with the minimum depth curvature.

[0070] A device for adaptive 3D object normal vector estimation in discontinuous regions includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] (1) This invention is an end-to-end normal vector estimation method, and the normal vector estimation accuracy is better than other traditional normal vector estimators.

[0073] (2) Since most objects in an image are irregular, the number of large-area planes on an object is limited, and the dominant method is usually based on known three-dimensional information, which involves a large amount of computation and is difficult to obtain. Therefore, the method of solving the normal vector through a plane has limitations. The end-to-end normal vector estimation method of this invention is fundamentally different from the dominant normal vector estimation algorithm. Instead of estimating the normal vector of each plane, it estimates the normal vector of each pixel. Therefore, the normal vector of each pixel can be compared, thereby effectively distinguishing whether the surfaces of different objects are on the same plane in a two-dimensional image. At the same time, the computational complexity of normal vector estimation in a two-dimensional image is greatly reduced compared to three-dimensional information, and it is more efficient.

[0074] (3) The present invention optimizes the normal vector estimation in discontinuous regions by calculating the depth curvature in each direction to represent the curvature of the surface, thus ensuring the robustness of the normal vector estimation at the edge.

[0075] (4) This invention only requires depth maps acquired by consumer-grade depth cameras, which is inexpensive. Attached Figure Description

[0076] Figure 1 This is a flowchart of the method of the present invention;

[0077] Figure 2 This is a schematic diagram of depth curvature;

[0078] Figure 3 This is a schematic diagram illustrating the normal vector estimation effect of one embodiment of the present invention. Detailed Implementation

[0079] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0080] This embodiment provides an adaptive method for estimating the normal vectors of 3D objects in discontinuous regions, such as... Figure 1 As shown, it includes the following steps:

[0081] Step S1: Based on the inverse depth information of the image and two gradient filters, initially calculate the normal vector of the 3D object in space. x , y Components in direction.

[0082] In this embodiment, the structured depth information of the target object needs to be known. The depth information can be stored in PNG, JPG or other image formats. At the same time, the intrinsic parameters of the consumer-grade depth camera used in the detection process need to be known.

[0083] The normal vector is approximated by performing horizontal and vertical image gradient filtering on the inverse depth image. , The components in the direction are denoted as follows: , :

[0084]

[0085] in, , The normal vectors are respectively in x , y Components in direction, For depth information, These are the pixel coordinates. and The camera is in , Focal length in direction, inverse depth information Partial derivatives in the horizontal and vertical directions , Approximate calculations are performed using horizontal and vertical image gradient filters, respectively.

[0086] Specifically, inverse depth information The partial derivatives in the horizontal and vertical directions are estimated by convolution of the inverse depth image with the image kernel:

[0087]

[0088] Where * denotes convolution, G x and G y These represent the image gradients of the convolution kernel in the horizontal and vertical directions, respectively. Different convolution kernels correspond to... G x and G y different.

[0089] In this embodiment, the convolution kernel can be a Sobel, Scharr, Prewitt, FD, or Roberts convolution kernel. Specifically, the Sobel, Scharr, Prewitt, and FD kernels convolve the image in both the horizontal and vertical directions, respectively using... The convolution kernel, represented by the Sobel kernel, shows the image gradients in the horizontal and vertical directions as follows:

[0090]

[0091] The image gradients of the Scharr kernel in the horizontal and vertical directions are as follows:

[0092]

[0093] The image gradients of the Prewitt kernel in the horizontal and vertical directions are as follows:

[0094]

[0095] The image gradients of the FD kernel in the horizontal and vertical directions are as follows:

[0096]

[0097] The Roberts convolution kernel performs convolution along the diagonal direction, and its image gradients along the main diagonal and secondary diagonal directions are as follows:

[0098]

[0099] When using the Roberts convolution kernel, because it performs convolution in the diagonal direction, but considering... , The calculations are performed in both the horizontal and vertical directions; therefore, after performing inverse depth image and image convolution kernel convolution, the... G u and G v Angle compensation is performed in the horizontal direction, applying the Roberts calculations to both the horizontal and vertical directions. The compensation angle is:

[0100]

[0101] Step S2: For the neighborhood points of the target point, preliminarily estimate the normal vector of the 3D object in space based on the central tendency filter. z Components in direction.

[0102] For the target point neighborhood points Using a central tendency filter to apply the normal vector in The components in the direction are estimated, denoted as... :

[0103]

[0104] in, , The normal vectors are respectively in x , y Components in direction, , The number of neighboring points. This represents a central tendency filter, employing different central tendency measurement methods for neighborhood points. The value is operated on, corresponding to the eight neighboring points. The values ​​are sorted by numerical value and represented as follows: Central tendency measurement methods include eight methods: mean method, median method, trimean method, midhinge method, truncated mean method, winsorized mean method, midrange method, and harmonic mean method.

[0105] The mean method calculates the average of all neighboring points. Value estimation target point normal vector in Components in direction .

[0106] Median method for neighborhood points The values ​​are sorted, and the target point is estimated by solving the median. .

[0107] Both the three-means method and the central pivot method calculate the quartiles, among which...

[0108] Three-means method:

[0109]

[0110] Central hub approach:

[0111]

[0112] The truncated mean method and the extreme value adjusted mean method calculate the mean by removing and replacing extreme values, respectively.

[0113] Truncated mean method:

[0114]

[0115] Methods for adjusting the average based on extreme values:

[0116]

[0117] The midpoint method calculates the target point using the maximum and minimum values. :

[0118]

[0119] The harmonic mean method is based on Calculate the target point by the reciprocal :

[0120]

[0121] Step S3: Calculate the depth curvature based on the depth information, where the depth curvature of the target point is the maximum value of the depth curvature of its neighboring points in each direction.

[0122] A schematic diagram of depth curvature is shown below. Figure 2 As shown. In this embodiment, the depth curvature is calculated based on depth information for the horizontal, vertical, main diagonal, and secondary diagonal directions, specifically including the following steps:

[0123] S31: Calculation depth at , First partial derivative in the direction:

[0124]

[0125] in, For depth information, and The camera is in , Focal length in direction, These are the pixel coordinates. Represents the origin of the image. , ;

[0126] S32: Calculate the main diagonal of the depth in the camera coordinate system and the second diagonal First partial derivative in the direction:

[0127]

[0128] S33: Calculation depth at , , , Second partial derivative in direction:

[0129]

[0130]

[0131] in, , ;

[0132] S34: Calculate pixel coordinates , Main diagonal and the second diagonal Depth curvature :

[0133]

[0134]

[0135]

[0136] S35: Determine the depth curvature of the target point p:

[0137]

[0138] Step S4: Compare the depth curvature values ​​of the target point and its neighboring points. Using the preliminary 3D object normal vector estimates obtained in steps S1 and S2, replace the 3D object normal vector estimate of the target point with the 3D object normal vector estimate of the point with the smallest depth curvature value.

[0139] Specifically, first determine the point with the minimum depth curvature value:

[0140]

[0141] in, q i The target point represents the neighborhood point, and p represents the pixel coordinates. This represents the depth curvature at point a.

[0142] Then, determine the target point normal vector estimate. ,in, n a This is the estimated normal vector value of the point with the minimum depth curvature.

[0143] In the above steps, steps S1 and S2 are the process of initially estimating the normal vector of the three-dimensional object using three filters, and steps S3 and S4 are the process of optimizing the normal vector of the three-dimensional object using the minimum value of depth curvature.

[0144] Figure 3 This example shows a comparison chart of normal vector estimation results. Figure 3 It can be observed that the normal vector estimation method described in this invention has higher estimation accuracy.

[0145] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for adaptive estimation of the normal vector of a three-dimensional object in a discontinuous region, characterized in that, Includes the following steps: Step S1: Based on the inverse depth information of the image and the gradient filter, initially calculate the normal vector of the 3D object in space. x , y Components in direction; Step S2: For the neighborhood points of the target point, preliminarily estimate the normal vector of the 3D object in space based on the central tendency filter. z Components in direction; Step S3: Calculate the depth curvature based on the depth information, where the depth curvature of the target point is the maximum value of the depth curvature of its neighboring points in each direction; Step S4: Compare the depth curvature values ​​of the target point and its neighboring points. Using the preliminary 3D object normal vector estimates obtained in steps S1 and S2, replace the 3D object normal vector estimate of the target point with the 3D object normal vector estimate of the point with the smallest depth curvature value.

2. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 1, characterized in that, In step S1, the normal vector of the three-dimensional object is... x , y The method for calculating the directional components is as follows: in, , The normal vectors are respectively in x , y Components in direction, For depth information, These are the pixel coordinates. and The camera is in , Focal length in direction, inverse depth information Partial derivatives in the horizontal and vertical directions , Approximate calculations are performed using horizontal and vertical image gradient filters, respectively.

3. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 2, characterized in that, The inverse depth information The partial derivatives in the horizontal and vertical directions are estimated by convolution of the inverse depth image with the image kernel: Where * denotes convolution, G x and G y These represent the image gradients of the convolution kernel in the horizontal and vertical directions, respectively. Different convolution kernels correspond to... G x and G y different.

4. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 3, characterized in that, The convolution kernel is a Sobel, Scharr, Prewitt, FD, or Roberts convolution kernel.

5. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 4, characterized in that, When using Roberts convolution kernels, considering , The calculations are performed in the horizontal and vertical directions. Therefore, after convolving the inverse depth image with the image kernel, the result is... G u and G v Angle compensation is performed in the horizontal direction, where the compensation angle is: 。 6. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 1, characterized in that, In step S2, for the target point Neighboring points Using a central tendency filter to apply the normal vector in The components in the direction are estimated, denoted as... : in, , The normal vectors are respectively in x , y Components in direction, , The number of neighboring points. This represents a central tendency filter, employing different central tendency measurement methods for neighborhood points. The value is operated on, corresponding to the eight neighboring points. The values ​​are sorted by numerical value and represented as follows: Central tendency measurement methods include eight methods: mean method, median method, three-mean method, central pivot method, truncated mean method, extreme value adjusted mean method, midpoint of range method, and harmonic mean method.

7. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 6, characterized in that, The mean method calculates the average of all neighboring points. Value estimation target point normal vector in Components in direction ; The median method is used for neighboring points. The values ​​are sorted, and the target point is estimated by solving the median. ; Both the three-mean method and the central pivot method calculate the quartiles. The calculation method for the three-mean method is as follows: The calculation method of the central pivot method is as follows: The truncated mean method and the extreme value adjusted mean method calculate the mean by removing and replacing extreme values, respectively. The calculation method for the truncated mean method is as follows: The calculation method for the extreme value adjusted average method is as follows: The midpoint range method calculates the target point using the maximum and minimum values. : The harmonic mean method is based on Calculate the target point by the reciprocal : 。 8. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 1, characterized in that, In step S3, the depth curvature is calculated based on the depth information for the horizontal, vertical, main diagonal, and secondary diagonal directions. This specifically includes the following steps: S31: Calculation depth is , First partial derivative in the direction: in, For depth information, and The camera is in , Focal length in direction, These are the pixel coordinates. Represents the origin of the image. , ; S32: Calculate the main diagonal of the depth in the camera coordinate system and the second diagonal First partial derivative in the direction: S33: Calculation depth at , , , Second partial derivative in direction: in, , ; S34: Calculate pixel coordinates , Main diagonal and the second diagonal Depth curvature : S35: Determine the depth curvature of the target point p: 。 9. The method for adaptive estimation of 3D object normal vectors in discontinuous regions according to claim 1, characterized in that, In step S4, the point with the minimum depth curvature value is: in, q i The target point represents the neighborhood point, and p represents the pixel coordinates. This represents the depth curvature at point a; Then the estimated value of the target point normal vector ,in, n a This is the estimated normal vector value of the point with the minimum depth curvature.

10. A device for adaptive estimation of normal vectors of a three-dimensional object in a discontinuous region, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-9.

Images