A method for estimating RGB-D surface normal based on fractional differential

Abstract

The application discloses an RGB-D surface normal vector estimation method based on fractional differential. The method firstly performs coordinate system conversion on pixel points of an image, then performs neighborhood point screening on each point in the image, removes hollow points and points with relatively large errors, then solves an algebraic expression of a normal vector by using the property that the normal vector is perpendicular to a plane where the pixel point is located, and finally solves fractional differential optimization gradient to improve the calculation precision of the surface normal vector.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an RGB-D surface normal vector estimation method based on fractional derivatives. Background Technology

[0002] With the widespread use of depth sensors, RGB-D data is becoming increasingly easy to acquire and apply to various visual computing tasks. Depth images provide geometric information such as scene depth, shape, and boundaries, complementing RGB images effectively and thus improving the scene understanding capabilities of visual computing models. Therefore, research on depth information is receiving increasing attention.

[0003] In the field of computer vision, image understanding is one of the most fundamental technologies and an indispensable task. Surface normals are a commonly used visual feature when extracting features from images for semantic understanding. Surface normals contain rich information and have strong interpretability, thus they are often used as auxiliary functions in other vision applications. This auxiliary function is often performed online at the front end of the overall functional framework; therefore, the calculation of surface normals requires high accuracy and high speed.

[0004] Surface normals can be estimated from color images, 3D point clouds, or depth / parallax images. Color images can be obtained using traditional methods such as photometric stereo methods or deep learning methods. For radar point cloud images, the point cloud data we acquire represents a set of fixed-point samples on the surface of a real object. Therefore, there are usually two solutions for solving surface normals. One is surface reconstruction technology, which obtains the surface corresponding to the sampling points from the acquired point cloud dataset and then calculates the surface normals from the surface model. However, although this method has high computational accuracy, it is computationally expensive and time-consuming, so it is rarely used in practical applications. The other is to approximate the surface normals from the point cloud data. Commonly, the problem of solving the normal of a single point on the surface is approximated as the problem of estimating the normal of a tangent plane on the surface, which is a least-squares plane fitting estimation problem. This is accomplished by solving the eigenvectors and eigenvalues ​​of the covariance matrix created by analyzing the nearest neighbors of the query point, but this also requires a lot of computation.

[0005] Therefore, recent research has generally focused on depth maps or disparity maps. Existing methods for calculating surface normal vectors can generally be divided into two categories: those using geometric algebraic properties and those using deep learning neural networks for regression. The former is easily affected by the noise in the image itself and is highly dependent on the calculation method and the optimization techniques used during the calculation process. The latter, on the other hand, requires a large amount of manually labeled datasets as a foundation, and training and final practical testing also require a significant amount of time and computational resources. Summary of the Invention

[0006] The purpose of this invention is to propose a method for calculating surface normal vectors based on fractional derivatives.

[0007] The technical solution adopted to achieve the purpose of this invention is as follows: an RGB-D surface normal vector estimation method based on fractional derivatives, the specific steps of which are as follows:

[0008] 1. Perform coordinate system transformation on the pixels in the image, transforming them from two-dimensional pixels in the image coordinate system to three-dimensional points in the world coordinate system;

[0009] 2. Filter the neighborhood points of pixels in the image to remove holes and neighborhood points with large errors;

[0010] 3. For each pixel in the image, use the perpendicular relationship between the normal vector and the plane containing the pixel to solve for its surface normal vector.

[0011] 4. Convert the obtained surface normal vectors into unit normal vector form in spherical coordinates;

[0012] 5. Combining the definitions of image and mask convolution, construct the mask corresponding to the image f(x,y) in the negative x and y directions, and optimize the gradient calculation method by combining fractional derivatives;

[0013] 6. Traverse the depth image, repeating steps 2-5 until the last pixel of the image, to obtain the complete surface normal vector result map.

[0014] Compared with the prior art, the significant advantages of this invention are:

[0015] (1) It effectively alleviates the image noise problem caused by objective factors such as cameras.

[0016] (2) For the difference between the neighboring point and the current pixel, the first-order gradient is usually used to solve the problem. However, the gradient solution has a large error. The fractional derivative solution is used instead. Compared with the gradient mask, the fractional derivative mask optimizes the gradient value through the diagonal pixels in the neighborhood, making the solved gradient more accurate and improving the calculation accuracy of the surface normal. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention.

[0018] Figure 2 It is a color image.

[0019] Figure 3 It is a surface normal diagram calculated using the method of this invention. Detailed Implementation

[0020] Combination Figure 1This embodiment provides a surface normal vector estimation method based on fractional derivatives, which includes the following steps:

[0021] Combination Figure 2 For a point P[X, Y, Z] in three-dimensional space, the corresponding point p = [x, y] in the image coordinate system is as follows:

[0022]

[0023] Where K is the camera intrinsic parameter matrix, (x0, y0) is the image center, and f x and f y These are the camera focal lengths in the x and y directions, respectively (in pixels).

[0024] For any pixel p in the image, let its 8 neighboring pixels be set Q = {q1, q2, ..., q...} i If the neighboring points q, ..., q8}, are... i Not equal to 0, and |pq i If |≤p*0.1, then retain the neighboring pixel q. i Where q i It refers to one of the eight points in the neighborhood of pixel p.

[0025] For any point P[X, Y, Z] in three-dimensional space, let its surface normal vector be n = [n x n y n z ] T Then the following relationship exists:

[0026] n x X+n y Y+n z Z+d=0

[0027] By combining the relationship between two-dimensional and three-dimensional coordinate systems, we can deduce that:

[0028]

[0029] If we take the derivative of the image in the x-direction, i.e. using a horizontal filter, we can obtain the following formula:

[0030]

[0031] If we take the derivative of the image in the y-direction, i.e., using a vertical filter, we can obtain the following formula:

[0032]

[0033] Combining the above three equations, we can obtain:

[0034] n x=-df x g x

[0035] n y =-df y g y

[0036]

[0037] Therefore, the surface normal vector can be expressed as:

[0038]

[0039] The commonly used form of a unit normal vector is shown below:

[0040]

[0041] Combined with n x With n y The expressions can be listed expression:

[0042]

[0043] To solve for θ, we can use the summation of the 8-neighborhood of p, which is achieved using a mean filter:

[0044]

[0045] Finally, it can be calculated The surface normal vector can be calculated using these two parameters.

[0046] The v-th order GL fractional derivative of signal s(t) can be defined as:

[0047]

[0048] Where s(t) ranges from [a, t], m = (ta) / h, and h is the step size. Γ(·) represents the gamma function, which is defined as:

[0049]

[0050] For an image, the smallest scale is one pixel, so h = 1. If s(t) is divided by h = 1, the difference of the fractional derivative can be expressed as:

[0051]

[0052] Where n is calculated based on the function and corresponds to the nth neighborhood value. For the image f(x, y), the above equation is extended to two dimensions. To simplify the calculation, the first three terms are used for approximation. The numerical implementations of the fractional partial derivatives in the x and y directions are expressed as follows:

[0053]

[0054]

[0055] Therefore, by combining the definitions of image and mask convolution, we can construct a mask corresponding to the fractional derivatives of the image f(x, y) in the negative x and y directions, in order to calculate the difference [ΔX] between the neighboring point and the current pixel. i ΔY i ΔZ i ].

[0056] The difference [ΔX] obtained by solving the above problem using fractional derivatives i ΔY i ΔZ i Substitute The surface normal vector at the current point can be obtained by solving the expression for θ.

[0057] By traversing the image and repeating the above operations, the final surface normal vector result map can be obtained, as shown below. Figure 3 As shown.

Claims

1. A method for estimating RGB-D surface normal vectors based on fractional derivatives, characterized in that, include: Step S1: Perform coordinate system transformation on the pixels in the image, transforming them from two-dimensional pixels in the image coordinate system to three-dimensional points in the world coordinate system; Step S2: Filter the neighborhood points of the pixels in the image to remove empty points and neighborhood points with large errors; Step S3: For each pixel in the image, use the perpendicular relationship between the normal vector and the plane containing the pixel to solve for its surface normal vector; Step S4: Convert the obtained surface normal vector into the form of a unit normal vector in spherical coordinates; Step S5: Construct the image by combining the definitions of image and mask convolution. The masks corresponding to the negative x and y directions are used, and the gradient calculation method is optimized by combining fractional derivatives; specifically including: Signal of The fractional derivative of order GL is defined as: in, The range is ,and Step size; The gamma function is represented by the following definition: For an image, the smallest scale is one pixel, so ;if according to Dividing by, the difference of fractional derivatives can be expressed as: in It is calculated based on a function, corresponding to the first... 1 neighborhood value; for an image The above formula is extended to two dimensions; the first three terms are used for approximate calculation; the numerical implementations of the fractional partial derivatives in the x and y directions are expressed as follows: By combining the definitions of image and mask convolution, we can construct images of both. The mask corresponding to the fractional derivatives in the negative x and y directions is used to calculate the difference between the neighboring points and the current pixel. ; Step S6: Traverse the depth image, repeating steps S2-S5 until the last pixel of the image, to obtain the complete surface normal vector result map.

2. The RGB-D surface normal vector estimation method based on fractional derivatives according to claim 1, characterized in that, Step S1 specifically includes: For a point in three-dimensional space The point corresponding to it in the image coordinate system The corresponding relationships are as follows: Where K is the camera intrinsic parameter matrix, It is the center of the image. and They are direction and The camera focal length in the direction, in pixels.

3. The RGB-D surface normal vector estimation method based on fractional derivatives according to claim 1, characterized in that, Step S2 specifically includes: For any pixel p in the image, let its 8 neighboring pixels be denoted as a set. If the neighboring points Not equal to 0, and Then retain the neighboring pixels. ;in It refers to one of the eight points in the neighborhood of pixel p.

4. The RGB-D surface normal vector estimation method based on fractional derivatives according to claim 1, characterized in that, Step S3 specifically includes: For any point P in three-dimensional space Let its surface normal vector be . Then the following relationship exists: By combining the relationship between two-dimensional and three-dimensional coordinate systems, we can deduce: If we take the derivative of the image in the x-direction, i.e. using a horizontal filter, we can obtain the following formula: If we take the derivative of the image in the y-direction, i.e., using a vertical filter, we can obtain the following formula: Combining the above three equations, we can obtain: Therefore, the surface normal vector is expressed as: 。 5. The RGB-D surface normal vector estimation method based on fractional derivatives according to claim 1, characterized in that, Step S4 specifically includes: The commonly used form of a unit normal vector is shown below: Combination and The expression, list expression: for The solution is obtained by summing the 8-neighborhood of p, i.e., by using a mean filter: Finally, it can be calculated The surface normal vector can be calculated using these two parameters.

Images