A 2.5d point cloud based normal vector estimation method

Abstract

The application discloses a normal vector estimation method based on a 2.5D point cloud, relates to the technical field of computer vision and computer graphics, and uses a 3D camera to collect a 2.5D point cloud of a target object, converts the 2.5D point cloud into a depth map, so that three-dimensional coordinates of the 2.5D point cloud are mapped into the depth map, each pixel grid of the depth map contains a group of coordinate information, neighborhood points of a target point are found based on the depth map, coordinate information in surrounding pixel grids is extracted with the pixel grid corresponding to the target point as a center, the extracted coordinate information corresponds to the coordinates of the neighborhood points, a directional vector between the neighborhood points and the target point is calculated according to the coordinates of the neighborhood points and the target point, adjacent directional vectors are cross-multiplied two by two and an average value is calculated, and the average value is taken as the normal vector of the target point; under the premise of guaranteeing the availability of a calculation result, the technical scheme of the application quickly finds out the neighborhood points and calculates the normal vector by using the depth map, and the problems of large calculation amount and long time consumption in finding the neighborhood points in the prior art are solved.

Description

Technical Field

[0001] This application relates to the technical fields of computer vision and computer graphics, and more specifically, to a method for estimating normal vectors based on 2.5D point clouds. Background Technology

[0002] Point cloud normal vector estimation is a crucial step in computer vision and computer graphics, applicable to object recognition, 3D reconstruction, surface reconstruction, and point cloud segmentation. Most traditional algorithms for obtaining normal vectors involve constructing a nearest neighbor search data structure to obtain the neighborhood of each point in the 3D data. Then, they use least squares (LS), moving least squares (MLS), or principal component analysis (PCA) to calculate the normal vector for each point. This computationally intensive approach is time-consuming and inefficient. Another example is the covariance matrix-based normal vector estimation method, which uses a KD-Tree (K-Dimensional Tree, a high-dimensional binary tree data structure) to calculate neighborhood points, dividing the point cloud into multiple local regions. Normal vectors are then estimated for each local region, and finally, the normal vectors of the entire point cloud are obtained by concatenating these local region normal vectors. This algorithm is suitable for obtaining point cloud normals in large-scale scenes, but its use in small-scale scenes typically increases computational cost and complexity.

[0003] In recent years, methods using deep learning to calculate normal vectors have yielded considerable results. For example, existing techniques propose a method using deep learning for point cloud upsampling and normal vector estimation to achieve more robust surface reconstruction. Using deep learning-based neural networks, point cloud features can be automatically learned, and problems such as irregular sampling and noise can be handled. However, this method is only suitable for small point cloud models with a limited number of points; for large-scale point clouds, the algorithm is time-consuming. Existing techniques use pre-trained semantic segmentation models to perform semantic segmentation on point cloud data, cluster the point cloud data based on the semantic segmentation results, select the neighborhood points corresponding to each point in the point cloud data based on the clustering results, and calculate the normal vector based on the corresponding neighborhood points. This method eliminates the need for a 3D data structure, improving efficiency to some extent, but the computational load remains significant. Therefore, it is necessary to design an efficient data structure for calculating normal vectors, reducing the heavy computational load and solving the problem of long processing times. Summary of the Invention

[0004] The purpose of this application is to: under the premise of ensuring the usability of the calculation results, the present invention uses a depth map conversion method to compress a three-dimensional data structure into a two-dimensional data structure, and quickly finds neighboring points and calculates normal vectors through the two-dimensional data structure, thereby solving the problem of large amount of calculation and long time consumption in the prior art for finding neighboring points.

[0005] The technical solution of this application is: to provide a normal vector estimation method based on 2.5D point clouds, the method comprising:

[0006] Step 1: Use a 3D camera to collect data from one side of the target object to obtain a 2.5D point cloud of the target object;

[0007] Step 2: Using a virtual telecentric camera, virtually photograph the 2.5D point cloud, convert the acquired 2.5D point cloud into a depth map, and map the three-dimensional coordinates of each point in the 2.5D point cloud onto the depth map so that each pixel of the depth map contains a set of coordinate information.

[0008] Step 3: Based on the depth map, find the neighboring points of the target point. Taking the pixel grid corresponding to the target point as the center, extract the coordinate information of the surrounding pixel grid and use the coordinate information of the surrounding pixel grid as the coordinates of the neighboring points.

[0009] Step 4: Calculate the normal vector of the target point based on the coordinates of the neighboring points and the target point.

[0010] Furthermore, step 2 specifically includes the following steps:

[0011] Step 21: Calculate the bounding box of the 2.5D point cloud, and calculate the length, width and height of the 2.5D point cloud based on the bounding box;

[0012] Step 22: Create a virtual telecentric lens camera model based on the length, width, and height of the 2.5D point cloud;

[0013] Step 23: Convert the 2.5D point cloud into a depth map based on the virtual telecentric lens camera model.

[0014] Further, in step 22, after constructing the bounding box of the 2.5D point cloud, a camera coordinate system is established with the length direction of the 2.5D point cloud as the x-axis, the width direction as the y-axis, and the height direction as the z-axis. The virtual telecentric lens camera model is set at the origin of the camera coordinate system, and the optical axis of the camera is made parallel to the z-axis. The 2.5D point cloud is translated into the camera coordinate system so that all points in the 2.5D point cloud are located in the shooting area below the virtual telecentric lens camera model. The center of the bounding box of the 2.5D point cloud is located directly below the virtual telecentric lens camera model. The coordinates of all points in the 2.5D point cloud are converted into coordinates corresponding to the camera coordinate system.

[0015] Furthermore, in step 23, the specific process of converting the 2.5D point cloud into a depth map based on the virtual telecentric lens camera model is as follows:

[0016] A virtual telecentric lens camera model is used to virtually capture images of a 2.5D point cloud. All points in the 2.5D point cloud are projected onto the xy-plane of the camera coordinate system, forming projection points. The 3D coordinates of the points in the 2.5D point cloud are matched and saved with their projection point positions to generate a depth map. The projection point positions correspond to the pixel positions in the depth map; the width of the depth map corresponds to the length of the 2.5D point cloud; and the height of the depth map corresponds to the width of the 2.5D point cloud. The width direction of the depth map corresponds to the horizontal direction of the image, and the height direction corresponds to the vertical direction of the image. The resolution of the depth map is:

[0017]

[0018]

[0019] in, This represents the horizontal resolution of the depth map. s is the vertical resolution of the depth map, L is the length of the 2.5D point cloud, W is the width of the 2.5D point cloud, s is the scale factor, w is the width of the depth map, and h is the height of the depth map.

[0020] Adjust the scaling factor s so that each pixel in the depth map corresponds to only one projection point. When generating the depth map, match and save the 3D coordinate information of the midpoint of the 2.5D point cloud with its corresponding pixel position. The position of the pixel in the depth map is (u, v). Each pixel contains a set of coordinate information, namely x value, y value and z value. The depth value of the pixel is represented by the z value. The coordinate information contained in the pixel corresponds to the coordinates of the midpoint of the 2.5D point cloud. When the pixel has no corresponding projection point, its coordinate information is set to (NAN, NAN, NAN).

[0021] Furthermore, step 3 specifically includes the following steps:

[0022] Step 31: Using the position (u, v) of the pixel grid corresponding to the target point in the depth map as the center, determine the position of the surrounding pixel grid;

[0023] Step 32: Extract the coordinate information from the surrounding pixel grid, where the coordinate information corresponds to the coordinates of the neighboring points.

[0024] Further, in step 31, the target point for which the normal vector needs to be calculated is determined. The position of the pixel grid corresponding to the target point in the depth map is (u, v). Taking the position of the pixel grid (u, v) as the center, the position of the pixel grid is extended by m pixels to the top, bottom, left and right boundaries to find the position of the surrounding pixel grid. Here, m=1, 2. When m=1, the surrounding 8 pixel grids are searched, and when m=2, the surrounding 24 pixel grids are searched.

[0025] Furthermore, step 4 specifically includes the following steps:

[0026] Step 41: Calculate the direction vector between the target point and its neighboring points using coordinate subtraction. The specific method is as follows:

[0027] The target point's coordinates are (x0, y0, z0), and the coordinates of its neighboring points are (x0, y0, z0). j y j , z j ), with direction vector k i = (x j -x0, y j -y0, z j -z0), where i = 1, 2, 3, ..., n, when m = 1, j = 8, n ≤ 8, when m = 2, j = 24, n ≤ 24, when the coordinate information extracted from the surrounding pixel grid is (NAN, NAN, NAN), the coordinate does not participate in the direction vector calculation;

[0028] Step 42: Perform pairwise cross products on adjacent direction vectors to obtain multiple normal vectors. The calculation formula is as follows:

[0029]

[0030] in, Let i be the normal vector, i = 1, 2, 3, ..., n;

[0031] Step 43: Calculate the average of all normal vectors and use the average as the normal vector of the target point. The calculation formula is as follows:

[0032]

[0033] Where c is the normal vector of the target point, and the normal vectors of all points are calculated according to the above method.

[0034] Furthermore, in step 21, the collected 2.5D point cloud includes ordered point cloud and unordered point cloud;

[0035] When the acquired 2.5D point cloud is an ordered point cloud, its AABB bounding box is calculated, and the center coordinates, as well as the length, width, and height of the point cloud, are calculated based on the AABB bounding box, as follows:

[0036] Traverse all points in the ordered point cloud, and find the minimum and maximum values ​​of x, y, and z coordinates for each point. The minimum and maximum values ​​of x, y, and z coordinates form the two diagonal points of the AABB bounding box. Calculate the center coordinates, length, width, and height of the AABB bounding box using the minimum and maximum coordinates. The length of the AABB bounding box is the difference between the maximum and minimum x values, the width is the difference between the maximum and minimum y values, and the height is the difference between the maximum and minimum z values. The length, width, and height of the bounding box correspond to the length, width, and height of the ordered point cloud, respectively.

[0037] When the acquired 2.5D point cloud is an unordered point cloud, its OBB bounding box is calculated, and the center coordinates, as well as the length, width, and height of the point cloud, are calculated based on the OBB bounding box. The specific steps are as follows:

[0038] Step 211: Decompose the x, y, and z components of all points in the disordered point cloud, and store the x, y, and z values ​​of all points in separate one-dimensional arrays.

[0039] Step 212: Calculate the covariance matrix of the three random variables x, y, and z;

[0040] Step 213: Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and construct the rotation matrix M based on the eigenvalues ​​and eigenvectors;

[0041] Step 214: Translate the disordered point cloud so that its geometric center is located at the origin of the current coordinate system. After translation, multiply the coordinates of all points in the disordered point cloud by the rotation matrix M to achieve the rotation transformation of the entire disordered point cloud and obtain the coordinates of all points after the transformation.

[0042] Step 215: Based on the coordinates of all points after rotation transformation, obtain the maximum and minimum values ​​of the disordered point cloud on the x-axis, y-axis, and z-axis, and then construct the OBB bounding box of the point cloud, and calculate the center coordinates, length, width, and height of the OBB bounding box. The length, width, and height of the bounding box correspond to the length, width, and height of the disordered point cloud, respectively.

[0043] Step 216: Multiply the center coordinates of the OBB bounding box by the inverse of the rotation matrix M to obtain the coordinates of the center before the point cloud is rotated.

[0044] Step 217: Translate the center coordinates of the OBB bounding box to the initial position to obtain its coordinate values ​​at the initial position.

[0045] Further, in step 1, a 3D camera is used to acquire a 2.5D point cloud of the target object. The position of the 3D camera is adjusted so that it can capture the shape of one side of the target object. The 3D camera is used to acquire data of the target object. Based on the acquired 3D data, a 2.5D point cloud of the target object is generated. The 3D coordinates of the points in the 2.5D point cloud correspond to the 3D coordinates of the points on the surface of the target object.

[0046] The beneficial effects of this application are:

[0047] The technical solution in this application converts a 2.5D point cloud into a depth map, so that the three-dimensional coordinate information of all points in the 2.5D point cloud is mapped onto the depth map in an orderly manner. Then, the advantages of the two-dimensional data structure of the depth map are used to find fast neighboring points. The direction vectors of the neighboring points and the target point are calculated based on their coordinates. The normal vector of the target point is obtained quickly by averaging the cross products of adjacent direction vectors. The technical solution in this application reduces the computational complexity and improves the speed of finding neighboring points while ensuring the accuracy of the calculation.

[0048] The technical solution in this application maps the coordinate information of a 2.5D point cloud to a depth map, and uses the depth map to quickly find neighboring points. Compared with the traditional method of finding neighboring points using Kdtree or Octree 3D data structures, the technical solution in this application makes full use of the depth map. Figure 2 The advantage of 3D data structures lies in extracting coordinate information from surrounding pixels centered on the target point, thus obtaining the coordinates of neighboring points and improving the efficiency of neighbor point search. Compared to methods using LS, MLS, and PCA to calculate normal vectors in 3D data structures, this reduces computational load and improves the speed of normal vector calculation while maintaining accuracy, solving the problems of complex and time-consuming algorithms in existing technologies.

[0049] The technical solution in this application can calculate the normal vectors of both ordered and unordered point clouds, and has a wide range of applications. When the point cloud is unordered, the technical solution of this application can restore the orderliness of the point cloud by converting it into a depth map. A coordinate system is established based on the directions of the calculated bounding box's length, width, and height, and the coordinates of all points in the point cloud are mapped sequentially to the pixel grids of the depth map based on this coordinate system, thus restoring the orderliness of the point cloud. When the presence of invalid points in the point cloud causes some pixel grids in the depth map to have missing information, the missing pixel grids are marked with specific values, overcoming the disorderliness of the point cloud.

[0050] The direction of the normal vector calculated using the technical solution of this application is directional. The direction of the calculated direction vector is from the target point to the neighboring point. The normal vector is obtained by taking the average of the cross products of adjacent direction vectors. For the target object as a whole, the normal vector is uniformly from the inside to the outside of the target object plane (or uniformly from the outside to the inside of the target object plane). In contrast, the normal vector obtained by the existing technology has two directions. Compared with the existing technology, the normal vector obtained by the technical solution of this application has the characteristic of being directional. Attached Figure Description

[0051] Figure 1 This is a schematic flowchart of a normal vector estimation method based on 2.5D point cloud according to an embodiment of this application;

[0052] Figure 2 This is a schematic diagram illustrating the search for the surrounding 8 pixels when m is 1 according to an embodiment of this application;

[0053] Figure 3 This is a schematic diagram illustrating the search for the surrounding 24 pixels when m is 2 according to an embodiment of this application;

[0054] Figure 4 This is a schematic diagram illustrating the determination of the position of surrounding pixel grids according to an embodiment of this application;

[0055] Figure 5 This is a normal vector difference map according to an embodiment of this application;

[0056] Figure 6 This is a location map of a target object during identification according to an embodiment of this application;

[0057] Figure 7 This is the result of identifying the target object using the MLS algorithm (the image is in color because color markings are needed to display the recognition results).

[0058] Figure 8 This is the result of identifying the target object using the method of the present invention (the image is a color image because color markings are needed to display the recognition effect). Detailed Implementation

[0059] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other.

[0060] In the following description, many specific details are set forth in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0061] like Figure 1 As shown, this embodiment provides a normal vector estimation method based on 2.5D point clouds, which includes:

[0062] Step 1: Use a 3D camera to collect data from one side of the target object to obtain a 2.5D point cloud of the target object.

[0063] Adjust the position of the 3D camera so that it can capture the shape of one side of the target object. Use the 3D camera to collect data on the target object. Generate a 2.5D point cloud of the target object based on the collected 3D data. The 3D coordinates of the points in the 2.5D point cloud correspond to the 3D coordinates of the points on the surface of the target object.

[0064] In this embodiment, the working principle of the 3D camera used includes, but is not limited to, structured light technology, binocular vision principle and time-of-flight method. For example, a 3D camera based on binocular vision principle includes multiple cameras. These cameras capture images of the target object from different perspectives in the same scene. Based on the images from different perspectives, the three-dimensional coordinates of each pixel on the surface of the object are calculated using the principle of triangulation. These coordinate data constitute a 2.5D point cloud.

[0065] In this embodiment, the 2.5D point cloud of the target object refers to the point cloud obtained by taking a picture (i.e., acquiring data) from one side of the target object using a 3D camera (the optical axis of a 3D camera diverges from the lens). The target object is an object in three-dimensional space. For an object, a single shot only obtains the point cloud corresponding to one face of the object. Each point in the obtained 2.5D point cloud has a set of three-dimensional coordinates, i.e. (x, y, z). In practical applications, when using a 3D camera to photograph a target object, it is necessary to select one or more faces of the object to photograph according to the specific application scenario, and then perform subsequent processing on each face separately. For example, when performing 3D reconstruction of the target object, a 3D camera is used to photograph each face to obtain the 2.5D point cloud of each face, and then these point clouds are processed.

[0066] Step 2: Use a virtual telecentric camera to virtually capture the 2.5D point cloud, convert the acquired 2.5D point cloud into a depth map, and map the three-dimensional coordinates of each point in the 2.5D point cloud onto the depth map so that each pixel in the depth map contains a set of coordinate information.

[0067] The acquired 2.5D point cloud of the target object is converted into a depth map, and the 3D coordinate information of the 2.5D point cloud is mapped to the depth map. The specific steps are as follows:

[0068] Step 21: Calculate the bounding box of the acquired 2.5D point cloud, and calculate the length, width, and height of the 2.5D point cloud based on the bounding box.

[0069] In this embodiment, the collected point clouds include ordered point clouds and unordered point clouds. Ordered point clouds are point cloud data organized according to certain rules or structures, in which the positions of points in the data structure are regular (such as sorted according to the size of coordinate values), and all points in ordered point clouds are valid points (valid points are points that have actual information and contribute to the task). Unordered point clouds are point cloud data that are arranged irregularly or contain invalid points, in which the arrangement of points is random.

[0070] A point cloud bounding box is a virtual rectangular box used to enclose a point cloud, and it's a data structure used to improve computational efficiency. Point cloud bounding boxes are divided into AABB bounding boxes and OBB bounding boxes. Based on the coordinate system of the current 2.5D point cloud, an AABB bounding box is a rectangular box parallel to the coordinate axes, meaning its boundaries are aligned with the coordinate axes. The x-axis value of the AABB bounding box represents the overall length of the 2.5D point cloud, the y-axis value represents the overall width of the 2.5D point cloud, and the z-axis value represents the overall height of the 2.5D point cloud. An OBB bounding box is a rectangular box that can rotate in any direction, and its boundaries are not necessarily parallel to the coordinate axes. OBB bounding boxes can flexibly adapt to the shape of the point cloud.

[0071] In this embodiment, when the acquired 2.5D point cloud is an ordered point cloud, the AABB bounding box of the 2.5D point cloud is calculated, and the center coordinates and the values ​​of length, width, and height are calculated based on the AABB bounding box. Using the AABB bounding box calculation method can more conveniently and quickly obtain the calculation results and apply them to the calculation in subsequent steps. When the acquired 2.5D point cloud is an unordered point cloud, the OBB bounding box of the 2.5D point cloud is calculated, and the center coordinates and the values ​​of length, width, and height are calculated based on the OBB bounding box. Because of the disorder and unstructured nature of the unordered point cloud, its pose may not be aligned with the coordinate axes of the global coordinate system. When calculating the bounding box of the unordered point cloud, it is necessary to calculate its minimum bounding box. Therefore, the OBB bounding box is more suitable for unordered point clouds.

[0072] The specific steps for calculating the AABB bounding box of an ordered point cloud are as follows:

[0073] Traverse all points in the ordered point cloud, and find the minimum and maximum values ​​of the x, y, and z coordinates for each point. The minimum and maximum values ​​of the x, y, and z coordinates form the two diagonal points of the AABB bounding box. Using the minimum and maximum coordinates of the AABB bounding box, calculate the center coordinates, length, width, and height of the bounding box. The length of the bounding box is the difference between the maximum and minimum x values, the width is the difference between the maximum and minimum y values, and the height is the difference between the maximum and minimum z values. The length, width, and height of the bounding box correspond to the length, width, and height of the ordered point cloud, respectively.

[0074] The specific steps for calculating the OBB bounding box of an unordered point cloud are as follows:

[0075] Step 211: Decompose the x, y, and z components of all points in the disordered point cloud, and store the x, y, and z values ​​of all points in separate one-dimensional arrays.

[0076] Step 212: Calculate the covariance matrix of the three random variables x, y, and z. This covariance matrix is ​​a 3×3 matrix representing the covariance relationship among the three variables.

[0077] Step 213: Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and construct a rotation matrix M based on the eigenvalues ​​and eigenvectors (a rotation matrix is ​​a matrix used to orthogonalize the point cloud). In this embodiment, the Jacobi iteration method is used to calculate the eigenvalues ​​and eigenvectors of the covariance matrix.

[0078] Step 214: Translate the disordered point cloud so that its geometric center (obtained by calculating the average of all point coordinates) is located at the origin of the current coordinate system (translation means adding the coordinates of all points in the disordered point cloud to the translation vector). After translation, multiply the coordinates of all points in the disordered point cloud by the rotation matrix M to achieve the rotation transformation of the entire disordered point cloud (rotation transformation is used to standardize and align the point cloud to facilitate the subsequent steps of calculating the bounding box), and obtain the coordinates of all points after transformation.

[0079] Step 215: Based on the coordinates of all points after the rotation transformation, obtain the maximum and minimum values ​​of the unordered point cloud on the x, y, and z axes, i.e. , , , , , Then, the OBB bounding box of the point cloud is constructed, and the center coordinates, length, width and height of the OBB bounding box are calculated (to obtain the position and size of the OBB bounding box). The length, width and height of the bounding box correspond to the length, width and height of the unordered point cloud, respectively.

[0080] Step 216: Multiply the center coordinates of the OBB bounding box by the inverse of the rotation matrix M to obtain the coordinates of the center before the point cloud was rotated.

[0081] Step 217: Translate the center coordinates of the OBB bounding box obtained in step 216 to the initial position to obtain its coordinate values ​​at the initial position (i.e., add the center coordinates of the OBB bounding box to the inverse vector of the translation vector to obtain the initial coordinate values).

[0082] The center coordinates, length, width, and height of the bounding box are obtained through the above calculation method. The length, width, and height of the bounding box correspond to the length, width, and height of the 2.5D point cloud, respectively.

[0083] Step 22: Create a virtual telecentric lens camera model based on the length, width, and height values ​​of the 2.5D point cloud.

[0084] After constructing the bounding box of the 2.5D point cloud, a camera coordinate system is established with the length direction of the 2.5D point cloud as the x-axis, the width direction as the y-axis, and the height direction as the z-axis. The virtual telecentric lens camera model is set at the origin of the camera coordinate system, and the camera's optical axis is parallel to the z-axis. The 2.5D point cloud is then translated into the camera coordinate system so that all points in the 2.5D point cloud are located in the shooting area below the virtual telecentric lens camera model. The center of the bounding box of the 2.5D point cloud is located directly below the virtual telecentric lens camera model (i.e., directly below the origin of the camera coordinate system). The coordinates of all points in the 2.5D point cloud are updated according to the camera coordinate system, that is, the coordinates of all points in the 2.5D point cloud are converted to coordinates corresponding to the camera coordinate system. At this point, the 2.5D point cloud is in a positive state relative to the camera coordinate system, and its boundaries are aligned with the coordinate axes; that is, the sides containing the length, width, and height of the 2.5D point cloud bounding box are parallel to the x-axis, y-axis, and z-axis of the camera coordinate system, respectively.

[0085] In this embodiment, the focal length of the virtual telecentric lens camera model is infinite, and the light rays are assumed to be parallel. When performing perspective projection on all points in the 2.5D point cloud based on the virtual telecentric lens camera, the influence of the focal length is ignored, simplifying the calculation. The coordinate values ​​in the depth map generated based on the virtual telecentric lens camera model are absolute coordinate values ​​corresponding to the coordinate system (i.e., the absolute distance from the point to the camera model).

[0086] Step 23: Convert the 2.5D point cloud into a depth map based on the virtual telecentric lens camera model.

[0087] A virtual telecentric lens camera model is used to virtually capture a 2.5D point cloud below it. All points in the 2.5D point cloud are projected onto the xy-plane of the camera coordinate system in order of their coordinate values, forming projection points. The 3D coordinate information of the points in the 2.5D point cloud is matched and saved with the positions of their projection points to generate a depth map. This involves mapping the 3D coordinates of all points in the 2.5D point cloud to the depth map, where the positions of the projection points correspond to the positions of the pixels in the depth map. The width of the depth map corresponds to the length of the 2.5D point cloud (i.e., the length of the OBB bounding box), and the height of the depth map corresponds to the width of the 2.5D point cloud. In terms of the image, the width direction of the depth map corresponds to the horizontal direction of the image, and the height direction corresponds to the vertical direction of the image. The resolution of the depth map is:

[0088]

[0089]

[0090] in, This represents the horizontal resolution of the depth map. L is the vertical resolution of the depth map, W is the length of the 2.5D point cloud, s is the width of the 2.5D point cloud, s is the scale factor (usually set to 1), w is the width of the depth map, and h is the height of the depth map. and This determines the spatial range of each pixel in the depth map, and each pixel corresponds to the actual size of the 2.5D point cloud. In this embodiment, and They are not necessarily equal. In this embodiment, the generated depth map is a two-dimensional data structure that contains the three-dimensional coordinate information of all points in the point cloud.

[0091] In the process of converting a 2.5D point cloud into a depth map based on a virtual telecentric lens camera model, the position of the projection point corresponds to the position of the pixel grid in the depth map. To facilitate subsequent calculations, the scaling factor s is adjusted so that each pixel grid in the depth map corresponds to only one projection point. When generating the depth map, the 3D coordinate information of the points in the point cloud is matched and saved with their corresponding pixel grid positions. At this point, the 3D coordinates of all points in the 2.5D point cloud are mapped to the depth map, and the position of the pixel grid in the depth map is (u, v). Each pixel grid contains a set of coordinate information, namely x, y, and z values. The depth value of the pixel grid is represented by the z value. The coordinate information in the pixel grid corresponds to the coordinates of the points in the 2.5D point cloud. When a pixel grid has no corresponding projection point, its coordinate information is set to (NAN, NAN, NAN).

[0092] This invention is mainly aimed at converting 2.5D point clouds acquired by 3D cameras using surface structured light or line structured light as light sources into depth maps. The scale factor s is set to 1. At this time, after converting the 2.5D point cloud into a depth map, the coordinate information of the point cloud can be completely preserved. The depth map and the original 2.5D point cloud are completely data-corresponding, with no data loss.

[0093] Step 3: Find the neighboring points of the target point based on the depth map. Taking the pixel grid corresponding to the target point as the center, extract the coordinate information of the surrounding pixel grid and use the coordinate information of the surrounding pixel grid as the coordinates of the neighboring points.

[0094] After converting the 2.5D point cloud into a depth map using the method in step 2, the generated depth map is used to find neighboring points. The specific steps are as follows:

[0095] Step 31: Using the position (u, v) of the pixel grid corresponding to the target point in the depth map as the center, determine the position of the surrounding pixel grid.

[0096] Typically, for each point in a point cloud, its normal vector needs to be calculated. The normal vector is an important attribute describing the geometry of the point cloud surface, indicating the orientation of the surface at that point. In this embodiment, the acquired 2.5D point cloud is first converted into a depth map. The depth map stores the three-dimensional coordinate information of all points in the original 2.5D point cloud. Then, the image advantage of the depth map is used to find neighboring points, which improves the search speed while ensuring search accuracy.

[0097] To determine the target point for which the normal vector needs to be calculated, in the depth map, the position of the corresponding pixel grid is (u, v), where (u, v) represents the pixel grid coordinates in the image, where u represents the horizontal direction (i.e., the column) and v represents the vertical direction (i.e., the row). For example... Figure 4 As shown, to determine the positions of surrounding pixels, we take the pixel position (u, v) as the center and extend m pixels to each of the four boundaries of the pixel (upper, lower, left, and right) to find the positions of the pixels surrounding the center pixel. That is, taking the pixel position (u, v) as the center, the surrounding pixels are the pixels adjacent to the center pixel in eight directions: upper left, upper, upper right, right, lower right, lower, lower left, and left. For example, in the depth map, taking the pixel position (2, 2) as the center and m = 1, the positions of the surrounding pixels are (1, 3), (2, 3), (3, 3), (3, 2), (3, 1), (2, 1), (1, 1), (1, 2).

[0098] In this embodiment, the method of determining neighboring points based on the depth map fully utilizes the advantages of two-dimensional images to quickly find neighboring points. Compared to methods that use three-dimensional data structures to calculate the neighborhood, there is no need to create a Kdtree or Octree, thus significantly reducing the time consumption. Figure 2 This is a diagram illustrating how to find the surrounding 8 pixels when m=1, as shown below. Figure 3 This is a diagram illustrating the process of finding the surrounding 24 pixels when m=2. The larger the value of m, the more neighboring points are found, and the more neighboring points there are, the higher the accuracy of normal vector estimation.

[0099] Step 32: Extract the coordinate information of the surrounding pixel grid, where the coordinate information corresponds to the coordinates of the neighboring points.

[0100] After obtaining the position of the surrounding pixel grid in the depth map, the coordinate information of these surrounding pixel grids is extracted. When m=1, 8 sets of coordinate information are obtained, and when m=2, 24 sets of coordinate information are obtained. These coordinate information correspond to the coordinates of the neighboring points in the 2.5D point cloud.

[0101] Step 4: Calculate the normal vector of the target point based on the coordinates of the neighboring points and the target point.

[0102] After traversing all points using the method described in step 3 and finding all neighboring points, the estimation of the normal vector begins. In this embodiment, the normal vector is calculated using the cross product of adjacent vectors. The specific steps are as follows:

[0103] Step 41: Calculate the direction vector between the target point and its neighboring points using the coordinate subtraction method.

[0104] The target point's coordinates are (x0, y0, z0), and the coordinates of its neighboring points are (x0, y0, z0). j y j , z j ), with direction vector k i = (x j -x0, y j -y0, z j -z0), where i = 1, 2, 3, ..., n, when m = 1, j = 8, n ≤ 8, when m = 2, j = 24, n ≤ 24.

[0105] The coordinates of the target point whose normal vector needs to be calculated are (x0, y0, z0). These coordinates correspond to the coordinate information contained in the center pixel grid in step 3. Let the coordinates of one of the neighboring points be (x1, y1, z1). These coordinates correspond to the coordinate information contained in the surrounding pixel grid. The direction vector obtained by subtracting the two sets of coordinates is k. i = (x1-x0, y1-y0, z1-z0).

[0106] When the coordinate information extracted from the surrounding pixel grid is (NAN, NAN, NAN), this coordinate does not participate in the direction vector calculation.

[0107] Step 42: Perform pairwise cross products on adjacent direction vectors to obtain multiple normal vectors. The calculation formula is as follows:

[0108]

[0109] in, Let be the normal vector (i = 1, 2, 3, ..., n). When there is a pixel in the surrounding pixel grid with coordinate information (NAN, NAN, NAN), the coordinates are not included in the direction vector calculation. The other direction vectors obtained are still calculated as normal vectors by cross product of adjacent vectors.

[0110] like Figure 2 As shown, assuming there is no coordinate information of (NAN, NAN, NAN), when m=1, there are 8 direction vectors, namely k1 to k8. When calculating the normal vector, k1 is multiplied by k2, k2 by k3, and so on until k8 is multiplied by k1, resulting in c1 to c8. Similarly, as... Figure 3 As shown, when m=2, there are 24 direction vectors, namely k1 to k 24 When calculating the normal vector, the cross product of k1 and k2 is used, the cross product of k2 and k3 is used, and so on up to k. 24 After the cross product with k1, we get c1 to c. 24 .

[0111] Step 43: Calculate the average of all normal vectors from step 42, and use this average as the normal vector of the target point. The calculation formula is as follows:

[0112]

[0113] Where c is the normal vector of the target point. The normal vectors of all points are calculated using the method in step 4. In this embodiment, the normal vectors calculated using the method in step 4 are directional, not random. Figure 2 and Figure 3 As shown, when calculating the direction vectors between the target point and its neighboring points, the direction of the vectors is always from the target point to the neighboring points. The normal vector is then obtained by taking the average of the cross products of adjacent direction vectors. For the target object as a whole, all normal vectors are obtained in the same way. The obtained normal vectors are uniformly directed from the inside to the outside of the target object plane (or uniformly directed from the outside to the inside of the target object plane), which shows that the normal vectors are directional. However, the normal vectors obtained by the existing methods have both of the above directions existing at the same time, that is, they are disordered.

[0114] The method of this invention converts a three-dimensional data structure into a two-dimensional data structure, that is, converts a 2.5D point cloud into a depth map, and uses the advantages of the depth map to find nearest neighbor points.

[0115] When acquiring point clouds, some 3D cameras capture unordered point cloud data (i.e., the points in the point cloud data are arranged irregularly or contain invalid points), and do not support storing depth maps; the resulting point cloud only stores valid points. For example, a 3D camera with a width of 2048 and a height of 1536 captures images with 2048×1536 pixels. When this camera captures an unordered point cloud, the points in the point cloud are arranged irregularly, or the number of valid points in the point cloud is less than 2048×1536.

[0116] The method described in this invention for calculating normal vectors can restore the orderliness of point clouds, specifically in the following two aspects:

[0117] Firstly, when the points in the point cloud are arranged irregularly, a bounding box calculation method is used to store the x, y, and z components of all points in the point cloud in independent arrays. Then, a coordinate system is established based on the direction of the length, width, and height of the bounding box. Based on this coordinate system, the coordinates of all points in the point cloud are mapped to the pixel grid of the depth map in order of coordinate value. In the generated depth map, the three-dimensional coordinates corresponding to all pixel grids are arranged in order of coordinate value. In this way, the orderliness of the point cloud is restored when converting the point cloud into a depth map.

[0118] Secondly, when invalid points exist in the point cloud, since the width and height of the camera are known and the number of pixels remains unchanged, after mapping the coordinates of all points in the point cloud to the pixel grid of the depth map, the pixel grid corresponding to the valid points contains three-dimensional coordinates and depth values. Invalid points, because they have no actual information, will not have a mapping relationship with the depth map. Therefore, some pixel grids in the depth map have information gaps. The method of this invention marks the pixel grids with information gaps as (NAN, NAN, NAN), so that each pixel grid in the depth map contains information. In this way, the order of the point cloud can be restored.

[0119] During the conversion of point clouds into depth maps, when the scaling factor s is set to 1, the depth map and the original point cloud have a complete data correspondence with no missing data. If it is set to a value greater than 1, the depth map converted from the point cloud will have a certain proportion of downsampling (i.e., reducing the density of the point cloud), similar to voxel downsampling (in this method, the point cloud is divided into small cubes, and then a representative point is selected in each cube as the representative point of that cube). When using the method of this invention to obtain the normal vector, the scaling factor s can be adjusted according to actual needs to decide whether to perform downsampling, and the depth map can be converted into a downsampled point cloud.

[0120] This embodiment uses a 3D camera with a resolution of 2048*1536 to perform 3D reconstruction of the target object, a bolt, and then identifies the point cloud after reconstruction. During the bolt identification process, the algorithm of this invention and the existing MLS algorithm are compared in terms of performance.

[0121] Normal vector comparison: Based on the result calculated by the MLS algorithm, the difference between the result calculated by the method of this invention and the result is as follows. Figure 5 As shown, the proportion is largest when the normal difference is 0.04, proving that the difference between the normal vector calculated by the method of this invention and the normal vector calculated by the MLS algorithm is very small, and it can be seen that the result is usable.

[0122] Time consumption comparison: The time consumption of a point cloud with more than 3.1 million points (more than 300,000 of which are invalid points) was compared. The time consumption of the MLS algorithm to calculate the normal vector was 2 seconds, while the time consumption of the method of this invention was 30 ms.

[0123] Recognition accuracy: Place the bolt within the camera's field of view, position as follows... Figure 6 As shown, bolts are identified, and the identification results are output. The algorithm of this invention is compared with the existing MLS algorithm. The normal vector calculation step of the MLS algorithm is used as part of the object recognition algorithm to identify the object. The result is: [39.1717mm, -2.3151mm, -3.77342mm, 10.4728°, 345.898°, 44.567°]. The recognition effect is as follows: Figure 7 As shown; the normal vector calculation step of the method of this invention is used as part of the object recognition algorithm to identify the object. The recognition result is: [38.7929mm, -1.92458mm, -3.67328mm, 9.87212°, 345.97°, 44.4146°], and the effect is as follows. Figure 8 As shown, when using the method of this invention for calculation, the recognition effect deviates by 0.3mm compared to the MLS algorithm. The deviation is small, proving that the recognition result calculated using the method of this invention is relatively accurate. In the step of obtaining the normal vector, the speed of the method of this invention is 6 to 7 times faster than the MLS algorithm. Overall, the method of this invention has strong practicality.

[0124] The steps in this application can be rearranged, combined, or deleted according to actual needs.

[0125] The units in the device of this application can be merged, divided, or deleted according to actual needs.

[0126] Although this application has been disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and not intended to limit the application of this application. The scope of protection of this application is defined by the appended claims and may include various variations, modifications, and equivalents of the invention without departing from the scope and spirit of this application.

Claims

1. A method for estimating normal vectors based on 2.5D point clouds, characterized in that, The normal vector estimation method based on 2.5D point clouds includes: Step 1: Use a 3D camera to collect data from one side of the target object to obtain a 2.5D point cloud of the target object; Step 2: Using a virtual telecentric camera, virtually photograph the 2.5D point cloud, convert the acquired 2.5D point cloud into a depth map, and map the 3D coordinates of each point in the 2.5D point cloud onto the depth map, so that each pixel in the depth map contains a set of coordinate information. The specific process of converting the 2.5D point cloud into a depth map is as follows: A virtual telecentric lens camera model is used to virtually capture a 2.5D point cloud. All points in the 2.5D point cloud are projected onto the xy plane of the camera coordinate system in order of their coordinate values, forming projection points. The 3D coordinate information of the points in the 2.5D point cloud is matched and saved with their projection point positions to generate a depth map. The projection point positions correspond to the pixel positions in the depth map, the width of the depth map corresponds to the length of the 2.5D point cloud, and the height of the depth map corresponds to the width of the 2.5D point cloud. The width direction of the depth map corresponds to the horizontal direction of the image, and the height direction corresponds to the vertical direction of the image. The resolution of the depth map is: ； ； in, This represents the horizontal resolution of the depth map. s is the vertical resolution of the depth map, L is the length of the 2.5D point cloud, W is the width of the 2.5D point cloud, s is the scale factor, w is the width of the depth map, and h is the height of the depth map. Adjust the scaling factor s so that each pixel in the depth map corresponds to only one projection point. When generating the depth map, match and save the 3D coordinate information of the midpoint of the 2.5D point cloud with its corresponding pixel position. The position of the pixel in the depth map is (u, v). Each pixel contains a set of coordinate information, namely x value, y value and z value. The depth value of the pixel is represented by the z value. The coordinate information contained in the pixel corresponds to the coordinates of the midpoint of the 2.5D point cloud. When the pixel has no corresponding projection point, its coordinate information is set to (NAN, NAN, NAN). Step 3: Find the neighboring points of the target point based on the depth map. Taking the pixel grid corresponding to the target point as the center, extract the coordinate information of the surrounding pixel grid and use the coordinate information of the surrounding pixel grid as the coordinates of the neighboring points. Step 4: Calculate the normal vector of the target point based on the coordinates of the neighboring points and the target point, including: Step 41: Calculate the direction vector between the target point and its neighboring points using coordinate subtraction. The specific method is as follows: The coordinates of the target point are (x0, y0, z0), and the coordinates of the neighboring points are (x0, y0, z0). j y j , z j The direction vector is k. i = (x j -x0, y j -y0, z j -z0), where i = 1, 2, 3, ..., n, when m = 1, j = 8, n ≤ 8, when m = 2, j = 24, n ≤ 24, when the coordinate information extracted from the surrounding pixel grid is (NAN, NAN, NAN), the coordinate does not participate in the direction vector calculation; Step 42: Perform pairwise cross products on adjacent direction vectors to obtain multiple normal vectors. The calculation formula is as follows: ； in, Let i be the normal vector, i = 1, 2, 3, ..., n; Step 43: Calculate the average of all normal vectors and use this average as the normal vector of the target point. The calculation formula is as follows: ； Where c is the normal vector of the target point, and the normal vectors of all points are calculated according to the above method.

2. The normal vector estimation method based on 2.5D point clouds as described in claim 1, characterized in that, Step 2 specifically includes the following steps: Step 21: Calculate the bounding box of the 2.5D point cloud, and calculate the length, width and height of the 2.5D point cloud based on the bounding box; Step 22: Create a virtual telecentric lens camera model based on the length, width, and height of the 2.5D point cloud; Step 23: Convert the 2.5D point cloud into a depth map based on the virtual telecentric lens camera model.

3. The normal vector estimation method based on 2.5D point clouds as described in claim 2, characterized in that, In step 22, after constructing the bounding box of the 2.5D point cloud, a camera coordinate system is established with the length direction of the 2.5D point cloud as the x-axis, the width direction as the y-axis, and the height direction as the z-axis. The virtual telecentric lens camera model is set at the origin of the camera coordinate system, and the optical axis of the camera is made parallel to the z-axis. The 2.5D point cloud is translated into the camera coordinate system so that all points in the 2.5D point cloud are located in the shooting area below the virtual telecentric lens camera model. The center of the bounding box of the 2.5D point cloud is located directly below the virtual telecentric lens camera model. The coordinates of all points in the 2.5D point cloud are converted into coordinates corresponding to the camera coordinate system.

4. The normal vector estimation method based on 2.5D point clouds as described in claim 1, characterized in that, Step 3 specifically includes the following steps: Step 31: Using the position (u, v) of the pixel grid corresponding to the target point in the depth map as the center, determine the position of the surrounding pixel grid; Step 32: Extract coordinate information from surrounding pixel grids, where the coordinate information corresponds to the coordinates of neighboring points.

5. The normal vector estimation method based on 2.5D point clouds as described in claim 4, characterized in that, In step 31, the target point for which the normal vector needs to be calculated is determined. The position of the pixel grid corresponding to the target point in the depth map is (u, v). Taking the position of the pixel grid (u, v) as the center, the position of the pixel grid is extended by m pixels to the top, bottom, left and right boundaries to find the position of the surrounding pixel grid. Here, m=1, 2. When m=1, the surrounding 8 pixel grids are searched. When m=2, the surrounding 24 pixel grids are searched.

6. The normal vector estimation method based on 2.5D point clouds as described in claim 2, characterized in that, In step 21, the collected 2.5D point cloud includes ordered point cloud and unordered point cloud; When the acquired 2.5D point cloud is an ordered point cloud, its AABB bounding box is calculated, and the center coordinates, as well as the length, width, and height of the point cloud, are calculated based on the AABB bounding box, as follows: Traverse all points in the ordered point cloud, and find the minimum and maximum values ​​of x, y, and z coordinates for each point. The minimum and maximum values ​​of x, y, and z coordinates form the two diagonal points of the AABB bounding box. Calculate the center coordinates, length, width, and height of the AABB bounding box using the minimum and maximum coordinates. The length of the AABB bounding box is the difference between the maximum and minimum x values, the width is the difference between the maximum and minimum y values, and the height is the difference between the maximum and minimum z values. The length, width, and height of the bounding box correspond to the length, width, and height of the ordered point cloud, respectively. When the acquired 2.5D point cloud is an unordered point cloud, its OBB bounding box is calculated, and the center coordinates, as well as the length, width, and height of the point cloud, are calculated based on the OBB bounding box. The specific steps are as follows: Step 211: Decompose the x, y, and z components of all points in the disordered point cloud, and store the x, y, and z values ​​of all points in separate one-dimensional arrays. Step 212: Calculate the covariance matrix of the three random variables x, y, and z; Step 213: Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and construct the rotation matrix M based on the eigenvalues ​​and eigenvectors; Step 214: Translate the disordered point cloud so that its geometric center is located at the origin of the current coordinate system. After translation, multiply the coordinates of all points in the disordered point cloud by the rotation matrix M to achieve the rotation transformation of the entire disordered point cloud and obtain the coordinates of all points after the transformation. Step 215: Based on the coordinates of all points after rotation transformation, obtain the maximum and minimum values ​​of the disordered point cloud on the x-axis, y-axis, and z-axis, and then construct the OBB bounding box of the point cloud, and calculate the center coordinates, length, width, and height of the OBB bounding box. The length, width, and height of the bounding box correspond to the length, width, and height of the disordered point cloud, respectively. Step 216: Multiply the center coordinates of the OBB bounding box by the inverse of the rotation matrix M to obtain the coordinates of the center before the point cloud is rotated. Step 217: Translate the center coordinates of the OBB bounding box to the initial position to obtain its coordinate values ​​at the initial position.

7. The normal vector estimation method based on 2.5D point clouds as described in claim 1, characterized in that, In step 1, a 3D camera is used to acquire a 2.5D point cloud of the target object. The position of the 3D camera is adjusted so that it can capture the shape of one side of the target object. The 3D camera is used to acquire data of the target object. A 2.5D point cloud of the target object is generated based on the acquired 3D data. The 3D coordinates of the points in the 2.5D point cloud correspond to the 3D coordinates of the points on the surface of the target object.

Images