A method for acquiring point cloud data of part surfaces based on 6D pose estimation

Abstract

The present application relates to the field of computer vision, and particularly relates to a part surface point cloud acquisition method based on 6D pose estimation, comprising the following steps: obtaining target object digital model data; estimating and establishing an initial pose relationship matrix between the part and the camera according to point cloud data obtained by a vision sensor; generating surface visible point cloud data under different pose matrices through rasterization technology; screening out a pose set with the least number of complete point cloud data through a minimum visual angle coverage algorithm; obtaining an optimal path when the pose produces motion through a motion cost minimum algorithm; and acquiring point cloud data under corresponding poses and splicing to generate complete point cloud data. The present application uses a deep learning algorithm for pose estimation, effectively realizes automatic and accurate modeling of the pose relationship between the part and the camera, solves the problems of difficult pose adjustment and low efficiency when manually acquiring point cloud data, and guarantees the completeness and rapidity of acquisition, and is particularly suitable for high-precision point cloud data acquisition of complex parts.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a method for acquiring point cloud data of part surfaces based on 6D pose estimation. Background Technology

[0002] In the field of computer vision, 3D object understanding is a challenging research area with applications in many fields, such as augmented reality and robotics. Acquiring the surface point cloud of an object requires accurate reconstruction and pose estimation. Currently, there are two acquisition methods: one is to move the camera around a stationary object, and the other is to rotate the object in front of a stationary camera. In recent years, visual sensors and robotics technologies have become increasingly mature. This invention primarily optimizes the acquisition method that involves placing a stationary object on a worktable and using a robotic arm to move a visual sensor around the object to acquire the surface point cloud.

[0003] Surface point cloud acquisition can be divided into two basic problems: initial pose estimation and subsequent pose point selection. Pose estimation is used to determine the pose relationship between the part and the camera, transforming the spatial states of the part and the camera to the same reference frame. The problem can be described as follows: given a CAD model of the part and partial point cloud data acquired by a vision sensor, calculate the transformation matrix between the part and the camera using an algorithm. Methods for pose estimation are divided into traditional methods and deep learning-based methods. Traditional methods utilize feature matching techniques to obtain the correspondence between feature points in the image or point cloud and the CAD model, and then use traditional numerical algorithms such as PnP or ICP to calculate pose parameters. However, these methods generally lack robustness to complex environmental changes, especially when facing challenges such as low texture, drastic lighting changes, or partial occlusion, where feature extraction and matching are prone to failure. Deep learning-based methods automatically extract features to output pose parameters by training a model, effectively improving the robustness of the algorithm. For example, the FoundationPose method renders a CAD model onto an image from a specific viewpoint. Combined with images captured by a camera, it extracts image features through an encoder and outputs pose refinement and pose score through a decoder to achieve pose estimation. Pose point selection determines subsequent camera pose parameters based on the initial pose to achieve complete acquisition of the surface point cloud. The problem can be described as: given visible point cloud data obtained from rendering a part's CAD model from different viewpoints, solve for an optimal trajectory that can achieve the most complete point cloud data acquisition of the part's surface with the fewest number of viewpoints.

[0004] Existing point cloud acquisition technologies have large errors in pose estimation, and their pose point selection schemes rely on human experience and lack systematic analysis of the geometric features of objects, resulting in redundant sampling paths or omission of key perspectives, leading to incomplete point clouds. At the same time, traditional trial-and-error point placement methods are time-consuming and cannot predict the time required to obtain a complete surface point cloud through quantitative models. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of the existing technology and to provide a method for acquiring point cloud data of part surfaces based on 6D pose estimation.

[0006] The technical solution adopted in this invention is as follows: a method for acquiring point cloud data of a part surface based on 6D pose estimation, the method comprising the following steps:

[0007] Obtain digital model data of the target object, wherein the digital model data of the target object contains M = {V, F}, where V represents a vertex and F represents a triangle;

[0008] The surface of the target object is scanned using a visual sensor to obtain point cloud data of the target object. The initial pose of the part is estimated based on the point cloud data and digital model data of the target object using a trained deep learning model for pose estimation, and the initial pose relationship matrix between the part and the camera is established.

[0009] Surface visible point cloud data under different pose matrices are generated by rasterization technology. Multiple pose matrices are set around the sphere of the part with the camera view facing the part. The pose matrices ensure that the part is covered from all angles. Surface visible point cloud data under these pose matrices are calculated using rasterization technology.

[0010] The minimum view coverage algorithm is used to select the minimum number of viewpoints to achieve complete point cloud data coverage of the part surface, thus obtaining the minimum pose set containing complete point cloud data.

[0011] The optimal path for motion generation during pose is determined using a motion cost minimization algorithm.

[0012] Based on the initial pose relationship matrix and the optimal path, the point cloud camera is controlled to move to each pose sequentially and collect data. An algorithm is used to stitch together the point cloud data collected in each pose to obtain complete point cloud data of the part surface.

[0013] Preferably, the target object digital model data is a CAD model of the target object.

[0014] Preferably, the deep learning model for pose estimation includes:

[0015] The input layer is used to receive point cloud data of the target object, as well as ideal point cloud data generated by rendering based on the digital model data of the target object under a uniformly distributed viewpoint.

[0016] The feature encoder layer extracts point cloud features from the target object point cloud data and the ideal point cloud data through a shared weight encoder. It then splices the two sets of collected point cloud features in the spatial dimension and fuses them in the channel dimension. The encoder simultaneously perceives the interaction relationship between the source point cloud and the target point cloud and outputs global features that cover the global context.

[0017] The output layer has three decoding branches. Global features are fed into the three decoding branches in parallel. Each branch compresses the features into a single vector through a multilayer perceptron and global average pooling. The first branch outputs a three-dimensional offset translation vector to correct the displacement error of the initial pose; the second branch outputs a three-dimensional offset rotation vector to correct the rotation error of the initial pose; and the third branch outputs a confidence score, which indicates the credibility of the current estimated pose, facilitating subsequent filtering.

[0018] Preferably, the target object point cloud data and ideal point cloud data received by the input layer are used as features with three-dimensional coordinates and additional attributes. The features of each point are upscaled point by point through a multilayer perceptron with shared weights, mapping the original low-dimensional geometric information to a high-dimensional feature space, resulting in two sets of upscaled features. The encoder with shared weights extracts point cloud features from the two sets of upscaled features.

[0019] Preferably, the minimum view coverage algorithm for selecting the pose set containing the minimum amount of complete point cloud data includes the following steps:

[0020] S1: Serialize and encode the vertex coordinates of the part's CAD model into an index set. , each camera pose matrix Visible point cloud data below Serialization encoding into an index set Serialization encoding refers to projecting point coordinates onto grid coordinates to obtain integer indices;

[0021] S2: Initialize the set of covered indices For an empty set, the index set subscripts covered are... An empty set, the set of indices not covered. for ;

[0022] S3: In the remaining set list In the middle, calculate each set in turn. With the currently uncovered set The intersection of data, that is, all data belonging to the set And belongs to a set The set consisting of the elements is selected, and the set with the largest intersection is added to the already covered set. In the middle, and update the already covered set;

[0023] S4: Repeat the previous step until the amount of covered data reaches the preset coverage requirement. At this point, the selected set... It can completely cover the point cloud data of the part surface, and obtain the most complete point cloud data coverage of the part surface with the fewest number of views.

[0024] Preferably, the algorithm for minimizing motion cost to find the optimal path when generating motion from pose includes the following steps:

[0025] S1: Use pose as state nodes. Each state node represents the position and attitude of the camera in a specific pose.

[0026] S2: Determine the transition relationship between adjacent poses, that is, the movement and rotation operations that the camera needs to perform when transitioning from one pose to another; calculate the motion cost corresponding to each state transition path;

[0027] S3: Select a starting pose as the initial state node, calculate the motion cost of transitioning from the starting pose to other adjacent poses, and record the corresponding path information;

[0028] S4: For each current state node, traverse all its reachable next state nodes, calculate the transition cost from the current node to the next node, and combine it with the previously recorded path cost to update the minimum cumulative motion cost of the next node and the corresponding optimal path; gradually expand the state nodes backward until all pose nodes are traversed.

[0029] S5: After all pose nodes have been processed, backtrack from the endpoint pose node and obtain the optimal path when the pose is generated based on the recorded optimal path information. This path minimizes the total motion cost of the entire pose transformation process.

[0030] The beneficial effects of this invention are as follows: This invention uses a deep learning algorithm for pose estimation, which effectively realizes the automatic and accurate modeling of the pose relationship between the part and the camera, improves the automation level of point cloud data acquisition, solves the problems of difficult pose adjustment and low efficiency when manually acquiring point cloud data, and ensures the integrity and speed of acquisition, especially suitable for high-precision point cloud data acquisition of complex parts. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0032] Figure 1 The flowchart illustrates a method for acquiring surface point clouds of a part based on 6D pose estimation, as provided in this embodiment of the invention.

[0033] Figure 2 This is an architecture diagram of a network model for pose estimation in a part surface point cloud acquisition method based on 6D pose estimation, provided in an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram illustrating the generation of a spherical surface pose using a part surface point cloud acquisition method based on 6D pose estimation, as provided in an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram of a point cloud acquisition path and a point cloud acquisition method based on 6D pose estimation for a part surface, provided in an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0037] like Figure 1 As shown in the figure, a method for acquiring point cloud data of a part surface based on 6D pose estimation is proposed in an embodiment of the present invention, including the following steps:

[0038] Step 1: Obtain the digital model data of the part: The digital model data contains M = {V, F}, where V represents a vertex and F represents a triangle.

[0039] The digital model can be a model based on various software, and digital model data containing vertex and triangle face data can be directly exported from the software. In this embodiment, it is specifically a CAD model. The file format that supports explicit storage of vertices and triangle faces is selected for export, such as obj format, stl format, and ply format. Then, the vertex coordinates (forming V) and the vertex indices of the triangle faces (forming F) are extracted according to the corresponding format rules.

[0040] Step 2: Estimate the pose relationship between the part and the camera based on the point cloud data acquired by the vision sensor: Use the vision sensor to scan the surface of the part to acquire point cloud data of the part surface. Through a deep learning model for pose estimation, estimate the initial pose of the part and establish the relationship matrix between the part and the camera.

[0041] The pose estimation deep learning model is trained on a large amount of point cloud data labeled with pose information. Its input is point cloud data, and its output is a pose relation matrix. The specific training process is as follows:

[0042] S1. Establish a large number of point cloud datasets labeled with pose information. In this embodiment, the dataset contains 15 different objects, a total of 148,938 training samples and 18,259 test samples. Each sample contains the CAD model of the object, the point cloud of the part surface obtained by the visual sensor and the corresponding pose information.

[0043] S2. Construct the deep learning model for the pose estimation, with the specific structure as follows: Figure 2 As shown, it has:

[0044] Input Layer: This layer first receives point clouds of the part surface acquired by a vision sensor, and then receives ideal point clouds generated from the rendering of the corresponding CAD model under a uniformly distributed viewpoint. Both sets of point clouds use 3D coordinates and additional attributes (such as normal vectors and color) as initial features. A multilayer perceptron with shared weights is used to progressively increase the dimensionality of each point's features, mapping the original low-dimensional geometric information to a high-dimensional feature space, providing the network with more discriminative feature representations. Using a multilayer perceptron with shared weights ensures that both sets of point clouds are processed using the exact same rules.

[0045] Feature Encoder Layer: The two sets of upscaled features are fed into a shared-weight encoder for further point cloud feature extraction. The encoder consists of farthest-point sampling, neighborhood query, and a multilayer perceptron. Farthest-point sampling downsamples both sets of point clouds separately, prioritizing the selection of the "point furthest from existing sampled points" to reduce computation while preserving the global geometric distribution of the point clouds (e.g., part contours, key protrusions / depressions). Neighborhood query finds local neighbors for each downsampled point using K-nearest neighbors (KNN) or sphere query (points within a fixed radius), capturing local geometric relationships between points (e.g., surface curvature, arrangement patterns of adjacent points). The multilayer perceptron performs a nonlinear transformation on each point's "self-features + neighborhood features," abstracting local geometric information into higher-dimensional "local semantic features." Afterward, the network concatenates the two downsampled point clouds spatially and fuses them in the channel dimension. The encoder then simultaneously perceives the interaction between the source and target point clouds, ultimately outputting high-dimensional features covering the global context.

[0046] Output layer: Global features are fed into three decoding branches in parallel. Each branch compresses the features into a single vector using a multilayer perceptron and global average pooling. The first branch outputs a 3D translation vector to correct for displacement errors in the initial pose; the second branch outputs a 3D rotation vector to correct for rotation errors in the initial pose; and the third branch outputs a confidence score, indicating the reliability of the current estimated pose, which facilitates subsequent filtering.

[0047] S3. Construct the loss function to optimize the model parameters. The loss function is described by the following formula:

[0048]

[0049] in , For the actual translation and rotation vectors, , For the predicted translation and rotation vectors, mean squared loss is used to evaluate the difference between the actual and predicted values. and These are represented as positive and negative sample scores, respectively. Positive samples are those whose actual values ​​are close to the predicted values, and negative samples are those whose actual values ​​are far from the predicted values. For comparison margin;

[0050] S4. Train the constructed model, select the AdamW optimizer to optimize the loss function, record the evaluation function on the validation set for each model iteration, and select the model parameters with the highest evaluation index. The evaluation function is described by the following formula:

[0051]

[0052] in , For the actual translation and rotation vectors, , For the predicted translation and rotation vectors, mean squared loss is used to evaluate the difference between the actual and predicted values. For all vertices on the object, This represents the number of samples.

[0053] Step 3: Generate visible point cloud data of the surface under different pose matrices using rasterization technology. Set multiple pose matrices around the spherical surface of the part, with the camera viewpoint facing the part. The pose matrices follow a uniform distribution principle to ensure coverage of the part from all angles. Calculate the visible point cloud data of the surface under these pose matrices using rasterization technology. The specific process is as follows:

[0054] S1. Establish a right-handed world coordinate system with the centroid O of the part as the origin, and set the sphere radius R so that the envelope sphere of the part is completely within the camera's view frustum;

[0055] S2. Perform uniform sampling on the sphere. Given the total number of samples N, let the index i = 0, 1, ..., N-1, and calculate the normalization parameter according to the formula. ,according to Find the polar angle ,according to Find the azimuth angle ,Will Mapping to a sphere yields the three-dimensional coordinates of the i-th sampling point. ;

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] S3. Construct the camera pose matrix for each sampling point. Calculate the camera pose matrix The camera's line-of-sight vector is calculated using the formula. Camera right direction vector Camera orientation vector And combine them into a 4×4 homogeneous transformation matrix. ;

[0061] ;

[0062] ;

[0063] ;

[0064] S4, Rendering Camera Pose Matrix Visible point cloud data below The vertex coordinates and triangular face information of the part's CAD model are read into memory, and then the camera intrinsics are used. and camera pose matrix Construct the camera-projection matrix. The vertices undergo a model-view-projection transformation to obtain homogeneous coordinates. Based on these homogeneous coordinates and triangular face information, obtain the centroid coordinates, normalized depth, and face index of the triangle for each pixel in the rendered image. Then, interpolate the vertex information for each pixel to finally obtain the camera pose matrix. Visible point cloud data below ;

[0065] Generate the spherical pose and render the visible point cloud portion, such as Figure 3 As shown.

[0066] Step 4: A minimum viewpoint coverage algorithm is used to select the fewest possible viewpoints to achieve the most complete point cloud data coverage of the part surface. This algorithm involves selecting the current optimal pose for each selection until a minimum set of poses that can completely cover the point cloud data of the part surface is found. The specific process is as follows:

[0067] S1. Serialize and encode the vertex coordinates of the part's CAD model into an index set. , each camera pose matrix Visible point cloud data below Serialization encoding into an index set Serialization encoding refers to projecting point coordinates onto grid coordinates to obtain integer indices;

[0068] S2, Initialize the covered index set For an empty set, the index set subscripts covered are... An empty set, the set of indices not covered. for ;

[0069] S3, in the remaining set list In the middle, calculate each set in turn. With the currently uncovered set The intersection of data, that is, all data belonging to the set And belongs to a set The set consisting of the elements is selected, and the set with the largest intersection is added to the already covered set. In the middle, and update the already covered set;

[0070] S4. Repeat the previous step until the amount of covered data reaches the preset coverage requirement. At this point, the selected set... It can completely cover the point cloud data of the part surface, and obtain the most complete point cloud data coverage of the part surface with the fewest number of views.

[0071] The pseudocode is as follows:

[0072]

[0073] Step 5: Find the optimal path for pose generation using the minimum motion cost algorithm: The minimum motion cost algorithm uses the shortest path between poses as the state transition equation. By solving the optimal path planning problem, it obtains the optimal path for pose generation, minimizing the total motion cost (such as path length, time, energy consumption, etc.). The specific process is as follows:

[0074] S1. Use pose as state nodes. Each state node represents the position and attitude of the camera in a specific pose.

[0075] S2. Determine the transition relationship between adjacent poses, that is, the movement and rotation operations that the camera needs to perform when transitioning from one pose to another; calculate the motion cost corresponding to each state transition path. The motion cost is comprehensively evaluated based on factors such as the camera's movement distance and stability during the movement process.

[0076] S3. Select a starting pose as the initial state node, calculate the motion cost of transitioning from the starting pose to other adjacent poses, and record the corresponding path information.

[0077] S4. For each current state node, traverse all its reachable next state nodes, calculate the transition cost from the current node to the next node, and combine it with the previously recorded path cost to update the minimum cumulative motion cost of the next node and the corresponding optimal path; gradually expand the state nodes backward until all pose nodes are traversed.

[0078] S5. After all pose nodes have been processed, backtrack from the endpoint pose node and obtain the optimal path when the pose is generated based on the recorded optimal path information. This path minimizes the total motion cost of the entire pose conversion process, thereby ensuring the efficiency of the acquisition process.

[0079] The pseudocode is as follows:

[0080]

[0081]

[0082] Step Six: Collect point cloud data under the corresponding pose and stitch them together to generate complete point cloud data: Based on the initial pose relationship matrix established in Step Two and the optimal path in Step Five, control the point cloud camera to move to each pose in sequence and collect data; use an algorithm to accurately stitch together the point cloud data collected under each pose to obtain complete point cloud data of the part surface.

[0083] like Figure 4 The diagram shown illustrates a data acquisition path and a data acquisition point cloud for a part surface point cloud acquisition method based on 6D pose estimation provided in an embodiment of the present invention.

[0084] This invention provides a simulation of a point cloud acquisition method for part surfaces based on 6D pose estimation. The specific process is as follows: The simulation experiment of this invention was tested on the linemod dataset, and compared with the classic ICP algorithm and the popular FoundationPose method. Table 1 shows the comparison between the method proposed in this invention and other methods on the test dataset.

[0085] Table 1 Evaluation metrics for the ADD-d dataset in the linemod dataset.

[0086]

[0087] Based on the comparison in Table 1, it can be determined that compared with the classic ICP algorithm and the popular FoundationPose method, the method of the present invention has the best average accuracy, and on the vast majority of objects, the ADD-d value is the smallest among the three, indicating better pose estimation accuracy.

[0088] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disk, etc.

[0089] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for acquiring a part surface point cloud based on 6D pose estimation, characterized in that, The method comprises the following steps: Obtain target object digital model data, the content of the target object digital model data includes M = {V, F}, wherein V represents a vertex, and F represents a triangular face; Scan the surface of the target object using a visual sensor to obtain target object point cloud data, estimate the initial pose of the part according to the target object point cloud data and the target object digital model data through a trained pose estimation deep learning model, and establish an initial pose relationship matrix between the part and the camera; Generate surface visible point cloud data under different pose matrices through rasterization technology, set a plurality of pose matrices around the part spherical surface and direct the camera view towards the part, the pose matrix ensures that the part is covered from various angles, and surface visible point cloud data under these pose matrices is calculated by using rasterization technology; Filter out the least number of viewing angles to realize complete point cloud data coverage of the part surface through a minimum viewing angle coverage algorithm to obtain a pose set containing complete point cloud data; Obtain the optimal path of the pose when the pose moves through a motion cost minimum algorithm; Based on the initial pose relationship matrix and the optimal path, control the point cloud camera to move to each pose in turn and collect data; and splice the point cloud data collected under each pose by using an algorithm to obtain complete part surface point cloud data. The pose estimation deep learning model comprises: An input layer for accepting target object point cloud data and ideal point cloud data generated by rendering according to target object digital model data under uniform distribution viewing angles; A feature encoder layer for extracting point cloud features of the target object point cloud data and the ideal point cloud data through a shared weight encoder, and splicing the two groups of collected point cloud features in spatial dimensions and fusing them in channel dimensions, so that the encoder simultaneously perceives the interaction between the source point cloud and the target point cloud, and outputs global features covering global context; An output layer provided with three decoding branches, the global features are sent into the three decoding branches in parallel, each branch compresses the features into a single vector through a multilayer perceptron and a global average pooling, wherein the first branch outputs a three-dimensional offset translation vector for correcting displacement error of the initial pose; the second branch outputs a three-dimensional offset rotation vector for correcting rotation error of the initial pose; and the third branch outputs a confidence score representing the credibility of the current estimated pose, facilitating subsequent screening.

2. The 6D pose estimation based part surface point cloud acquisition method according to claim 1, wherein: The target object digital model data is a CAD model of the target object.

3. The method of claim 1, wherein: The target object point cloud data and the ideal point cloud data accepted by the input layer use three-dimensional coordinates and additional attributes as features, a shared weight multilayer perceptron is used to perform point-by-point dimensionality increase on the features of each point, the original low-dimensional geometric information is mapped to a high-dimensional feature space, and two groups of features after dimensionality increase are obtained, and a shared weight encoder extracts point cloud features from the two groups of features after dimensionality increase.

4. The method of claim 1, wherein: The minimum viewing angle coverage algorithm for screening the pose set containing complete point cloud data includes the following steps: S1: Serialize and encode the vertex coordinates of the part's CAD model into an index set. , each camera pose matrix Visible point cloud data below Serialization encoding into an index set Serialization encoding refers to projecting point coordinates onto grid coordinates to obtain integer indices; S2: initialize the covered index set is empty, the covered index set subscript is empty, the uncovered index set is ; S3: In the remaining set list , each set is calculated in turn with the current uncovered set , the intersection of the data, i.e. the set of elements belonging to both sets and to set , the set with the largest intersection is selected to be added to the covered set and the covered set is updated; S4: repeat the previous step until the amount of covered data reaches the preset coverage requirement, at which time the selected set The part surface point cloud data can be completely covered, and the least number of view angles is used to achieve the most complete point cloud data coverage of the part surface.

5. The method of claim 1, wherein: The motion cost minimum algorithm for obtaining the optimal path of the pose when the pose moves includes the following steps: S1: Pose as a state node, each state node represents the camera in a certain position and attitude of a specific pose; S2: Determine the transition relationship between adjacent poses, that is, when moving from one pose to another, the camera needs to move and rotate; Calculate the motion cost corresponding to each state transition path; S3: Select a starting pose as the initial state node, calculate the motion cost from the starting pose to other adjacent poses, and record the corresponding path information; S4: For each current state node, traverse all the next state nodes that can be reached, calculate the transition cost from the current node to the next node, and update the minimum cumulative motion cost and the corresponding optimal path of the next node by combining the previously recorded path cost; Gradually extend the state nodes backward until all pose nodes are traversed; S5: When all pose nodes have been processed, backtrack from the end pose node, and according to the recorded optimal path information, obtain the optimal path when the pose generates motion, which makes the total motion cost of the entire pose conversion process minimum.

Images