Circuit breaker assembly point cloud data registration method and device, medium and product

Through random sampling consistency algorithm, statistical filtering and improved voxelization downsampling, combined with ISS and Harris3D algorithm, the problem of redundant points and outliers in point cloud registration of circuit breaker components is solved, and the registration accuracy is improved and efficiency, achieving more accurate component pose determination.

CN119027469APending Publication Date: 2024-11-26TECH INST OF WENZHOU UNIV YUEQING
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411124413.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

During the point cloud registration process of circuit breaker components, the prior art has the problems of messy point cloud data, redundant points and outlier points, resulting in low matching efficiency and poor accuracy.

Method used

The random sampling consistency algorithm is used to remove redundant points, the statistical filtering method removes outliers, and the improved voxelization downsampling method is used to streamline data, and combine ISS and Harris3D with a two-step solution of rough registration first and then accurate registration. The algorithm extracts key points and uses the method curvature constraint 4PCS algorithm to improve registration accuracy.

Benefits of technology

Effectively remove irrelevant points, improve the accuracy and matching accuracy of point cloud data, and enhance the certainty of the pose of the circuit breaker component.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027469A_ABST
    Figure CN119027469A_ABST
Patent Text Reader

Abstract

The invention discloses a circuit breaker assembly point cloud data registration method and device, a medium and a product, and relates to the technical field of circuit breaker assembly, and the method comprises the steps: removing redundant points and outliers in original point cloud data, and obtaining the point cloud data after the outliers are removed; processing the point cloud data after the outliers are removed by adopting an improved voxelization down-sampling method to obtain filtered point cloud data; the improved points of the improved voxelization downsampling method comprise the steps of constraining the side length of voxels through point cloud density in a unit voxel, and replacing the center of gravity in each voxel by adopting a nearest neighbor search algorithm; dividing the filtered point cloud data into a plurality of point cloud clusters, wherein each point cloud cluster corresponds to one circuit breaker assembly; and on the basis of the template point cloud of each circuit breaker assembly, registering each point cloud cluster by adopting a two-step registration scheme of firstly rough registration and then accurate registration to obtain a rigid transformation matrix for registering each circuit breaker assembly. According to the invention, the accuracy of point cloud data registration of the circuit breaker assembly is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of circuit breaker assembly, and in particular to a circuit breaker component point cloud data registration method, equipment, medium and product. Background Art

[0002] A circuit breaker is a switching device that can close, carry and disconnect current under normal circuit conditions and can close, carry and disconnect current under abnormal circuit conditions within a specified time. Circuit breakers are widely used in industrial and civil fields. In the flexible assembly process of automated manufacturing of circuit breakers, it is necessary to visually identify and locate the circuit breaker components to guide the robot to grab and install them. The point cloud registration technology of components based on stereo vision can solve the problems of metal reflection and blurred edges in two-dimensional vision. At present, in the point cloud registration process of circuit breaker components, the point cloud data obtained by camera scanning has the difficulties of messy point cloud data and unclear component information, which easily leads to problems such as low matching efficiency and poor matching accuracy. Summary of the invention

[0003] The object of the present invention is to provide a circuit breaker component point cloud data registration method, device, medium and product, which improve the accuracy of circuit breaker component point cloud data registration.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A circuit breaker component point cloud data registration method, comprising:

[0006] The original point cloud data is fitted by a random sampling consistency algorithm to obtain point cloud data after redundant points are removed; the original point cloud data is point cloud data including a plurality of circuit breaker components;

[0007] Using a statistical filtering method to remove outliers in the point cloud data after removing redundant points, to obtain point cloud data after removing outliers;

[0008] The point cloud data after the outliers are removed is processed by an improved voxel downsampling method to obtain filtered point cloud data; the improved voxel downsampling method includes constraining the voxel side length by the point cloud density within the unit voxel, and replacing the centroid point within each voxel by a nearest neighbor search algorithm;

[0009] Segmenting the filtered point cloud data into a plurality of point cloud clusters, each of the point cloud clusters corresponding to a circuit breaker component;

[0010] Based on the template point cloud of each circuit breaker component, a two-step registration scheme of rough registration followed by precise registration is adopted to register each point cloud cluster, and a rigid transformation matrix for registration of each circuit breaker component is obtained; the rigid transformation matrix is ​​used to determine the position and posture of the corresponding circuit breaker component.

[0011] A computer device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the circuit breaker component point cloud data registration method.

[0012] A computer-readable storage medium stores a computer program, which implements the circuit breaker component point cloud data registration method when executed by a processor.

[0013] A computer program product comprises a computer program. When the computer program is executed by a processor, the circuit breaker component point cloud data registration method is implemented.

[0014] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0015] The present invention removes redundant points and discrete points from the original point cloud data of the circuit breaker component, and then simplifies the point cloud data through improved voxel downsampling, thereby removing the influence of irrelevant points on subsequent registration, and improving the accuracy of the point cloud data to be registered. In addition, by merging the key points extracted by the ISS algorithm and the Harris3D algorithm, the obtained key point set combines the advantages of the key points extracted by the ISS algorithm and the Harris3D algorithm, and improves the accuracy of the coarse registration. The normal curvature of the point is used to constrain the 4PCS algorithm, and the point pairs with obvious differences in normal curvature are eliminated, thereby improving the accuracy of the fine registration, and then improving the accuracy of determining the position and posture of the circuit breaker component. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic flow chart of a circuit breaker component point cloud data registration method provided by an embodiment of the present invention;

[0018] Figure 2 A point cloud image in the preprocessing process provided by an embodiment of the present invention;

[0019] Figure 3A schematic diagram of an STL model file and a template point cloud provided in an embodiment of the present invention;

[0020] Figure 4 A schematic diagram of a segmentation flow chart provided by an embodiment of the present invention;

[0021] Figure 5 A segmentation visualization schematic diagram provided by an embodiment of the present invention;

[0022] Figure 6 The overall framework diagram of the point cloud registration solution provided by the embodiment of the present invention;

[0023] Figure 7 A schematic diagram of affine invariant coplanar four points provided by an embodiment of the present invention;

[0024] Figure 8 This is a schematic diagram of common mismatching;

[0025] Fig. 9 Schematic diagram of the improved rough registration algorithm provided by the embodiment of the present invention

[0026] Fig.10 A schematic diagram of the geometric relationship of variables provided in an embodiment of the present invention;

[0027] Fig.11 A schematic diagram of an experimental process provided by an embodiment of the present invention;

[0028] Fig.12 A schematic diagram of extracting key points of an arc extinguishing chamber provided in an embodiment of the present invention;

[0029] Fig.13 Schematic diagram of key point extraction of other components provided by embodiments of the present invention;

[0030] Fig.14 Schematic diagram of the registration results of various registration methods provided in the embodiments of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The object of the present invention is to provide a circuit breaker component point cloud data registration method, device, medium and product, which improve the accuracy of circuit breaker component point cloud data registration.

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Example 1

[0035] like Figure 1 As shown, a circuit breaker component point cloud data registration method in this embodiment includes the following steps.

[0036] Step 101: using a random sampling consistency algorithm to fit original point cloud data, to obtain point cloud data after removing redundant points; the original point cloud data is point cloud data including a plurality of circuit breaker components.

[0037] Step 102: using a statistical filtering method to remove outlier points in the point cloud data after removing redundant points, to obtain the point cloud data after removing outlier points.

[0038] Step 103: using an improved voxelization downsampling method to process the point cloud data after removing outliers to obtain filtered point cloud data; the improvements of the improved voxelization downsampling method include constraining the voxel side length by the point cloud density within the unit voxel, and replacing the center of gravity point within each voxel by using the nearest neighbor search algorithm.

[0039] Step 104: Segment the filtered point cloud data into multiple point cloud clusters, each of the point cloud clusters corresponding to a circuit breaker component.

[0040] Step 105: Based on the template point cloud of each circuit breaker component, a two-step registration scheme of rough registration followed by precise registration is adopted to register each point cloud cluster to obtain a rigid transformation matrix for registering each circuit breaker component; the rigid transformation matrix is ​​used to determine the position and posture of the corresponding circuit breaker component.

[0041] In this embodiment, the circuit breaker is specifically a small circuit breaker.

[0042] Among them, step 101-step 104 is the process of preprocessing and segmenting the point cloud data of the circuit breaker component. First, the characteristics of the original point cloud data are analyzed. In view of the problem that the existence of irrelevant points makes the point cloud data messy and the component information is not prominent, a method for simplifying the original point cloud data is proposed to remove irrelevant points. After processing the original point cloud, the 3D CAD model of each component is converted into a surface point cloud model as a matching template for extracting point cloud features, laying the foundation for subsequent alignment work. In addition, the simplified point cloud data contains multiple different types of components. In order to separate a single point cloud cluster from the point cloud data so that each point cloud cluster corresponds to a circuit breaker component, the simplified point cloud data is segmented.

[0043] Among them, the preprocessing includes the processing of the original point cloud data and the three-dimensional CAD model of the component. The scanning accuracy of the three-dimensional camera used in this embodiment to collect the original point cloud data is as high as 0.035mm, which makes the amount of original point cloud data huge. In addition, the component itself is made of metal. The laser emitted by the camera during scanning and imaging will be reflected to other positions in the space, which will cause a large number of points far away from the target in the obtained point cloud data, that is, outliers; the stage background within the camera's field of view will also be presented in the source point cloud (original point cloud data), forming a large number of data points that are irrelevant to the component and are large in number, that is, redundant points. The large number and density of the original point cloud will cause the subsequent data processing process to take too long, and the presence of outliers and redundant points will affect the accuracy of data processing. For CAD models, they have a higher resolution than the model point cloud, and will also contain redundant points after being converted into point cloud data.

[0044] Among them, step 101 uses the random sampling consensus algorithm (RANSAC) to fit the plane in the source point cloud to remove redundant points, which specifically includes:

[0045] 1) Select any three points from the source point cloud to fit a plane (plane model).

[0046] 2) Calculate the distance from the remaining points (points in the source point cloud except the above three points) to the plane. If the distance from the point (remaining point) to the plane is less than the threshold, the point is a point in the plane (inner point). Otherwise, it does not belong to the plane, and count the number of inner points under the plane model.

[0047] 3) Repeat the above two steps until the plane model with the largest number of internal points is found and marked, and the plane model parameters are saved.

[0048] 4) Finally, traverse all points in the source point cloud, calculate the distance from the point to the fitted plane, set a threshold, and retain the point index whose distance is outside the threshold range to obtain the source point cloud after removing redundant points.

[0049] Among them, the statistical filtering method in step 102 determines whether it is an outlier point by density, specifically including: calculating the mean and standard deviation of the Euclidean distance from each point in the point cloud data after removing redundant points to each point in its neighborhood. The area of ​​a point refers to the range within a set distance centered on the point.

[0050] For the i-th point in the point cloud data after removing redundant points, if the average value of the Euclidean distances of each point in its neighborhood is greater than the distance threshold corresponding to the i-th point, the i-th point is removed as an outlier; the distance threshold corresponding to the i-th point is determined according to the mean and standard deviation of the Euclidean distances from the i-th point to each point in its neighborhood, and i is an integer greater than 0 and less than or equal to the number of points in the point cloud data after removing redundant points.

[0051] Step 103 is a process of reducing the number of point clouds.

[0052] The voxel downsampling method is used to reduce the number of point clouds, where a voxel is a small cube in three-dimensional space, which can be understood as a pixel in a two-dimensional image. This method takes the minimum voxel side length as the only input parameter and uses the centroid of all points in the voxel to approximate the other points in the voxel.

[0053] There are two problems with the voxel downsampling method: 1. The voxel side length needs to be determined by empirical values, which requires a lot of repeated experiments and has no basis. If it is set too long, the number of points contained in the unit voxel will be too large. If too many points are replaced by only one point, local features will be lost. If it is set too short, the filtering effect will not be obvious. 2. The center of gravity obtained is not necessarily the point in the original point cloud, and subtle features will be lost.

[0054] The present invention improves the traditional voxel filtering in two aspects in view of these two problems.

[0055] 1. The present invention proposes a method of constraining the voxel side length by the point cloud density within the unit voxel and iteratively estimating the voxel side length to simplify the process of establishing the parameter.

[0056] 2. Use the k-nearest neighbor search algorithm to search for the point closest to the above-mentioned center of gravity in the original point cloud data to replace the center of gravity and form a new point set.

[0057] Wherein, step 103 specifically includes:

[0058] 1) According to the coordinates of the point cloud data after removing the outliers, the volume of the minimum bounding box of the point cloud data after removing the outliers is calculated, specifically: according to the coordinates of the point cloud data after removing the outliers, the maximum value x on the X, Y, and Z axes is obtained. max ,y max 、z max and the minimum value x min ,y min 、z min , so as to obtain the minimum bounding box side lengths L X , L Y , L Z .

[0059] The volume V of the minimum bounding box is: V = L x L y L z .

[0060] 2) Determine the point cloud density of the point cloud data after removing outliers according to the volume of the minimum bounding box, and estimate the voxel side length according to the point cloud density, specifically:

[0061] Assuming the total number of point clouds is N, the point cloud density is ρ, the voxel side length is L, and the average number of point clouds per unit voxel is n, then we have the equation:

[0062]

[0063] 0×L 3 =n

[0064] Where N is a known quantity, the voxel side length L' can be estimated as:

[0065] 3) Divide the minimum bounding box into voxels according to the estimated voxel side length, specifically:

[0066] The bounding box is divided into M = u × v × w voxels by the estimated voxel side length L', where u, v, and w are the number of voxels in the X, Y, and Z directions of the bounding box, respectively, and the values ​​are:

[0067]

[0068] in, The floor symbol.

[0069] 4) Calculate the index of the voxel where each point in the point cloud data after removing the outliers is located.

[0070] The index J of the voxel grid where each point pi is located:

[0071]

[0072] Among them, J≤M, if and only if each voxel contains a point, J=M.

[0073] 5) Based on the index of the voxel where each point is located, calculate the average density of the point cloud within the unit voxel containing the point, specifically:

[0074] Sort all elements in J from small to large, and count the number of points inside each voxel, denoted as n j , then the average density of the point cloud within the unit voxel containing the point is for:

[0075]

[0076] Among them, n k Represents the number of internal points of the k-th voxel.

[0077] The estimated voxel side length is adjusted according to the average density and the point cloud density to obtain the adjusted voxel side length, specifically: This means that L' is too large, then L'=L'-λ; if This means that L' is too small and the filtering effect may be poor. Then L'=L'+λ, where λ is the adjustment factor of the side length. until So far, let L=L'.

[0078] 6) Calculate the centroid of all points in each voxel according to the adjusted voxel side length. Specifically, substitute the adjusted voxel side length L into steps 3) and 4) to calculate the centroid of all points in the voxel:

[0079]

[0080] in, represents the centroid of the j-th voxel.

[0081] 7) Using the nearest neighbor search algorithm to search for the point closest to each of the centroids in the point cloud data after removing the outliers, the points closest to each of the centroids are used to replace the corresponding centroids, and a new point cloud is formed to obtain the filtered point cloud data. Therefore, the points in the original point cloud are used to form the filtered point cloud, which achieves the maximum retention of the original point cloud information.

[0082] Figure 2 The process of point cloud preprocessing is shown. Figure 2 (a) is the original point cloud data, (b) is the redundant point cloud data removed, (c) is the outlier point cloud data removed, and (b) is the voxel filtered point cloud data. The original point cloud data contains 3186816 points, the point cloud data after removing redundant points contains 396833 points, the point cloud data after removing outliers contains 383140 points, and the point cloud data after voxel filtering contains 71428 points. The final point cloud is simplified by 97.7%.

[0083] The circuit breaker component point cloud data registration method further includes: downsampling the STL model files of each circuit breaker component to obtain the template point cloud of each circuit breaker component.

[0084] The present invention converts the 3D CAD model of each component into a surface point cloud model as a matching template for extracting point cloud features. In order to reduce the resolution of the CAD model and achieve matching between point cloud pairs, the component STL model file provided by the circuit breaker manufacturer is downsampled. The CAD model is converted into template point cloud data at a sampling interval of the camera resolution (0.055mm). Figure 3 The template point cloud data of five types of components are displayed. Figure 3(a) is the STL model file, and (b) is the converted point cloud data (template point cloud). Compared with the traditional point cloud method, the template point cloud data sampled from the component design model contains richer information and more complete features, which improves the accuracy of point cloud matching and enables the type, position and posture of components to be accurately matched in scenes with multiple random posture components.

[0085] The irrelevant points are eliminated by simplifying the original point cloud data. Since the data contains different types of components, searching for local features of each component in the entire point cloud will miss key details. It is necessary to segment the preprocessed data and divide the entire point cloud into pairs of single point cloud clusters, so that each point cloud cluster corresponds to a circuit breaker component.

[0086] The present invention adopts a clustering algorithm based on Euclidean distance metric and creates a KD tree topology structure to quickly traverse the entire point cloud.

[0087] Wherein, step 104 specifically includes:

[0088] Any point in the filtered point cloud data is used as a target point.

[0089] The k-nearest neighbor search algorithm is used to search for all points within a set threshold from the target point, and the target point and the searched points are classified into the point set O.

[0090] Select points other than the target point from the point set O as the target point in turn, return to the step of using the k-nearest neighbor search algorithm to search for all points within the set threshold from the target point, and classify the target point and the searched points into the point set O, until all points in the point set O are traversed and no new points are added to the point set O. The search ends, the segmentation of a point cloud cluster is completed, and the points in the point set O are marked as processed.

[0091] Clear the point set O, take any unmarked point in the filtered point cloud data as the target point, return to the step of searching for all points within a set threshold from the target point using the k-nearest neighbor search algorithm, and classify the target point and the searched points into the point set O, until every point in the filtered point cloud data is marked, and the segmentation is completed.

[0092] The segmentation process is as follows Figure 4 As shown, the steps are simply:

[0093] 1) For any point p in the simplified point cloud, use the k-nearest neighbor search algorithm to find all the points in its neighborhood and classify them into the point set O, keeping the index of the point in the original structure unchanged;

[0094] 2) Select points other than point p from the point set O and repeat step 1) until all points in the point set O are traversed and no new points are added to the point set. The search ends, the segmentation of a point cloud cluster is completed, and the points in the point set O are marked as processed;

[0095] 3) Continue to select unmarked points and repeat steps 1) and 2) until all data points in the entire point cloud are marked, and the segmentation is completed.

[0096] For the simplified source point cloud, the points of a point cloud cluster will be classified into one category, corresponding to one component. There are five target components in the present invention. The segmented point cloud data is as follows: Figure 5 As shown in the figure, each color represents a point cloud cluster. The correspondence between the point cloud cluster and the template point cloud is determined in the registration. The template file is stored in advance. Using the 5*5 loop programming idea, a single point cloud cluster is registered with all template point clouds. The lowest registration error means the correspondence is determined. The template number and the corresponding component name can be output and the result page is displayed.

[0097] For point cloud data preprocessing and segmentation, the present invention first analyzes the characteristics of the original point cloud data, uses a random sampling consistency algorithm to fit the plane, removes redundant points, uses a statistical filtering method to remove outliers, and then uses improved voxelization downsampling to complete the simplification of the point cloud data. Secondly, the STL model file is converted into template point cloud data to lay the foundation for subsequent alignment work. Finally, the streamlined point cloud data is segmented by Euclidean clustering to separate a single point cloud cluster from the overall point cloud data, so that each point cloud cluster corresponds to a component.

[0098] The present invention adopts 4PCS (4-Points Congruent Sets) algorithm for point cloud coarse registration, the purpose of which is to find the best rigid transformation between two point sets P and Q at arbitrary initial positions, so that the number of points in P and Q with a distance between two points less than a certain threshold is the largest. The 4PCS algorithm constructs four coplanar points in P and Q as a base pair, uses affine invariance to find all eligible point pairs in the Q point set, and finally finds the optimal rigid transformation matrix through the LCP (Largest Common Pointset) strategy to obtain the optimal matching result. The specific steps of the 4PCS algorithm are as follows:

[0099] 1) Randomly select three points in the point set P, estimate the distance of the fourth point from the above three points according to the degree of overlap between the point cloud cluster and the template point cloud, and find the fourth point under the distance constraint to form a relatively ideal coplanar four-point basis B.

[0100] 2) Calculate the affine invariant ratios r1 and r2 of the four points a, n, c, and d in B, as well as the lengths d1 and d2 of the line segments an and cd:

[0101]

[0102] d1=||ab||

[0103] d2=||cd||

[0104] In the formula, e is the intersection point of line segments an and cd, such as Figure 7 As shown, Figure 7 (a) is the coplanar four-point basis {a, b, c, d}, Figure 7 (b) is the affine corresponding point {q1, q2, q3, q4, q5}.

[0105] 3) Set a certain threshold δ and find two sets of point pairs R1 and R2 in the point set Q:

[0106] R1={(q i ,q j )|q i ,q j ∈Q}, ||q i -q j ‖∈[d1-δ,d1+δ]

[0107] R2={(q i ,q j )|q i ,q j ∈Q}, ||q i -q j ‖∈[d2-δ,d2+δ]

[0108] 4) Using the ratios r1 and r2 obtained in step 2), calculate the midpoint e of all point pairs:

[0109] e1=q1+r1(q2-q1)

[0110] e2=q1+r2(q2-q1)

[0111] The midpoint of all pairs of points in R1 Build a structure tree and retrieve points All points within a certain neighborhood (The midpoint of each point pair in R2), if the distance between the two points is less than a certain threshold, it means that the two points are approximately coincident, and the two point pairs corresponding to the two midpoints can form a coplanar four-point basis. Figure 7 As shown in (b), {q1, q3, q4, q5} are the affine corresponding points of {a, b, c, d}. Find all the coplanar four-point bases that meet the conditions and form a set U.

[0112] 5) For any point U i , through B and U i The relationship between Ui A rigid transformation matrix T that approximates a coplanar four-point basis B in the least squares sense i , use T for the point set P i Perform the transformation and count the number of points that are close enough to the point set Q. This number is used to characterize the quality of T. The transformation matrix corresponding to the largest number of points is the transformation matrix T with the highest matching overlap.

[0113] 6) For different base pairs B i , repeat step 5), and finally obtain the optimal transformation matrix from point set P to point set Q.

[0114] During the experiment, the above 4PCS algorithm was used to match the point cloud cluster of each component of the circuit breaker with the template point cloud. It was found that the distance between the two point clouds still did not meet the initial value requirement of the next stage of precise matching; the probability of mismatching was high. Figure 8 shows common mismatches, Figure 8 (a) ignores detail textures. Figure 8 (b) matches the opposite.

[0115] The overall framework of the improved registration is as follows Figure 6 As shown, the rough registration in step 105 specifically includes:

[0116] For each point cloud cluster, the ISS algorithm and the Harris3D algorithm are used to extract key points respectively, and the key points extracted by the ISS algorithm and the Harris3D algorithm are merged to obtain the key point set of each point cloud cluster.

[0117] An improved 4PCS algorithm is used to match the source point set and the target point set to obtain a rough alignment result; the source point set is a set of key points of any point cloud cluster, and the target point set is a template point cloud corresponding to the source point set; the improvement of the improved 4PCS algorithm is to use the normal curvature constraint to eliminate the point pairs in the coplanar four-point basis set U.

[0118] The normal curvature constraint is used to remove the point pairs in the coplanar four-point basis set U, including:

[0119] The normal curvature is obtained by fitting the normal section curve at a point, and the Gaussian curvature of the point is calculated by establishing the relationship between the normal curvature and the principal curvature through the local coordinate system;

[0120] Eliminate the point pairs whose Gaussian curvature difference is greater than the set difference in the coplanar four-point basis set.

[0121] The rough registration method adopted in this embodiment, that is, the improved rough registration algorithm flow is as follows: Fig. 9 The improvements shown include extraction of key points and normal curvature constraints.

[0122] 1. Extracting key points: Before step 1) of the 4PCS algorithm, the key points of the source point cloud P (source point set) are extracted, so that the basic pairs B are found in the key point set rather than the original point cloud, in order to improve the computational efficiency.

[0123] 2. For normal curvature constraints. Due to the existence of affine invariant ratios, the four point pairs are one-to-one corresponding. After step 4) of the 4PCS algorithm, the point curvature is used to eliminate the point pairs with large differences in the set U, in order to further simplify the number of possible point pairs and make the results more reliable.

[0124] The present invention uses the ISS algorithm and the Harris3D algorithm to extract key points. The present invention uses these two algorithms to extract key points with different geometric meanings, and then merges the indexes of the key points extracted by the two algorithms in the input point cloud, and finally finds the key point set extracted by combining the ISS and Harris3D algorithms through the index.

[0125] The main algorithm steps of the ISS algorithm are as follows:

[0126] 1) Set a search radius r for each point, and then calculate the target point cloud for each point p i The weights of all points in the area with radius r are ω ij , whose expression is:

[0127]

[0128] Among them, ||p i -p j ||<r.

[0129] 2) Solve the covariance matrix for each point Its expression is:

[0130]

[0131] Where T is the transpose.

[0132] 3) Calculate the eigenvalues ​​of the covariance matrix of each point and sort them by size to get λ0>λ1>λ2, set the threshold, and the initial key point is the one that meets the following rules:

[0133]

[0134] Among them, ε1<1, ε2<1, ε1 and ε2 are constants, and λ0, λ1, λ2 are eigenvalues.

[0135] 4) Perform non-maximum suppression on the minimum eigenvalue of the initial key point within a certain radius, and retain the point with the largest change in the eigenvector direction corresponding to the minimum eigenvalue as the final key point.

[0136] The main steps of the Harris3D algorithm are as follows:

[0137] 1) Construct the residual equation:

[0138] E(u,v,w)=∑[f(x0+u,y0+v,z0+w)-f(x0,y0,z0)]

[0139] Among them, u, v, w are all increments, E(u, v, w) is the residual value, f(x, y, z) is the surface equation of the point cloud, which is approximately replaced by the plane equation within the local range Ω of the point. Therefore, the plane equation can be obtained by calculating the normal vector of the point. Let the normal vector of the point be expressed as n = [n x , n y ,n z ] T , (x0, y0, z0) are the coordinates of the point, then f(x, y, z) is expressed as: x X+n y Y+n z Z+d=0.

[0140] Organized: E(u, v, w) = ∑ [u, v, w] M [u, v, w] T .

[0141] In the formula

[0142] 2) Calculate the corner point response value R, the formula is: R = detM-α(traceM) 2 , where α is a constant, usually 0.04.

[0143] 3) Set the response threshold ε, and obtain the final feature points through non-maximum suppression after comparison.

[0144] For the normal curvature constraint, the curvature of the 3D point cloud surface is estimated by fitting the normal section curve. The curvature of the normal section curve is the normal curvature. The specific operations are as follows:

[0145] 1) For each point p in the point cloud (point cloud cluster), N is the normal vector of point p, and point p has m adjacent points q i (i=1,2,...,m),q i The normal vector is M i , let {p, X, Y, N} be the local coordinate system L of point p, X and Y be orthogonal unit vectors, in the local coordinate system L, p(0, 0, 0), q i (x i ,y i , z i ), M i (n x,i , n y,i , n z,i). Make an osculating circle of point p to estimate the normal curvature of point p Fig.10 Shows the geometric relationship of variables.

[0146] Then the normal curvature of point p relative to qi is:

[0147] Where α is the vector N and pq i is the angle between vectors N and M, β is the angle between vectors N and M, |pq i | is point p, q i The Euclidean distance between .

[0148] The approximate equation for the normal curvature above is:

[0149]

[0150] in n z =n z,i .

[0151] 2) Convert the coordinates of all adjacent points into coordinates in the local coordinate system L, and obtain the relationship between the normal curvature and the principal curvature according to the Euler formula:

[0152]

[0153] Among them, k1, k2 are the principal curvatures of point p, corresponding to Fig.10 The main direction at the midpoint p is e1, e2, θ is the angle between the vectors e1 and e2, θ i is the X-axis and pQ i The angle between them, where pQ i It is pq i Projection on plane S. θ i +θ is point p passing through point q i The angle between the tangent line of the normal section curve and the main direction.

[0154] Arranging the above equations for the relationship between normal curvature and principal curvature, we get:

[0155]

[0156] Among them, U i Indicates an intermediate parameter.

[0157] 3) Use least squares optimization to obtain the maximum value k1 and minimum value k2 of the normal curvature, that is, the principal curvature of point p:

[0158]

[0159] in:

[0160]

[0161] μ=(A,B,C) T

[0162]

[0163] In the formula, k1, k2 are matrices The characteristic value of .

[0164] The present invention calculates Gaussian curvature to compare the curvature between two points. Gaussian curvature K describes the total curvature degree at a point in the point cloud data, K = k1 × k2. Find the absolute value X of the Gaussian curvature difference of all points in the set B, R1, and R2, and calculate (q i ,q j )∈R1, (a, b)∈B, then X(q i ,q j )≈X(a,b), for (q i ,q j )∈R2、(c,d)∈B has X(q i ,q j )≈X(c, d), that is, X(q i ,q j ) and X(c, d) are extremely small, less than the set minimum value, and any point pair that meets the above conditions is retained. Use the further screened point pairs to update the set U. Among them, X() represents the absolute value of the Gaussian curvature difference between the two points.

[0165] The precise registration specifically includes:

[0166] The rough registration results are registered using the Point-Plane ICP algorithm to obtain the accurately registered source point cloud and template point cloud. The Point-Plane ICP algorithm is a Point-Plane ICP algorithm that minimizes the distance between the tangent plane of each roughly registered source point and its corresponding target point (template point cloud). The specific steps are as follows:

[0167] 1) For each point in the source point cloud P′ after the rough registration transformation Calculate the corresponding surface normal (k represents the number of iterations).

[0168] 2) Take The point closest to the line and the target point cloud Q As a point Corresponding point in point cloud Q, and fit a plane at that point

[0169] 3) Calculate the rigid transformation matrix T between point clouds P and Q k .

[0170] 4) Calculate the error function e of the kth iteration k for:

[0171]

[0172] Where ds is the point To distance.

[0173] 5) Let the minimum error threshold be ε min , the iteration ends when the following conditions are met and the point cloud registration is completed:

[0174]

[0175] Where N is the point The number of .

[0176] Through registration, a rigid transformation matrix describing the position and posture information of the components can be obtained.

[0177] The present invention also includes component pose information acquisition. Through registration, the rigid transformation matrix T between the source point cloud and the template point cloud can be obtained. The matrix consists of four parts and is defined as:

[0178]

[0179] Among them, R is the rotation matrix, H is the translation matrix, and S=1 is the scale factor.

[0180] The rotation matrix R can be used to obtain the Euler angles θ that rotate around the x, y, and z axes in accordance with the right-hand screw rule. x ,θ y ,θ z :

[0181]

[0182] The coordinates of the clamping points can be obtained from the rotation matrix R and the translation matrix H. Assume that the coordinates of the clamping points of the template point cloud are (x, y, z) and the coordinates of the clamping points of the source point cloud are (x′, y′, z′), then:

[0183] (x′, y′, z′) T =R(x, y, z) T +H.

[0184] The present invention uses the ISS algorithm to extract edge points from point cloud data, uses the Harris3D algorithm to extract corner points, and obtains a key point set from the original point cloud based on the indexes of the edge points and corner points. It proposes a method that uses the key point set rather than the original point cloud cluster as the input of the registration algorithm. In order to improve the accuracy of coarse registration, it proposes a method that uses the normal curvature of the point to constrain the 4PCS algorithm, and eliminates point pairs with obvious differences in normal curvature.

[0185] In order to verify the feasibility of the circuit breaker component point cloud data registration method of the present invention, the original point cloud data was obtained through the built hardware platform, and the designed component positioning and posture estimation method was experimentally verified. The verification process includes three parts: the results and analysis of key point extraction, the results and analysis of registration, and the acquisition of component posture information.

[0186] The component positioning and attitude estimation experimental system consists of multiple functional modules, including point cloud data acquisition, point cloud data preprocessing, point cloud data segmentation, data key point extraction, point cloud registration, etc. It can simultaneously complete the real-time positioning and attitude estimation of five types of components in random scenes.

[0187] Select Microsoft Visual Studio 2019 software, call PCL (Point Cloud Library) 1.12.0 point cloud library, program in C++ programming language, complete the design of functional modules and the connection between modules.

[0188] Verification process such as Fig.11 As shown:

[0189] (1) Point cloud data acquisition

[0190] The point cloud data acquisition module mainly includes the networking of the camera and the industrial control, calling the camera scanning API and checking whether the camera can be controlled to scan through the operation of the industrial control.

[0191] (2) Point cloud data preprocessing

[0192] The point cloud data preprocessing module mainly processes the scanned point cloud and the component template point cloud used for registration. The preprocessing of the scanned point cloud includes removing redundant points, removing outliers, and streamlining the point cloud. The preprocessing of the component template point cloud used for registration is to convert the component STL file provided by the circuit breaker manufacturer into a point cloud PCD file. The design of this module can shorten the time-consuming subsequent data processing process and provide a basis for subsequent processing.

[0193] (3) Point cloud data segmentation

[0194] The point cloud data segmentation module segments the overall point cloud data containing all components in the scene into single point cloud clusters, so that a point cloud cluster corresponds to only one component.

[0195] (4) Data key point extraction

[0196] The point cloud data key point extraction module searches for key points with component characteristics in the segmented point cloud cluster. The key points include edge points that can describe the component contour and corner points with detailed information, providing distinctive input data for subsequent point cloud registration.

[0197] (5) Point cloud registration

[0198] The point cloud registration module registers the point cloud cluster with the template point cloud. It first constructs four coplanar points in the key points of the point cloud cluster, and then searches for all affine-invariant coplanar points that meet the conditions in the template point cloud. Rough registration is completed based on this geometric feature matching to obtain the initial value required for precise registration. The root mean square error of the sum of the squared distances between the corresponding points in the two point clouds is then optimized through continuous iteration until the error meets the requirements. This completes the precise registration and finally obtains the rigid transformation matrix between the two registered point clouds.

[0199] 1. Results and analysis of key point extraction

[0200] Taking the arc extinguishing chamber of a component as an example, the key points are extracted using the ISS and Harris3D algorithms respectively. Fig.12 As shown in (a) and (b), Fig.12 (a) is the Harris3D detection corner point ε=0.00045, Fig.12 (b) shows the corner points detected by ISS, ε1 = 0.9, ε2 = 0.8. It can be clearly seen that the Harris3D algorithm is more accurate in detecting corner points, so the Harris3D algorithm is used to detect corner points.

[0201] The eigenvector corresponding to the eigenvalue obtained from the covariance matrix of a certain point is the local coordinate system of that point.

[0202] The experiment proves that the ISS algorithm is better than the corner point detection in detecting the edge points of the experimental object. Therefore, the ISS algorithm is used to detect the edge points. ε1<0.5,ε2>0.5 can be set. The effect is as follows: Fig.12 As shown in (c), Fig.12 (c) shows the ISS detected edge points ε1=0.645, ε2=0.45. Fig.12 (d) is the key point effect diagram finally extracted after combining the ISS algorithm with the Harris3D algorithm. Fig.13 (a), (b), (c) and (d) are the key point extraction effects of other components.

[0203] 2. Registration results and analysis

[0204] The rotation matrix and translation matrix of the point cloud cluster and the template point cloud after registration by the method of the present invention are obtained through experiments. The specific data are shown in Table 1.

[0205] Table 1 Rigid transformation matrix obtained by registration

[0206]

[0207]

[0208] In order to further verify the effectiveness of the method in this paper, the root mean square error (RMSE) of the average value of the squared sum of the nearest point distances corresponding to the registered point cloud is used as the effect evaluation. 50 repeated comparative experiments were conducted with scattered and different types of component point cloud data, and the registration results of each component under the traditional 4PCS algorithm and the improved 4PCS algorithm were recorded, as shown in Table 2. At the same time, the three registration methods proposed by the research group, namely the SAC-IA and ICP registration algorithms based on ISS key points and FPFH feature descriptors, the Super4PCS and ICP registration algorithms, and the traditional 4PCS and ICP registration algorithms, were compared with this method, and the registration results are shown in Table 3. Among them, the algorithm of the present invention is the improved 4PCS algorithm for coarse registration and the Point-Plane ICP algorithm for fine registration.

[0209] Table 2 Experimental results before and after the improvement of the rough registration algorithm

[0210]

[0211] Table 3 Experimental results before and after the improvement of the overall registration algorithm

[0212]

[0213] Fig.14 Results of four registration algorithms are shown, among which Fig.14 Among them, (a) is the registration result of ISS+FPFH+SAC-IA+ICP algorithm, (b) is the registration result of Super4PCS+ICP algorithm, (c) is the registration result of 4PCS+ICP algorithm, and (d) is the registration result of the method of the present invention.

[0214] According to the above point cloud registration results, the registration effect, registration accuracy and registration time were analyzed:

[0215] 1. Registration effect. Fig.14It can be seen that, although both registration methods (a) and (b) use the two-step registration method, they cannot achieve perfect registration of all components and have high matching errors for individual components. Although the registration result of method (c) has a high degree of overlap, there are mismatches. In method (d), i.e. the method of the present invention, the point cloud cluster and the template point cloud are highly overlapped in space.

[0216] 2. Registration accuracy. As can be seen from Table 1, compared with the traditional algorithms mentioned above, the registration accuracy of the algorithm of the present invention is high, and the registration error is small: for the arc extinguishing chamber, the error is reduced by 75.2%; for the U-shaped rod, the error is reduced by 48.0%; for the handle, the error is reduced by 79.5%; for the yoke, the error is reduced by 65.9%; for the magnetic system, the error is reduced by 69.1%. And the probability of the algorithm of the present invention successfully identifying the type of component (except the U-shaped rod) is 100%. As can be seen from Table 2, the average registration error of each component of the SAC-IA and ICP registration algorithms based on ISS key points and FPFH feature descriptors is 0.561mm, the average registration error of the Super4PCS and ICP registration algorithms is 0.224mm, the average registration error of the traditional 4PCS and ICP registration algorithms is 0.465mm, and the average registration error of the algorithm of the present invention is 0.132mm.

[0217] 3. Registration time. As shown in Tables 1 and 2, the SAC-IA and ICP registration algorithms and Super4PCS and ICP registration algorithms that integrate the FPFH feature descriptor take a long time to register, and their real-time performance is poor. The algorithm of the present invention takes longer time than the traditional 4PCS and ICP registration algorithms, with an average increase of 0.669s. However, for practical applications, the increased time will not have an impact.

[0218] Finally, considering the magnitude of the point cloud data of the experimental object and the fact that the improved algorithm not only has a higher recognition accuracy but also reduces the error to a large extent, the algorithm of the present invention has certain practicality.

[0219] For the acquisition of six-degree-of-freedom pose information of the circuit breaker assembly, since the pose and coordinates of the template parts are known, the pose and coordinates of the target assembly can be known as long as the pose information of the target assembly relative to the template assembly is known. The relative position relationship can be calculated according to the rigid transformation matrix in Table 1. When the target is grasped, the pose information is determined by the obtained coordinate transformation matrix, and the coordinate transformation is converted from the camera space to the world coordinate system, and then to the robot coordinate system. The robot finally completes the grasping action according to the located grasping point and the path planning algorithm, and puts the assembly into the specified position. Table 4 gives the six-degree-of-freedom pose information of all components obtained by the algorithm of the present invention, including the translation distance of the target assembly relative to the template assembly and the rotation angles along the X-axis, Y-axis, and Z-axis.

[0220] Table 4 Six-degree-of-freedom posture information obtained by the algorithm of the present invention

[0221]

[0222]

[0223] During the verification process, the design of the software functional modules was completed, and combined with the hardware platform, the key point extraction method proposed above was first experimentally verified and analyzed, and then all the above functional modules were connected in series, and the RMSE error and registration time of the registration before and after the improvement were compared. Finally, the six-degree-of-freedom pose information of the component was calculated based on the rigid transformation matrix. The experimental results show that this method can correctly identify the types of circuit breaker components, and the average RMSE error of each component is less than 0.2mm. Compared with the registration results of the existing traditional algorithm, the registration error is reduced by 68% on average. It can simultaneously and accurately realize the positioning and posture estimation of multiple components, improve the intelligent level of electronic device assembly in the circuit breaker manufacturing workshop, and has certain practicality.

[0224] Example 2

[0225] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the circuit breaker component point cloud data registration method described in Embodiment 1.

[0226] Example 3

[0227] A computer program product includes a computer program, which implements the circuit breaker component point cloud data registration method described in embodiment 1 when executed by a processor.

[0228] Example 4

[0229] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the circuit breaker component point cloud data registration method described in Embodiment 1.

[0230] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0231] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A circuit breaker component point cloud data registration method, characterized in that: include: The random sampling consistency algorithm is used to fit the original point cloud data to obtain the point cloud data after removing redundant points; The original point cloud data is point cloud data including a plurality of circuit breaker components; Using a statistical filtering method to remove outliers in the point cloud data after removing redundant points, to obtain point cloud data after removing outliers; The point cloud data after the outliers are removed is processed by an improved voxel downsampling method to obtain filtered point cloud data; the improved voxel downsampling method includes constraining the voxel side length by the point cloud density within the unit voxel, and replacing the centroid point within each voxel by a nearest neighbor search algorithm; Segmenting the filtered point cloud data into a plurality of point cloud clusters, each of the point cloud clusters corresponding to a circuit breaker component; Based on the template point cloud of each circuit breaker component, a two-step registration scheme of rough registration followed by precise registration is adopted to register each point cloud cluster, and a rigid transformation matrix for registration of each circuit breaker component is obtained; the rigid transformation matrix is ​​used to determine the position and posture of the corresponding circuit breaker component.

2. The circuit breaker component point cloud data registration method according to claim 1, characterized in that: The outlier points in the point cloud data after the redundant points are removed are removed by using a statistical filtering method to obtain the point cloud data after the outlier points are removed, specifically including: Calculate the mean and standard deviation of the Euclidean distances from each point in the point cloud data after removing redundant points to each point in its neighborhood; For the i-th point in the point cloud data after removing redundant points, if the average value of the Euclidean distances of each point in its domain is greater than the distance threshold corresponding to the i-th point, the i-th point is removed as an outlier; the distance threshold corresponding to the i-th point is determined according to the mean and standard deviation of the Euclidean distances from the i-th point to each point in its neighborhood, and i is an integer greater than 0 and less than or equal to the number of points in the point cloud data after removing redundant points.

3. The circuit breaker component point cloud data registration method according to claim 1, characterized in that: The point cloud data after removing outliers is processed by an improved voxel downsampling method to obtain filtered point cloud data, which specifically includes: Calculating the volume of the minimum bounding box of the point cloud data after removing the outliers according to the coordinates of the point cloud data after removing the outliers; Determine the point cloud density of the point cloud data after removing outliers according to the volume of the minimum bounding box, and estimate the voxel side length according to the point cloud density; Dividing the minimum bounding box into voxels according to the estimated voxel side length; Calculating the index of the voxel where each point in the point cloud data after removing the outliers is located; Based on the index of the voxel where each point is located, calculate the average density of the point cloud within the unit voxel containing the point; Adjusting the estimated voxel side length according to the average density and the point cloud density to obtain an adjusted voxel side length; Calculate the centroid of all points in each voxel based on the adjusted voxel side length; A nearest neighbor search algorithm is used to search for the point closest to each of the centroids in the point cloud data after removing the outliers, and the points closest to each of the centroids are used to replace the corresponding centroids to form a new point cloud to obtain the filtered point cloud data.

4. The circuit breaker component point cloud data registration method according to claim 1, characterized in that: Also includes: The STL model files of each circuit breaker component are downsampled to obtain the template point cloud of each circuit breaker component.

5. The circuit breaker component point cloud data registration method according to claim 1, characterized in that: The filtered point cloud data is divided into multiple point cloud clusters, specifically including: Taking any point in the filtered point cloud data as a target point; Use the k-nearest neighbor search algorithm to search for all points within a set threshold from the target point, and classify the target point and the searched points into the point set O; Select points other than the target point from the point set O as the target point, return to the step of searching all points within the set threshold of the target point using the k-nearest neighbor search algorithm, and classify the target point and the searched points into the point set O, until all points in the point set O are traversed and no new points are added to the point set O, then the search ends, the segmentation of a point cloud cluster is completed, and the points in the point set O are marked as processed; Clear the point set O, take any unmarked point in the filtered point cloud data as the target point, return to the step of searching for all points within a set threshold from the target point using the k-nearest neighbor search algorithm, and classify the target point and the searched points into the point set O, until every point in the filtered point cloud data is marked, and the segmentation is completed.

6. The circuit breaker component point cloud data registration method according to claim 1, characterized in that: The rough registration specifically includes: For each point cloud cluster, the ISS algorithm and the Harris3D algorithm are used to extract key points respectively, and the key points extracted by the ISS algorithm and the Harris3D algorithm are merged to obtain the key point set of each point cloud cluster; The improved 4PCS algorithm is used to match the source point set and the target point set to obtain a rough registration result; the source point set is a key point set of any point cloud cluster, and the target point set is a template point cloud corresponding to the source point set; the improved point of the improved 4PCS algorithm is to use the normal curvature constraint to eliminate the point pairs in the coplanar four-point basis set; The normal curvature constraint is used to remove the point pairs in the coplanar four-point basis set U, including: The normal curvature is obtained by fitting the normal section curve at a point, and the Gaussian curvature of the point is calculated by establishing the relationship between the normal curvature and the principal curvature through the local coordinate system; Eliminate the point pairs whose Gaussian curvature difference is greater than the set difference in the coplanar four-point basis set.

7. The circuit breaker component point cloud data registration method according to claim 6, characterized in that: The precise registration specifically includes: The rough registration result is registered using the Point-Plane ICP algorithm to obtain the accurately registered source point cloud and template point cloud, wherein the Point-Plane ICP algorithm minimizes the distance between the tangent plane of each source point and its corresponding target point after rough registration.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the circuit breaker component point cloud data registration method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the circuit breaker component point cloud data registration method described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the circuit breaker component point cloud data registration method described in any one of claims 1 to 7 is implemented.