Product assembly stage identification method based on point cloud geometric primitive information

Abstract

The invention provides a product assembly stage identification method based on point cloud geometric primitive information, and belongs to the technical field of computer-aided assembly and simulation. The method comprises the following steps: respectively acquiring a model point cloud corresponding to a workpiece model in each assembly stage, and acquiring an actual point cloud corresponding to an actual workpiece; geometric primitives are utilized to divide and construct an actual primitive library and a plurality of model primitive libraries, all the model primitive libraries are utilized to construct a model point cloud picture corresponding to each model point cloud, and the actual primitive library is utilized to construct an actual point cloud picture; coarse registration of each model point cloud and the actual point cloud is completed; and according to all coarse registration results, fine registration is carried out on the actual point cloud and each model point cloud, and the assembly stage of the actual workpiece is identified according to all fine registration results. According to the invention, the method can achieve the precise recognition of the assembly stage of an unmarked product.

Description

Technical Field

[0001] The present application relates to the technical field of computer-aided assembly and simulation, and in particular to a product assembly stage identification method based on point cloud geometric primitive information. Background Art

[0002] Augmented assembly technology is a key enabling technology for virtual-reality interaction and virtual fusion simulation of assembly objects. It is crucial for verifying the feasibility of product assembly processes and improving assembly efficiency. The mechanism for pushing process information during the assembly process is not only a core component of augmented assembly technology but is also deeply integrated with each assembly stage of a product. Therefore, relying solely on manual switching between different assembly stages will significantly impact assembly efficiency. Therefore, it is essential to be able to automatically and rapidly identify each assembly stage of a product.

[0003] A Chinese invention patent entitled "A method for identifying assembly status without marking based on product depth image point cloud" with publication number CN114842221A states that: by encoding the actual collected point cloud and the point cloud generated by the digital model, and constructing corresponding shape vector descriptors, and calculating the cosine similarity between the shape vector descriptors, the product assembly status can be identified. When constructing the shape vector descriptor of the point cloud, this method requires the point cloud to be greater than or equal to times of point-pair distance sampling. Especially when processing a digital model composed of point clouds from multiple perspectives, the above sampling process must be performed on the point clouds from any perspective, resulting in an increase in computing resources and time consumption as the number of perspectives increases exponentially. The shape vector descriptor can only record the distance between randomly sampled point pairs, which will inevitably lose some point cloud information. In addition, the shape vector descriptor classifies and counts the distance information between point pairs according to the distance range in the form of an equidistant histogram and stores it, which further weakens the point cloud information contained in the shape vector descriptor, so that the shape vector descriptor obtained by this method can only describe less point cloud information.

[0004] Another Chinese invention patent, titled "A Method for Recognizing Assembly Status Based on Three-Dimensional Feature Points," with publication number CN113111741A, states: All feature points are screened based on the rounded corners of each node in the point cloud, the spatial positional relationships of all feature points are recorded as descriptors of the point cloud, and the registration between point clouds is completed by establishing a mapping relationship between all feature points in different descriptors. The registration results are then used to assist in the identification of the assembly status of the product. Because this method screens feature points based on the circumference of each node, these feature points tend to cluster near corners or edges, largely ignoring point clouds that appear macroscopically as planes or cylinders. This results in point cloud loss in the descriptor. Furthermore, the clustering of these feature points causes some feature points to have similarities in the included point cloud, resulting in poor discrimination. Furthermore, when searching for the correspondence between all feature points, this method uses a traversal mode, which reduces the efficiency of assembly status recognition to a certain extent.

[0005] It can be seen that for general mechanical products, during their design and processing stages, it is necessary to require the products to contain a certain degree of basic geometric primitive information such as planes and cylinders, so that when scanning point clouds, such products contain point clouds with these geometric primitive information on a macro scale. The methods used in the above two patent documents have ignored the characteristic that mechanical products generally contain geometric primitive information, and treat the point clouds of mechanical products as general types of point clouds. The descriptors used only focus on the relationship between points and point pairs, without considering the characteristics of mechanical products and their point clouds, making the descriptors complex and insufficient in describing the features of point clouds, further increasing the difficulty of finding the corresponding relationship between the descriptors, and directly affecting the judgment results in the final assembly stage.

[0006] Therefore, it is necessary to propose a solution to improve one or more problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0008] The present application provides a method for identifying the assembly stage of a product based on point cloud geometric primitive information, the method comprising the following steps:

[0009] Obtaining the model point cloud corresponding to the workpiece model at each assembly stage and the actual point cloud corresponding to the actual workpiece, wherein each model point cloud contains model normal vector information and each actual point cloud contains actual normal vector information;

[0010] Divide the actual point cloud and all model point clouds into geometric primitives respectively, store all the obtained actual primitive point clouds into the actual primitive library respectively, and store all the obtained model primitive point clouds into the corresponding model primitive library respectively;

[0011] Utilize all model primitive libraries to construct the model point cloud map corresponding to each model point cloud, and use the actual primitive library to construct the actual point cloud map;

[0012] Compare the similarity of each model point cloud with the actual point cloud and complete the corresponding coarse registration;

[0013] According to all the coarse registration results, the actual point cloud is finely registered with each model point cloud, and the assembly stage of the actual workpiece is identified based on all the fine registration results.

[0014] Furthermore, the steps of respectively obtaining the model point cloud corresponding to the workpiece model at each assembly stage and obtaining the actual point cloud corresponding to the actual workpiece include:

[0015] Generate a bounding box that fits each workpiece model according to the length, width, and height of all workpiece models;

[0016] The center of each bounding box is used as the sphere center, and the preset multiple of the distance from the sphere center to the farthest corner point of the corresponding bounding box is used as the radius to construct an outer sphere that is adapted to each workpiece model;

[0017] Multiple sampling points are evenly set on each outer sphere, and a virtual camera is set at each sampling point;

[0018] All virtual cameras are used to sample the model point cloud of the corresponding workpiece model, and all the obtained model sampling points are combined into the corresponding model point cloud;

[0019] Perform actual point cloud sampling on the actual workpiece, and compose the actual point cloud with all the actual sampling points obtained.

[0020] Furthermore, the steps of dividing the actual point cloud into geometric primitives and storing all the obtained actual primitive point clouds in the actual primitive library include:

[0021] Randomly select an actual sampling point from the actual point cloud as the initial actual growth point, record the actual normal vector of the initial actual growth point, and store the initial actual growth point and its actual normal vector in the actual growth point set;

[0022] Calculate the actual Euclidean distances between the initial actual growth point and all other actual sampling points respectively, and take all actual sampling points whose actual Euclidean distances are less than the actual distance threshold as the actual neighborhood points of the initial actual growth point;

[0023] Calculate the actual angle between the normal vector of each actual neighborhood point and the actual normal vector of the initial actual growth point respectively, store all the actual neighborhood points whose actual angle value is less than the actual angle threshold into the actual growth point set, and mark the initial actual growth point as used;

[0024] Each actual neighborhood point in the actual growth point set is taken as an actual growth point in turn, and the above steps of calculating and comparing the actual Euclidean distance and the actual angle are iteratively performed in turn, and the marked initial actual growth point and all actual growth points are excluded in each iteration until all actual growth points are marked as used, thereby obtaining a final actual growth point set corresponding to the initial actual growth point;

[0025] Iterate all the above steps until the final actual growth point set corresponding to each initial actual growth point is obtained;

[0026] The initial actual growth point and all actual growth points extracted from each final actual growth point set are respectively used as actual plane primitive points, all actual plane primitive points respectively form corresponding actual plane primitive point clouds, and all actual plane primitive point clouds are stored in the actual primitive library;

[0027] Respectively taking out all actual plane primitive point clouds containing all actual plane primitive points whose number is less than a threshold value of the number of actual plane primitive points from the actual primitive library and using them as actual new point clouds;

[0028] Using a plane fitting method based on random sampling consistency, the actual plane equation corresponding to each remaining actual plane primitive point cloud is obtained from the actual primitive library after all actual new point clouds are taken out;

[0029] Perform Euclidean distance clustering on all actual new point clouds to obtain multiple actual clusters;

[0030] Principal component analysis is performed on the normal vectors of all actual new point clouds in each actual cluster, and all the obtained minimum actual eigenvectors are used to project the corresponding actual new point clouds. Roundness detection is performed on all projection results, and all actual new point clouds corresponding to actual roundness less than the actual roundness threshold are used as the actual cylindrical primitive point clouds corresponding to each actual cluster. All actual cylindrical primitive point clouds are stored in the actual primitive library. Each actual cylindrical primitive point cloud contains multiple actual cylindrical primitive points, and each actual cylindrical primitive point is an actual new sampling point in the actual new point cloud.

[0031] The types of actual primitive point clouds include actual plane primitive point clouds and actual cylindrical primitive point clouds.

[0032] Furthermore, the steps of dividing all model point clouds into geometric primitives and storing all obtained model primitive point clouds into corresponding model primitive libraries include:

[0033] Randomly select a model sampling point from the first model point cloud as the initial model growth point, record the model normal vector of the initial model growth point, and store the initial model growth point and its model normal vector in the model growth point set;

[0034] Calculate the model Euclidean distances between the initial model growth point and all other model sampling points respectively, and take all model sampling points whose model Euclidean distances are less than the model distance threshold as the model neighborhood points of the initial model growth point;

[0035] Calculate the model angle between the normal vector of each model neighborhood point and the model normal vector of the initial model growth point respectively, store all model neighborhood points whose model angle value is less than the model angle threshold into the model growth point set, and mark the initial model growth point as used;

[0036] Each model neighborhood point in the model growth point set is used as a model growth point in turn, and the steps of calculating and comparing the model Euclidean distance and the model angle are iteratively performed in turn, and the marked initial model growth point and all model growth points are excluded in each iteration until all model growth points are marked as used, thereby obtaining the final model growth point set corresponding to the initial model growth point;

[0037] Iterate all the above steps until the final model growth point set corresponding to each initial model growth point is obtained;

[0038] The initial model growth points and all model growth points extracted from each final model growth point set are respectively used as model plane primitive points, and all model plane primitive points respectively form corresponding model plane primitive point clouds, and all model plane primitive point clouds are stored in the corresponding model primitive library;

[0039] Respectively taking out all model plane primitive point clouds containing all model plane primitive points whose number is less than the model plane primitive point number threshold from the corresponding model primitive library, and respectively taking them as new model point clouds;

[0040] Using a plane fitting method based on random sampling consistency, the model plane equation corresponding to each remaining model plane primitive point cloud is obtained from the corresponding model primitive library after all new model point clouds are taken out;

[0041] Perform Euclidean distance clustering on all new point clouds of the model to obtain multiple model clusters;

[0042] Perform principal component analysis on the normal vectors of all new model point clouds in each model cluster respectively, use all obtained minimum model eigenvectors to project the corresponding new model point clouds respectively, perform roundness detection on all projection results respectively, and use all new model point clouds corresponding to model roundness less than the model roundness threshold as the model cylindrical primitive point clouds corresponding to each model cluster respectively, store all model cylindrical primitive point clouds in the corresponding model primitive library, each model cylindrical primitive point cloud contains multiple model cylindrical primitive points, and each model cylindrical primitive point is a new model sampling point in the new model point cloud;

[0043] Iterate all the above steps until all the model cylindrical primitive point clouds corresponding to each model cluster are stored in the corresponding model primitive library;

[0044] The types of model primitive point clouds include model plane primitive point clouds and model cylinder primitive point clouds.

[0045] Furthermore, the steps of using all model primitive libraries to construct a model point cloud map corresponding to each model point cloud, and using the actual primitive library to construct the actual point cloud map include:

[0046] Traverse the actual primitive library, use all actual primitive point clouds in the actual primitive library as actual graph vertices, and establish actual connection edge relationships between any two actual primitive point clouds to construct an actual point cloud graph;

[0047] Traverse all model primitive libraries, take each model primitive point cloud in each model primitive library as a model graph vertex, and establish a model connection edge relationship between any two model primitive point clouds in each model primitive library, and construct a model point cloud graph corresponding to each model point cloud.

[0048] Furthermore, the actual primitive library is traversed, all actual primitive point clouds in the actual primitive library are used as actual graph vertices, and actual connection edge relationships are established between any two actual primitive point clouds. The steps of constructing the actual point cloud graph include:

[0049] In the actual primitive library, any two actual primitive point clouds are respectively grouped into actual primitive point cloud groups, and in each actual primitive point cloud group, all actual minimum Euclidean distances from each actual primitive point of one actual primitive point cloud to the other actual primitive point cloud are respectively calculated;

[0050] Set a first distance threshold and a first point threshold;

[0051] Two actual primitive point clouds in all actual primitive point cloud groups whose actual minimum Euclidean distance is less than the first distance threshold and whose number of actual primitive points is greater than the first point number threshold are considered to be adjacent to each other, and the value of the actual adjacency attribute of the two adjacent actual primitive point clouds in each actual primitive point cloud group is marked as 1;

[0052] If both actual primitive point clouds in the actual primitive point cloud group are actual plane primitive point clouds, then the actual normal vector angle between the actual plane equations corresponding to the two actual primitive point clouds is calculated; when the value of the actual normal vector angle is less than the actual normal vector angle threshold, the two actual primitive point clouds are considered to be parallel, and the values ​​of the actual parallel attributes of the two actual primitive point clouds are marked as 1; when the sum of the absolute values ​​of the differences between the coefficients contained in the actual plane equations corresponding to the two actual primitive point clouds is less than the actual coefficient threshold, the two actual primitive point clouds are considered to be coplanar, and the values ​​of the actual coplanar attributes of the two actual primitive point clouds are marked as 1;

[0053] If both actual primitive point clouds in the actual primitive point cloud group are actual cylindrical primitive point clouds, the two actual primitive point clouds are considered non-parallel and non-coplanar, and the values ​​of the actual parallel attribute, the actual coplanar attribute, and the actual normal vector angle of the two actual primitive point clouds are marked as 0;

[0054] If one of the actual primitive point clouds in the actual primitive point cloud group is an actual plane primitive point cloud, and the other actual primitive point cloud is an actual cylindrical primitive point cloud, the values ​​of the actual normal vector angles, the values ​​of the actual parallel attributes, and the values ​​of the actual coplanar attributes of the two actual primitive point clouds are all marked as 0;

[0055] Each actual primitive point cloud in the actual primitive library is used as an actual graph vertex, and any two actual graph vertices are connected to each other to form corresponding actual connection edges, thereby constructing an actual point cloud graph corresponding to the actual point cloud;

[0056] Among them, each actual vertex in the actual point cloud is numbered according to Arabic numerals; the value of each actual connecting edge consists of six digits, each digit in the units place is the value of the actual coplanar attribute, each digit in the tens place is the value of the actual parallel attribute, each digit in the hundreds place is the value of the actual adjacency attribute, and each digit from the hundredth place to the thousandth place is the value of the actual normal vector angle;

[0057] The types of actual graph vertices include actual planar graph vertices and actual cylindrical graph vertices.

[0058] Furthermore, all model primitive libraries are traversed, each model primitive point cloud in each model primitive library is used as a model graph vertex, and a model connection edge relationship is established between any two model primitive point clouds in each model primitive library. The steps of constructing a model point cloud graph corresponding to each model point cloud include:

[0059] From the first model primitive library, any two model primitive point clouds are respectively grouped into model primitive point cloud groups, and in each model primitive point cloud group, the minimum Euclidean distance of all models from each model primitive point of one model primitive point cloud to the other model primitive point cloud is respectively calculated;

[0060] Set a second distance threshold and a second point threshold;

[0061] Two model primitive point clouds in all model primitive point cloud groups whose minimum Euclidean distance is less than the second distance threshold and whose number of all model primitive points is greater than the second point number threshold are considered to be adjacent to each other, and the value of the model adjacency attribute of the two adjacent model primitive point clouds in each model primitive point cloud group is marked as 1;

[0062] If both model primitive point clouds in the model primitive point cloud group are model plane primitive point clouds, then the model normal vector angle between the model plane equations corresponding to the two model primitive point clouds is calculated; when the value of the model normal vector angle is less than the model normal vector angle threshold, the two model primitive point clouds are considered to be parallel, and the values ​​of the model parallel attributes of the two model primitive point clouds are marked as 1; when the sum of the absolute values ​​of the differences between the coefficients contained in the model plane equations corresponding to the two model primitive point clouds is less than the model coefficient threshold, the two model primitive point clouds are considered to be coplanar, and the values ​​of the model coplanar attributes of the two model primitive point clouds are marked as 1;

[0063] If both model primitive point clouds in the model primitive point cloud group are model cylindrical primitive point clouds, the two model primitive point clouds are considered non-parallel and non-coplanar, and the values ​​of the model parallel attribute, model coplanar attribute, and model normal vector angle of the two model primitive point clouds are marked as 0;

[0064] If one of the model primitive point clouds in the model primitive point cloud group is a model plane primitive point cloud, and the other model primitive point cloud is a model cylinder primitive point cloud, the values ​​of the model normal vector angle, the values ​​of the model parallel attribute, and the values ​​of the model coplanar attribute of the two model primitive point clouds are all marked as 0;

[0065] Each model primitive point cloud in the first model primitive library is used as a model graph vertex, and any two model graph vertices are connected to each other to form corresponding model connection edges, thereby constructing a model point cloud graph corresponding to each model point cloud;

[0066] In the model point cloud, each model vertex is numbered with Arabic numerals; the value of each model connection edge consists of six digits, each digit in the units place is the value of the model coplanarity attribute, each digit in the tens place is the value of the model parallelism attribute, each digit in the hundreds place is the value of the model adjacency attribute, and each digit from the hundredth place to the thousandth place is the value of the model normal vector angle;

[0067] Iterate all the above steps until the model point cloud maps corresponding to all model point clouds are constructed respectively;

[0068] The types of model graph vertices include model plane graph vertices and model cylindrical graph vertices.

[0069] Furthermore, the steps of comparing the similarity of each model point cloud image with the actual point cloud image and completing the corresponding coarse registration include:

[0070] Compare the actual point cloud with the first model point cloud. If the number of all model point cloud vertices of a certain type in the first model point cloud is less than the number of all actual point cloud vertices of the same type in the actual point cloud, skip the first model point cloud and continue to compare the actual point cloud with the next model point cloud. Otherwise, continue to the subsequent steps.

[0071] All actual connected edges marked with the value of the actual adjacency attribute of 1 in the actual point cloud graph are regarded as actual real edges, and all actual real edges are respectively counted into the actual degree of at least one corresponding actual graph vertex;

[0072] All model connection edges marked with a value of 1 for the model adjacency attribute in the first model point cloud graph are regarded as model real edges, and all model real edges are counted into the model degree of at least one corresponding model graph vertex;

[0073] Arrange all actual graph vertices in the actual point cloud graph in descending order according to the number of all actual degrees they contain, and arrange all model graph vertices in the first model point cloud graph in descending order according to the number of all model degrees they contain;

[0074] According to the descending order of the actual point cloud graph, count the type of each actual graph vertex and the number of all actual degrees contained;

[0075] In the first model point cloud graph, all model graph vertices of the same type as each actual graph vertex and containing a number of model degrees greater than or equal to the number of actual degrees are selected as the candidate model graph vertex set corresponding to each actual graph vertex;

[0076] Construct a mapping array with the same number of all actual graph vertices in the actual point cloud image, and construct a tag array with the same number of all model graph vertices in the first model point cloud image;

[0077] Traverse all candidate model graph vertex sets, put all candidate model graph vertices in each candidate model graph vertex set into the mapping array as mapping model graph vertices, and mark them accordingly in the marking array;

[0078] When there are at least two mapping model graph vertices in the mapping array, check whether the values ​​of the model adjacency attribute, the model coplanarity attribute, and the model parallelity attribute between any two mapping model graph vertices are all one-to-one corresponding to the values ​​of the actual adjacency attribute, the actual coplanarity attribute, and the actual parallelity attribute between the corresponding two actual graph vertices, and whether the absolute value of the difference between the model normal vector angle between any two mapping model graph vertices and the actual normal vector angle between the corresponding two actual graph vertices is within the absolute value threshold. If there is a non-corresponding equality or the absolute value of the difference is not within the absolute value threshold, the mapping relationship is considered unreasonable. At this time, the mapping array is restored to a reasonable mapping relationship state, and the next candidate model graph vertex is selected to execute the above mapping process until a reasonable mapping relationship set corresponding to each actual graph vertex in the actual point cloud is found.

[0079] According to all reasonable mapping relationship sets, record the frequency of each actual graph vertex being mapped to all corresponding mapping model graph vertices, and calculate the sum of the frequencies corresponding to each actual graph vertex;

[0080] Divide each frequency by the corresponding sum of frequencies to obtain the mapping probability of each actual graph vertex being mapped to the corresponding vertices of all mapping model graphs;

[0081] Traverse all actual graph vertices, and form graph vertex pairs with each actual graph vertex and at least one corresponding mapping model graph vertex with the highest mapping probability, and record the corresponding mapping probabilities;

[0082] Arrange all actual graph vertices in descending order according to the corresponding mapping probabilities, and select at least one graph vertex pair that ranks first as a candidate graph vertex pair;

[0083] Calculate the actual mean center and actual mean normal vector of the actual primitive point cloud corresponding to the actual graph vertex, and the model mean center and model mean normal vector of the model primitive point cloud corresponding to the mapped model graph vertex for each candidate graph vertex pair;

[0084] Each actual mean center is translated to the corresponding model mean center, and each actual mean normal vector is rotated to the corresponding model mean normal vector to obtain the coarse registration transformation matrix corresponding to the first model point cloud;

[0085] Multiply the coarse registration transformation matrix by the actual point cloud to obtain the corresponding actual coarse registration transformation point cloud. The actual coarse registration transformation point cloud is the coarse registration result between the first model point cloud and the actual point cloud.

[0086] All the above steps are repeated iteratively until all the coarse registration results are obtained.

[0087] Furthermore, the steps of respectively performing fine registration on the actual point cloud and each model point cloud based on all the coarse registration results, and identifying the assembly stage of the actual workpiece based on all the fine registration results include:

[0088] All actual coarse registration transformed point clouds are taken as source point clouds, and all model point clouds are taken as target point clouds;

[0089] Input the first source point cloud and all target point clouds into the iterative closest point algorithm to obtain the actual precise registration transformation point cloud set;

[0090] Iterate the input process of the source point cloud and all target point clouds until the actual fine registration transformation point cloud set corresponding to each source point cloud is obtained; each actual fine registration transformation point cloud set is the corresponding fine registration result;

[0091] Calculate the root mean square error between any model point cloud and the corresponding actual precise registration transformation point cloud in each actual precise registration transformation point cloud set;

[0092] When the RMS error is greater than the RMS error threshold, it is considered that the corresponding model point cloud is not the assembly stage of the actual point cloud;

[0093] When the RMS error is less than or equal to the RMS error threshold, perform the following operations:

[0094] Calculate the nearest Euclidean distance between all actual fine-registration transformation points in each actual fine-registration transformation point cloud and the nearest model sampling point in the corresponding model point cloud;

[0095] Count the total number of all actual fine-aligned transformation points whose nearest Euclidean distance is greater than the Euclidean distance threshold in each actual fine-aligned transformation point cloud, and calculate the proportion of all the total number in the corresponding actual fine-aligned transformation point cloud;

[0096] If the quantity ratio is greater than the ratio threshold, it is considered that the corresponding model point cloud is not the assembly stage of the actual point cloud;

[0097] If the quantity ratio is less than or equal to the ratio threshold, the corresponding model point cloud is considered to be the assembly stage of the actual point cloud.

[0098] Furthermore, the expression for calculating the root mean square error is:

[0099] (1)

[0100] in, represents the root mean square error, Represents the number of all model sampling points in the model point cloud, Indicates the first Model sampling points, Indicates the actual precise registration transformation point cloud and the corresponding model point cloud The actual precise registration transformation point with the closest distance between the model sampling points is Indicates precise alignment, represents the Euclidean norm, express and The Euclidean distance between them.

[0101] This application provides a product assembly stage identification method based on point cloud geometric primitive information, which has at least the following beneficial effects:

[0102] (1) This application uses the geometric primitive information of the product itself as the global descriptor of the product point cloud, thereby avoiding the influence of the multi-angular characteristics of the product on the selection of feature point positions and alleviating the inadequacy of general descriptors in describing point clouds in flat areas. At the same time, the topological and positional relationship between geometric primitive information is used as a descriptor, thereby avoiding the problem of requiring multiple descriptors to describe the same primitive point cloud and simplifying the number and dimension of descriptors.

[0103] (2) This application reduces the impact of the simultaneous existence of multiple vertex correspondences on the correct correspondence by constructing a credibility mechanism for the correspondence between the vertices of the actual point cloud map and the model point cloud map. The correspondence between the vertices of the actual point cloud map and the model point cloud map is used to achieve coarse registration of the point cloud, reducing the possibility of falling into local optimality in the fine registration stage, thereby completing the recognition of different assembly stages of the product and reducing the impact of frequent manual switching of assembly stages on product assembly efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0105] Figure 1 A schematic diagram showing the steps of a product assembly stage identification method based on point cloud geometric primitive information in an exemplary embodiment of the present application is shown;

[0106] Figure 2 A schematic flow chart showing a method for identifying a product assembly stage based on point cloud geometric primitive information in an exemplary embodiment of the present application is shown;

[0107] Figure 3 Schematic diagrams showing workpiece models at different assembly stages in an exemplary embodiment of the present application;

[0108] Figure 4 A schematic diagram showing different numbers of virtual cameras set on the bounding box of the workpiece model in the fourth assembly stage in an exemplary embodiment of the present application;

[0109] Figure 5 A schematic diagram showing the collection of model point clouds of a workpiece model in the fourth assembly stage by multiple virtual cameras in an exemplary embodiment of the present application;

[0110] Figure 6 A schematic diagram showing an actual point cloud corresponding to an actual workpiece in an exemplary embodiment of the present application;

[0111] Figure 7 A schematic diagram showing an actual point cloud image in an exemplary embodiment of the present application;

[0112] Figure 8 A schematic diagram showing a model point cloud corresponding to the fourth assembly stage in an exemplary embodiment of the present application;

[0113] Figure 9 A schematic diagram showing the poses of an actual point cloud that has not undergone coarse registration and a model point cloud in the fourth assembly stage in an exemplary embodiment of the present application;

[0114] Figure 10 A schematic diagram showing the poses of the actual point cloud after coarse registration and the model point cloud in the fourth assembly stage in an exemplary embodiment of the present application is shown;

[0115] Figure 11 A schematic diagram showing the result of fine registration of the actual point cloud and the model point cloud of the fourth assembly stage without coarse registration in an exemplary embodiment of the present application, but directly using the iterative closest point algorithm;

[0116] Figure 12 A schematic diagram showing the result of coarse registration of the actual point cloud and the model point cloud of the fourth assembly stage in an exemplary embodiment of the present application, followed by fine registration using an iterative closest point algorithm;

[0117] Figure 13 A schematic diagram showing the results of identifying different assembly stages of an actual workpiece in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0118] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0119] In addition, the accompanying drawings are merely schematic illustrations of the present application and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0120] Below, a product assembly state recognition method based on point cloud geometric primitive information proposed in an embodiment of the present application will be described in more detail.

[0121] In order to solve the problem that general descriptors do not make sufficient use of geometric primitive information of mechanical products, this exemplary embodiment proposes a product assembly state recognition method based on point cloud geometric primitive information, such as Figure 1 and Figure 2 As shown, the method may include the following steps:

[0122] In this embodiment, step S101: Figure 3 As shown, the model point cloud corresponding to the workpiece model of each assembly stage is obtained respectively, and the actual point cloud corresponding to the actual workpiece is obtained.

[0123] Figure 3 a in the figure shows the workpiece model at the first assembly stage. Figure 3 b in the figure shows the workpiece model at the second assembly stage. Figure 3 Figure c shows the workpiece model at the third assembly stage. Figure 3 Figure d shows the workpiece model at the fourth assembly stage.

[0124] Furthermore, each model point cloud includes model normal vector information, and each actual point cloud includes actual normal vector information.

[0125] Furthermore, if Figure 4 and Figure 5 As shown, step S101 in this embodiment may include the following sub-steps:

[0126] Sub-step S1011: generating a bounding box adapted to each workpiece model according to the length, width and height of all workpiece models.

[0127] Sub-step S1012: Construct an outer sphere that fits each workpiece model, with the center of each bounding box as the sphere center and a preset multiple of the distance from the sphere center to the farthest corner of the corresponding bounding box as the radius. In this embodiment, the preset multiple is preferably 1.5 times.

[0128] Sub-step S1013: evenly setting a plurality of sampling points on each outer sphere, and setting a virtual camera at each sampling point.

[0129] Figure 4 The situations of setting different numbers of virtual cameras on the outer sphere of the workpiece model in the fourth assembly stage are given. Figure 4 The a in the table contains only one virtual camera. Figure 4 b in the figure contains 11 virtual cameras. Figure 4 The c in the figure contains 15 virtual cameras.

[0130] Figure 4 and Figure 5 In FIG, 101 represents a virtual camera set at each sampling point, 102 represents a model point cloud, 103 represents a workpiece model, 104 represents a bounding box, and 105 represents an outer sphere.

[0131] Sub-step S1014: using all virtual cameras to perform model point cloud sampling on the corresponding workpiece model, and forming the corresponding model point cloud from all the obtained model sampling points.

[0132] Sub-step S1015: Figure 6 As shown, a point cloud acquisition device is generally used to sample the actual point cloud of the actual workpiece, and all the actual sampling points obtained are combined into an actual point cloud.

[0133] In this embodiment, step S102: divide the actual point cloud and all model point clouds into geometric primitives, store all the obtained actual primitive point clouds into the actual primitive library, and store all the obtained model primitive point clouds into the corresponding model primitive library. In this embodiment, step S102 may include the following sub-steps:

[0134] Sub-step S1021, dividing the actual point cloud into geometric primitives, and storing all the obtained actual primitive point clouds into the actual primitive library. The specific process includes:

[0135] Step 1: Randomly select an actual sampling point from the actual point cloud as the initial actual growth point, record the actual normal vector of the initial actual growth point, and store the initial actual growth point and its actual normal vector in the actual growth point set.

[0136] Step 2: Calculate the actual Euclidean distances between the initial actual growth point and all other actual sampling points, and take all actual sampling points whose actual Euclidean distances are less than the actual distance threshold as the actual neighborhood points of the initial actual growth point.

[0137] Step 3: Calculate the actual angle between the normal vector of each actual neighborhood point and the actual normal vector of the initial actual growth point respectively, store all actual neighborhood points whose actual angle value is less than the actual angle threshold into the actual growth point set, and mark the initial actual growth point as used.

[0138] Step 4: Take each actual neighborhood point in the actual growth point set as the actual growth point in turn, and iterate the above steps of calculating and comparing the actual Euclidean distance and the actual angle. In each iteration, the marked initial actual growth point and all actual growth points are excluded until all actual growth points are marked as used, and the final actual growth point set corresponding to the initial actual growth point is obtained.

[0139] Step 5: Iterate all the above steps until the final actual growth point set corresponding to each initial actual growth point is obtained.

[0140] Step 6: The initial actual growth points and all actual growth points extracted from each final actual growth point set are respectively used as actual plane primitive points. All actual plane primitive points respectively form corresponding actual plane primitive point clouds, and all actual plane primitive point clouds are stored in the actual primitive library.

[0141] Step 7: All actual plane primitive point clouds containing all actual plane primitive points whose number is less than the actual plane primitive point number threshold are taken out from the actual primitive library and used as actual new point clouds respectively.

[0142] Step 8: Using a plane fitting method based on random sampling consistency, obtain the actual plane equation corresponding to each remaining actual plane primitive point cloud from the actual primitive library after taking out all actual new point clouds.

[0143] Step 9: Perform Euclidean distance clustering on all actual new point clouds to obtain multiple actual clusters.

[0144] Step 10: Perform principal component analysis on the normal vectors of all actual new point clouds in each actual cluster respectively, use all the obtained minimum actual eigenvectors to project the corresponding actual new point clouds respectively, perform roundness detection on all projection results respectively, and use all actual new point clouds corresponding to actual roundness less than the actual roundness threshold as the actual cylindrical primitive point clouds corresponding to each actual cluster respectively, and store all actual cylindrical primitive point clouds in the actual primitive library. Each actual cylindrical primitive point cloud contains multiple actual cylindrical primitive points, and each actual cylindrical primitive point is an actual new sampling point in the actual new point cloud.

[0145] Here, the types of actual primitive point clouds include actual planar primitive point clouds and actual cylindrical primitive point clouds.

[0146] Sub-step S1022: Divide all model point clouds into geometric primitives, and store all obtained model primitive point clouds into corresponding model primitive libraries. The specific process includes:

[0147] Step 1: Randomly select a model sampling point from the first model point cloud as the initial model growth point, record the model normal vector of the initial model growth point, and store the initial model growth point and its model normal vector in the model growth point set.

[0148] Step 2: Calculate the model Euclidean distances between the initial model growth point and all other model sampling points, and use all model sampling points whose model Euclidean distances are less than the model distance threshold as the model neighborhood points of the initial model growth point.

[0149] Step 3: Calculate the model angle between the normal vector of each model neighborhood point and the model normal vector of the initial model growth point respectively, store all model neighborhood points whose model angle value is less than the model angle threshold into the model growth point set, and mark the initial model growth point as used.

[0150] Step 4: Take each model neighborhood point in the model growth point set as a model growth point in turn, and iterate the above steps of calculating and comparing the model Euclidean distance and model angle. In each iteration, the marked initial model growth point and all model growth points are excluded until all model growth points are marked as used, and the final model growth point set corresponding to the initial model growth point is obtained.

[0151] Step 5: Iterate all the above steps until the final model growth point set corresponding to each initial model growth point is obtained.

[0152] Step 6: The initial model growth points and all model growth points extracted from each final model growth point set are respectively used as model plane primitive points. All model plane primitive points respectively form corresponding model plane primitive point clouds, and all model plane primitive point clouds are stored in the corresponding model primitive library.

[0153] Step 7: All model plane primitive point clouds containing all model plane primitive points whose number is less than the model plane primitive point number threshold are taken out from the corresponding model primitive library and used as new model point clouds.

[0154] Step 8: Using a plane fitting method based on random sampling consistency, obtain the model plane equation corresponding to each remaining model plane primitive point cloud from the corresponding model primitive library after taking out all the new model point clouds.

[0155] Step 9: Perform Euclidean distance clustering on all new model point clouds to obtain multiple model clusters.

[0156] Step 10: Perform principal component analysis on the normal vectors of all new model point clouds in each model cluster respectively, use all the obtained minimum model eigenvectors to project the corresponding new model point clouds respectively, perform roundness detection on all projection results respectively, and use all new model point clouds corresponding to model roundness less than the model roundness threshold as the model cylindrical primitive point clouds corresponding to each model cluster respectively, and store all model cylindrical primitive point clouds in the corresponding model primitive library. Each model cylindrical primitive point cloud contains multiple model cylindrical primitive points, and each model cylindrical primitive point is a new model sampling point in the new model point cloud.

[0157] Step 11: Iterate all the above steps until all model cylindrical primitive point clouds corresponding to each model cluster are stored in the corresponding model primitive library.

[0158] Here, the types of model primitive point clouds include model plane primitive point clouds and model cylinder primitive point clouds.

[0159] In this embodiment, step S103: using all model primitive libraries, constructing the model point cloud map corresponding to each model point cloud, and using the actual primitive library to construct the actual point cloud map. In this embodiment, step S103 may include the following sub-steps:

[0160] Sub-step S1031: Figure 7 As shown, the actual primitive library is traversed, all actual primitive point clouds in the actual primitive library are used as actual graph vertices, and actual connection edge relationships are established between any two actual primitive point clouds to construct the actual point cloud graph. The specific process is as follows:

[0161] Step 1: In the actual primitive library, any two actual primitive point clouds are grouped into actual primitive point cloud groups, and all actual minimum Euclidean distances from each actual primitive point in one actual primitive point cloud to the other actual primitive point cloud in each actual primitive point cloud group are calculated.

[0162] Step 2: Set the first distance threshold and the first point threshold.

[0163] Step 3: The two actual primitive point clouds in all actual primitive point cloud groups corresponding to the actual minimum Euclidean distance being less than the first distance threshold and the number of all actual primitive points being greater than the first point number threshold are considered to be adjacent to each other, and the value of the actual adjacency attribute of the two adjacent actual primitive point clouds in each actual primitive point cloud group is marked as 1.

[0164] Step 4: If both actual primitive point clouds in the actual primitive point cloud group are actual plane primitive point clouds, then calculate the actual normal vector angle between the actual plane equations corresponding to the two actual primitive point clouds; when the value of the actual normal vector angle is less than the actual normal vector angle threshold, the two actual primitive point clouds are considered to be parallel, and the values ​​of the actual parallel attributes of the two actual primitive point clouds are marked as 1; when the sum of the absolute values ​​of the differences between the coefficients contained in the actual plane equations corresponding to the two actual primitive point clouds is less than the actual coefficient threshold, the two actual primitive point clouds are considered to be coplanar, and the values ​​of the actual coplanar attributes of the two actual primitive point clouds are marked as 1.

[0165] Step 5: If both actual primitive point clouds in the actual primitive point cloud group are actual cylindrical primitive point clouds, the two actual primitive point clouds are considered non-parallel and non-coplanar, and the values ​​of the actual parallel attributes, actual coplanar attributes, and actual normal vector angles of the two actual primitive point clouds are marked as 0.

[0166] Step 6: If one of the actual primitive point clouds in the actual primitive point cloud group is an actual plane primitive point cloud, and the other actual primitive point cloud is an actual cylindrical primitive point cloud, then the values ​​of the actual normal vector angles, the values ​​of the actual parallel attributes, and the values ​​of the actual coplanar attributes of the two actual primitive point clouds are all marked as 0.

[0167] Step 7: Take each actual primitive point cloud in the actual primitive library as an actual graph vertex, and connect any two actual graph vertices to form corresponding actual connection edges, and construct an actual point cloud graph corresponding to the actual point cloud.

[0168] Among them, each actual vertex in the actual point cloud map is numbered according to Arabic numerals; the value of each actual connecting edge consists of six digits, each unit digit is the value of the actual coplanar attribute, each tens digit is the value of the actual parallel attribute, each hundreds digit is the value of the actual adjacency attribute, and each digit from the hundredth digit to the thousandth digit is the value of the actual normal vector angle.

[0169] Here, the types of actual graph vertices include actual planar graph vertices and actual cylindrical graph vertices.

[0170] Sub-step S1032: Figure 8 As shown, all model primitive libraries are traversed, and each model primitive point cloud in each model primitive library is used as a model graph vertex. A model connection edge relationship is established between any two model primitive point clouds in each model primitive library, and a model point cloud graph corresponding to each model point cloud is constructed. The specific process is as follows:

[0171] Step 1: From the first model primitive library, any two model primitive point clouds are grouped into model primitive point cloud groups, and in each model primitive point cloud group, the minimum Euclidean distance of all models from each model primitive point in one model primitive point cloud to the other model primitive point cloud is calculated.

[0172] Step 2: Set the second distance threshold and the second point threshold.

[0173] Step 3: The two model primitive point clouds in all model primitive point cloud groups whose minimum Euclidean distance is less than the second distance threshold and whose number of all model primitive points is greater than the second point count threshold are considered to be adjacent to each other, and the value of the model adjacency attribute of the two adjacent model primitive point clouds in each model primitive point cloud group is marked as 1.

[0174] Step 4: If both model primitive point clouds in the model primitive point cloud group are model plane primitive point clouds, then calculate the model normal vector angle between the model plane equations corresponding to the two model primitive point clouds; when the value of the model normal vector angle is less than the model normal vector angle threshold, the two model primitive point clouds are considered to be parallel, and the values ​​of the model parallel attributes of the two model primitive point clouds are marked as 1; when the sum of the absolute values ​​of the differences between the coefficients contained in the model plane equations corresponding to the two model primitive point clouds is less than the model coefficient threshold, the two model primitive point clouds are considered to be coplanar, and the values ​​of the model coplanar attributes of the two model primitive point clouds are marked as 1.

[0175] Step 5: If both model primitive point clouds in the model primitive point cloud group are model cylindrical primitive point clouds, the two model primitive point clouds are considered non-parallel and non-coplanar, and the values ​​of the model non-parallel attribute, the value of the model coplanar attribute, and the value of the human model normal vector angle of the two model primitive point clouds are all marked as 0.

[0176] Step 6: If one of the model primitive point clouds in the model primitive point cloud group is a model plane primitive point cloud and the other model primitive point cloud is a model cylinder primitive point cloud, then the values ​​of the model normal vector angle, the values ​​of the model parallel attribute, and the values ​​of the model coplanar attribute of these two model primitive point clouds are all marked as 0.

[0177] Step 7: Use each model primitive point cloud in the first model primitive library as a model graph vertex, and connect any two model graph vertices to form corresponding model connection edges, and construct the model point cloud graph corresponding to each model point cloud.

[0178] Among them, each model vertex in the model point cloud is numbered according to Arabic numerals; the value of each model connecting edge consists of six digits, each unit digit is the value of the model coplanar attribute, each tens digit is the value of the model parallel attribute, each hundreds digit is the value of the model adjacency attribute, and each digit from the hundredth digit to the thousandth digit is the value of the model normal vector angle.

[0179] Step 8: Iterate all the above steps until the model point cloud maps corresponding to all model point clouds are constructed.

[0180] Here, the types of model graph vertices include model plane graph vertices and model cylindrical graph vertices.

[0181] In this embodiment, step S104: Figure 9 and Figure 10 As shown, each model point cloud image is compared with the actual point cloud image for similarity, and the corresponding coarse registration is completed. In this embodiment, step S104 may include the following sub-steps:

[0182] Sub-step S1041: Compare the actual point cloud map with the first model point cloud map. If the number of all model map vertices of a certain type in the first model point cloud map is less than the number of all actual map vertices of the same type in the actual point cloud map, skip the first model point cloud map and continue to compare the actual point cloud map with the next model point cloud map. Otherwise, continue with the subsequent steps.

[0183] Sub-step S1042: All actual connected edges marked with a value of 1 for the actual adjacency attribute in the actual point cloud graph are regarded as actual real edges, and all actual real edges are respectively counted into the actual degree of the corresponding at least one actual graph vertex;

[0184] All model connection edges marked with the value of the model adjacency attribute of 1 in the first model point cloud graph are regarded as model real edges, and all model real edges are counted into the model degree of each corresponding model graph vertex.

[0185] Sub-step S1043: Arrange all actual graph vertices in the actual point cloud graph in descending order according to the number of all actual degrees contained in each of them, and arrange all model graph vertices in the first model point cloud graph in descending order according to the number of all model degrees contained in each of them;

[0186] According to the descending order of the actual point cloud graph, count the type of each actual graph vertex and the number of all actual degrees contained;

[0187] In the first model point cloud graph, all model graph vertices that are of the same type as each actual graph vertex and contain a number of all model degrees greater than or equal to the number of all actual degrees are selected as the candidate model graph vertex set corresponding to each actual graph vertex.

[0188] Sub-step S1044: construct a mapping array with the same number as all actual graph vertices in the actual point cloud image, and construct a label array with the same number as all model graph vertices in the first model point cloud image.

[0189] Sub-step S1045: traverse all candidate model graph vertex sets, put all candidate model graph vertices in each candidate model graph vertex set into the mapping array respectively as mapping model graph vertices, and mark them accordingly in the marking array;

[0190] When there are at least two mapping model graph vertices in the mapping array, check whether the values ​​of the model adjacency attributes, the values ​​of the model coplanarity attributes, and the values ​​of the model parallelity attributes between any two mapping model graph vertices are all one-to-one corresponding to the values ​​of the actual adjacency attributes, the values ​​of the actual coplanarity attributes, and the values ​​of the actual parallelity attributes between the corresponding two actual graph vertices, and whether the absolute value of the difference between the model normal vector angle between any two mapping model graph vertices and the actual normal vector angle between the corresponding two actual graph vertices is within the absolute value threshold. If there is a non-correspondence or the absolute value of the difference is no longer within the absolute value threshold, the mapping relationship is considered unreasonable. At this time, the mapping array is restored to a reasonable mapping relationship state, and the next candidate model graph vertex is selected to execute the above mapping process until a reasonable mapping relationship set corresponding to each actual graph vertex in the actual point cloud is found.

[0191] Sub-step S1046: Based on all reasonable mapping relationship sets, record the frequency of each actual graph vertex being mapped to all corresponding mapping model graph vertices, and calculate the sum of the frequencies corresponding to each actual graph vertex;

[0192] Divide each obtained frequency by the corresponding sum of frequencies to obtain the mapping probability of each actual graph vertex being mapped to all corresponding mapping model vertices.

[0193] Sub-step S1047: traverse all actual graph vertices, respectively form graph vertex pairs with each actual graph vertex and at least one corresponding mapping model graph vertex with the highest mapping probability, and record the corresponding mapping probabilities;

[0194] Arrange all actual graph vertices in descending order according to the corresponding mapping probabilities, and select at least one graph vertex pair that ranks first as a candidate graph vertex pair;

[0195] Calculate the actual mean center and actual mean normal vector of the actual primitive point cloud corresponding to the actual graph vertex, and the model mean center and model mean normal vector of the model primitive point cloud corresponding to the mapped model graph vertex for each candidate graph vertex pair;

[0196] Each actual mean center is translated to the corresponding model mean center, and each actual mean normal vector is rotated to the corresponding model mean normal vector to obtain the coarse registration transformation matrix corresponding to the first model point cloud.

[0197] Sub-step S1048: multiplying the coarse registration transformation matrix by the actual point cloud to obtain the corresponding actual coarse registration transformation point cloud. The actual coarse registration transformation point cloud is the coarse registration result of the first model point cloud and the actual point cloud.

[0198] Sub-step S1049: Iterate all the above steps until all the coarse registration results are obtained.

[0199] Figure 9 The poses of the actual point cloud before coarse registration and the model point cloud in the fourth assembly stage are given; it can be seen that the two basically do not overlap and the pose difference is also quite large. Figure 10 The poses of the actual point cloud after coarse registration and the model point cloud in the fourth assembly stage are shown. It can be seen that the two have a high degree of overlap and are relatively close in pose.

[0200] In this embodiment, step S105: Figure 11 、 Figure 12 and Figure 13 As shown, based on all the coarse registration results, the actual point cloud is finely registered with each model point cloud, and the assembly stage of the actual workpiece is identified based on all the fine registration results. In this embodiment, step S105 may include the following sub-steps:

[0201] Sub-step S1051: taking all actual coarse registration transformed point clouds as source point clouds and taking all model point clouds as target point clouds.

[0202] Sub-step S1052: Input the first source point cloud and all target point clouds into the iterative closest point algorithm to obtain a set of actual precisely registered transformed point clouds.

[0203] Sub-step S1053: iteratively inputting the source point cloud and all target point clouds until obtaining a set of actual fine-registration transformed point clouds corresponding to each source point cloud; each set of actual fine-registration transformed point clouds is a corresponding fine-registration result.

[0204] Sub-step S1054: Calculate the root mean square error between any model point cloud and the corresponding actual fine registration transformed point cloud in each actual fine registration transformed point cloud set.

[0205] When the root mean square error is greater than the root mean square error threshold, it is considered that the corresponding model point cloud is not the assembly stage of the actual point cloud.

[0206] When the RMS error is less than or equal to the RMS error threshold, perform the following operations:

[0207] Calculate the nearest Euclidean distance between all actual fine-registration transformation points in each actual fine-registration transformation point cloud and the nearest model sampling point in the corresponding model point cloud;

[0208] Count the total number of all actual fine-aligned transformation points whose nearest Euclidean distance is greater than the Euclidean distance threshold in each actual fine-aligned transformation point cloud, and calculate the proportion of all the total number in the corresponding actual fine-aligned transformation point cloud;

[0209] If the quantity ratio is greater than the ratio threshold, it is considered that the corresponding model point cloud is not the assembly stage of the actual point cloud;

[0210] If the quantity ratio is less than or equal to the ratio threshold, the corresponding model point cloud is considered to be the assembly stage of the actual point cloud.

[0211] Furthermore, the expression for calculating the root mean square error is:

[0212] (1)

[0213] in, represents the root mean square error, Represents the number of all model sampling points in the model point cloud, Indicates the first Model sampling points, Indicates the actual precise registration transformation point cloud and the corresponding model point cloud The actual precise registration transformation point with the closest distance between the model sampling points is Indicates precise alignment, represents the Euclidean norm, express and The Euclidean distance between them.

[0214] Figure 11 In the figure, a represents the initial pose of the actual point cloud and the model point cloud in the fourth assembly stage, and b represents the pose after fine registration of the actual point cloud and the model point cloud in the fourth assembly stage using the iterative nearest algorithm without coarse registration. Figure 11 It can be seen that at this time, in the fine registration results, the overlap between the actual point cloud and the model point cloud in the fourth assembly stage is not high, and the pose difference is large.

[0215] Figure 12 Where a represents the initial pose of the actual point cloud and the model point cloud in the fourth assembly stage, b represents the pose of the actual point cloud and the model point cloud in the fourth assembly stage after coarse registration, and c represents the pose of the actual point cloud and the model point cloud in the fourth assembly stage after coarse registration and fine registration using the iterative nearest neighbor algorithm. Figure 12 It can be seen that after coarse alignment and then fine alignment using the iterative closest point algorithm, the overlap between the actual point cloud and the model point cloud in the fourth assembly stage is improved, and the poses of the two are relatively close.

[0216] Figure 13 The recognition results of the actual point cloud and the model point cloud at each assembly stage are given. Figure 13 As can be seen, the smaller the RMS error and the smaller the percentage of points, the greater the likelihood of a match at that assembly stage. The assembly stage likelihood indicates the degree of match between the actual point cloud and the corresponding assembly stage of the model point cloud. 0% indicates no match, and 100% indicates a perfect match between the actual point cloud and the model point cloud.

[0217] exist Figure 9 、 Figure 10 、 Figure 11 a and b, and Figure 12 In a, b and c, 201 represents the actual point cloud and 202 represents the model point cloud.

[0218] It should be noted that the various types of thresholds involved in this embodiment are all adjusted and determined based on the actual results obtained in each experimental operation.

[0219] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise clearly specified.

[0220] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0221] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of the present application.

[0222] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

Claims

1. A product assembly stage recognition method based on point cloud geometric primitive information, characterized in that: The method comprises the following steps: Obtaining a model point cloud corresponding to the workpiece model at each assembly stage, and obtaining an actual point cloud corresponding to the actual workpiece, wherein each of the model point clouds includes model normal vector information, and each of the actual point clouds includes actual normal vector information; Performing geometric primitive division on the actual point cloud and all the model point clouds, respectively, storing all the obtained actual primitive point clouds in actual primitive libraries, and storing all the obtained model primitive point clouds in corresponding model primitive libraries; Using all the model primitive libraries, respectively construct a model point cloud map corresponding to each of the model point clouds, and using the actual primitive library to construct an actual point cloud map; Comparing the similarity of each model point cloud image with the actual point cloud image respectively, and completing the corresponding coarse registration; According to all the coarse registration results, the actual point cloud is respectively finely registered with each of the model point clouds, and the assembly stage of the actual workpiece is identified according to all the fine registration results.

2. The product assembly stage identification method based on point cloud geometric primitive information according to claim 1, characterized in that: The steps of respectively obtaining the model point cloud corresponding to the workpiece model at each assembly stage and obtaining the actual point cloud corresponding to the actual workpiece include: generating a bounding box adapted to each of the workpiece models according to the length, width and height of all the workpiece models; Taking the center of each bounding box as the sphere center and a preset multiple of the distance from the sphere center to the farthest corner point of the corresponding bounding box as the radius, a peripheral sphere adapted to each workpiece model is constructed; Evenly setting a plurality of sampling points on each of the outer spheres, and setting a virtual camera at each of the sampling points; Using all the virtual cameras to perform model point cloud sampling on the corresponding workpiece model, and forming the corresponding model point cloud from all the obtained model sampling points; Actual point cloud sampling is performed on the actual workpiece, and all the obtained actual sampling points are combined into the actual point cloud.

3. The product assembly stage identification method based on point cloud geometric primitive information according to claim 2, characterized in that: The step of dividing the actual point cloud into geometric primitives and storing all the obtained actual primitive point clouds in the actual primitive library comprises: Randomly selecting one of the actual sampling points from the actual point cloud as an initial actual growth point, recording an actual normal vector of the initial actual growth point, and storing the initial actual growth point and its actual normal vector in an actual growth point set; Calculating actual Euclidean distances between the initial actual growth point and all other actual sampling points respectively, and taking all the actual sampling points corresponding to the actual Euclidean distances being less than an actual distance threshold as actual neighboring points of the initial actual growth point; respectively calculating an actual angle between the normal vector of each actual neighborhood point and the actual normal vector of the initial actual growth point, storing all the actual neighborhood points corresponding to the actual angle values ​​being less than an actual angle threshold into the actual growth point set, and marking the initial actual growth point as used; Taking each of the actual neighborhood points in the actual growth point set as an actual growth point in turn, and iteratively performing the steps of calculating and comparing the actual Euclidean distance and the actual angle, and excluding the marked initial actual growth point and all the actual growth points in each iteration, until all the actual growth points are marked as used, to obtain a final actual growth point set corresponding to the initial actual growth point; Iterate all the above steps until the final actual growth point set corresponding to each initial actual growth point is obtained; The initial actual growth point and all the actual growth points extracted from each of the final actual growth point sets are respectively used as actual plane primitive points, all the actual plane primitive points respectively form corresponding actual plane primitive point clouds, and all the actual plane primitive point clouds are stored in the actual primitive library; Respectively taking out all the actual plane primitive point clouds containing all the actual plane primitive points whose number is less than a threshold value of the number of actual plane primitive points from the actual primitive library, and taking them as actual new point clouds respectively; Using a plane fitting method based on random sampling consistency, respectively obtaining the actual plane equation corresponding to each remaining actual plane primitive point cloud from the actual primitive library after all the actual new point clouds are taken out; Performing Euclidean distance clustering on all the actual new point clouds to obtain multiple actual cluster clusters; Performing principal component analysis on normal vectors of all the actual new point clouds in each actual cluster respectively, projecting the corresponding actual new point clouds using all obtained minimum actual eigenvectors respectively, performing roundness detection on all projection results respectively, and taking all the actual new point clouds corresponding to actual roundness less than an actual roundness threshold as actual cylindrical primitive point clouds corresponding to each actual cluster respectively, storing all the actual cylindrical primitive point clouds in the actual primitive library, each of the actual cylindrical primitive point clouds containing multiple actual cylindrical primitive points, and each of the actual cylindrical primitive points being an actual new sampling point in the actual new point cloud; The types of the actual primitive point cloud include the actual plane primitive point cloud and the actual cylindrical primitive point cloud.

4. The product assembly stage identification method based on point cloud geometric primitive information according to claim 3 is characterized in that: The steps of dividing all the model point clouds into geometric primitives and storing all the obtained model primitive point clouds into corresponding model primitive libraries include: Randomly selecting a model sampling point from the first model point cloud as an initial model growth point, recording a model normal vector of the initial model growth point, and storing the initial model growth point and its model normal vector in a model growth point set; Calculating the model Euclidean distances between the initial model growth point and all other model sampling points respectively, and taking all the model sampling points corresponding to the model Euclidean distances being less than a model distance threshold as model neighborhood points of the initial model growth point; Calculating the model angle between the normal vector of each model neighborhood point and the model normal vector of the initial model growth point respectively, storing all the model neighborhood points corresponding to the model angle values ​​being less than a model angle threshold into the model growth point set, and marking the initial model growth point as used; Each of the model neighborhood points in the model growth point set is sequentially used as a model growth point, and the steps of calculating and comparing the model Euclidean distance and the model angle are iteratively performed in sequence, and the marked initial model growth point and all the model growth points are excluded in each iteration until all the model growth points are marked as used, thereby obtaining a final model growth point set corresponding to the initial model growth point; Iterate all the above steps until the final model growth point set corresponding to each initial model growth point is obtained; The initial model growth points and all the model growth points extracted from each of the final model growth point sets are respectively used as model plane primitive points, all the model plane primitive points respectively form corresponding model plane primitive point clouds, and all the model plane primitive point clouds are stored in the corresponding model primitive library; Respectively taking out all the model plane primitive point clouds containing all the model plane primitive points whose number is less than a model plane primitive point number threshold from the corresponding model primitive library, and respectively taking them as new model point clouds; Using a plane fitting method based on random sampling consistency, respectively obtaining the model plane equation corresponding to each remaining model plane primitive point cloud from the corresponding model primitive library after all the new model point clouds are taken out; Performing Euclidean distance clustering on all new point clouds of the model to obtain multiple model clusters; Performing principal component analysis on the normal vectors of all the new model point clouds in each of the model clusters, projecting the corresponding new model point clouds using all the obtained minimum model eigenvectors, performing roundness detection on all the projection results, and taking all the new model point clouds corresponding to model roundness less than a model roundness threshold as the model cylindrical primitive point clouds corresponding to each of the model clusters, storing all the model cylindrical primitive point clouds in the corresponding model primitive library, each of the model cylindrical primitive point clouds containing a plurality of model cylindrical primitive points, and each of the model cylindrical primitive points being a new model sampling point in the new model point cloud; Iterate all the above steps until all the model cylindrical primitive point clouds corresponding to each model cluster are stored in the corresponding model primitive library; The types of the model primitive point cloud include the model plane primitive point cloud and the model cylinder primitive point cloud.

5. The product assembly stage identification method based on point cloud geometric primitive information according to claim 4 is characterized in that: The steps of using all the model primitive libraries to construct the model point cloud map corresponding to each of the model point clouds, and using the actual primitive library to construct the actual point cloud map include: Traversing the actual primitive library, taking all the actual primitive point clouds in the actual primitive library as actual graph vertices, and establishing actual connection edge relationships between any two actual primitive point clouds to construct the actual point cloud graph; Traverse all the model primitive libraries, take each model primitive point cloud in each model primitive library as a model graph vertex, and establish a model connection edge relationship between any two model primitive point clouds in each model primitive library, and construct the model point cloud graph corresponding to each model point cloud.

6. The product assembly stage identification method based on point cloud geometric primitive information according to claim 5, characterized in that: The step of traversing the actual primitive library, taking all the actual primitive point clouds in the actual primitive library as actual graph vertices, and establishing actual connection edge relationships between any two actual primitive point clouds to construct the actual point cloud graph includes: In the actual primitive library, any two actual primitive point clouds are respectively grouped into an actual primitive point cloud group, and all actual minimum Euclidean distances from each actual primitive point of one actual primitive point cloud to another actual primitive point cloud in each actual primitive point cloud group are respectively calculated; Set a first distance threshold and a first point threshold; The two actual primitive point clouds in all the actual primitive point cloud groups corresponding to the actual minimum Euclidean distance being less than the first distance threshold and the number of all the actual primitive points being greater than the first point number threshold are considered to be adjacent to each other, and the value of the actual adjacency attribute of the two adjacent actual primitive point clouds in each actual primitive point cloud group is marked as 1; If two of the actual primitive point clouds in the actual primitive point cloud group are both the actual plane primitive point clouds, then the actual normal vector angle between the actual plane equations corresponding to the two actual primitive point clouds is calculated; when the value of the actual normal vector angle is less than the actual normal vector angle threshold, the two actual primitive point clouds are considered to be parallel, and the values ​​of the actual parallel attributes of the two actual primitive point clouds are marked as 1; when the sum of the absolute values ​​of the differences between the coefficients contained in the actual plane equations corresponding to the two actual primitive point clouds is less than the actual coefficient threshold, the two actual primitive point clouds are considered to be coplanar, and the values ​​of the actual coplanar attributes of the two actual primitive point clouds are marked as 1; If two actual primitive point clouds in the actual primitive point cloud group are both actual cylindrical primitive point clouds, the two actual primitive point clouds are considered to be non-parallel and non-coplanar, and the values ​​of the actual parallel attribute, the actual coplanar attribute, and the actual normal vector angle of the two actual primitive point clouds are all marked as 0; If one of the actual primitive point clouds in the actual primitive point cloud group is the actual plane primitive point cloud, and the other actual primitive point cloud is the actual cylindrical primitive point cloud, the values ​​of the actual normal vector angle, the values ​​of the actual parallel attribute, and the values ​​of the actual coplanar attribute of the two actual primitive point clouds are all marked as 0; Each actual primitive point cloud in the actual primitive library is used as a vertex of the actual graph, and any two actual graph vertices are connected to each other to form corresponding actual connection edges, thereby constructing the actual point cloud graph corresponding to the actual point cloud; Wherein, each vertex of the actual point cloud is numbered according to Arabic numerals; the value of each actual connecting edge consists of six digits, each digit in the units place is the value of the actual coplanar attribute, each digit in the tens place is the value of the actual parallel attribute, each digit in the hundreds place is the value of the actual adjacency attribute, and each digit from the hundreds of thousands place to the thousands place is the value of the actual normal vector angle; The types of the actual graph vertices include actual planar graph vertices and actual cylindrical graph vertices.

7. The product assembly stage identification method based on point cloud geometric primitive information according to claim 6, characterized in that: The steps of traversing all the model primitive libraries, taking each model primitive point cloud in each model primitive library as a model graph vertex, establishing a model connection edge relationship between any two model primitive point clouds in each model primitive library, and constructing the model point cloud graph corresponding to each model point cloud include: From the first model primitive library, any two model primitive point clouds are respectively grouped into a model primitive point cloud group, and in each model primitive point cloud group, the minimum Euclidean distance of all models from each model primitive point of one model primitive point cloud to another model primitive point cloud is respectively calculated; Set a second distance threshold and a second point threshold; Two of the model primitive point clouds in all the model primitive point cloud groups corresponding to the model minimum Euclidean distance being less than the second distance threshold and the number of all the model primitive points being greater than the second point number threshold are considered to be adjacent to each other, and the value of the model adjacency attribute of the two adjacent model primitive point clouds in each model primitive point cloud group is marked as 1; If two of the model primitive point clouds in the model primitive point cloud group are both the model plane primitive point clouds, then the model normal vector angle between the model plane equations corresponding to the two model primitive point clouds is calculated; when the value of the model normal vector angle is less than the model normal vector angle threshold, the two model primitive point clouds are considered to be parallel, and the values ​​of the model parallel attributes of the two model primitive point clouds are marked as 1; when the sum of the absolute values ​​of the differences between the coefficients contained in the model plane equations corresponding to the two model primitive point clouds is less than the model coefficient threshold, the two model primitive point clouds are considered to be coplanar, and the values ​​of the model coplanar attributes of the two model primitive point clouds are marked as 1; If two of the model primitive point clouds in the model primitive point cloud group are both the model cylindrical primitive point clouds, the two model primitive point clouds are considered to be non-parallel and non-coplanar, and the values ​​of the model parallel attribute, the values ​​of the model coplanar attribute, and the values ​​of the model normal vector angle of the two model primitive point clouds are all marked as 0; If one of the model primitive point clouds in the model primitive point cloud group is the model plane primitive point cloud and the other model primitive point cloud is the model cylinder primitive point cloud, the values ​​of the model normal vector angle, the values ​​of the model parallel attribute and the values ​​of the model coplanar attribute of the two model primitive point clouds are all marked as 0; Each of the model primitive point clouds in the first model primitive library is used as a vertex of the model graph, and any two of the model graph vertices are connected to form corresponding model connection edges, thereby constructing the model point cloud graph corresponding to each of the model point clouds; Wherein, each vertex of the model graph in the model point cloud graph is numbered according to Arabic numerals; the value of each model connecting edge consists of six digits, each digit in the units place is the value of the coplanar attribute of the model, each digit in the tens place is the value of the parallel attribute of the model, each digit in the hundreds place is the value of the adjacency attribute of the model, and each digit from the hundreds of thousands place to the thousands place is the value of the angle between the normal vectors of the model; Iterate all the above steps until the model point cloud images corresponding to all the model point clouds are constructed respectively; The types of the model graph vertices include model plane graph vertices and model cylinder graph vertices.

8. The product assembly stage identification method based on point cloud geometric primitive information according to claim 7 is characterized in that: The steps of respectively comparing the similarity of each of the model point cloud images with the actual point cloud image and completing the corresponding coarse registration include: Comparing the actual point cloud image with the first model point cloud image, if the number of all model point cloud vertices of a certain type in the first model point cloud image is less than the number of all actual point cloud vertices of the same type in the actual point cloud image, skipping the first model point cloud image and continuing to compare the actual point cloud image with the next model point cloud image, otherwise continuing to the subsequent steps; All the actual connected edges marked with a value of 1 in the actual adjacency attribute in the actual point cloud graph are regarded as actual real edges, and all the actual real edges are respectively counted into the actual degree of the corresponding at least one vertex in the actual graph; All the model connection edges marked with a value of 1 in the model adjacency attribute in the first model point cloud graph are regarded as model real edges, and all the model real edges are respectively counted into the model degree of the corresponding at least one vertex in the model graph; Arrange all the actual graph vertices in the actual point cloud graph in descending order according to the number of all the actual degrees contained in each of them, and arrange all the model graph vertices in the first model point cloud graph in descending order according to the number of all the model degrees contained in each of them; According to the descending order of the actual point cloud graph, the type of each vertex of the actual graph and the number of all the actual degrees contained therein are counted respectively; Selecting, in the first model point cloud graph, all the model graph vertices of the same type as each actual graph vertex and containing a number of all the model degrees greater than or equal to a number of all the actual degrees as a candidate model graph vertex set corresponding to each actual graph vertex; Constructing a mapping array having the same number as all the actual point cloud vertices in the actual point cloud image, and constructing a label array having the same number as all the model point cloud vertices in the first model point cloud image; Traversing all the candidate model graph vertex sets, respectively putting all the candidate model graph vertices in each candidate model graph vertex set into the mapping array as mapping model graph vertices, and marking them accordingly in the marking array; When there are at least two mapping model graph vertices in the mapping array, check whether the values ​​of the model adjacency attribute, the model coplanarity attribute, and the model parallelity attribute between any two mapping model graph vertices are all one-to-one corresponding to the values ​​of the actual adjacency attribute, the actual coplanarity attribute, and the actual parallelity attribute between the corresponding two actual graph vertices, and whether the absolute value of the difference between the model normal vector angle between any two mapping model graph vertices and the actual normal vector angle between the corresponding two actual graph vertices is within the absolute value threshold. If there is a non-corresponding equality or the absolute value of the difference is not within the absolute value threshold, the mapping relationship is considered unreasonable. At this time, the mapping array is restored to a reasonable mapping relationship state, and the next candidate model graph vertex is selected to execute the above mapping process until a reasonable mapping relationship set corresponding to each actual graph vertex in the actual point cloud is found. According to all the reasonable mapping relationship sets, respectively record the frequency of each actual graph vertex being mapped to all corresponding mapping model graph vertices, and respectively calculate the sum of the frequencies corresponding to each actual graph vertex; Dividing each of the frequencies by the corresponding sum of the frequencies respectively, to obtain a mapping probability of each of the actual graph vertices being mapped to all corresponding mapping model graph vertices; Traversing all the actual graph vertices, respectively forming graph vertex pairs with each actual graph vertex and at least one corresponding mapping model graph vertex with the highest mapping probability, and recording the corresponding mapping probabilities; Arrange all the actual graph vertices in descending order according to the corresponding mapping probabilities, and use at least one of the graph vertex pairs that ranks first as candidate graph vertex pairs; Calculating, for each pair of candidate graph vertices, the actual mean center and the actual mean normal vector of the actual primitive point cloud corresponding to the actual graph vertex, and the model mean center and the model mean normal vector of the model primitive point cloud corresponding to the mapped model graph vertex; Translating each of the actual average centers to the corresponding model average center, and rotating each of the actual average normal vectors to the corresponding model average normal vector, to obtain a coarse registration transformation matrix corresponding to the first model point cloud; Multiplying the coarse registration transformation matrix by the actual point cloud to obtain a corresponding actual coarse registration transformation point cloud, wherein the actual coarse registration transformation point cloud is the coarse registration result of the first model point cloud and the actual point cloud; All the above steps are iterated until all the coarse registration results are obtained.

9. The product assembly stage identification method based on point cloud geometric primitive information according to claim 8, characterized in that: The steps of respectively performing fine registration on the actual point cloud and each of the model point clouds based on all the coarse registration results, and identifying the assembly stage of the actual workpiece based on all the fine registration results include: All the actual coarse registration transformation point clouds are respectively used as source point clouds, and all the model point clouds are used as target point clouds; Inputting the first source point cloud and all the target point clouds into an iterative closest point algorithm to obtain a set of actual precisely registered transformed point clouds; Iteratively inputting the source point cloud and all the target point clouds until the actual fine registration transformed point cloud set corresponding to each source point cloud is obtained; each of the actual fine registration transformed point cloud sets is the corresponding fine registration result; Calculating the root mean square error between any of the model point clouds in each of the actual fine registration transformed point cloud sets and the corresponding actual fine registration transformed point cloud; When the root mean square error is greater than a root mean square error threshold, it is considered that the corresponding model point cloud is not the assembly stage of the actual point cloud; When the root mean square error is less than or equal to the root mean square error threshold, perform the following operations: respectively calculating the closest Euclidean distances between all actual precisely registered transformed points in each actual precisely registered transformed point cloud and the closest model sampling point in the corresponding model point cloud; Counting the total number of all the actual fine-registration transformation points corresponding to the nearest Euclidean distance greater than the Euclidean distance threshold in each of the actual fine-registration transformation point clouds, and calculating the proportion of all the total number in the corresponding actual fine-registration transformation point cloud; If the number ratio is greater than the ratio threshold, it is considered that the corresponding model point cloud is not the assembly stage of the actual point cloud; If the quantity ratio is less than or equal to the ratio threshold, it is considered that the corresponding model point cloud is in the assembly stage of the actual point cloud.

10. The product assembly stage identification method based on point cloud geometric primitive information according to claim 9, characterized in that: The expression for calculating the root mean square error is: (1) in, represents the root mean square error, Represents the number of all model sampling points in the model point cloud, Indicates the first Model sampling points, Indicates the actual precise registration transformation point cloud and the corresponding model point cloud The actual precise registration transformation point with the closest distance between the model sampling points is Indicates precise alignment, represents the Euclidean norm, express and The Euclidean distance between them.

Images