Product assembly stage recognition method based on point cloud geometric primitive information

Abstract

The application provides a product assembly stage recognition method based on point cloud geometric primitive information, and belongs to the technical field of computer-aided assembly and simulation. The method comprises the following steps: acquiring model point clouds corresponding to workpiece models of each assembly stage respectively, and acquiring actual point clouds corresponding to an actual workpiece; constructing an actual primitive library and a plurality of model primitive libraries by using geometric primitive division; constructing model point cloud graphs corresponding to each model point cloud by using all the model primitive libraries, and constructing an actual point cloud graph by using the actual primitive library; completing coarse registration of each model point cloud and the actual point cloud respectively; performing fine registration of the actual point cloud and each model point cloud according to all coarse registration results, and recognizing the assembly stage of the actual workpiece according to all fine registration results. The application can realize accurate recognition of the assembly stage of a product without identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer-aided assembly and simulation, and particularly relates to a product assembly stage identification method based on point cloud geometric primitive information. BACKGROUND

[0002] Augmented assembly technology is an important enabling technology for realizing virtual-real interaction and virtual fusion simulation of assembly objects. It is of great significance to carry out product assembly process feasibility verification and improve product assembly efficiency. The pushing mechanism of process information in the assembly process is not only one of the core processes of augmented assembly technology, but also deeply bound with each assembly stage of the product. Therefore, if only relying on manual switching of different assembly stages of the product, the assembly efficiency of the product will be affected to some extent. Therefore, it is necessary to automatically and quickly identify each assembly stage of the product.

[0003] A Chinese invention patent with the title of "A non-identification assembly state recognition method based on product depth image point cloud" and the publication number of CN114842221A points out that: by encoding the actually collected point cloud and the point cloud generated by the digital model respectively, and constructing the corresponding shape vector descriptor, the cosine similarity between each shape vector descriptor is calculated, so as to complete the recognition of the product assembly state. When constructing the shape vector descriptor of the point cloud, the point cloud needs to be sampled more than or equal to times of point pair distance. Especially when dealing with a digital model composed of point clouds from multiple perspectives, the above sampling process needs to be performed on the point cloud under any perspective, resulting in the consumption of computing resources and time increasing exponentially with the number of perspectives. The shape vector descriptor can only record the distance between randomly sampled point pairs, which inevitably loses part of the point cloud information, and the shape vector descriptor is stored by classifying and counting the distance information between point pairs according to the distance range in the form of equal interval histogram, which further weakens the point cloud information contained in the shape vector descriptor, so that the shape vector descriptor obtained by using the method can only describe less point cloud information.

[0004] Another Chinese invention patent with the title of "An assembly state recognition method based on three-dimensional feature points" and the publication number CN113111741A points out that all feature points are selected based on the fillets of each node in the point cloud, the spatial position relationship of all feature points is recorded as the descriptor of the point cloud, and the registration between point clouds is completed by establishing the mapping relationship of all feature points in different descriptors, and the registration result is used to assist the recognition of the product assembly state. Since this method is based on the fillet of each node to select feature points, these feature points are gathered near the corner points or edges, which largely ignores the point clouds that are macroscopically planar or cylindrical. This causes the descriptor to lose some point clouds. In addition, when searching for the correspondence of all feature points, this method uses a traversal mode to search, which reduces the assembly state recognition efficiency to some extent.

[0005] Therefore, for general mechanical products, it is necessary to include certain geometric primitive information such as planes and cylinders during the design and processing stages, so that the point clouds of such products macroscopically contain these geometric primitive information. The methods used in the above two patent documents ignore the characteristic that mechanical products generally contain geometric primitive information, and treat the point cloud of the mechanical product as a general type of point cloud. The descriptor used only focuses on the relationship between points and point pairs, without considering the characteristics of mechanical products and their point clouds, making the descriptor complex and lacking in feature description capability for point clouds, further increasing the difficulty of finding the correspondence between descriptors, and directly affecting the judgment result in the final assembly stage.

[0006] Therefore, it is necessary to propose a scheme to improve one or more problems in the above-mentioned related technical solutions.

[0007] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0008] The present application provides a product assembly stage recognition method based on point cloud geometric primitive information, which comprises the following steps:

[0009] Obtain the model point cloud corresponding to each assembly stage workpiece model respectively, and obtain the actual point cloud corresponding to the actual workpiece, wherein each model point cloud respectively contains model normal vector information, and each actual point cloud respectively contains actual normal vector information;

[0010] The actual point cloud and all the model point clouds are respectively divided into geometric primitives, all the obtained actual primitive point clouds are respectively stored in an actual primitive library, and all the obtained model primitive point clouds are respectively stored in a corresponding model primitive library;

[0011] All the model primitive libraries are used to respectively construct a model point cloud graph corresponding to each model point cloud, and the actual primitive library is used to construct an actual point cloud graph;

[0012] Each model point cloud graph is respectively compared with the actual point cloud graph in terms of similarity, and a corresponding coarse registration is completed;

[0013] According to all the coarse registration results, the actual point cloud is respectively fine registered with each model point cloud, and the assembly stage of the actual workpiece is identified according to all the fine registration results.

[0014] Further, the steps of respectively acquiring a model point cloud corresponding to each assembly stage workpiece model and acquiring an actual point cloud corresponding to the actual workpiece include:

[0015] According to the length, width and height of all the workpiece models, a bounding box adapted to each workpiece model is respectively generated;

[0016] The center of each bounding box is taken as the center of a sphere, and a preset multiple of the distance from the center of the sphere to the farthest corner point on the corresponding bounding box is taken as the radius, to construct a peripheral sphere adapted to each workpiece model;

[0017] A plurality of sampling points are respectively set on each peripheral sphere, and a virtual camera is respectively set on each sampling point;

[0018] All the virtual cameras are used to sample the corresponding workpiece model to obtain a model point cloud;

[0019] The actual workpiece is sampled to obtain an actual point cloud.

[0020] Further, the step of performing geometric primitive division on the actual point cloud and storing all the obtained actual primitive point clouds in the actual primitive library respectively includes:

[0021] An initial actual growth point is randomly selected from the actual point cloud, the actual normal vector of the initial actual growth point is recorded, and the initial actual growth point and the actual normal vector are stored in an actual growth point set;

[0022] The actual Euclidean distances between the initial actual growth point and all the other actual sampling points are respectively calculated, and all the actual sampling points corresponding to an actual Euclidean distance less than an actual distance threshold are taken as the actual neighborhood points of the initial actual growth point;

[0023] respectively, and the initial actual growing point is marked as used;

[0024] respectively, and the initial actual growing point is marked as used;

[0025] respectively, and the initial actual growing point is marked as used;

[0026] respectively, and the initial actual growing point is marked as used;

[0027] respectively, and the initial actual growing point is marked as used;

[0028] respectively, and the initial actual growing point is marked as used;

[0029] respectively, and the initial actual growing point is marked as used;

[0030] respectively, and the initial actual growing point is marked as used;

[0031] respectively, and the initial actual growing point is marked as used;

[0032] Further, the step of respectively performing geometric primitive division on all model point clouds, and respectively storing all obtained model primitive point clouds into corresponding model primitive libraries comprises:

[0033] randomly selecting one model sampling point from the first model point cloud as an initial model growing point, recording a model normal vector of the initial model growing point, and storing the initial model growing point and the model normal vector into a model growing point set;

[0034] respectively calculating model Euclidean distances between the initial model growing point and all other model sampling points, and taking all model sampling points corresponding to model Euclidean distances less than a model distance threshold as model neighborhood points of the initial model growing point;

[0035] respectively calculating model included angles between normal vectors of each model neighborhood point and the model normal vector of the initial model growing point, storing all model neighborhood points corresponding to model included angles less than a model included angle threshold into the model growing point set, and marking the initial model growing point as used;

[0036] respectively taking each model neighborhood point in the model growing point set as a model growing point, and iteratively performing the above steps of calculating and comparing model Euclidean distances and model included angles, and excluding the initial model growing point marked as used and all model growing points at each iteration, until all model growing points are marked as used, to obtain a final model growing point set corresponding to the initial model growing point;

[0037] iteratively performing all the above steps until a final model growing point set corresponding to each initial model growing point is obtained;

[0038] respectively taking initial model growing points and all model growing points extracted from each final model growing point set as model plane primitive points, all model plane primitive points respectively forming corresponding model plane primitive point clouds, and storing all model plane primitive point clouds into corresponding model primitive libraries;

[0039] respectively taking all model plane primitive point clouds containing less than a model plane primitive point quantity threshold of model plane primitive points from corresponding model primitive libraries, and respectively as model new point clouds;

[0040] respectively obtaining model plane equations corresponding to each model plane primitive point cloud remaining in the corresponding model primitive library after taking out all model new point clouds, by using a plane fitting method based on random sample consensus;

[0041] performing Euclidean distance clustering on all model new point clouds to obtain multiple model clustering clusters;

[0042] respectively, and respectively projecting all the model new point clouds in each model clustering cluster onto the corresponding model base element library by using the minimum model feature vectors obtained by the principal component analysis of the normal vectors of all the model new point clouds in each model clustering cluster, respectively performing roundness detection on all the projection results, and respectively taking all the model new point clouds corresponding to the model roundness less than the model roundness threshold as the model cylindrical primitive point clouds corresponding to each model clustering cluster, and storing all the model cylindrical primitive point clouds into the corresponding model base element library, each of the model cylindrical primitive point clouds containing a plurality of model cylindrical primitive points, and each of the model cylindrical primitive points being a model new sampling point in the model new point cloud.

[0043] Iteratively performing all the above steps until all the model cylindrical primitive point clouds corresponding to each model clustering cluster are stored into the corresponding model base element library.

[0044] The types of the model primitive point clouds include model planar primitive point clouds and model cylindrical primitive point clouds.

[0045] Further, the model point cloud graph corresponding to each model point cloud is respectively constructed by using all the model base element libraries, and the step of constructing the actual point cloud graph by using the actual base element library includes:

[0046] Traversing the actual base element library, taking all the actual primitive point clouds in the actual base element library as actual graph vertices, and respectively establishing actual connection edge relationships between any two actual primitive point clouds to construct the actual point cloud graph.

[0047] Traversing all the model base element libraries, taking each model primitive point cloud in each model base element library as a model graph vertex, and respectively establishing model connection edge relationships between any two model primitive point clouds in each model base element library to respectively construct the model point cloud graph corresponding to each model point cloud.

[0048] Further, the step of traversing the actual base element library, taking all the actual primitive point clouds in the actual base element library as actual graph vertices, and respectively establishing actual connection edge relationships between any two actual primitive point clouds to construct the actual point cloud graph includes:

[0049] In the actual base element library, any two actual primitive point clouds are respectively taken as an actual primitive point cloud group, and all the actual minimum Euclidean distances from each actual primitive point in one of the actual primitive point clouds to the other actual primitive point cloud are respectively calculated.

[0050] A first distance threshold and a first point number threshold are set.

[0051] If the actual minimum Euclidean distance is less than the first distance threshold value and the number of all actual primitive points is greater than the first point number threshold value, two actual primitive point clouds in all actual primitive point cloud groups corresponding to the actual primitive point cloud are considered to be adjacent to each other, and the value of the actual adjacent attribute of the two actual primitive point clouds adjacent to each other in each actual primitive point cloud group is marked as 1 respectively;

[0052] If the two actual primitive point clouds in the actual primitive point cloud group are both actual plane primitive point clouds, the actual normal vector included between the actual plane equations corresponding to the two actual primitive point clouds is calculated; when the value of the actual normal vector is less than the actual normal vector threshold value, the two actual primitive point clouds are considered to be parallel, and the value of the actual parallel attribute of the two actual primitive point clouds is marked as 1; when the sum of the absolute values of the differences of the coefficients included in the actual plane equations corresponding to the two actual primitive point clouds is less than the actual coefficient threshold value, the two actual primitive point clouds are considered to be coplanar, and the value of the actual coplanar attribute of the two actual primitive point clouds is marked as 1;

[0053] If the two actual primitive point clouds in the actual primitive point cloud group are both actual cylindrical primitive point clouds, the two actual primitive point clouds are considered to be non-parallel and non-coplanar, and the value of the actual parallel attribute, the value of the actual coplanar attribute and the value of the actual normal vector included between the two actual primitive point clouds are all marked as 0;

[0054] If one actual primitive point cloud in the actual primitive point cloud group is an actual plane primitive point cloud and the other actual primitive point cloud is an actual cylindrical primitive point cloud, the value of the actual normal vector included between the two actual primitive point clouds, the value of the actual parallel attribute and the value of the actual coplanar attribute are all marked as 0;

[0055] Each actual primitive point cloud in the actual primitive library is taken as an actual graph vertex, and any two actual graph vertices are connected to each other to form a corresponding actual connection edge, so as to construct an actual point cloud graph corresponding to the actual point cloud;

[0056] Each actual graph vertex in the actual point cloud graph is labeled according to Arabic numerals; the value of each actual connection edge is composed of six digits, each individual digit is the value of the actual coplanar attribute, each ten digit is the value of the actual parallel attribute, each hundred digit is the value of the actual adjacent attribute, and each ten million digit to thousand digit is the value of the actual normal vector included;

[0057] The type of the actual graph vertex includes an actual plane graph vertex and an actual cylindrical graph vertex.

[0058] Further, the step of constructing the model point cloud graph corresponding to each model point cloud respectively includes:

[0059] From the first model primitive library, any two model primitive point clouds are grouped into a model primitive point cloud group, and the minimum model Euclidean distance from each model primitive point of one model primitive point cloud to all model primitive points of the other model primitive point cloud in each model primitive point cloud group is calculated respectively;

[0060] A second distance threshold and a second point threshold are set;

[0061] If the minimum model Euclidean distance is less than the second distance threshold, and the number of all model primitive points is greater than the second point threshold, the two model primitive point clouds in the corresponding all model primitive point cloud group are considered to be adjacent to each other, and the value of the model adjacent attribute of the two model primitive point clouds adjacent to each other in each model primitive point cloud group is marked as 1 respectively;

[0062] If the two model primitive point clouds in the model primitive point cloud group are both model plane primitive point clouds, the model normal vector included between the model plane equations corresponding to the two model primitive point clouds is calculated; when the value of the model normal vector is less than the model normal vector threshold, the two model primitive point clouds are considered to be parallel, and the value of the model parallel attribute of the two model primitive point clouds is marked as 1; when the sum of the absolute values of the differences of the coefficients contained in the model plane equations corresponding to the two model primitive point clouds is less than the model coefficient threshold, the two model primitive point clouds are considered to be coplanar, and the value of the model coplanar attribute of the two model primitive point clouds is marked as 1;

[0063] If the two model primitive point clouds in the model primitive point cloud group are both model cylindrical primitive point clouds, the two model primitive point clouds are considered to be non-parallel and non-coplanar, and the value of the model parallel attribute, the value of the model coplanar attribute and the value of the model normal vector included between the two model primitive point clouds are all marked as 0;

[0064] If one model primitive point cloud in the model primitive point cloud group is a model plane primitive point cloud and the other model primitive point cloud is a model cylindrical primitive point cloud, the value of the model normal vector included between the two model primitive point clouds, the value of the model parallel attribute and the value of the model coplanar attribute are all marked as 0;

[0065] Each model primitive point cloud in the first model primitive library is taken as a model graph vertex, and any two model graph vertices are connected to each other to form a corresponding model connection edge, and a model point cloud graph corresponding to each model point cloud is constructed respectively;

[0066] Wherein, each model graph vertex in the model point cloud graph is labeled according to Arabic numerals; the value of each model connection edge consists of six digits, each single digit is the value of the model coplanar attribute, each ten digit is the value of the model parallel attribute, each hundred digit is the value of the model adjacency attribute, and each ten-thousandth digit to the thousandth digit is the value of the model normal vector angle;

[0067] Iteratively performing all the above steps until the model point cloud graphs corresponding to all the model point clouds are constructed respectively;

[0068] The type of the model graph vertex includes a model plane graph vertex and a model cylindrical graph vertex.

[0069] Further, the step of comparing the similarity of each model point cloud graph with the actual point cloud graph and completing the corresponding coarse registration includes:

[0070] Comparing the actual point cloud graph with the first model point cloud graph, if the number of all model graph vertices of a certain type in the first model point cloud graph is less than the number of all actual graph vertices of the same type in the actual point cloud graph, then skip the first model point cloud graph and continue to compare the actual point cloud graph with the next model point cloud graph, otherwise continue the subsequent steps;

[0071] Marking all actual connection edges in the actual point cloud graph with the value of the actual adjacency attribute as 1 as actual solid edges, and counting all actual solid edges into the actual degree of the corresponding at least one actual graph vertex respectively;

[0072] Marking all model connection edges in the first model point cloud graph with the value of the model adjacency attribute as 1 as model solid edges, and counting all model solid edges into the model degree of the corresponding at least one model graph vertex respectively;

[0073] Arranging all actual graph vertices in the actual point cloud graph in descending order according to the number of all actual degrees contained by each actual graph vertex, and arranging all model graph vertices in the first model point cloud graph in descending order according to the number of all model degrees contained by each model graph vertex;

[0074] According to the descending order arrangement of the actual point cloud graph, respectively counting the type of each actual graph vertex and the number of all actual degrees contained;

[0075] Selecting all model graph vertices in the first model point cloud graph which have the same type as each actual graph vertex and contain all model degrees whose number is greater than or equal to the number of all actual degrees as the candidate model graph vertex set of each actual graph vertex respectively;

[0076] Constructing a mapping array with the same number of actual graph vertices in the actual point cloud graph, and constructing a marking array with the same number of model graph vertices in the first model point cloud graph.

[0077] Traverse all candidate model graph vertex sets, respectively put all candidate model graph vertices in each candidate model graph vertex set into the mapping array as mapping model graph vertices respectively, and mark in the marking array respectively;

[0078] When there are at least two mapping model graph vertices in the mapping array, check whether the values of the model adjacency attribute, the model coplanar attribute, and the model parallel attribute between any two mapping model graph vertices are respectively equal to the values of the actual adjacency attribute, the actual coplanar attribute, and the actual parallel attribute between the corresponding two actual graph vertices, and whether the absolute value of the difference between the model normal vector included angle between any two mapping model graph vertices and the actual normal vector included angle between the corresponding two actual graph vertices is within the absolute value threshold, if there is a situation that is not equal or the absolute value of the difference is not within the absolute value threshold, it is considered that the mapping relationship is unreasonable, at this time, the mapping array is restored to the reasonable mapping relationship state, and the next candidate model graph vertex is selected to continue the above mapping process until a reasonable mapping relationship set corresponding to each actual graph vertex in the actual point cloud graph is found respectively;

[0079] According to all reasonable mapping relationship sets, record the frequency of each actual graph vertex being mapped to the corresponding all mapping model graph vertices respectively, and calculate the frequency sum corresponding to each actual graph vertex respectively;

[0080] Divide each frequency by the corresponding frequency sum to obtain the mapping probability of each actual graph vertex being mapped to the corresponding all mapping model graph vertices;

[0081] Traverse all actual graph vertices, respectively form graph vertex pairs by each actual graph vertex and at least one mapping model graph vertex with the highest corresponding mapping probability, and record the corresponding mapping probability;

[0082] Arrange all actual graph vertices in descending order according to the size of the corresponding mapping probability, and arrange at least one graph vertex pair in the first place as a candidate graph vertex pair respectively;

[0083] Calculate the actual average center and the actual average normal vector of the actual primitive point cloud corresponding to the actual graph vertex, and the model average center and the model average normal vector of the model primitive point cloud corresponding to the mapping model graph vertex in each candidate graph vertex pair respectively;

[0084] Translate each actual average center to the corresponding model average center, and rotate each actual average normal vector to the corresponding model average normal vector to obtain a coarse registration transformation matrix corresponding to the first model point cloud;

[0085] The coarse registration transformation matrix is multiplied by the actual point cloud to obtain a corresponding actual coarse registration transformation point cloud, which is the coarse registration result of the first model point cloud and the actual point cloud.

[0086] The above steps are iteratively performed until all coarse registration results are obtained.

[0087] Further, according to all coarse registration results, the actual point cloud is respectively fine registered with each model point cloud, and the assembly stage of the actual workpiece is identified according to all fine registration results. The steps include:

[0088] All actual coarse registration transformation point clouds are respectively taken as source point clouds, and all model point clouds are taken as target point clouds.

[0089] The first source point cloud and all target point clouds are input into an iterative closest point algorithm to obtain an actual fine registration transformation point cloud set.

[0090] The input process of the source point cloud and all target point clouds is iteratively performed until the actual fine registration transformation point cloud set corresponding to each source point cloud is obtained; each actual fine registration transformation point cloud set is a corresponding fine registration result.

[0091] The root mean square error between any model point cloud in each actual fine registration transformation point cloud set and the corresponding actual fine registration transformation point cloud is respectively calculated.

[0092] When the root mean square error is greater than a root mean square error threshold, it is considered that the corresponding model point cloud is not the assembly stage in which the actual point cloud is located.

[0093] When the root mean square error is less than or equal to the root mean square error threshold, the following operations are performed:

[0094] The nearest Euclidean distance between all actual fine registration transformation points in each actual fine registration transformation point cloud and the nearest model sampling point in the corresponding model point cloud is respectively calculated.

[0095] The total number of all actual fine registration transformation points in each actual fine registration transformation point cloud, whose nearest Euclidean distance is greater than a Euclidean distance threshold, is respectively counted, and the proportion of the number in the corresponding actual fine registration transformation point cloud is respectively calculated.

[0096] If the proportion is greater than a proportion threshold, it is considered that the corresponding model point cloud is not the assembly stage in which the actual point cloud is located.

[0097] If the proportion is less than or equal to the proportion threshold, it is considered that the corresponding model point cloud is the assembly stage in which the actual point cloud is located.

[0098] Further, the expression for calculating the root mean square error is:

[0099] Further, the expression for calculating the root mean square error is: (1)

[0100] wherein, denotes the root mean square error, denotes the number of all model sampling points in the model point cloud, denotes the i-th model sampling point in the model point cloud, denotes the actual fine registration transformation point closest to the i-th model sampling point in the corresponding model point cloud in the actual fine registration transformation point cloud, denotes the fine registration, denotes the Euclidean norm, denotes the and Euclidean distance between and. The application provides a product assembly stage recognition method based on point cloud geometric primitive information, and has at least the following beneficial effects:

[0101] (1) The application uses the geometric primitive information of the product itself as the global descriptor of the product point cloud, avoiding the influence of the multi-angled characteristics of the product on the selection of feature point positions, reducing the insufficient description of general descriptors for flat area point clouds, and using the topological and positional relationship between geometric primitive information as a descriptor, avoiding the need for multiple descriptors to describe the same primitive point cloud, simplifying the number and dimension of the descriptors;

[0102] (2) The application establishes a credibility mechanism for the correspondence relationship between the graph vertices of the actual point cloud graph and the model point cloud graph, to a certain extent, reducing the influence of multiple graph vertex correspondence relationships on the correct correspondence relationship, realizing the coarse registration of the point cloud through the correspondence relationship between the graph vertices of the actual point cloud graph and the model point cloud graph, reducing the possibility of falling into local optimization in the fine registration stage, thereby completing the recognition of different assembly stages of the product, and reducing the influence of frequent manual switching of assembly stages on the product assembly efficiency.

[0103] (2) The application establishes a credibility mechanism for the correspondence relationship between the graph vertices of the actual point cloud graph and the model point cloud graph, to a certain extent, reducing the influence of multiple graph vertex correspondence relationships on the correct correspondence relationship, realizing the coarse registration of the point cloud through the correspondence relationship between the graph vertices of the actual point cloud graph and the model point cloud graph, reducing the possibility of falling into local optimization in the fine registration stage, thereby completing the recognition of different assembly stages of the product, and reducing the influence of frequent manual switching of assembly stages on the product assembly efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0104] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application. It is apparent that the accompanying drawings in the following description are only some embodiments of the application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0105] Figure 1 The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application. It is apparent that the accompanying drawings in the following description are only some embodiments of the application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0106] ​Figure 2 Fig. 1 shows a flowchart of a product assembly stage recognition method based on point cloud geometric primitive information according to an example embodiment of the present application;

[0107] Figure 3 Fig. 2 shows a schematic diagram of workpiece models in different assembly stages according to an example embodiment of the present application;

[0108] Figure 4 Fig. 3 shows a schematic diagram of setting different numbers of virtual cameras on the bounding box of a workpiece model in the fourth assembly stage according to an example embodiment of the present application;

[0109] Figure 5 Fig. 4 shows a schematic diagram of collecting model point clouds by multiple virtual cameras for a workpiece model in the fourth assembly stage according to an example embodiment of the present application;

[0110] Figure 6 Fig. 5 shows a schematic diagram of actual point clouds corresponding to an actual workpiece according to an example embodiment of the present application;

[0111] Figure 7 Fig. 6 shows a schematic diagram of an actual point cloud map according to an example embodiment of the present application;

[0112] Figure 8 Fig. 7 shows a schematic diagram of a model point cloud map corresponding to the fourth assembly stage according to an example embodiment of the present application;

[0113] Figure 9 Fig. 8 shows a schematic diagram of the poses of actual point clouds and model point clouds in the fourth assembly stage without coarse registration according to an example embodiment of the present application;

[0114] Figure 10 Fig. 9 shows a schematic diagram of the poses of actual point clouds and model point clouds in the fourth assembly stage after coarse registration according to an example embodiment of the present application;

[0115] Figure 11 Fig. 10 shows a schematic diagram of the result of directly performing fine registration on actual point clouds and model point clouds in the fourth assembly stage without coarse registration according to an example embodiment of the present application;

[0116] Figure 12 Fig. 11 shows a schematic diagram of the result of performing fine registration on actual point clouds and model point clouds in the fourth assembly stage after coarse registration according to an example embodiment of the present application;

[0117] Figure 13 Fig. 12 shows a schematic diagram of the result of recognizing different assembly stages of an actual workpiece according to an example embodiment of the present application. DETAILED DESCRIPTION

[0118] Example implementations are now described with reference to the drawings. Example implementations can, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these examples are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.

[0119] In addition, the accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application. In the drawings:

[0120] In the following, a product assembly state recognition method based on point cloud geometric primitive information proposed in the embodiments of the present application will be described in more detail.

[0121] In view of the problem that general descriptors do not make full use of the geometric primitive information of mechanical products, a product assembly state recognition method based on point cloud geometric primitive information is proposed in the example embodiments, as shown in Figure 1 and Figure 2 The method can include the following steps:

[0122] The embodiment step S101: as shown in Figure 3 , the model point cloud corresponding to each assembly stage of the workpiece model is obtained respectively, and the actual point cloud corresponding to the actual workpiece is obtained.

[0123] Figure 3 a in the above formula (1) shows the workpiece model of the first assembly stage, Figure 3 b in the above formula (2) shows the workpiece model of the second assembly stage, Figure 3 c in the above formula (3) shows the workpiece model of the third assembly stage, Figure 3 d in the above formula (4) shows the workpiece model of the fourth assembly stage.

[0124] Further, each model point cloud respectively contains model normal vector information, and each actual point cloud respectively contains actual normal vector information.

[0125] Further, as shown in Figure 4 and Figure 5 , the embodiment step S101 can include the following sub-steps:

[0126] Sub-step S1011: According to the length, width and height of all workpiece models, a bounding box adapted to each workpiece model is respectively generated.

[0127] Sub-step S1012: A peripheral sphere adapted to each workpiece model is respectively constructed with the center of each bounding box as the center of the sphere and with the distance from the center of the sphere to the farthest corner point on the corresponding bounding box as the radius of the sphere. In this embodiment, the preset multiple is preferably 1.5 times.

[0128] Sub-step S1013: A plurality of sampling points are respectively and uniformly arranged on each peripheral sphere, and a virtual camera is respectively arranged on each sampling point.

[0129] Figure 4 The case of setting different numbers of virtual cameras on the peripheral sphere of the workpiece model in the fourth assembly stage is given. Figure 4 In a of a, only 1 virtual camera is included, Figure 4 In b of a, 11 virtual cameras are included, Figure 4 In c of a, 15 virtual cameras are included.

[0130] Figure 4 And Figure 5 In the above figures, 101 represents the virtual camera respectively arranged on each sampling point, 102 represents the model point cloud, 103 represents the workpiece model, 104 represents the bounding box, and 105 represents the peripheral sphere.

[0131] Sub-step S1014: The corresponding workpiece model is respectively sampled by using all virtual cameras to obtain a model point cloud.

[0132] Sub-step S1015: As shown in Figure 6 , the actual workpiece is generally sampled by using a point cloud acquisition device to obtain an actual point cloud.

[0133] In this embodiment, step S102: The actual point cloud and all model point clouds are respectively divided into geometric primitives, and all actual primitive point clouds obtained are respectively stored in an actual primitive library, and all model primitive point clouds obtained are respectively stored in a corresponding model primitive library. The step S102 of this embodiment can include the following sub-steps:

[0134] Sub-step S1021: The actual point cloud is divided into geometric primitives, and all actual primitive point clouds obtained are respectively stored in an actual primitive library, and the specific process includes:

[0135] First step: randomly select an actual sampling point from the actual point cloud as an initial actual growing point, record the actual normal vector of the initial actual growing point, and store the initial actual growing point and its actual normal vector into the actual growing point set.

[0136] Second step: calculate the actual Euclidean distance between the initial actual growing point and all other actual sampling points respectively, and all actual sampling points corresponding to an actual Euclidean distance less than the actual distance threshold are taken as the actual neighborhood points of the initial actual growing point.

[0137] Third step: calculate the actual included angle between the normal vector of each actual neighborhood point and the actual normal vector of the initial actual growing point respectively, and all actual neighborhood points corresponding to an actual included angle value less than the actual included angle threshold are stored into the actual growing point set, and the initial actual growing point is marked as used.

[0138] Fourth step: each actual neighborhood point in the actual growing point set is taken as an actual growing point in turn, and the above steps of calculating and comparing actual Euclidean distance and actual included angle are iteratively performed, and the initial actual growing point and all actual growing points marked as used are excluded in each iteration, until all actual growing points are marked as used, to obtain the final actual growing point set corresponding to the initial actual growing point.

[0139] Fifth step: iteratively perform all the above steps until the final actual growing point set corresponding to each initial actual growing point is obtained.

[0140] Sixth step: the initial actual growing point and all actual growing points extracted from each final actual growing point set are taken as actual plane primitive points respectively, and all actual plane primitive points form corresponding actual plane primitive point clouds, and all actual plane primitive point clouds are stored into the actual primitive library.

[0141] Seventh step: all actual plane primitive point clouds containing less than the actual plane primitive point number threshold of actual plane primitive points are taken out from the actual primitive library respectively, and are taken as actual new point clouds respectively.

[0142] Eighth step: using the plane fitting method based on random sample consensus, the actual plane equation corresponding to each actual plane primitive point cloud remaining in the actual primitive library after taking out all actual new point clouds is obtained.

[0143] Ninth step: perform Euclidean distance clustering on all actual new point clouds to obtain multiple actual clustering clusters.

[0144] Step 10: Perform principal component analysis on the normal vectors of all actual new point clouds in each actual clustering cluster respectively, project the corresponding actual new point clouds using all the minimum actual feature vectors obtained respectively, perform roundness detection on all the projection results respectively, and take all the actual new point clouds with actual roundness less than the actual roundness threshold as the actual cylindrical primitive point clouds corresponding to each actual clustering cluster respectively, and store all the actual cylindrical primitive point clouds in the actual primitive library. Each actual cylindrical primitive point cloud contains a plurality of actual cylindrical primitive points, and each actual cylindrical primitive point is an actual new sampling point in the actual new point cloud.

[0145] Here, the types of actual primitive point clouds include actual planar primitive point clouds and actual cylindrical primitive point clouds.

[0146] Sub-step S1022: Perform geometric primitive division on all model point clouds respectively, and store all the model primitive point clouds obtained in the corresponding model primitive library. The specific process includes:

[0147] Step 1: Randomly select a model sampling point from the first model point cloud as an initial model growth point, record the model normal vector of the initial model growth point, and store the initial model growth point and its model normal vector in the model growth point set.

[0148] Step 2: Calculate the model Euclidean distance between the initial model growth point and all other model sampling points respectively, and take all the model sampling points with model Euclidean distance less than the model distance threshold as the model neighborhood points of the initial model growth point.

[0149] Step 3: Calculate the model included angle between the normal vector of each model neighborhood point and the model normal vector of the initial model growth point, and store all the model neighborhood points with model included angle less than the model included angle threshold in the model growth point set, and mark the initial model growth point as used.

[0150] Step 4: Take each model neighborhood point in the model growth point set as a model growth point in turn, and iteratively perform the steps of calculating and comparing the model Euclidean distance and the model included angle described above, and exclude the initial model growth point marked as used and all model growth points at each iteration, until all model growth points are marked as used, to obtain the final model growth point set corresponding to the initial model growth point.

[0151] Step 5: Iteratively perform all the steps described above until the final model growth point set corresponding to each initial model growth point is obtained.

[0152] Step 6: Extract initial model growth points from each final model growth point set and all model growth points as model plane primitive points, respectively, and all model plane primitive points form corresponding model plane primitive point clouds, respectively, and store all model plane primitive point clouds in corresponding model primitive libraries.

[0153] Step 7: Take all model plane primitive point clouds containing less than the model plane primitive point quantity threshold from the corresponding model primitive library, respectively, and use them as model new point clouds.

[0154] Step 8: Use the plane fitting method based on random sample consensus to obtain the model plane equation corresponding to each remaining model plane primitive point cloud from the corresponding model primitive library after taking all model new point clouds.

[0155] Step 9: Perform Euclidean distance clustering on all model new point clouds to obtain multiple model clustering clusters.

[0156] Step 10: Perform principal component analysis on the normal vectors of all model new point clouds in each model clustering cluster, respectively, and use all obtained minimum model feature vectors to project the corresponding model new point clouds, respectively, perform roundness detection on all projection results, and take all model new point clouds with model roundness less than the model roundness threshold as corresponding model cylindrical primitive point clouds for each model clustering cluster, respectively, and store all model cylindrical primitive point clouds in corresponding model primitive libraries, each of which contains multiple model cylindrical primitive points, each of which is a model new sampling point in the model new point cloud.

[0157] Step 11: Iterate through all the above steps until all model cylindrical primitive point clouds corresponding to each model clustering cluster are stored in the corresponding model primitive library.

[0158] Here, the types of model primitive point clouds include model plane primitive point clouds and model cylindrical primitive point clouds.

[0159] Step S103 of the present embodiment: Use all model primitive libraries to construct a model point cloud graph corresponding to each model point cloud, respectively, and use the actual primitive library to construct an actual point cloud graph. Step S103 of the present embodiment can include the following sub-steps:

[0160] Sub-step S1031: As shown in Figure 7 , traverse the actual primitive library, take all actual primitive point clouds in the actual primitive library as actual graph vertices, respectively, and establish actual connection edge relationships between any two actual primitive point clouds, respectively, to construct an actual point cloud graph. The specific process is as follows:

[0161] First step: In the actual primitive library, respectively, any two actual primitive point clouds are composed into an actual primitive point cloud group, and the actual minimum Euclidean distance from each actual primitive point in one actual primitive point cloud to all actual primitive points in the other actual primitive point cloud is calculated.

[0162] Second step: Set the first distance threshold and the first point number threshold.

[0163] Third step: The two actual primitive point clouds in all actual primitive point cloud groups corresponding to the actual minimum Euclidean distance less than the first distance threshold and the number of all actual primitive points greater than the first point number threshold are considered to be mutually adjacent, and the value of the actual adjacent attribute of each actual primitive point cloud in each actual primitive point cloud group is marked as 1.

[0164] Fourth step: If the two actual primitive point clouds in the actual primitive point cloud group are both actual plane primitive point clouds, the actual normal vector angle between the actual plane equations corresponding to the two actual primitive point clouds is calculated; when the value of the actual normal vector angle is less than the actual normal vector angle threshold, the two actual primitive point clouds are considered to be parallel, and the value of the actual parallel attribute of the two actual primitive point clouds is marked as 1; when the sum of the absolute values of the differences of the coefficients contained in the actual plane equations corresponding to the two actual primitive point clouds is less than the actual coefficient threshold, the two actual primitive point clouds are considered to be coplanar, and the value of the actual coplanar attribute of the two actual primitive point clouds is marked as 1.

[0165] Fifth step: If the two actual primitive point clouds in the actual primitive point cloud group are both actual cylindrical primitive point clouds, the two actual primitive point clouds are considered to be non-parallel and non-coplanar, and the value of the actual parallel attribute, the value of the actual coplanar attribute and the value of the actual normal vector angle of the two actual primitive point clouds are all marked as 0.

[0166] Sixth step: If one actual primitive point cloud in the actual primitive point cloud group is an actual plane primitive point cloud and the other actual primitive point cloud is an actual cylindrical primitive point cloud, the value of the actual normal vector angle, the value of the actual parallel attribute and the value of the actual coplanar attribute of the two actual primitive point clouds are all marked as 0.

[0167] Seventh step: Each actual primitive point cloud in the actual primitive library is respectively taken as an actual graph vertex, and any two actual graph vertices are connected to each other to form a corresponding actual connection edge, thereby constructing an actual point cloud graph corresponding to the actual point cloud.

[0168] Wherein, each actual graph vertex in the actual point cloud graph is labeled according to Arabic numerals; the value of each actual connection edge consists of six digits, each individual digit is the value of the actual coplanar attribute, each ten digit is the value of the actual parallel attribute, each hundred digit is the value of the actual adjacency attribute, and each ten million digit to thousand digit is the value of the actual normal vector angle.

[0169] Here, the type of the actual graph vertex includes an actual planar graph vertex and an actual cylindrical graph vertex.

[0170] Sub-step S1032: As shown in the figure, all model primitive libraries are traversed, each model primitive point cloud in each model primitive library is taken as a model graph vertex, a model connection edge relationship is established between any two model primitive point clouds in each model primitive library, and a model point cloud graph corresponding to each model point cloud is constructed. The specific process is as follows: Figure 8 First step: from the first model primitive library, any two model primitive point clouds are taken to form a model primitive point cloud group, and all model minimum Euclidean distances from each model primitive point in one model primitive point cloud to another model primitive point cloud are calculated.

[0171] Second step: a second distance threshold and a second point number threshold are set.

[0172] Third step: two model primitive point clouds in all model primitive point cloud groups whose model minimum Euclidean distances are less than the second distance threshold and whose number of all model primitive points is greater than the second point number threshold are considered to be adjacent to each other, and the value of the model adjacency attribute of each model primitive point cloud in each model primitive point cloud group is marked as 1.

[0173] Fourth step: if the two model primitive point clouds in the model primitive point cloud group are both model planar primitive point clouds, the model normal vector angle between the model plane equations corresponding to the two model primitive point clouds is calculated; when the value of the model normal vector angle is less than the model normal vector angle threshold, the two model primitive point clouds are considered to be parallel, and the value of the model parallel attribute of the two model primitive point clouds is marked as 1; when the sum of the absolute values of the differences of the coefficients contained in the model plane equations corresponding to the two model primitive point clouds is less than the model coefficient threshold, the two model primitive point clouds are considered to be coplanar, and the value of the model coplanar attribute of the two model primitive point clouds is marked as 1.

[0174]

[0175] ​Step 5: If both of the model primitive point clouds in the model primitive point cloud group are model cylinder primitive point clouds, it is considered that the two model primitive point clouds are non-parallel and non-coplanar, and the value of the model non-parallel attribute, the value of the model coplanar attribute and the value of the model normal vector angle of the two model primitive point clouds are all marked as 0.

[0176] Step 6: If one of the model primitive point clouds in the model primitive point cloud group is a model plane primitive point cloud and the other is a model cylinder primitive point cloud, the value of the model normal vector angle, the value of the model parallel attribute and the value of the model coplanar attribute of the two model primitive point clouds are all marked as 0.

[0177] Step 7: Each model primitive point cloud in the first model primitive library is taken as a model graph vertex, and any two model graph vertices are connected to each other to form a corresponding model connection edge, so as to construct a corresponding model point cloud graph for each model point cloud.

[0178] In the model point cloud graph, each model graph vertex is labeled according to Arabic numerals; the value of each model connection edge consists of six digits, each individual digit is the value of the model coplanar attribute, each ten-digit is the value of the model parallel attribute, each hundred-digit is the value of the model adjacent attribute, and each ten-thousand-digit to thousand-digit is the value of the model normal vector angle.

[0179] Step 8: All the above steps are iteratively performed until the model point cloud graphs corresponding to all model point clouds are constructed.

[0180] Here, the types of model graph vertices include model plane graph vertices and model cylinder graph vertices.

[0181] In step S104 of the embodiment, as shown in FIGS. 10 and 11, each model point cloud graph is compared with the actual point cloud graph for similarity, and the corresponding coarse registration is completed. Figure 9 and Figure 10 In step S104 of the embodiment, as shown in FIGS. 10 and 11, each model point cloud graph is compared with the actual point cloud graph for similarity, and the corresponding coarse registration is completed.

[0182] Sub-step S1041: The actual point cloud graph is compared with the first model point cloud graph, if the number of all model graph vertices of a certain type in the first model point cloud graph is less than the number of all actual graph vertices of the same type in the actual point cloud graph, the first model point cloud graph is skipped, and the actual point cloud graph is compared with the next model point cloud graph, otherwise the subsequent steps are continued.

[0183] Sub-step S1042: All actual connection edges with the value of the actual adjacent attribute marked as 1 in the actual point cloud graph are taken as actual real edges, and all actual real edges are respectively counted into the actual degrees of the corresponding at least one actual graph vertex.

[0184] All model connecting edges in which the value of the model adjacency attribute of the first model point cloud graph is marked as 1 are regarded as model edges, and all model edges are respectively added to the model degree of each corresponding model graph vertex.

[0185] Sub-step S1043: all actual graph vertices in the actual point cloud graph are arranged in descending order according to the number of all actual degrees contained by each actual graph vertex, and all model graph vertices in the first model point cloud graph are arranged in descending order according to the number of all model degrees contained by each model graph vertex;

[0186] According to the descending order arrangement of the actual point cloud graph, the type and the number of all actual degrees contained by each actual graph vertex are respectively counted.

[0187] In the first model point cloud graph, all model graph vertices with the same type as each actual graph vertex and the number of all model degrees contained by each model graph vertex being greater than or equal to the number of all actual degrees are respectively selected as the candidate model graph vertex set of each corresponding actual graph vertex.

[0188] Sub-step S1044: a mapping array with the same number of actual graph vertices in the actual point cloud graph is constructed, and a marking array with the same number of model graph vertices in the first model point cloud graph is constructed.

[0189] Sub-step S1045: all candidate model graph vertex sets are traversed, and all candidate model graph vertices in each candidate model graph vertex set are respectively put into the mapping array as mapping model graph vertices and are respectively marked in the marking array.

[0190] When there are at least two mapping model graph vertices in the mapping array, the value of the model adjacency attribute, the value of the model coplanar attribute, the value of the model parallel attribute between any two mapping model graph vertices are respectively checked, and the value of the actual adjacency attribute, the value of the actual coplanar attribute, the value of the actual parallel attribute between the corresponding two actual graph vertices are respectively checked, whether they are all one-to-one corresponding and equal, and whether the absolute value of the difference between the model normal vector included angle between any two mapping model graph vertices and the actual normal vector included angle between the corresponding two actual graph vertices is within the absolute value threshold, if there is a situation that is not corresponding and equal or the absolute value of the difference is not within the absolute value threshold, it is considered that the mapping relationship is unreasonable, at this time, the mapping array is restored to a reasonable mapping relationship state, and the next candidate model graph vertex is selected to continue the above mapping process until a reasonable mapping relationship set corresponding to each actual graph vertex in the actual point cloud graph is found.

[0191] Sub-step S1046: according to all reasonable mapping relationship sets, the frequency of each actual graph vertex being mapped to all corresponding mapping model graph vertices is respectively recorded, and the frequency sum corresponding to each actual graph vertex is respectively calculated.

[0192] Divide each frequency by the corresponding frequency sum respectively to obtain the mapping probability of each actual graph vertex being mapped to all the corresponding mapping model vertices.

[0193] Sub-step S1047: Traverse all the actual graph vertices, and respectively form a graph vertex pair by each actual graph vertex and the at least one mapping model graph vertex with the highest mapping probability corresponding to the actual graph vertex, and record the corresponding mapping probability.

[0194] Arrange all the actual graph vertices in descending order according to the size of the corresponding mapping probability, and take the at least one graph vertex pair arranged in the first place as a candidate graph vertex pair.

[0195] Calculate the actual average center and the actual average normal vector of the actual primitive point cloud corresponding to the actual graph vertex, and the model average center and the model average normal vector of the model primitive point cloud corresponding to the mapping model graph vertex in each candidate graph vertex pair respectively.

[0196] Translate each actual average center to the corresponding model average center, and rotate each actual average normal vector to the corresponding model average normal vector to obtain a coarse registration transformation matrix corresponding to the first model point cloud.

[0197] Sub-step S1048: Multiply the coarse registration transformation matrix by the actual point cloud to obtain the corresponding actual coarse registration transformation point cloud, which is the coarse registration result of the first model point cloud and the actual point cloud.

[0198] Sub-step S1049: Iteratively perform all the above steps until all the coarse registration results are obtained.

[0199] Figure 9 The poses of the actual point cloud and the model point cloud of the fourth assembly stage before coarse registration are shown. It can be seen that the two do not substantially coincide, and the poses are quite different. Figure 10 The poses of the actual point cloud and the model point cloud of the fourth assembly stage after coarse registration are shown. It can be seen that the two have a high degree of coincidence, and the poses are relatively close.

[0200] The step S105 of the embodiment is shown in FIGS. 1 5A and 15B. Figure 11 、 Figure 12 and Figure 13 According to all the coarse registration results, the actual point cloud is respectively fine registered with each model point cloud, and the assembly stage of the actual workpiece is identified according to all the fine registration results.

[0201] Sub-step S1051: Take all the actual coarse registration transformation point clouds as source point clouds, and take all the model point clouds as target point clouds.

[0202] Sub-step S1052: Input the first source point cloud and all target point clouds into the iterative nearest point algorithm to obtain the actual fine registration transformation point cloud set.

[0203] Sub-step S1053: Iterate through the input process of the source point cloud and all target point clouds until the actual fine registration transformation point cloud set corresponding to each source point cloud is obtained; each actual fine registration transformation point cloud set is the corresponding fine registration result.

[0204] Sub-step S1054: Calculate the root mean square error between any model point cloud and the corresponding actual fine registration transform point cloud in each actual fine registration transform point cloud set.

[0205] When the root mean square error is greater than the root mean square error threshold, the corresponding model point cloud is considered not to be in the assembly stage of the actual point cloud.

[0206] When the root mean square error is less than or equal to the root mean square error threshold, the following operation is performed:

[0207] Calculate the nearest Euclidean distance between each actual fine-registration transformation point in the actual fine-registration transformation point cloud and the nearest model sampling point in the corresponding model point cloud;

[0208] For each actual fine-registration transform point cloud, count the total number of all actual fine-registration transform points whose nearest Euclidean distance is greater than the Euclidean distance threshold, and calculate the proportion of all total points in the corresponding actual fine-registration transform point cloud.

[0209] If the proportion of the number is greater than the proportion threshold, it is considered that the corresponding model point cloud is not in the assembly stage of the actual point cloud.

[0210] If the proportion of the number is less than or equal to the proportion threshold, then the corresponding model point cloud is considered to be in the assembly stage of the actual point cloud.

[0211] Furthermore, the expression for calculating the root mean square error is:

[0212] (1)

[0213] in, This represents the root mean square error. This represents the total number of model sampling points in the model point cloud. The first point in the model point cloud Each model sampling point, This represents the actual fine-registration transformed point cloud and the corresponding model point cloud. The closest actual fine-registration transform point between each model sampling point Indicates precise registration. Denotes the Euclidean norm. express and The Euclidean distance between them.

[0214] Figure 11 In the diagram, 'a' represents the initial pose of the actual point cloud and the model point cloud from the fourth assembly stage, while 'b' represents the pose after fine registration using the iterative nearest neighbor algorithm, without coarse registration. Figure 11 As can be seen, at this point, the overlap between the actual point cloud and the model point cloud in the fourth assembly stage is not high, and the pose difference is large.

[0215] Figure 12 In the diagram, 'a' represents the initial pose of the actual point cloud and the model point cloud in the fourth assembly stage; 'b' represents the pose of the actual point cloud and the model point cloud in the fourth assembly stage after coarse registration; and 'c' represents the pose of the actual point cloud and the model point cloud in the fourth assembly stage after coarse registration followed by fine registration using the iterative nearest neighbor algorithm. Figure 12 As can be seen, after coarse registration and fine registration using the iterative nearest point algorithm, the overlap between the actual point cloud and the model point cloud in the fourth assembly stage is improved, and their poses are relatively close.

[0216] Figure 13 The identification results of the actual point cloud and the model point cloud at each assembly stage are presented. Figure 13 It can be seen that the smaller the root mean square error and the smaller the proportion of errors, the greater the probability of matching at that assembly stage. The assembly stage probability represents the degree of matching between the actual point cloud and the model point cloud at the corresponding assembly stage. 0% represents a mismatch, and 100% represents a perfect match between the actual point cloud and the model point cloud.

[0217] exist Figure 9 , Figure 10 , Figure 11 a and b, and Figure 12 In a, b, and c, 201 represents the actual point cloud, and 202 represents the model point cloud.

[0218] It should be noted that the various types of thresholds involved in this embodiment are all adjusted and determined based on the actual results obtained in each experimental operation.

[0219] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0220] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0221] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements shall be included in the protection scope of the present application.

[0222] Other embodiments of the present application will be readily apparent to those skilled in the art upon considering the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present application following the general principles of the present application and including common knowledge or conventional technical means in the art not disclosed by the present application.

Claims

1. A product assembly stage identification method based on point cloud geometric primitive information, characterized in that, The method comprises the following steps: Respectively acquire model point clouds corresponding to workpiece models of each assembly stage, and acquire actual point clouds corresponding to actual workpieces, wherein each of the model point clouds respectively comprises model normal vector information, and each of the actual point clouds respectively comprises actual normal vector information; Respectively perform geometric primitive division on the actual point clouds and all the model point clouds, store all the obtained actual primitive point clouds into an actual primitive library, and store all the obtained model primitive point clouds into corresponding model primitive libraries; Respectively construct model point cloud graphs corresponding to each of the model point clouds by using all the model primitive libraries, and construct an actual point cloud graph by using the actual primitive library, comprising: Traverse the actual primitive library, take all the actual primitive point clouds in the actual primitive library as actual graph vertices, and establish actual connection edge relationships between any two of the actual primitive point clouds to construct the actual point cloud graph; Traverse all the model primitive libraries, take each of the model primitive point clouds in each of the model primitive libraries as a model graph vertex, and establish model connection edge relationships between any two of the model primitive point clouds in each of the model primitive libraries to respectively construct the model point cloud graphs corresponding to each of the model point clouds; Respectively compare each of the model point cloud graphs with the actual point cloud graph, and complete corresponding coarse registration, comprising: Compare the actual point cloud graph with a first model point cloud graph, if the number of all the model graph vertices of a certain type in the first model point cloud graph is less than the number of all the actual graph vertices of the same type in the actual point cloud graph, skip the first model point cloud graph, and continue to compare the actual point cloud graph with a next model point cloud graph, otherwise, continue to perform subsequent steps; Take all the actual connection edges in which the value of an actual adjacency attribute is marked as 1 as actual real edges, and count all the actual real edges into the actual degrees of corresponding at least one actual graph vertex; Take all the model connection edges in which the value of a model adjacency attribute is marked as 1 as model real edges, and count all the model real edges into the model degrees of corresponding at least one model graph vertex; Arrange all the actual graph vertices in the actual point cloud graph in descending order according to the number of all the actual degrees contained by each of the actual graph vertices, and arrange all the model graph vertices in the first model point cloud graph in descending order according to the number of all the model degrees contained by each of the model graph vertices; Respectively count the type of each of the actual graph vertices and the number of all the actual degrees contained by each of the actual graph vertices according to the descending arrangement order of the actual point cloud graph; In the first model point cloud graph, select all the model graph vertices which have the same type as each of the actual graph vertices and contain all the model degrees which are greater than or equal to the number of all the actual degrees as a candidate model graph vertex set corresponding to each of the actual graph vertices; Respectively count the type of each of the actual graph vertices and the number of all the actual degrees contained by each of the actual graph vertices according to the descending arrangement order of the actual point cloud graph; In the first model point cloud graph, select all the model graph vertices which have the same type as each of the actual graph vertices and contain all the model degrees which are greater than or equal to the number of all the actual degrees as a candidate model graph vertex set corresponding to each of the actual graph vertices; constructing a mapping array with the same number of elements as the number of all the actual graph vertices in the actual point cloud graph, and constructing a label array with the same number of elements as the number of all the model graph vertices in the first model point cloud graph; traversing all the candidate model graph vertex sets, and respectively placing all the candidate model graph vertices in each candidate model graph vertex set into the mapping array as mapping model graph vertices and respectively marking in the label array; when there are at least two mapping model graph vertices in the mapping array, respectively checking whether the values of the model adjacency attribute, the values of the model coplanar attribute, the values of the model parallel attribute between any two mapping model graph vertices are respectively equal to the values of the actual adjacency attribute, the values of the actual coplanar attribute, the values of the actual parallel attribute between the corresponding two actual graph vertices, and whether the absolute value of the difference between the model normal vector included angle between any two mapping model graph vertices and the actual normal vector included angle between the corresponding two actual graph vertices is within an absolute value threshold, if there is a case of not corresponding equal or the absolute value of the difference is not within the absolute value threshold, it is considered that the mapping relationship is unreasonable, at this time, the mapping array is restored to a reasonable mapping relationship state, and the next candidate model graph vertex is selected to continue the above mapping process until a reasonable mapping relationship set corresponding to each actual graph vertex in the actual point cloud graph is found; according to all the reasonable mapping relationship sets, respectively recording the frequency of each actual graph vertex being mapped to all the corresponding mapping model graph vertices, and respectively calculating the frequency sum corresponding to each actual graph vertex; respectively dividing each frequency by the corresponding frequency sum to obtain the mapping probability of each actual graph vertex being mapped to all the corresponding mapping model graph vertices; traversing all the actual graph vertices, respectively forming a graph vertex pair by taking each actual graph vertex and at least one mapping model graph vertex with the highest mapping probability corresponding to the actual graph vertex as a pair, and recording the corresponding mapping probability; arranging all the actual graph vertices in descending order according to the corresponding mapping probability, and taking at least one graph vertex pair arranged in the first place as a candidate graph vertex pair; respectively calculating the actual average center and the actual average normal vector of the actual primitive point cloud corresponding to the actual graph vertex in each candidate graph vertex pair, and the model average center and the model average normal vector of the model primitive point cloud corresponding to the mapping model graph vertex in the candidate graph vertex pair; translating each actual average center to the corresponding model average center, and rotating each actual average normal vector to the corresponding model average normal vector to obtain a coarse registration transformation matrix corresponding to the first model point cloud; multiplying the coarse registration transformation matrix by the actual point cloud to obtain a corresponding actual coarse registration transformation point cloud, which is the coarse registration result of the first model point cloud and the actual point cloud; iterating all the above steps until all the coarse registration results are obtained; According to all coarse registration results, respectively, the actual point cloud is fine registration with each model point cloud, and the assembly stage of the actual workpiece is identified according to all fine registration results, comprising: Respectively, all the actual coarse registration transformation point clouds are taken as source point clouds, and all the model point clouds are taken as target point clouds; The first source point cloud and all the target point clouds are input into the iterative closest point algorithm to obtain an actual fine registration transformation point cloud set; The input process of the source point cloud and all the target point clouds is iteratively performed until the actual fine registration transformation point cloud set corresponding to each source point cloud is obtained; each actual fine registration transformation point cloud set is the corresponding fine registration result; The root mean square error between any model point cloud in each actual fine registration transformation point cloud set and the corresponding actual fine registration transformation point cloud is calculated respectively; When the root mean square error is greater than the root mean square error threshold, it is considered that the corresponding model point cloud is not the assembly stage where the actual point cloud is located; When the root mean square error is less than or equal to the root mean square error threshold, the following operations are performed: The nearest Euclidean distance between all actual fine registration transformation points in each actual fine registration transformation point cloud and the nearest model sampling point in the corresponding model point cloud is calculated respectively; The total number of all actual fine registration transformation points corresponding to the nearest Euclidean distance greater than the Euclidean distance threshold in each actual fine registration transformation point cloud is counted respectively, and the number ratio of all the total number in the corresponding actual fine registration transformation point cloud is calculated respectively; If the number ratio is greater than the ratio threshold, it is considered that the corresponding model point cloud is not the assembly stage where the actual point cloud is located; If the number ratio is less than or equal to the ratio threshold, it is considered that the corresponding model point cloud is the assembly stage where the actual point cloud is located.

2. The product assembly phase identification method based on point cloud geometric primitive information according to claim 1, characterized in that, The steps of respectively obtaining the model point cloud corresponding to each assembly stage of the workpiece model and obtaining the actual point cloud corresponding to the actual workpiece comprise: According to the length, width and height of all the workpiece models, a bounding box suitable for each workpiece model is generated respectively; The center of each bounding box is taken as the center of the sphere, and the distance from the center of the sphere to the farthest corner point on the corresponding bounding box is taken as the radius of the sphere, and a peripheral sphere suitable for each workpiece model is constructed; A plurality of sampling points are uniformly arranged on each peripheral sphere, and a virtual camera is arranged on each sampling point; All the virtual cameras are used to sample the model point cloud of the corresponding workpiece model, and all the model sampling points obtained are combined to form the corresponding model point cloud; The actual point cloud of the actual workpiece is sampled, and all the actual sampling points obtained are combined to form the actual point cloud.

3. The product assembly phase identification method based on point cloud geometric primitive information according to claim 2, characterized in that, The steps of performing the geometric primitive division on the actual point cloud and storing all the actual primitive point clouds obtained in the actual primitive library comprise: randomly selecting one of the actual sampling points from the actual point cloud as an initial actual growing point, recording an actual normal vector of the initial actual growing point, and storing the initial actual growing point and the actual normal vector into an actual growing point set; calculating actual Euclidean distances between the initial actual growing point and all other actual sampling points respectively, and taking all the actual sampling points corresponding to the actual Euclidean distances less than an actual distance threshold as actual neighborhood points of the initial actual growing point; calculating actual included angles between the actual normal vector of each actual neighborhood point and the actual normal vector of the initial actual growing point respectively, and storing all the actual neighborhood points corresponding to the actual included angles less than an actual included angle threshold into the actual growing point set, and marking the initial actual growing point as used; taking each actual neighborhood point in the actual growing point set as an actual growing point in turn, and iteratively performing the steps of calculating and comparing the actual Euclidean distances and the actual included angles, and excluding the initial actual growing point marked as used and all actual growing points in each iteration, until all actual growing points are marked as used, to obtain a final actual growing point set corresponding to the initial actual growing point; iteratively performing all the above steps until the final actual growing point set corresponding to each initial actual growing point is obtained; extracting the initial actual growing point and all actual growing points from each final actual growing point set as actual plane primitive points, and grouping all the actual plane primitive points into corresponding actual plane primitive point clouds, and storing all the actual plane primitive point clouds into an actual primitive library; taking all the actual plane primitive point clouds containing less than an actual plane primitive point number threshold from the actual primitive library as actual new point clouds respectively; using a plane fitting method based on random sample consensus to obtain an actual plane equation corresponding to each actual plane primitive point cloud remaining in the actual primitive library after taking out all the actual new point clouds; performing Euclidean distance clustering on all the actual new point clouds to obtain multiple actual clustering clusters; performing principal component analysis on the normal vectors of all the actual new point clouds in each actual clustering cluster respectively, projecting the corresponding actual new point clouds using all the minimum actual feature vectors obtained respectively, performing roundness detection on all the projection results respectively, and taking all the actual new point clouds corresponding to actual roundness less than an actual roundness threshold as actual cylindrical primitive point clouds corresponding to each actual clustering cluster respectively, and storing all the actual cylindrical primitive point clouds into the actual primitive library, each of which contains multiple actual cylindrical primitive points, each of which is an actual new sampling point in the actual new point cloud. The types of the actual primitive point clouds include the actual plane primitive point clouds and the actual cylindrical primitive point clouds.

4. The product assembly phase identification method based on point cloud geometric primitive information according to claim 3, characterized in that, The step of performing the geometric primitive division on all the model point clouds respectively, and storing all the obtained model primitive point clouds into the corresponding model primitive library respectively, comprises: randomly selecting one model sampling point from the first model point cloud as an initial model growing point, recording a model normal vector of the initial model growing point, and storing the initial model growing point and the model normal vector into a model growing point set; calculating model Euclidean distances between the initial model growing point and all other model sampling points respectively, and taking all the model sampling points corresponding to the model Euclidean distances less than a model distance threshold as model neighborhood points of the initial model growing point; calculating model included angles between a normal vector of each model neighborhood point and the model normal vector of the initial model growing point, storing all the model neighborhood points corresponding to the model included angles less than a model included angle threshold into the model growing point set, and marking the initial model growing point as used; taking each model neighborhood point in the model growing point set as a model growing point in turn, and iteratively performing the above steps of calculating and comparing the model Euclidean distances and the model included angles, and excluding the initial model growing point marked as used and all the model growing points in each iteration, until all the model growing points are marked as used, to obtain a final model growing point set corresponding to the initial model growing point; iteratively performing all the above steps until the final model growing point set corresponding to each initial model growing point is obtained; extracting the initial model growing point and all the model growing points from each final model growing point set as model plane primitive points, all the model plane primitive points forming corresponding model plane primitive point clouds respectively, and storing all the model plane primitive point clouds into the corresponding model primitive library; taking all the model plane primitive point clouds containing less than a model plane primitive point number threshold of model plane primitive points from the corresponding model primitive library as model new point clouds respectively; using a plane fitting method based on random sampling consensus, obtaining a model plane equation corresponding to each model plane primitive point cloud from the corresponding model primitive library after taking out all the model new point clouds; performing Euclidean distance clustering on all the model new point clouds to obtain multiple model clustering clusters; and performing the above steps on all the model point clouds, and obtaining multiple model clustering clusters. respectively, and respectively projecting all the model new point clouds in each of the model clustering clusters onto all the minimum model feature vectors obtained by performing principal component analysis on the normal vectors of all the model new point clouds in each of the model clustering clusters, respectively performing roundness detection on all the projection results, and respectively taking all the model new point clouds corresponding to model roundness less than a model roundness threshold as model cylindrical primitive point clouds corresponding to each of the model clustering clusters, and storing all the model cylindrical primitive point clouds into the model primitive library corresponding thereto, each of the model cylindrical primitive point clouds comprising a plurality of model cylindrical primitive points, each of the model cylindrical primitive points being a model new sampling point in the model new point cloud; iteratively performing all the steps above until all the model cylindrical primitive point clouds corresponding to each of the model clustering clusters are stored into the model primitive library corresponding thereto; wherein the types of the model primitive point clouds include the model planar primitive point clouds and the model cylindrical primitive point clouds.

5. The product assembly phase identification method based on point cloud geometric primitive information according to claim 4, characterized in that, The step of traversing the actual primitive library, taking all the actual primitive point clouds in the actual primitive library as actual graph vertices, and establishing an actual connection edge relationship between any two of the actual primitive point clouds to construct the actual point cloud graph comprises: In the actual primitive library, any two of the actual primitive point clouds are taken as an actual primitive point cloud group, and all the actual minimum Euclidean distances from each actual primitive point in one of the actual primitive point clouds in each actual primitive point cloud group to another actual primitive point cloud are calculated; a first distance threshold and a first point number threshold are set; two actual primitive point clouds in each actual primitive point cloud group corresponding to the actual minimum Euclidean distances less than the first distance threshold and the number of all the actual primitive points greater than the first point number threshold are considered to be adjacent to each other, and the value of an actual adjacency attribute of the two actual primitive point clouds adjacent to each other in each actual primitive point cloud group is marked as 1; if the two actual primitive point clouds in the actual primitive point cloud group are both the actual planar primitive point clouds, an actual normal vector included between the actual plane equations corresponding to the two actual primitive point clouds is calculated; when the value of the actual normal vector is less than an actual normal vector threshold, the two actual primitive point clouds are considered to be parallel, and the value of an actual parallel attribute of the two actual primitive point clouds is marked as 1; when the sum of the absolute values of the differences between the coefficients included in the actual plane equations corresponding to the two actual primitive point clouds is less than an actual coefficient threshold, the two actual primitive point clouds are considered to be coplanar, and the value of an actual coplanar attribute of the two actual primitive point clouds is marked as 1; if the two actual primitive point clouds in the actual primitive point cloud group are both the actual cylindrical primitive point clouds, the two actual primitive point clouds are considered to be non-parallel and non-coplanar, and the value of the actual parallel attribute, the value of the actual coplanar attribute, and the value of the actual normal vector included between the two actual primitive point clouds are all marked as 0; If one of the actual primitive point clouds in the actual primitive point cloud group is the actual plane primitive point cloud and the other is the actual cylinder primitive point cloud, the value of the actual normal vector included angle, the value of the actual parallel attribute and the value of the actual coplanar attribute of the two actual primitive point clouds are all marked as 0; Each actual primitive point cloud in the actual primitive library is taken as an actual graph vertex, and any two actual graph vertices are connected to form a corresponding actual connection edge, so as to construct the actual point cloud graph corresponding to the actual point cloud; The actual graph vertices are labeled according to Arabic numerals; the value of each actual connection edge is composed of six digits, each individual digit is the value of the actual coplanar attribute, each ten digit is the value of the actual parallel attribute, each hundred digit is the value of the actual adjacency attribute, and each ten thousand digit to thousand digit is the value of the actual normal vector included angle; The types of the actual graph vertices include actual plane graph vertices and actual cylinder graph vertices.

6. The product assembly phase identification method based on point cloud geometric primitive information according to claim 5, characterized in that, The step of traversing all the model primitive libraries, taking each model primitive point cloud in each model primitive library as a model graph vertex, and establishing a model connection edge relationship between any two model primitive point clouds in each model primitive library to construct the model point cloud graph corresponding to each model point cloud includes: In the first model primitive library, any two model primitive point clouds are taken as a model primitive point cloud group, and the model minimum Euclidean distance from each model primitive point in one of the model primitive point clouds to the other model primitive point cloud is calculated; A second distance threshold and a second point number threshold are set; If the model minimum Euclidean distance is less than the second distance threshold and the number of all model primitive points is greater than the second point number threshold, the two model primitive point clouds in the model primitive point cloud group are considered to be adjacent to each other, and the value of the model adjacency attribute of the two model primitive point clouds in each model primitive point cloud group that are adjacent to each other is marked as 1; If the two model primitive point clouds in the model primitive point cloud group are both model plane primitive point clouds, the model normal vector included angle between the model plane equations corresponding to the two model primitive point clouds is calculated; when the value of the model normal vector included angle is less than a model normal vector included angle threshold, the two model primitive point clouds are considered to be parallel, and the value of the model parallel attribute of the two model primitive point clouds is marked as 1; when the sum of the absolute values of the differences of the coefficients contained in the model plane equations corresponding to the two model primitive point clouds is less than a model coefficient threshold, the two model primitive point clouds are considered to be coplanar, and the value of the model coplanar attribute of the two model primitive point clouds is marked as 1; If both of the model primitive point cloud in the model primitive point cloud group are the model cylindrical primitive point cloud, it is considered that the two model primitive point clouds are non-parallel and non-coplanar, and the values of the model parallel attribute, the model coplanar attribute and the model normal vector angle of the two model primitive point clouds are all marked as 0; If one of the model primitive point cloud in the model primitive point cloud group is the model plane primitive point cloud and the other is the model cylindrical primitive point cloud, the values of the model normal vector angle, the model parallel attribute and the model coplanar attribute of the two model primitive point clouds are all marked as 0; Each model primitive point cloud in the first model primitive library is taken as a model graph vertex, and any two model graph vertices are connected to form a corresponding model connection edge, thereby constructing a model point cloud graph corresponding to each model point cloud; Each model graph vertex in the model point cloud graph is labeled according to Arabic numerals; the value of each model connection edge consists of six digits, each individual digit is the value of the model coplanar attribute, each ten digit is the value of the model parallel attribute, each hundred digit is the value of the model adjacency attribute, and each ten-thousandth digit to thousandth digit is the value of the model normal vector angle; All the above steps are iteratively performed until the model point cloud graphs corresponding to all the model point clouds are constructed; The types of the model graph vertices include model plane graph vertices and model cylindrical graph vertices.

7. The product assembly phase identification method based on point cloud geometric primitive information according to claim 6, characterized in that, The expression for calculating the root mean square error is: (1) in, This represents the root mean square error. This represents the number of all model sampling points in the model point cloud. The first point in the model point cloud Each model sampling point, This represents the actual fine-registration transformed point cloud and the corresponding model point cloud. The closest actual fine-registration transform point between each model sampling point Indicates precise registration. Describes the Euclidean norm. express and The Euclidean distance between them.

Images