Methods, equipment, and devices for detecting part assembly errors based on augmented reality technology

Abstract

This invention discloses a method, device, and apparatus for detecting assembly errors in parts based on augmented reality (AR) technology. The method includes: acquiring point clouds of virtual models of the parts to be assembled and virtual model point clouds of the assembly; acquiring point clouds of the actual parts to be assembled; registering the point clouds of the model to be assembled to the point clouds of the actual parts to be assembled; visually displaying the registered virtual image of the assembly and guiding assembly personnel to complete the assembly task to obtain the actual assembly parts; acquiring the current pose point cloud of the actual assembly parts; and detecting assembly errors. This method for detecting assembly errors based on AR technology introduces augmented reality technology, which blends virtual and real elements, into the assembly process. It detects errors generated by assembly personnel during part assembly and guides them to make timely adjustments based on error feedback results, thereby improving assembly accuracy and facilitating training for new employees and quality inspection.

Description

Technical Field

[0001] This invention belongs to the field of augmented reality technology, and more specifically, relates to a method, equipment, and apparatus for detecting assembly errors of parts based on augmented reality technology. Background Technology

[0002] Augmented Reality (AR) is a technology that calculates the position and angle of a camera image in real time and adds corresponding images. AR integrates real-world and virtual-world information, simulating and overlaying real-world visual, auditory, olfactory, and tactile information through computer simulation. This allows the virtual information to be applied to the real world, enabling real-time overlay of the real environment and virtual objects onto the same screen or space. Users can then see this overlaid scene using a head-mounted display (HMD).

[0003] In today's industrial automation environment, although industrial robots can replace manual labor in assembling some equipment, when assembling complex and precision products, such as phased array antennas, the high dimensional accuracy, large number of parts to be assembled, and complex assembly processes, coupled with the lack of intuitiveness in paper assembly manuals, lead to long manual assembly cycles, errors in the assembly process, and poor product consistency. Furthermore, some parts have irregular shapes, and the production cycle often needs to be adjusted according to requirements during the assembly process. If fully automated equipment is used for assembly, it will be difficult and costly to achieve.

[0004] Therefore, there is an urgent need to design a technical solution to guide assembly and detect deviations in real time during the assembly process, so as to reduce the cumulative error generated throughout the assembly process. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a part assembly error detection method based on augmented reality technology. This method introduces augmented reality technology, which blends virtual and real elements, into the assembly process to detect errors generated by assembly personnel during part assembly. Based on the error feedback results, it guides assembly personnel to make timely adjustments, thereby improving assembly accuracy and facilitating training for new employees and quality inspection.

[0006] To achieve the above objectives, the present invention provides a method for detecting part assembly errors based on augmented reality technology, comprising:

[0007] Obtain the point cloud of the virtual model of the part to be assembled and the point cloud of the virtual model of the assembly;

[0008] Obtain the point cloud of the actual parts to be assembled;

[0009] Register the point cloud of the model to be assembled to the point cloud of the actual part to be assembled.

[0010] The registered virtual image of the assembly to be assembled is visualized and displayed to guide the assembly personnel to complete the assembly task of the assembly to be assembled, so as to obtain the real assembly parts.

[0011] Obtain the current pose point cloud of the actual assembly parts;

[0012] By comparing the point cloud of the virtual model of the assembly with the point cloud of the current pose of the actual assembly parts, the assembly error of the parts is detected.

[0013] Optionally, obtaining the virtual model point cloud of the part to be assembled and the virtual model point cloud of the assembly includes:

[0014] A convolutional neural network model is constructed to identify the parts to be assembled.

[0015] The convolutional neural network model, the virtual model of the part to be assembled, the point cloud of the virtual model of the part to be assembled, the virtual model of the assembly parts, and the point cloud of the virtual model of the assembly parts are read using an augmented reality device.

[0016] Optionally, the step of using a convolutional neural network to identify the parts to be assembled and constructing a convolutional neural network model includes:

[0017] Convolutional neural network model based on the YOLO v3 framework was constructed.

[0018] Collect image data of parts and assemblies to be assembled under different external environments and establish a dataset;

[0019] The images in the dataset are pre-labeled to generate training and testing data;

[0020] The training data is input into the convolutional neural network model for training.

[0021] The trained convolutional neural network model is tested using the test data to obtain multiple test labels. If the accuracy of any of the test labels is greater than the judgment threshold, it is considered qualified; otherwise, the training is repeated.

[0022] Optionally, obtaining the point cloud of the actual part to be assembled includes:

[0023] Use an RGB-D camera to collect data on site and obtain RGB images of the parts to be assembled;

[0024] The RGB image is identified based on a convolutional neural network model. The data stream acquired by the RGB-D camera is converted into a depth frame map. The depth image is transformed into a 2D-3D coordinate system to obtain the actual point cloud of the part to be assembled in the world coordinate system.

[0025] Optionally, registering the point cloud of the model to be assembled to the point cloud of the actual part to be assembled includes:

[0026] The SAC-IA algorithm based on FPFH coarsely registers the model to be assembled to the actual point cloud of the parts to be assembled.

[0027] After coarse registration, the ICP algorithm is used for fine registration to register the model to be assembled with the point cloud of the actual parts to be assembled, thus obtaining the actual virtual model of the parts to be assembled.

[0028] Optionally, it also includes:

[0029] The ICP algorithm is repeatedly used for fine registration, so that the overlap between the point cloud of the model to be assembled and the actual part to be assembled is higher than the set value, thus obtaining the actual virtual model of the part to be assembled.

[0030] Optionally, the step of detecting assembly errors of parts by comparing the point cloud of the virtual model of the assembly with the point cloud of the current pose of the actual assembly parts includes:

[0031] The SAC-IA algorithm based on FPFH coarsely registers the virtual model of the assembly to the current pose point cloud of the real assembly parts.

[0032] After coarse registration, the ICP algorithm is used for fine registration to register the virtual model of the assembly with the current pose point cloud of the real assembly parts.

[0033] The non-overlapping deviation between the point cloud of the virtual assembly model and the point cloud of the current pose of the real assembly parts is obtained.

[0034] Optionally, it also includes: repeatedly using the registered virtual image of the assembly to be assembled for visualization and guiding the assembly personnel to complete the assembly task of the assembly to be assembled, so that the non-overlap deviation is less than a set value.

[0035] An electronic device, comprising:

[0036] Memory, which stores executable instructions;

[0037] A processor that executes the executable instructions in the memory to implement the above-described part assembly error detection method based on augmented reality technology.

[0038] A parts assembly error detection device based on augmented reality technology includes:

[0039] Data acquisition module;

[0040] A storage module, wherein the storage module stores the above-mentioned part assembly error detection method based on augmented reality technology;

[0041] A helmet display, electrically connected to the storage module and the acquisition module, the helmet display comprising...

[0042] This invention provides a method for detecting assembly errors in parts based on augmented reality technology, the advantages of which are:

[0043] 1. This part assembly error detection method based on augmented reality technology introduces augmented reality technology that blends virtual and real elements into the assembly process, detects the errors generated by assembly personnel when assembling parts, and guides assembly personnel to make timely adjustments based on the error feedback results, thereby improving assembly accuracy and facilitating teaching for new employees and quality inspection.

[0044] 2. This part assembly error detection method based on augmented reality technology can solve problems such as relying on paper manuals for assembly guidance, frequent misassembly or omission of parts, and inability to measure and provide feedback on part assembly accuracy in real time. Based on augmented reality technology, it provides assembly guidance in a virtual environment and detects and records deviations in the real assembly process in real time, thereby guiding assembly personnel to make adjustments. This can effectively reduce the cumulative error generated in the entire real assembly process and improve assembly accuracy and efficiency.

[0045] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0046] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.

[0047] Figure 1 A flowchart of a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown.

[0048] Figure 2 A schematic diagram of the coordinate system definition in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown.

[0049] Figure 3 A schematic diagram showing the relationship between pixel coordinates and image coordinate system in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown.

[0050] Figure 4 A schematic diagram illustrating the relationship between the image coordinate system and the camera coordinate system in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown.

[0051] Figure 5The point cloud influence area in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown. Detailed Implementation

[0052] Preferred embodiments of the invention will now be described in more detail. While preferred embodiments of the invention are described below, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0053] Figure 1 A flowchart of a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown; Figure 2 A schematic diagram of the coordinate system definition in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown; Figure 3 This diagram illustrates the relationship between pixel coordinates and the image coordinate system in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention. Figure 4 This diagram illustrates the relationship between the image coordinate system and the camera coordinate system in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention. Figure 5 The point cloud influence area in a part assembly error detection method based on augmented reality technology according to an embodiment of the present invention is shown.

[0054] like Figure 1-5 As shown, the part assembly error detection method based on augmented reality technology includes:

[0055] Obtain the point cloud of the virtual model of the part to be assembled and the point cloud of the virtual model of the assembly;

[0056] Obtain the point cloud of the actual parts to be assembled;

[0057] Register the point cloud of the model to be assembled to the point cloud of the actual part to be assembled.

[0058] The registered virtual image of the assembly to be assembled is visualized and displayed to guide the assembly personnel to complete the assembly task of the assembly to be assembled, so as to obtain the real assembly parts.

[0059] Obtain the current pose point cloud of the actual assembly parts;

[0060] By comparing the point cloud of the virtual model of the assembly with the point cloud of the current pose of the actual assembly parts, the assembly error of the parts is detected.

[0061] Specifically, by enhancing the equipment to combine virtual and reality, the assembly process is guided by virtual models of the parts to be assembled and the assembly parts. Then, by comparing and analyzing the point clouds of the parts to be assembled and the assembly parts, errors can be analyzed in real time, which can provide auxiliary teaching, adapt to the site in a timely manner, and facilitate the learning and progress of operators.

[0062] Furthermore, virtual models of the parts to be assembled, virtual models of the assembly, and assembly demonstration animations are pre-imported, with the model format being CAD models.

[0063] In this embodiment, obtaining the virtual model point cloud of the part to be assembled and the virtual model point cloud of the assembly includes:

[0064] A convolutional neural network model is constructed to identify the parts to be assembled.

[0065] The convolutional neural network model, the virtual model of the part to be assembled, the point cloud of the virtual model of the part to be assembled, the virtual model of the assembly parts, and the point cloud of the virtual model of the assembly parts are read using an augmented reality device.

[0066] Specifically, by using neural networks to intelligently process data, it can be easily adapted to different production uses and improve teaching effectiveness.

[0067] In this embodiment, the step of using a convolutional neural network to identify the parts to be assembled and constructing a convolutional neural network model includes:

[0068] Convolutional neural network model based on the YOLO v3 framework was constructed.

[0069] Collect image data of parts and assemblies to be assembled under different external environments and establish a dataset;

[0070] The images in the dataset are pre-labeled to generate training and testing data;

[0071] The training data is input into the convolutional neural network model for training.

[0072] The trained convolutional neural network model is tested using the test data to obtain multiple test labels. If the accuracy of any of the test labels is greater than the judgment threshold, it is considered qualified; otherwise, the training is repeated.

[0073] Specifically, the neural network is trained and tested to adapt to usage needs and fit actual production applications.

[0074] Furthermore, if there are labels with a target score greater than 0.95 among multiple test labels, the maximum value is selected as the final predicted label, and the training model is verified to be qualified; if there are no labels with a target score greater than 0.95 among a series of predicted labels, training continues.

[0075] In this embodiment, obtaining the point cloud of the actual part to be assembled includes:

[0076] Use an RGB-D camera to collect data on site and obtain RGB images of the parts to be assembled;

[0077] The RGB image is identified based on a convolutional neural network model. The data stream acquired by the RGB-D camera is converted into a depth frame map. The depth image is then transformed into a 2D-3D coordinate system to obtain the actual point cloud of the part to be assembled in the world coordinate system.

[0078] Specifically, 2D-3D coordinate transformations include:

[0079] Define a left-handed coordinate system. A depth camera has four coordinate systems: world coordinate system P... w Camera coordinate system P c Image coordinate system P i Pixel coordinate system P uv ;

[0080] O w -X w Y w Z w : World coordinate system, used to describe the camera's position, unit: meters;

[0081] O C -X C Y C Z C Camera coordinate system, origin at the optical center, unit: m;

[0082] o-xy: Image coordinate system, with the midpoint of the image as the optical center, unit: mm;

[0083] uv: pixel coordinates, originating from the top left corner of the image, unit: pixel;

[0084] P: A point in the world coordinate system, that is, a point in space;

[0085] p: The image point of point P in the image, with coordinates (x, y) in the image coordinate system and (u, v) in the pixel coordinate system;

[0086] f: Camera focal length equal to o and O C The distance between them;

[0087] Transform a point P(u, v) in the depth image from the pixel coordinate system to the image coordinate system.

[0088] Specifically, both the pixel coordinate system and the image coordinate system are on the imaging plane, but their origins and units are different. The origin of the image coordinate system is the intersection of the camera's optical axis and the imaging plane, which is chosen here as the midpoint of the imaging plane. The unit of the image coordinate system is mm, while the origin of the pixel coordinate system is the upper left corner of the image, and the unit is pixels.

[0089] Furthermore, the conversion between these two is as follows:

[0090] Where dx and dy represent the number of millimeters (mm) for each column and each row, respectively, i.e., 1 pixel = dx mm. The pixel coordinate system, image coordinate system, and the position of pixel p are referenced. Figure 3 .

[0091] according to Figure 3 The positional relationship in the middle is transformed by the following formula:

[0092]

[0093]

[0094] Where: (u, v) are the coordinates of point p in the pixel coordinate system, (u0, v0) are the coordinates of the origin of the image coordinate system in the pixel coordinate system, and the transformed coordinates of point p(x, y) in the image coordinate system.

[0095] Transforming the point p(x, y) from the image coordinate system to the camera coordinate system involves perspective projection, a conversion from 2D to 3D. The positional relationship between the image coordinate system, the camera coordinate system, and the point p is referenced below. Figure 4 .

[0096] According to the principle of similar triangles:

[0097] ΔABO C ~ΔoCO C

[0098] ΔPBO C ~ΔpCO C

[0099] The proportional relationship can be obtained as follows:

[0100]

[0101] The transformation formula is as follows:

[0102]

[0103]

[0104]

[0105] Point P(X) in the camera coordinate systemC Y C Z C Transforming from the camera coordinate system to the world coordinate system involves rotation and translation. By rotating around different coordinate axes by different angles, the corresponding rotation matrix is ​​obtained.

[0106] In this embodiment, registering the point cloud of the model to be assembled to the point cloud of the actual part to be assembled includes:

[0107] The SAC-IA algorithm based on FPFH coarsely registers the model to be assembled to the actual point cloud of the parts to be assembled.

[0108] After coarse registration, the ICP algorithm is used for fine registration to register the model to be assembled with the point cloud of the actual parts to be assembled, thus obtaining the actual virtual model of the parts to be assembled.

[0109] Specifically, FPFH stands for FastPoint Feature Histogram, and SAC-IA is a sampling consistency algorithm.

[0110] In this embodiment, it also includes:

[0111] The ICP algorithm is repeatedly used for fine registration, so that the overlap between the point cloud of the model to be assembled and the actual part to be assembled is higher than the set value, thus obtaining the actual virtual model of the part to be assembled.

[0112] Specifically, by repeatedly using the ICP algorithm for fine registration, the point cloud error between the model to be assembled and the actual parts to be assembled is reduced, thereby improving assembly accuracy.

[0113] In this embodiment, the step of detecting assembly errors of parts by comparing the point cloud of the virtual model of the assembly with the point cloud of the current pose of the actual assembly parts includes:

[0114] The SAC-IA algorithm based on FPFH coarsely registers the virtual model of the assembly to the current pose point cloud of the real assembly parts.

[0115] After coarse registration, the ICP algorithm is used for fine registration to register the virtual model of the assembly with the current pose point cloud of the real assembly parts.

[0116] The non-overlapping deviation between the point cloud of the virtual assembly model and the point cloud of the current pose of the real assembly parts is obtained.

[0117] Specifically, the Hausdorff distance between the point cloud of the virtual assembly model and the point cloud of the current pose of the real assembly parts is calculated, which is the assembly error.

[0118] In this embodiment, the method further includes: repeatedly using the registered virtual image of the assembly to be assembled for visualization and guiding the assembly personnel to complete the assembly task of the assembly to be assembled, so that the non-overlapping deviation is less than a set value.

[0119] Specifically, repeated demonstrations facilitate adjustments and eliminate errors.

[0120] When using the part assembly error detection method based on augmented reality technology in this embodiment, taking the use of a convolutional neural network built on a head-mounted display as an example, the specific steps are as follows:

[0121] Please refer to Figure 1 , Figure 1 The flowchart illustrates the part assembly error detection method based on augmented reality technology provided in the embodiments of the present invention.

[0122] Step S1: Object recognition based on convolutional neural network;

[0123] Step S1.1: Collect image data of the parts to be assembled and the assembly under different external environments and establish a dataset;

[0124] Step S1.2: Perform labeling preprocessing on the images in the dataset to generate training data and test data;

[0125] Step S1.3: Build a convolutional neural network model based on the YOLO v3 framework;

[0126] Step S1.4: Input the training data obtained in step S1.2 into the convolutional neural network model built in step S1.3 to start training;

[0127] Step S1.5: Input the test data obtained in step S1.2 into the trained model obtained in step S1.4 to obtain a series of test labels. If there is a label with a target score greater than 0.95 in the series of predicted labels, select the maximum value as the final predicted label and verify that the trained model obtained in step S1.4 is qualified; if there is no label with a target score greater than 0.95 in the series of predicted labels, return to step S1.4 to continue training.

[0128] Step S2: Assembly is performed based on augmented reality technology;

[0129] Step S2.1: Import the assembly animations of different parts to be assembled and the corresponding assembly parts into the augmented reality device;

[0130] Step S2.2: Import the convolutional neural network model for recognizing the parts to be assembled, which was trained in step S1, into the augmented reality device.

[0131] Step S2.3: Import all CAD models of the parts to be assembled that are related to the assembly into the augmented reality device;

[0132] Step S2.4: Wear the augmented reality device and use the RGB-D camera within it to collect on-site data;

[0133] Step S2.5: Use the convolutional neural network trained in step S1 to identify the parts to be assembled in the RGB image captured by the current RGB-D camera.

[0134] Step S2.6: Extract the depth image region corresponding to the RGB image of the part P1 to be assembled identified by the convolutional neural network, and store the corresponding CAD model of the part to be assembled into the cache.

[0135] Step S2.7: Obtain the pose of the point cloud Q1 of the part to be assembled in the world coordinate system through the 2D-3D coordinate transformation of the depth image. The coordinate transformation steps are as follows;

[0136] Step S2.7.1, define the left-handed coordinate system reference. Figure 2 A depth camera has four coordinate systems: world coordinate system P w Camera coordinate system P c Image coordinate system P i Pixel coordinate system P uv ;

[0137] O w -X w Y w Z w : World coordinate system, used to describe the camera's position, unit: meters;

[0138] O C -Y C Y c Z c Camera coordinate system, origin at the optical center, unit: m;

[0139] o-xy: Image coordinate system, with the midpoint of the image as the optical center, unit: mm;

[0140] uv: pixel coordinates, originating from the top left corner of the image, unit: pixel;

[0141] P: A point in the world coordinate system, that is, a point in space;

[0142] p: The image point of point P in the image, with coordinates (x, y) in the image coordinate system and (u, v) in the pixel coordinate system;

[0143] f: Camera focal length equal to o and O c The distance between them;

[0144] Step S2.7.2 transforms a point P(u, v) in the depth image from the pixel coordinate system to the image coordinate system. Both the pixel coordinate system and the image coordinate system lie on the imaging plane, but their origins and units differ. The origin of the image coordinate system is the intersection of the camera's optical axis and the imaging plane, here chosen as the midpoint of the imaging plane. The unit of the image coordinate system is mm. The origin of the pixel coordinate system is the upper left corner of the image, and the unit is pixels.

[0145] The conversion between these two is as follows: where dx and dy represent the number of millimeters in each column and each row, respectively, i.e., 1 pixel = dx mm. Reference for pixel coordinate system, image coordinate system, and the position of pixel p. Figure 3 .

[0146] according to Figure 3 The positional relationship in the middle is transformed by the following formula:

[0147]

[0148]

[0149] Where: (u, v) are the coordinates of point p in the pixel coordinate system, (u0, v0) are the coordinates of the origin of the image coordinate system in the pixel coordinate system, and the transformed coordinates of point p(x, y) in the image coordinate system.

[0150] Step S2.7.3 transforms the point p(x, y) from the image coordinate system to the camera coordinate system. This is a perspective projection relationship, converting from 2D to 3D. The positional relationship between the image coordinate system, the camera coordinate system, and the point p is referenced. Figure 4 .

[0151] According to the principle of similar triangles:

[0152] ΔABO C ~ΔoCO C (1-3)

[0153] ΔPBO C ~ΔpCO C (1-4)

[0154] The proportional relationship can be obtained as follows:

[0155]

[0156] The transformation formula is as follows:

[0157]

[0158]

[0159]

[0160] Step S2.7.4, transfer point P(X) in the camera coordinate system C Y C Z C Transforming from the camera coordinate system to the world coordinate system involves rotation and translation. Rotating around different coordinate axes by different angles yields the corresponding rotation matrix.

[0161] Step S3, Assembly error detection:

[0162] Step S3.1: Use the SAC-IA algorithm (sample consistency algorithm) based on FPFH (FastPoint Feature Histogram) to coarsely register the model P1 to be assembled to the point cloud Q1 of the actual part to be assembled.

[0163] Step S3.1.1: First, the Fast Point Feature Histogram (FPFH) algorithm is used to extract the point feature histograms of the assembly model P1 and the actual point cloud Q1 of the parts to be assembled. The implementation process is as follows:

[0164] refer to Figure 5 This displays a center point p in the virtual point cloud P1 of the model to be assembled. z The SPFH (Simplified Point Feature Histogram) is used to calculate the region of influence, where point p z Let be the center point of a sphere with radius r in three-dimensional space. This center point and all k neighboring points inside the sphere with radius r form a network.

[0165] First, regarding point p... z Calculate point p z SPFH (Simplified Point Feature Histogram). Here, point p... z The simplified feature is defined as point p z With a point p in its domain k The relationship between the three angles Triple group The definition is as follows:

[0166] Given two points p z and p k And their respective normal directions are and And in p z A local coordinate system is defined at the point, consisting of three unit vectors. The rules for establishing it are as follows:

[0167]

[0168] in, Representing two points p Z and pk The Euclidean distance between them, and then using the above... coordinate system and The differences can be viewed from three angles The definitions of each angle are as follows:

[0169]

[0170] The next step is to redetermine the neighborhood of each point k and use the SPFH values ​​of neighboring points to calculate the final p. z The FPFH eigenvalues. Then point p z The FPFH eigenvalues ​​are:

[0171]

[0172] Where the weight ω k The value is the nearest neighbor p. k Triples that are counted repeatedly Increment the number of times by 1, i.e., refer to Figure 5 p k Add 1 to the number of the thickest black lines emitted.

[0173] Step S3.1.2: After calculating the point cloud feature histograms of the virtual model to be assembled P1 and the actual part to be assembled Q1, the SAC-IA algorithm (Sample Consistency Algorithm) is used for coarse registration, so that the virtual model to be assembled P1 and the actual part to be assembled Q1 are approximately coincident and registered. The steps are as follows:

[0174] (1) Select n sampling points from the point cloud P to be registered. In order to ensure that the sampled points have different FPFH features, the distance between each pair of sampling points should be greater than the pre-given minimum distance threshold δ.

[0175] (2) Find points in the target point cloud Q that are similar to the FPFH feature values ​​of the sampling points selected in step (1), and use them as the corresponding points of the sampling points of the point cloud to be registered in the target point cloud.

[0176] (3) Based on the correspondence between the point clouds determined in the first two steps, calculate the rotation and translation transformation matrices between them, and calculate the registration error through the transformation matrix. Evaluate the current registration result. Here, the Huber penalty function is used to calculate the registration error. The formula is as follows:

[0177]

[0178] Where δ is a preset value, and d i The distance difference after transformation of the corresponding points in the i-th group

[0179] (4) Repeat the first three steps until the registration error is minimized, which is to obtain the required optimal transformation matrix.

[0180] (5) The transformation matrix obtained in step (4) is subjected to nonlinear optimization using the LM (Levenverg-Marquardt) algorithm.

[0181] Step S3.2: After coarse registration of the virtual assembly model point cloud P1 and the actual assembly part point cloud Q1 using the SAC-IA algorithm, the point clouds basically overlap. Based on this, the traditional ICP algorithm is used for fine registration. The ICP algorithm process is as follows:

[0182] (1) Find the nearest point set corresponding to the target point cloud Q in the point cloud P to be registered, and obtain the initial correspondence between the two sets of point cloud data.

[0183] (2) Based on the correspondence obtained in step (1), calculate the rotation matrix R and translation vector T of the two sets of point clouds, and substitute them into the following formula to calculate the error;

[0184]

[0185] In the formula: Q k P is a point in the target point cloud Q; k It is a point in the target point cloud P;

[0186] (3) Update the position of the point cloud to be registered according to the transformation matrix calculated in step (2), and repeat step (1) to obtain the new corresponding nearest point set P′. k ;

[0187] (4) Calculate the newly obtained cloud of points to be registered, P′ k With the target point cloud Q k The average distance between them;

[0188]

[0189] (5) If d is less than the set threshold, or the number of iterations completed at this time is greater than the set number of iterations, then stop the calculation; otherwise, return to step (2) until the condition position is met.

[0190] Step S3.3: The augmented reality device displays the registered virtual image of the assembly to be assembled based on the registered assembly pose, current assembly progress and preset assembly relationship, to guide the assembly personnel to complete the assembly task of the assembly to be assembled.

[0191] Step S3.4: During the assembly process, the assembler uses the RGB-D camera in the augmented reality device to collect real-time data of the assembled assembly.

[0192] Step S3.5: Convert the data stream acquired by the RGB-D camera into a depth frame map, and then use step S2.7 to transform the depth images into 2D-3D coordinates to obtain the real position of the target point cloud Q2 of the real assembly.

[0193] Step S3.6: According to the SAC-IA algorithm based on FPFH in step S3.1, the virtual assembly model point cloud P2 is coarsely registered to the current pose point cloud Q2 of the actual assembly in real time. Here, the FPFH value is mainly collected from the feature values ​​of the parts to be assembled.

[0194] Step S3.7: According to step S3.2, the virtual assembly model point cloud P2 and the current pose point cloud Q2 of the actual assembly have basically overlapped after coarse registration by the SAC-IA algorithm. On this basis, the iterative nearest point (ICP) algorithm is used to perform real-time fine registration of the parts to be assembled in the assembly, so that the point cloud of the parts to be assembled in the virtual assembly model is accurately registered with the point cloud of the parts to be assembled in the real assembly.

[0195] In the point cloud images after registration in steps S3.8 and S3.7, the point clouds of the parts to be assembled overlap, while the assembled parts may not overlap due to reasons such as the operation of the assemblers, which results in assembly error. The magnitude of the assembly error is represented by Hausdorff distance, which is defined as follows.

[0196] Let the point cloud of the ideal assembly model be P2 = {P21, ..., P2i}, and the point cloud of the current pose of the real assembly part be Q2 = {Q21, ..., Q2k}. Then the Hay distance between the virtual assembly model point cloud P2 and the real assembly part's current pose point cloud Q2 is defined as:

[0197] H(P2, Q2)=max(h(P2, Q2), h(Q2, P2)) (1-14)

[0198] in:

[0199]

[0200] ||·|| is the distance paradigm between point sets P2 and Q2.

[0201] h(P2, Q2) is actually first represented by each point P in the virtual assembly model point cloud P2. 2i and distance from this point P 2i Each point Q in the current pose point cloud of the most recent real assembly part. 2k The distance between ||P 2i -Q 2kFind the minimum value of || and then take the maximum value of that distance as the value of h(P2, Q2); similarly, h(Q2, P2) can be obtained. Then, select the maximum value from h(P2, Q2) and h(Q2, P2) as the Howe distance between the virtual assembly model point cloud P2 and the current pose point cloud Q2 of the real assembly part.

[0202] Step S3.9: The assembler adjusts the assembly position of the parts according to the assembly error;

[0203] In step S3.10, the assembler uses the RGB-D camera in the augmented reality device to collect real-time data of the assembled assembly again, and repeats steps S3.4 to S3.9 until the Howe distance calculated in step S3.8 is less than the specified threshold, which indicates that the assembly is successful.

[0204] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for detecting a part assembly error based on augmented reality technology, characterized by, The method comprises the following steps: Obtain the virtual model point cloud of the parts to be assembled and the virtual model point cloud of the assembly; Obtain the actual point cloud of the parts to be assembled; Register the virtual model point cloud of the parts to be assembled to the actual point cloud of the parts to be assembled; Visualize the registered virtual image of the assembly to be assembled and guide the assembly personnel to complete the assembly task of the assembly to be assembled, and obtain the real assembly parts; Obtain the current pose point cloud of the real assembly parts; Detect the assembly error of the parts by comparing the virtual model point cloud of the assembly with the current pose point cloud of the real assembly parts; The visualization display of the registered virtual image of the assembly to be assembled and the guidance of the assembly personnel to complete the assembly task of the assembly to be assembled comprises: Import the assembly animation of different parts to be assembled and corresponding assembly parts into the augmented reality device, and the augmented reality device is provided with a trained convolutional neural network model for recognizing the parts to be assembled; Import all assembly-related CAD models of the parts to be assembled into the augmented reality device; The detection of the assembly error of the parts by comparing the virtual model point cloud of the assembly with the current pose point cloud of the real assembly parts comprises: Coarsely register the virtual model point cloud of the assembly to the current pose point cloud of the real assembly parts based on the SAC-IA algorithm of FPFH; After coarse registration, use the ICP algorithm for fine registration to register the virtual model point cloud of the assembly to the current pose point cloud of the real assembly parts; Obtain the non-coincidence deviation between the virtual model point cloud of the assembly and the current pose point cloud of the real assembly parts; Further comprising: repeatedly visualizing the registered virtual image of the assembly to be assembled and guiding the assembly personnel to complete the assembly task of the assembly to be assembled, so that the non-coincidence deviation is less than a set value.

2. The augmented reality technology-based part assembly error detection method according to claim 1, characterized by, The method comprises the following steps: Identify the parts to be assembled by using a convolutional neural network, and construct a convolutional neural network model; Read the convolutional neural network model, the virtual model of the parts to be assembled, the virtual model point cloud of the parts to be assembled, the virtual model of the assembly parts, and the virtual model point cloud of the assembly parts by using the augmented reality device.

3. The augmented reality technology-based part assembly error detection method according to claim 2, characterized by, The method comprises the following steps: Build the convolutional neural network model based on the YOLO v3 framework; Collect image data of the parts to be assembled and the assembly under different external environments and establish a data set; Label the pictures in the data set for preprocessing to generate training data and test data; Input the training data into the convolutional neural network model for training; Test the trained convolutional neural network model using the test data to obtain multiple test labels. If the accuracy of any test label is greater than a determination threshold, it is qualified, otherwise, repeat the training.

4. The augmented reality technology-based part assembly error detection method according to claim 1, characterized by, The method comprises the following steps: Use an RGB-D camera to collect on-site data and obtain an RGB image of the parts to be assembled; The RGB image is recognized based on a convolutional neural network model, data flow collected by the RGB-D camera is converted into a depth frame number graph, and a depth image is converted into an actual to-be-assembled part point cloud corresponding to the to-be-assembled part in a world coordinate system through 2D-3D coordinate transformation.

5. The augmented reality technology-based part assembly error detection method according to claim 1, characterized by, The to-be-assembled part virtual model point cloud is registered to the actual to-be-assembled part point cloud, comprising: The to-be-assembled model is coarsely registered to the actual to-be-assembled part point cloud based on an FPFH SAC-IA algorithm; After coarse registration, fine registration is performed using an ICP algorithm, so that the to-be-assembled model is registered to the actual to-be-assembled part point cloud, and an actual to-be-assembled part virtual model is obtained.

6. The augmented reality technology-based part assembly error detection method according to claim 5, characterized by, Further comprising: Fine registration is repeatedly performed using the ICP algorithm, so that the coincidence degree of the to-be-assembled model and the actual to-be-assembled part point cloud is higher than a set value, and an actual to-be-assembled part virtual model is obtained.

7. An electronic device, characterized by Comprising: A memory storing executable instructions; A processor running the executable instructions in the memory to implement the part assembly error detection method based on augmented reality technology according to any one of claims 1-6.

8. A part assembly error detection device based on augmented reality technology, characterized by, Comprising: An acquisition module; A storage module storing the part assembly error detection method based on augmented reality technology according to any one of claims 1-6; A helmet display electrically connected to the storage module and the acquisition module.

Images