An assisted assembly method based on virtual pose and scene-aware matching learning

Abstract

This invention provides an assisted assembly method based on virtual pose and scene-aware matching learning, belonging to the field of intelligent manufacturing technology. To perceive the scene, this invention utilizes a deep learning model to identify assembly components and their assembly areas on the assembly substrate. The corner information output by the deep learning model is used to calculate the corresponding homography matrix, thereby rendering the identified assembly components into the corresponding assembly areas to provide assembly guidance to the user. Simultaneously, a correct installation pose matching template for the components is generated in the virtual environment. The pose information of the assembly components is compared with the matching template, and a similarity score is calculated to obtain state correction information. Using the technical solution of this invention can reduce the user's cognitive burden in assembly, achieve rapid product assembly, and improve assembly efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing, and more specifically, to an assisted assembly method based on virtual pose and scene perception matching learning. Background Technology

[0002] As market demands become more personalized and diversified, products undergo rapid transformation and upgrading. The manual assembly of complex products changes constantly with product requirements. In traditional manual assembly, workers need to refer to two-dimensional paper assembly processes, resulting in fragmented assembly information and low visualization, leading to low efficiency in understanding and assembly operations. Augmented reality, as a new form of human-computer interaction and information visualization, can effectively solve these problems.

[0003] However, most existing mature augmented reality-assisted assembly systems focus on the research of the overall architecture and the implementation of methods, paying little attention to the status detection of parts after assembly. Yet, status detection of assembled parts is crucial for improving assembly success rates. Furthermore, augmented reality-assisted assembly largely requires manual triggering of control processes, increasing the cognitive burden on users and making it difficult to adapt to complex assembly scenarios.

[0004] Therefore, realizing an intelligent augmented reality-assisted assembly method that can proactively push assembly assistance information and correct status errors is of great significance for reducing the cognitive threshold of assembly, improving assembly efficiency and success rate, and promoting the development of the assembly manufacturing industry. Summary of the Invention

[0005] To address the aforementioned issues, this invention aims to disclose an assisted assembly method based on virtual pose and scene-aware matching learning. This method utilizes deep learning algorithms to match assembly parts with assembly areas, while simultaneously generating matching templates in a virtual environment to detect assembly quality, thereby improving assembly efficiency and success rate.

[0006] To achieve the above objectives, the technical solution adopted by the present invention mainly includes the following processes:

[0007] Step 1: Collect the following information about the assembly object: the three-dimensional workpiece model of the assembly base, the three-dimensional workpiece model of each assembly component, the relative pose of the assembly components when they are correctly installed on the assembly base, and the assembly process information corresponding to each assembly component during assembly.

[0008] Step 2: Import the information collected in Step 1 into the virtual assembly environment, set a virtual camera with the same parameters as the real camera, and use the virtual camera to acquire images of the assembly parts in the correct installation pose from different angles to form a set of pose matching templates of the assembly parts correctly installed on the assembly base.

[0009] Step 3: Repeat step 2 to acquire images of different assembly parts to form a pose matching template library;

[0010] Step 4: Use a real camera to capture images of different assembly parts, and label the captured images with the corresponding assembly part names to create an assembly part image dataset.

[0011] Step 5: Use a real camera to acquire images of the assembly base, mark corner points on the acquired images, two corner points enclose a rectangular area to be assembled, and label the area with the name of the parts to be assembled, thus establishing an assembly base image dataset.

[0012] Step 6: Using the assembly component images in the assembly component image dataset as input and the name labels in the assembly component image dataset as output, construct Deep Learning Model I;

[0013] The data from the assembly component image dataset is input into the deep learning model I for training;

[0014] Step 7: Using the assembly base image in the assembly base image dataset as input, and the corner positions and corresponding assembly component name labels in the assembly base image dataset as output, construct Deep Learning Model II;

[0015] The data from the assembly substrate image dataset is input into the deep learning model II for training;

[0016] Step 8: Collect images of the assembly work site using a real camera and input the image information into Deep Learning Model I and Deep Learning Model II. Determine the assembly parts selected by the user based on the output of Deep Learning Model I, and determine the assembly area and corresponding corner information of the assembly parts selected by the user on the assembly base based on the output of Deep Learning Model II.

[0017] Step 9: Based on the workpiece model and assembly process information of the assembly parts selected by the user, and using the corresponding corner point information as the alignment information, render the 3D workpiece model and assembly process information of the assembly parts into the real assembly operation scene to provide assembly guidance for the user.

[0018] Step 10: After the user completes the corresponding operation of the current assembly part, the image of the completed assembly is matched with the corresponding matching template in the pose matching template library in Step 3. The similarity score between the matching template and the image of the completed assembly is calculated. If the similarity score is greater than the set threshold, the assembly of the assembly part is completed. If it is less than the threshold, Step 11 is executed.

[0019] Step 11: Estimate the pose information of the currently assembled parts using 2D and depth information from the real scene, compare the obtained pose with the pose corresponding to the matching template in Step 3, calculate the pose difference, and re-render the pose difference to provide users with assembly error correction information for reassembly.

[0020] Step 12: Repeat steps 8 to 11 until all assembly parts are assembled.

[0021] Preferably, in step 2, the correctly installed pose matching template includes the following information: color gradient information obtained using edge detection technology, point cloud normal vector information obtained using fitting technology, pose information of the assembled parts in the virtual camera coordinate system when they are correctly installed, pose information of the assembled base in the virtual camera coordinate system, and pose information of the virtual camera image acquisition in the world coordinate system.

[0022] Preferably, in step 9, the three-dimensional workpiece model of the assembly parts and the assembly process information are rendered into the real assembly operation scene. The specific method is as follows: the homography matrix is ​​calculated by using the three-dimensional coordinates of the corner points and the pixel coordinates obtained by the deep learning model II in step 8, and the three-dimensional model of the parts is projected onto the pixel plane through the homography matrix, thereby superimposing it onto the real scene.

[0023] Preferably, in step 10, the similarity score between the matching template and the assembled image is calculated by: cropping the ROI region of the assembly operation site image, extracting key matching points from the cropped image, converting the extracted key matching points into feature vectors, and calculating the Euclidean distance between the extracted feature vectors and the feature vectors in the template image. The smaller the Euclidean distance, the higher the similarity.

[0024] Preferably, in step 11, the pose difference value is re-rendered, and the rendering method includes, but is not limited to, the rendering method with text annotation, the rendering method with static arrow guidance, and the rendering method with dynamic 3D animation. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below. However, it should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the scope of the invention.

[0026] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains, and the use of terms as used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention.

[0027] Step 1: Collect the following information about the assembly object: the three-dimensional workpiece model of the assembly base, the three-dimensional workpiece model of each assembly component, the relative pose of the assembly components when they are correctly installed on the assembly base, and the assembly process information corresponding to each assembly component during assembly.

[0028] Step 2: Import the information collected in Step 1 into the virtual assembly environment, set a virtual camera with the same parameters as the real camera, and use the virtual camera to acquire images of the assembly parts in the correct installation pose from different angles to form a set of pose matching templates of the assembly parts correctly installed on the assembly base.

[0029] Preferably, the correctly installed pose matching template includes the following information: color gradient information obtained by edge detection technology, point cloud normal vector information obtained by fitting technology, pose information of the assembled parts in the virtual camera coordinate system when they are correctly installed, pose information of the assembled base in the virtual camera coordinate system, and pose information of the virtual camera image in the world coordinate system.

[0030] Step 3: Repeat step 2 to acquire images of different assembly parts to form a pose matching template library;

[0031] Step 4: Use a real camera to capture images of different assembly parts, and label the captured images with the corresponding assembly part names to create an assembly part image dataset.

[0032] Preferably, the image is acquired by taking pictures with a camera, and the image content is in RGB and RGB-D formats, and the saved image is in PNG format;

[0033] Preferably, the assembly part image dataset includes all parts to be assembled and their corresponding labels, with each assembly part consisting of 200 images and its corresponding label;

[0034] Step 5: Use a real camera to acquire images of the assembly base, mark corner points on the acquired images, two corner points enclose a rectangular area to be assembled, and label the area with the name of the parts to be assembled, thus establishing an assembly base image dataset.

[0035] Preferably, the assembly base image dataset includes assembly base images at different assembly stages and overall images of the assembled base after assembly, containing a total of 300 images and corresponding corner information. The saved images are in PNG format, and the saved corner information is in TXT format.

[0036] Step 6: Using the assembly component images in the assembly component image dataset as input and the name labels in the assembly component image dataset as output, construct Deep Learning Model I;

[0037] The data from the assembly component image dataset is input into the deep learning model I for training;

[0038] As a preferred embodiment, the deep learning model I adopts the AlexNet network structure. After the input undergoes nonlinear transformation through 5 convolutional layers and 3 fully connected layers, the final result is obtained by a softmax classifier in the output layer. The activation function is the ReLU function, and the loss function is the mean squared error function.

[0039] Step 7: Using the assembly base image in the assembly base image dataset as input, and the corner positions and corresponding assembly component name labels in the assembly base image dataset as output, construct Deep Learning Model II;

[0040] The data from the assembly substrate image dataset is input into the deep learning model II for training;

[0041] Preferably, the deep learning model II adopts the YOLOv5 network structure, which is divided into four modules: input, backbone, neck, and head. The input module adjusts the size of the input image to support the mining and detection of fine-grained features of the image. The backbone module, as the core feature extractor, has 72 convolutional layers with a kernel size of 3×3 and a stride of 2. The neck module is used for feature fusion and contains three max pooling layers with kernel sizes of 3×3, 9×9, and 13×13, respectively. The head module outputs the final target object and category probability, and the loss function adopted is CIOULoss.

[0042] Step 8: Collect images of the assembly work site using a real camera and input the image information into Deep Learning Model I and Deep Learning Model II. Determine the assembly parts selected by the user based on the output of Deep Learning Model I, and determine the assembly area and corresponding corner information of the assembly parts selected by the user on the assembly base based on the output of Deep Learning Model II.

[0043] Step 9: Based on the workpiece model and assembly process information of the assembly parts selected by the user, and using the corresponding corner point information as the alignment information, render the 3D workpiece model and assembly process information of the assembly parts into the real assembly operation scene to provide assembly guidance for the user.

[0044] Preferably, the three-dimensional workpiece model of the assembly parts and the assembly process information are rendered into a real assembly operation scene. The specific method is as follows: the homography matrix is ​​calculated by using the three-dimensional coordinates of the corner points and the pixel coordinates obtained by the deep learning model II in step 8, and the three-dimensional model of the parts is projected onto the pixel plane through the homography matrix, thereby superimposing it onto the real scene.

[0045] Step 10: After the user completes the corresponding operation of the current assembly part, the image of the completed assembly is matched with the corresponding matching template in the pose matching template library in Step 3. The similarity score between the matching template and the image of the completed assembly is calculated. If the similarity score is greater than the set threshold, the assembly of the assembly part is completed. If it is less than the threshold, Step 11 is executed.

[0046] Preferably, the method for calculating the similarity score between the matching template and the assembled image is as follows: cropping the ROI region of the assembly operation site image, extracting key matching points from the cropped image, converting the extracted key matching points into feature vectors, and calculating the Euclidean distance between the extracted feature vectors and the feature vectors in the template image. The smaller the Euclidean distance, the higher the similarity.

[0047] Step 11: Estimate the pose information of the currently assembled parts using 2D and depth information from the real scene, compare the obtained pose with the pose corresponding to the matching template in Step 3, calculate the pose difference, and re-render the pose difference to provide users with assembly error correction information for reassembly.

[0048] Preferably, the pose difference value is re-rendered, and the rendering method includes, but is not limited to, the rendering method with text annotation, the rendering method with static arrow guidance, and the rendering method with dynamic 3D animation.

[0049] Step 12: Repeat steps 8 to 11 until all assembly parts are assembled.

[0050] The above are merely embodiments of the present invention. Commonly known structures and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the structure of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. An assisted assembly method based on virtual pose and scene-aware matching learning, characterized in that, Includes the following steps: Step 1: Collect the following information about the assembly object: the three-dimensional workpiece model of the assembly base, the three-dimensional workpiece model of each assembly component, the relative pose of the assembly components when they are correctly installed on the assembly base, and the assembly process information corresponding to each assembly component during assembly. Step 2: Import the information collected in Step 1 into the virtual assembly environment, set a virtual camera with the same parameters as the real camera, and use the virtual camera to acquire images of the assembly parts in the correct installation pose from different angles to form a set of pose matching templates of the assembly parts correctly installed on the assembly base. Step 3: Repeat step 2 to acquire images of different assembly parts to form a pose matching template library; Step 4: Use a real camera to capture images of different assembly parts, and label the captured images with the corresponding assembly part names to create an assembly part image dataset. Step 5: Use a real camera to acquire images of the assembly base, mark corner points on the acquired images, two corner points enclose a rectangular area to be assembled, and label the area with the name of the parts to be assembled, thus establishing an assembly base image dataset. Step 6: Using the assembly component images in the assembly component image dataset as input and the name labels in the assembly component image dataset as output, construct Deep Learning Model I; The data from the assembly parts image dataset is input into the deep learning model I for training; Step 7: Using the assembly base image in the assembly base image dataset as input, and the corner positions and corresponding assembly component name labels in the assembly base image dataset as output, construct Deep Learning Model II; The data from the assembly substrate image dataset is input into the deep learning model II for training; Step 8: Collect images of the assembly work site using a real camera and input the image information into Deep Learning Model I and Deep Learning Model II. Determine the assembly parts selected by the user based on the output of Deep Learning Model I, and determine the assembly area and corresponding corner information of the assembly parts selected by the user on the assembly base based on the output of Deep Learning Model II. Step 9: Based on the workpiece model and assembly process information of the assembly parts selected by the user, and using the corresponding corner point information as the alignment information, render the 3D workpiece model and assembly process information of the assembly parts into the real assembly operation scene to provide assembly guidance for the user. Step 10: After the user completes the corresponding operation of the current assembly part, the image of the completed assembly is matched with the corresponding matching template in the pose matching template library in Step 3. The similarity score between the matching template and the image of the completed assembly is calculated. If the similarity score is greater than the set threshold, the assembly of the assembly part is completed. If it is less than the threshold, Step 11 is executed. Step 11: Estimate the pose information of the currently assembled parts using 2D and depth information from the real scene, compare the obtained pose with the pose corresponding to the matching template in Step 3, calculate the pose difference, and re-render the pose difference to provide users with assembly error correction information for reassembly. Step 12: Repeat steps 8 to 11 until all assembly parts are assembled.

2. The assisted assembly method based on virtual pose and scene-aware matching learning as described in claim 1, characterized in that: The correctly installed pose matching template described in step 2 includes the following information: color gradient information obtained using edge detection technology, point cloud normal vector information obtained using fitting technology, pose information of the assembled parts in the virtual camera coordinate system when they are correctly installed, pose information of the assembled base in the virtual camera coordinate system, and pose information of the virtual camera image in the world coordinate system.

3. The assisted assembly method based on virtual pose and scene-aware matching learning as described in claim 1, characterized in that: In step 9, the three-dimensional workpiece model of the assembly parts and the assembly process information are rendered into the real assembly operation scene. The specific method is as follows: the homography matrix is ​​calculated by using the three-dimensional coordinates of the corner points and the pixel coordinates obtained by the deep learning model II in step 8, and the three-dimensional model of the parts is projected onto the pixel plane through the homography matrix, thereby superimposing it onto the real scene.

4. The assisted assembly method based on virtual pose and scene-aware matching learning as described in claim 1, characterized in that: In step 10, the similarity score between the matching template and the assembled image is calculated by cropping the ROI region of the assembly operation site image, extracting key matching points from the cropped image, converting the extracted key matching points into feature vectors, and calculating the Euclidean distance between the extracted feature vectors and the feature vectors in the template image. The smaller the Euclidean distance, the higher the similarity.

5. The assisted assembly method based on virtual pose and scene-aware matching learning as described in claim 1, characterized in that: In step 11, the pose difference value is re-rendered, and the rendering methods include: a rendering method with text annotation, a rendering method with static arrow guidance, and a rendering method with dynamic 3D animation.