An assembly detection method based on augmented reality technology

Abstract

An assembly detection method based on augmented reality technology, comprising: 1) performing 3D modeling on the assembly object in the physical world to obtain a 3D model and its source point cloud, and attaching an assembly animation through the unity3D engine; 2) acquiring the depth information of the physical world based on the HoloLens2 researcher mode to obtain a three-dimensional point cloud, and preprocessing the target point cloud; 3) performing point cloud registration on the source point cloud and the target point cloud to obtain a pose transformation matrix, matching the virtual model and the physical model through the transformation matrix, and setting a spatial anchor point; 4) making a UI interface based on Unity3D to control the assembly animation of the virtual model and prompt the detailed assembly information. 5) Using the Octree structure to search for overlapping point clouds and calculate the point cloud overlap degree, calculating the relative error through the number of overlapping points, so as to evaluate the assembly completion degree.

Description

Technical Field

[0001] The present invention relates to the technical field of digital twins, and particularly to an assembly detection method based on augmented reality technology. Background Art

[0002] Augmented reality technology is a new technology that "seamlessly" integrates real-world information and virtual-world information. It is a technology that superimposes entity information (visual information, sound, taste, touch, etc.) that is difficult to experience within a certain time and space range in the real world through scientific and technological means such as computers, and then applies the virtual information to the real world, which can be perceived by the human senses, so as to achieve a sensory experience beyond reality. The Hololens2 device officially released by Microsoft is a comfortable-to-wear mixed reality device, which is applied to modernize factories to adapt to the rapid changes of Industry 4.0, build innovative, efficient and sustainable production environments, and in industrial automation and employee training, to bring productivity to a new level.

[0003] Assembly refers to the process of assembling parts according to specified technical requirements and making them into qualified products through debugging and inspection. However, the diversity of products and the complexity of the assembly process lead to situations where assembly steps are missing or parts are not installed in place, which will reduce product safety and the benefits of enterprises. Traditional assembly training is that experienced technical workers guide and train new employees. However, due to limited human energy and the complexity of guiding the assembly process, the effect is not good, and new employees need to continuously make mistakes to obtain better experience.

[0004] To solve the above problems, augmented reality assembly technology is continuously being developed. Zhou Guanghui proposed an assembly guidance method and system based on Hololens depth data (Zhou Guanghui, An assembly guidance method and system based on Hololens depth data [P]. Chinese Patent, CN113706689A, 2021.11.26). This system realizes the collection of environmental data using the depth sensor built in Hololens to achieve assembly guidance and improve assembly efficiency. However, this system does not detect the assembly effect after assembly is completed. Yang Kangkang proposed research and application of key augmented reality technologies for complex product assembly (Yang Kangkang. Research and application of key augmented reality technologies for complex product assembly [D]. Jiangsu: Nanjing University of Aeronautics and Astronautics, 2019.). This technology performs assembly through Kinect tracking calibration and three-dimensional registration of physical models, and HoloLens. However, this method requires data transmission between Kinect and HoloLens, which affects real-time performance. And the present method adopts an assembly detection method based on augmented reality technology, which has real-time feedback on the assembly effect, and adopts interaction between HoloLens2 and the PC side to reduce time consumption and improve real-time performance. Summary of the Invention

[0005] The present invention aims to solve the above-mentioned technical problems existing in the prior art, and provides an assembly detection method based on augmented reality technology.

[0006] Assembly refers to the process of assembling parts according to specified technical requirements, and through debugging and inspection to make them into qualified products. However, the diversity of products and the complexity of the assembly process lead to situations where assembly steps are missing or parts are not installed in place, which will reduce product safety and the efficiency of enterprises. Traditional assembly training is that skilled technical workers with rich technical experience conduct guiding training for new employees. However, due to limited human energy and the complexity of guiding the assembly process, good results cannot be obtained, and new employees need to continuously make mistakes to gain better experience.

[0007] Augmented reality technology is a new technology that "seamlessly" integrates real-world information and virtual-world information. It is to simulate and superimpose entity information (visual information, sound, taste, touch, etc.) that is difficult to experience within a certain time and space range in the real world through scientific and technological means such as computers, and then apply the virtual information to the real world, which can be perceived by human senses, so as to achieve a sensory experience beyond reality. The Hololens2 device officially released by Microsoft is a comfortable-to-wear mixed reality device, which is applied to modernize factories to adapt to the rapid changes of Industry 4.0 and build an innovative, efficient and sustainable production environment, as well as in industrial automation and employee training, so as to bring productivity to a new level.

[0008] The present invention overcomes the above problems of the prior art and proposes an assembly detection method based on augmented reality technology.

[0009] The present invention collects data based on the Hololens2 device, processes the data on the PC side, makes an assembly animation through SolidWorks and attaches it to the model, realizes the assembly interaction between the virtual and real worlds through the Unity3D engine, and detects the assembly effect through the point cloud coincidence degree. An assembly detection method based on augmented reality technology of the present invention mainly includes five parts: model preparation, data collection, point cloud processing, virtual-real fusion, and effect detection.

[0010] In order to solve the above technical problems, the present invention provides an assembly detection method based on augmented reality technology, including the following steps:

[0011] Step 1, perform 3D modeling on the assembly object in the physical world to obtain a 3D model and its source point cloud, and attach an assembly animation through the Unity3D engine.

[0012] Step 2: Use Hololens2 researcher mode to collect depth information of the physical world to obtain a three-dimensional point cloud, and pre-process the target point cloud.

[0013] Step 3: Perform point cloud registration on the source point cloud and the target point cloud to obtain the pose transformation matrix. The virtual model and the physical model are matched through the transformation matrix, and the spatial anchor point is set.

[0014] Step 4: Create a UI interface based on Unity3D to control the virtual model assembly animation and provide detailed assembly information.

[0015] Step 5: Use the Octree structure to search for overlapping point clouds and calculate the point cloud overlap. The relative error is calculated based on the number of overlapping points to evaluate the assembly completion.

[0016] Wherein, the step 1 specifically includes:

[0017] First, obtain the assembly step instructions of the object to be assembled, then use SolidWorks software to perform proportional 3D modeling on the assembly model, and import the created STEP file format model file into 3DMax software for mapping and rendering. Then save the processed model file as an FBX format file. The obtained FBX model first needs to be imported into Python to call the Open3D library to convert it into a point cloud model in PCD format, and save it to the database for easy call. Secondly, it needs to be imported into Unity3D to produce assembly animation and step information.

[0018] The production of Unity3D assembly animation uses the engine's built-in Animation system. First, select the initial state model of the object to be assembled, and then add the animation information of the parts assembly based on the initial state model. The parts assembly process must comply with the step information in the assembly manual. It is necessary to add Rigidbody to the sub-objects of the assembly object, cancel the Use Gravity option, select the Is Kinematic option, and check the Loop Time option for the Animation object so that the assembly object animation can serve as a real-time reminder during the assembly process.

[0019] Wherein, the step 2 specifically includes:

[0020] Before proceeding to step 2, you need to enable Hololens 2's Researcher Mode. Researcher Mode is an application used by Hololens to access key sensors. Hololens 2 adds data access to visible light environment tracking cameras, depth cameras, and more. After enabling Researcher Mode, depth data is acquired through the Hololens 2 depth camera and transmitted to the PC for processing.

[0021] The processing of depth data adopts the camera calibration principle, that is, the process of obtaining the final projection matrix P from the world coordinate system to the camera coordinate system and then to the image coordinate system. The process from the world coordinate system to the camera coordinate system is to solve the external parameter matrix, which is obtained by the PrintDepthExtrinsics class in HL2ResearchMode. The process from the camera coordinate system to the image coordinate system is to solve the internal parameter matrix, which is obtained by the Zhang Zhengyou checkerboard calibration method. Finally, the three-dimensional point cloud is obtained through the matrix operation of formula (1).

[0022]

[0023] Among them, u and v are arbitrary coordinate points in the image coordinate system, u0 and v0 are the central coordinates of the image respectively, x w , y w , z w represent the three-dimensional coordinate points in the world coordinate system. z c represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera. R and T are the 3x3 rotation matrix and 3x1 translation matrix in the external parameter matrix respectively.

[0024] Among them, the specific steps of step three include:

[0025] First, preprocess the target point cloud obtained in step two. Since there is wall and clutter information around the object to be assembled, it is necessary to cut the point cloud. Here, the selectionpolygonVolume class in the Open3D library in Python is used to crop the useful area in the point cloud to obtain the target point cloud. Then, the processed target point cloud and the source point cloud need to be registered by the registration_icp class in the Open3D library, and the pose transformation matrix from the source point cloud to the target point cloud is output.

[0026] The pose transformation matrix is sent to HoloLens2 through TCP / IP communication. HoloLens2 calls the relevant model and uses the pose transformation matrix to transform its position in its world coordinate system to achieve the effect of virtual-real fusion. Since the virtual model will shift when the camera shakes, a spatial anchor is added to the fused virtual model. The spatial anchor in HoloLens2 is a method that can keep the object in a specific position and rotation state. The virtual model is fixed on the physical model through the spatial anchor technology, which is convenient for subsequent assembly animation demonstration.

[0027] Among them, the specific steps of step four include:

[0028] The buttons that need to be added first are as follows: two forward and backward buttons for controlling the front and back frame assembly steps of the animation; a button for adding and deleting spatial anchors for flexible operation of the spatial anchors; an assembly evaluation button for evaluating the completion of the assembly. The text boxes that need to be added are as follows: one for describing the detailed information of the parts required for the assembly to find the corresponding assembly parts; one for describing the detailed text information of the assembly steps to assist the assembly operation.

[0029] When adding buttons, the PressableButtonHoloLens2 script in the MRTK package needs to be added to endow the button with the pressable attribute, the interactable script to endow the interactive attribute, and the NearInteractionTouchable script to endow the near-interactive touch attribute. When adding text components, the text information needs to be corresponded to the assembly steps one by one. For the entire UI interface, the SolverHandler script and the RadialView script in the MRTK package need to be added to make the GameObject follow the user's gaze. Among them, the SolverHandler script is used to set the object for tracking reference. Here, the reference point is selected as the Head, that is, the main camera, and the RadialView script is a trailing component used to keep a specific part of the GameObject within the cone of the user's field of view.

[0030] Among them, the specific content of Step 5 includes:

[0031] Under the animation and text step prompts in Step 4, assemble the assembly to be assembled. After each step is completed, the assembly completion degree needs to be evaluated. The evaluation uses Octree to calculate the overlap degree between two point clouds. The specific evaluation method is as follows: First, the source point cloud of the virtual assembly part has been generated in Step 1. The target point cloud needs to be obtained by collecting the depth data of the physical model that has been assembled according to Step 2. Then, call the PCL library to search for the overlapping part of the source point cloud and the target point cloud using the Octree structure, and then calculate the relative error of the number of overlapping points between the source point cloud and the target point cloud. Finally, set a threshold to judge the assembly completion degree.

[0032] The specific steps of the Octree structure search are to first build an octree for the source point cloud, and then traverse all points in the target point cloud to query whether there are the same point clouds in the voxels corresponding to the source point cloud. If there are, they are overlapping point clouds.

[0033] The specific steps for the point cloud operation of the overlapping part and the total point cloud are as follows: collect depth data from N angles of the assembled parts to obtain N point clouds at different angles, then use the Octree structure to search for the overlapping parts of these point clouds with the source point cloud respectively, calculate the relative error value between the overlapping part points and the source point cloud, and finally take the average of these N results as the experimental result. The calculation formula is shown in formula (2), where is the number of points in the overlapping part, P is the number of points in the source point cloud, is the relative error result.

[0034]

[0035] For the selection of the threshold, corresponding experimental data collection and calculation are carried out before the assembly experiment. For the detection of the assembly, it is to detect whether the assembled parts meet the requirements of the assembly steps. Select the average of the relative error of the coincidence degree calculated from the target point cloud and the source point cloud that are successfully assembled at K steps and N different angles, and finally select this average value as the upper limit of the assembly completion evaluation value. The calculation formula is shown in formula (3).

[0036]

[0037] Finally, the judgment standard for the assembly completion degree is shown in formula (4), where is the relative error during the assembly process, is the relative error obtained from the preliminary experiment. If the UI interface displays completed, if the UI interface displays uncompleted.

[0038]

[0039] An assembly detection method based on augmented reality technology of the present invention collects data based on a Hololens2 device, processes data on a PC side, makes an assembly animation through SolidWorks and attaches it to a model, realizes assembly interaction between the virtual and real worlds through the Unity3D engine, and detects the assembly effect through the point cloud coincidence degree. It is mainly divided into five parts: model preparation, data collection, point cloud processing, virtual-real fusion, and effect detection. First, 3D modeling is performed on the assembly object in the physical world to obtain a 3D model and its source point cloud, and an assembly animation is attached through the Unity3D engine. Then, the depth information of the physical world is collected in the researcher mode of Hololens2 to obtain a three-dimensional point cloud, and the target point cloud is preprocessed. Next, the source point cloud and the target point cloud are registered to obtain a pose transformation matrix, the virtual model and the physical model are matched through the transformation matrix, and a spatial anchor point is set. To improve the operation experience of HoloLens2, a UI interface is made based on Unity3D to control the assembly animation of the virtual model and prompt the detailed assembly information. Finally, the Octree structure is used to search for overlapping point clouds and calculate the point cloud overlap degree, and the relative error is calculated through the number of overlapping points, so as to evaluate the assembly completion degree. Therefore, the present invention can not only achieve the effect of virtual-real fusion of the assembly environment, but also evaluate the assembly completion degree and feedback it to the operator, realizing a good assembly operation experience and a complete assembly process.

[0040] The advantages of the present invention are as follows: Attaching an assembly animation to the 3D model through the Animation animation system in Unity3D is beneficial for the operator to receive assembly signals in real time and make assembly operations. The integrated process of using Hololens2 to obtain depth data and perform assembly operations, and setting a spatial anchor point for the virtual model is beneficial for the stability of the virtual model in the physical world and reducing the equipment requirement cost. Using the Octree structure to search for overlapping point clouds and calculate the point cloud overlap degree, and calculating the relative error through the number of overlapping points, so as to evaluate the assembly completion degree and feedback it to the operator, is beneficial for improving the correctness and experience of the operator's assembly. Description of the Drawings

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, obtaining other drawings based on these drawings still belongs to the scope of the present invention.

[0042] Figure 1 It is a block diagram for obtaining a pose matrix of the assembly detection method based on augmented reality technology provided by an embodiment of the present invention.

[0043] Figure 2 This is the effect detection block diagram of the assembly detection method based on augmented reality technology provided by the embodiments of the present invention. Detailed implementation manners

[0044] The present invention will be further described in detail below with reference to the accompanying drawings.

[0045] As Figures 1 to 2 shown, this is an embodiment of the present invention, providing an assembly detection method based on augmented reality technology. This method is constructed based on Hololens2 and mainly includes five parts: model preparation, data acquisition, point cloud processing, virtual-real fusion, and effect detection. Figure 1 Describe the three parts of model preparation, data acquisition, and point cloud processing. Figure 2 Describe the two parts of virtual-real fusion and effect detection.

[0046] The three parts of model preparation, data acquisition, and point cloud processing are mainly divided into online state and offline state. The online state is mainly data acquisition and point cloud processing, and the offline state is mainly model preparation. In the offline state, according to the physical assembly model, 3D modeling is performed using SolidWorks and saved in STEP format and transmitted to 3DMAX software. The virtual model is rendered and textured through 3DMAX software to achieve similarity with the physical model's color information, and the rendered and textured virtual model is saved in FBX file format. Finally, the model file is imported into Python, and the Open3D library is called to convert the FBX file into a PCD source point cloud file.

[0047] In the online state, Hololens2 and a PC are mainly used. First, the depth data of the model to be assembled is collected by the depth camera of Hololens2, and the depth data is transmitted to the Python side in the form of a depth image in PNG format through TCP / IP communication. The Python side calls Open3D and the internal and external parameter matrices to convert the depth data into point clouds. The internal parameter matrix is calculated by the Zhang Zhengyou checkerboard calibration method, and the external parameter matrix is obtained from the PrintDepthExtrinsics class in HL2ResearchMode. The point clouds collected by the depth camera contain irrelevant information such as walls, floors, and sundries, and need to be segmented to obtain the required point cloud information and saved as a target point cloud in PCD format. Finally, the source point cloud obtained in the offline state is registered with the target point cloud obtained in the online state to obtain a pose transformation matrix and transmit it to the assembly scene.

[0048] The two parts of virtual-real fusion and effect detection consist of Hololens2, a local computer, and a database. By Figure 1The obtained pose transformation matrix is transmitted to the Hololens2 assembly scene through TCP / IP communication and the virtual model in the database is called, so that the virtual model undergoes pose transformation in the scene and the spatial anchor technology in Hololens2 is added to fix the virtual model in the assembly scene.

[0049] When performing the assembly operation, it is necessary to call the assembly animation in the database and create the UI interface. The assembly animation is made by the Animation plugin in Unity3D. The buttons to be added to the UI interface are as follows: two forward and backward buttons for controlling the assembly steps of the front and back frames of the animation; a button for adding and deleting spatial anchors for flexible operation of the spatial anchors; an assembly evaluation button for evaluating the completion of the assembly. The text boxes to be added are as follows: one for describing the detailed information of the parts required for assembly to find the corresponding assembly parts; one for describing the detailed text information of the assembly steps to assist the assembly operation. When adding buttons, it is necessary to add the PressableButtonHoloLens2 script in the MRTK package to endow the pressable button attribute, the Interactable script to endow the interactive attribute, and the NearInteractionTouchable script to endow the near-interactive touch attribute. When adding text components, it is necessary to correspond the text information to the assembly steps one by one.

[0050] After the parts are installed, the Hololens2 is used to collect the depth data of the four angles of east, south, west, and north of the scene, and the depth data of these four angles is transmitted to the Python side in the form of a PNG depth image through TCP / IP communication. The Python side calls Open3D and the internal and external parameter matrices to convert the depth data into a point cloud. The internal parameter matrix is calculated by the Zhang Zhengyou checkerboard calibration method, and the external parameter matrix is obtained from the PrintDepthExtrinsics class in HL2ResearchMode. The point cloud obtained by the depth camera acquisition contains irrelevant information such as walls, floors, and sundries, and needs to be cut to obtain the required point cloud information and save the target point cloud in the PCD format.

[0051] The Octree structure search is performed on these four target point clouds and the source point cloud saved in the database to obtain the overlapping point cloud. The relative error value is calculated between the overlapping points and the source point cloud, and finally the average value of these four results is taken as the experimental result. As shown in formula (5), where is the number of overlapping points, P is the number of points in the source point cloud, and X is the relative error result.

[0052]

[0053] The pre-experimental error in the database is the average value of the relative error of the coincidence between the target point cloud and the source point cloud after five repeated experiments on the target step and the successful assembly from four different angles before the experiment. This average value is finally selected as the upper limit of the assembly completion evaluation value, as shown in formula (6), where is the number of overlapping points, P is the number of source point cloud points, and Y is the relative error result.

[0054]

[0055] The final evaluation criteria for assembly completion are shown in formula (7), where X is the relative error in the assembly process, Y is the relative error obtained from the preliminary experiment, where the Y value is 0.75, and f is the evaluation result. If X <Y在UI界面显示装配成功,如果X≥Y在UI界面显示装配失败。

[0056]

[0057] The above is a specific implementation method of the entire invention. The present invention obtains a point cloud through the depth data of Hololens2, and performs point cloud registration with the source point cloud to obtain a posture transformation matrix. The obtained matrix is used to integrate the virtual model and the physical model and perform assembly operations. The assembled point cloud and the source point cloud are overlapped to obtain a relative error, and then the assembly result is obtained by comparing it with the pre-experimental error.

[0058] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. An assembly detection method based on augmented reality technology, comprising the following steps: Step 1, perform 3D modeling on the physical world assembly object to obtain a 3D model and its source point cloud, and attach an assembly animation through the unity3D engine; Step 2, collect the physical world depth information through the HoloLens2 researcher mode to obtain a three-dimensional point cloud, and preprocess the target point cloud; Step 3, perform point cloud registration on the source point cloud and the target point cloud to obtain a pose transformation matrix, match the virtual model and the physical model through the transformation matrix, and set a spatial anchor point; Step 4, make a UI interface based on Unity3D to control the assembly animation of the virtual model and prompt the assembly details; Step 5, use the Octree structure to search for overlapping point clouds and calculate the point cloud overlap degree, calculate the relative error through the number of overlapping points, so as to evaluate the assembly completion degree; specifically including: Under the animation and text step prompts in Step 4, assemble the parts to be assembled. After each step is completed, it is necessary to evaluate the assembly completion degree. The evaluation uses the Octree to calculate the coincidence degree between two point clouds for evaluation. The specific evaluation method is as follows: First, the source point cloud of the virtual assembly part has been generated in Step 1. The target point cloud needs to be obtained by collecting the depth data of the physically assembled model according to Step 2. Then, call the PCL library to search for the overlapping part of the source point cloud and the target point cloud with the Octree structure, and then calculate the relative error of the number of overlapping points between the source point cloud and the target point cloud. Finally, judge the assembly completion degree by setting a threshold; The specific steps of the Octree structure search are to first build an octree for the source point cloud, and then traverse all points in the target point cloud to query whether there are the same point clouds in the voxels corresponding to the source point cloud. If so, it is an overlapping point cloud; The specific steps for the operation and processing of the overlapping points and the total point cloud are to collect depth data from N angles of the assembled parts to obtain N point clouds at different angles, and then use the Octree structure to search for the overlapping parts of these point clouds and the source point cloud respectively. Calculate the relative error value of the overlapping points and the source point cloud, and finally take the average value of these N results as the experimental result; The calculation formula is as shown in formula (2), where is the number of points in the overlapping part, P is the number of points in the source point cloud, is the relative error result; The selection of the threshold is to collect and calculate the corresponding experimental data before the assembly experiment. For the detection of the assembly, it is to detect whether the assembled parts meet the requirements of the assembly steps. Select the average value of the relative error of the coincidence degree calculated from the target point cloud and the source point cloud that are successfully assembled at K steps and N different angles, and finally select this average value as the upper limit of the assembly completion degree evaluation value; The calculation formula is as shown in formula (3): Finally, the evaluation criteria for the assembly completion degree are shown in formula (4), where is the relative error during the assembly process, is the relative error obtained from the preliminary experiment. If , the UI interface displays "completed". If , the UI interface displays "uncompleted"; 2. The method according to claim 1, characterized in that, The specific content of Step 1 includes: First, obtain the assembly step instructions for the object to be assembled. Then use SolidWorks software to perform proportional 3D modeling on the assembly model. Import the created STEP file format model file into 3DMax software for mapping and rendering. Then save the processed model file as an FBX format file. The obtained FBX model first needs to be imported into Python to call the Open3D library to convert it into a PCD format point cloud model and save it to the database for easy access. Then it needs to be imported into Unity to produce assembly animation and step information. The production of Unity3D assembly animation uses the engine's built-in Animation system. First, select the initial state model of the object to be assembled, and then add the animation information of the parts assembly based on the initial state model. The parts assembly process must comply with the step information in the assembly manual. It is necessary to add Rigidbody to the sub-objects of the assembly object, cancel the UseGravity option, select the Is Kinematic option, and check the Loop Time option for the Animation object so that the assembly object animation can serve as a real-time reminder during the assembly process.

3. The method according to claim 1, wherein The second step specifically includes: Before proceeding to step 2, you need to enable HoloLens 2's Researcher Mode. Researcher Mode is an application used by HoloLens to access key sensors. HoloLens 2 adds data access to visible light environment tracking cameras, depth cameras, etc. on the basis of HoloLens 1. After turning on Researcher Mode, depth data is obtained through the HoloLens 2 depth camera and transmitted to the PC for processing. The processing of depth data adopts the camera calibration principle, that is, the process of obtaining the final projection matrix P from the world coordinate system to the camera coordinate system and then to the image coordinate system; the process of obtaining the external parameter matrix from the world coordinate system to the camera coordinate system is obtained by the PrintDepthExtrinsics class in HL2ResearchMode, and the process of obtaining the internal parameter matrix from the camera coordinate system to the image coordinate system is obtained by the Zhang Zhengyou checkerboard calibration method. Finally, the three-dimensional point cloud is obtained through the matrix operation of formula (1); where u and v are arbitrary coordinate points in the image coordinate system, u0 and v0 are the central coordinates of the image respectively, x w , y w , z w represent the three-dimensional coordinate points in the world coordinate system; z c represents the z-axis value of the camera coordinate, that is, the distance from the target to the camera; R and T are the 3x3 rotation matrix and the 3x1 translation matrix in the external parameter matrix respectively.

4. The method according to claim 1, characterized in that, The step three specifically includes: First, pre-process the target point cloud obtained in step 2. Since there are walls and debris around the object to be assembled, the target point cloud needs to be cut. Here, the selectionpolygonVolume class in the Open3D library in Python is used to crop the useful area in the point cloud to obtain the target point cloud; then, the processed target point cloud and the source point cloud need to be ICP-registered using the registration_icp class in the Open3D library, and the pose transformation matrix from the source point cloud to the target point cloud is output; Send the pose transformation matrix to HoloLens 2 via TCP / IP communication. HoloLens 2 calls the relevant model and uses the pose transformation matrix to transform its position in its world coordinate system to achieve the effect of virtual-real fusion. Since the virtual model will shift under camera jitter, a spatial anchor is added to the fused virtual model. The spatial anchor in HoloLens 2 is a method that can keep an object in a specific position and rotation state. The virtual model is fixed to the physical model through the spatial anchor technology, which facilitates the subsequent assembly animation demonstration.

Images