MR equipment virtual-real fusion method based on point cloud registration

Abstract

The invention relates to an MR equipment virtual-real fusion method based on point cloud registration, belongs to the technical field of mixed reality, and solves the problems of long time consumption and insufficient alignment precision of existing virtual-real fusion. Comprising the following steps: constructing a real-time updated point cloud map, and loading a historical point cloud map at the same time; registering the current real-time updated point cloud map with a historical point cloud map, and if registration succeeds, fusing the historical point cloud map in the current real-time updated point cloud map to obtain a to-be-registered point cloud map; otherwise, continuing to update the current real-time updated point cloud map to obtain a to-be-registered point cloud map; acquiring a model point cloud according to the three-dimensional grid model file of the real training equipment, registering the model point cloud with a to-be-registered point cloud map, and obtaining a pose of the model point cloud in a world coordinate system after successful registration; and correcting the pose of the MR equipment at each moment according to the pose so as to render a training application of virtual-real alignment. And rapid and efficient virtual-real fusion is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of mixed reality, and in particular to a method for virtual-real fusion of an MR device based on point cloud registration. Background Art

[0002] MR (Mixed Reality) technology provides users with a highly immersive experience by seamlessly integrating the virtual world with the real world. It can not only superimpose virtual objects onto real scenes but also enable real-time interaction between the virtual and the real.

[0003] Manual alignment is a commonly used method for virtual-real fusion in MR devices. It mainly adjusts the position and orientation of virtual objects manually to align them with objects in the real world. The advantage of this method is simplicity and directness, but it has high operation difficulty, low efficiency, high requirements for the operator's experience, and it is difficult to achieve precise alignment in complex three-dimensional scenes. There are also some semi-automatic alignment methods that still require manual intervention to complete the final adjustment, and the operation is cumbersome.

[0004] Moreover, existing methods either rely on historical point cloud maps or construct point cloud maps in real time, without considering the fusion of the two, and are easily affected by environmental dynamic changes, unable to improve the efficiency of virtual-real fusion. Summary of the Invention

[0005] In view of the above analysis, embodiments of the present invention aim to provide a method for virtual-real fusion of an MR device based on point cloud registration to solve the problems of long time consumption and insufficient alignment accuracy in existing virtual-real fusion.

[0006] Embodiments of the present invention provide a method for virtual-real fusion of an MR device based on point cloud registration, including the following steps: Construct a real-time updated point cloud map according to the collected environmental data, and at the same time load the historical point cloud map; register the currently real-time updated point cloud map with the historical point cloud map. If the registration is successful, fuse the historical point cloud map in the currently real-time updated point cloud map to obtain a point cloud map to be registered; otherwise, continue to update the currently real-time updated point cloud map to obtain a point cloud map to be registered; Obtain model point clouds according to the three-dimensional mesh model file of the real training device, register the model point clouds with the point cloud map to be registered, and obtain the pose of the model point clouds in the world coordinate system after successful registration; Correct the poses of the MR device at each moment according to the pose of the model point clouds in the world coordinate system, obtain the poses of the MR device at each moment in the model coordinate system where the model point clouds are located, and then render a virtual-real aligned training application.

[0007] Based on further improvements to the above method, the historical point cloud map is the latest real-time updated point cloud map saved when the MR device was shut down last time; the method further includes: calculating the most probable position of the MR device in the historical point cloud map; the most probable position is obtained by regularly collecting the position points of the MR device at non-stationary moments during the use of the MR device to form a set of position points; dividing the space of the set of position points into voxels, and calculating the average position of all position points in the top N voxels with the largest number of points in the set, where N > 1; the non-stationary moment is when both the position difference and the attitude difference in the poses of the MR device at adjacent moments are greater than the corresponding thresholds.

[0008] Based on further improvements to the above method, registering the current real-time updated point cloud map with the historical point cloud map includes: According to the most probable position of the MR device in the historical point cloud map and the current pose of the MR device, constructing multiple first poses of the MR device in the historical point cloud map; according to each first pose and the current pose of the MR device, obtaining multiple second poses of the current real-time updated point cloud map relative to the historical point cloud map; Based on each second pose respectively, registering the current real-time updated point cloud map with the historical point cloud map to obtain each third pose and the first registration score; if the largest first registration score is not less than the score threshold, the registration is successful, otherwise, the registration fails.

[0009] Based on further improvements to the above method, constructing multiple first poses of the MR device in the historical point cloud map according to the most probable position of the MR device in the historical point cloud map and the current pose of the MR device includes: Taking the pitch angle and roll angle in the current pose of the MR device as the pitch angle and roll angle of the MR device in the historical point cloud map; and, setting the heading angle to increase from 0 at interval angles to obtain multiple first rotation matrices; obtaining the first translation vector according to the most probable position of the MR device in the historical point cloud map; Combining the multiple first rotation matrices with the first translation vector respectively to form multiple first transformation matrices to obtain multiple first poses.

[0010] Based on further improvements to the above method, the multiple second poses are calculated by the following formula: , where, represents the k-th second pose, represents the k-th first pose; represents the current pose of the MR device.

[0011] Based on the further improvement of the above method, the point cloud map to be registered is obtained by fusing the historical point cloud map with the currently real-time updated point cloud map. The historical point cloud map is transformed according to the third pose corresponding to the maximum first registration score. The transformed historical point cloud map is matched with the currently real-time updated point cloud map using a point cloud registration algorithm. The unmatched historical point cloud is added to the currently real-time updated point cloud map, and after voxel filtering, the fused point cloud map is obtained as the point cloud map to be registered.

[0012] Based on the further improvement of the above method, the model point cloud is registered with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system, including: Segment the point cloud map to be registered, and obtain the sub-region with the smallest distance from the MR device as the target point cloud; Take the model point cloud as the source point cloud, and construct multiple initial poses for the source point cloud in the world coordinate system; Based on each initial pose respectively, register the source point cloud with the target point cloud respectively to obtain each optimized pose and the second registration score; if the maximum second registration score is not less than the score threshold, the corresponding optimized pose is the pose of the model point cloud in the world coordinate system; otherwise, update the point cloud map to be registered according to the latest real-time point cloud map, and register the model point cloud with the point cloud map to be registered again until the maximum second registration score is not less than the score threshold.

[0013] Based on the further improvement of the above method, segment the point cloud map to be registered, and obtain the sub-region with the smallest distance from the MR device as the target point cloud, including: Segment the point cloud map to be registered according to the volume of the model point cloud and the minimum volume threshold, calculate the volume of each sub-region and compare it with the volume of the model point cloud, and retain the sub-regions within the error range of the volume of the model point cloud as candidate regions; calculate the distance between each candidate region and the current MR device, and select the candidate region corresponding to the minimum distance as the target point cloud.

[0014] Based on the further improvement of the above method, construct multiple initial poses for the source point cloud in the world coordinate system, including: incrementally set the heading angle from 0 at intervals, and set the pitch angle and roll angle to 0 degrees to obtain multiple second rotation matrices; obtain the second translation vector according to the position of the target point cloud; combine multiple second rotation matrices with the second translation vector respectively to form multiple second transformation matrices to obtain multiple initial poses.

[0015] Based on the further improvement of the above method, correct the pose of the MR device at each moment according to the pose of the model point cloud in the world coordinate system to obtain the pose of the MR device at each moment in the model coordinate system where the model point cloud is located. The formula is as follows: , Among them, represents the pose of the MR device in the model coordinate system at time t, represents the pose of the MR device at time t, represents the pose of the model point cloud in the world coordinate system.

[0016] Compared with the prior art, the present invention can at least achieve one of the following beneficial effects: 1. According to the registration result of the real-time updated point cloud map and the historical point cloud map, reasonably utilize the historical point cloud map to supplement the global environmental information of the real-time updated point cloud map, optimize the real-time mapping process, reduce the consumption of computing resources and time, facilitate quickly entering the virtual-real fusion registration process, and improve the virtual-real fusion efficiency.

[0017] 2. Through operations such as registration and coordinate system transformation of the point cloud of the three-dimensional mesh model of the real training device and the real-time point cloud map, adjust the pose of the MR device in real time, avoid alignment deviation caused by device movement, achieve fully automatic and high-precision virtual-real fusion, reduce operation steps and time, and enable users to enter the immersive experience more quickly.

[0018] 3. By constructing initial poses in multiple initial orientations to cover various possible poses of the point cloud in space, thereby increasing the possibility of finding the correct match, effectively improving the registration success rate, accuracy, and robustness, especially suitable for applications in complex scenarios and with high-precision requirements.

[0019] In the present invention, the above technical solutions can also be combined with each other to achieve more preferred combination schemes. Other features and advantages of the present invention will be described in the subsequent specification, and some advantages can be made obvious from the specification, or understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained through the content specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings are only for the purpose of showing specific embodiments and are not considered as limiting the present invention. Throughout the drawings, the same reference signs denote the same components; Figure 1 is a flowchart of a method for virtual-real fusion of an MR device based on point cloud registration in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The following will specifically describe the preferred embodiments of the present invention with reference to the drawings, where the drawings form a part of the present application and are used together with the embodiments of the present invention to explain the principles of the present invention, rather than to limit the scope of the present invention.

[0022] A specific embodiment of the present invention discloses a virtual-real fusion method for an MR device based on point cloud registration, as Figure 1 shown, which includes the following steps: S1. Construct a real-time updated point cloud map based on the collected environmental data, and at the same time load the historical point cloud map; register the currently real-time updated point cloud map with the historical point cloud map. If the registration is successful, fuse the historical point cloud map in the currently real-time updated point cloud map to obtain the point cloud map to be registered; otherwise, continue to update the currently real-time updated point cloud map to obtain the point cloud map to be registered.

[0023] It should be noted that the MR device in this embodiment is a head-mounted display device, including: 1 binocular color camera, 2 binocular black-and-white cameras, and an IMU (Inertial Measurement Unit) sensor. The binocular color camera is used to capture real-scene images, and the binocular black-and-white cameras and the IMU are used to track the MR device in real time and calculate the pose of the MR device in the world coordinate system.

[0024] A training application is deployed in the MR device to provide a virtual-real fusion scenario for the user to conduct simulation training. Exemplarily, the training application is a driver training application. In the real scene (real world), there is a real cockpit training device, and the training application renders a virtual cockpit training device and the external scene of the cockpit in the virtual scene (virtual world); after the virtual and real are aligned, through the MR device, a scene where the real cockpit is fused with the virtual external scene of the cockpit can be seen, and various operations of the operator in the real cockpit can be reflected in the driver training application.

[0025] In this embodiment, the environmental data where the real training device is located is collected by wearing the MR device. An SLAM algorithm is deployed in the MR device to output the six-degree-of-freedom (Six Degrees of Freedom, abbreviated as 6DOF) pose data at the exposure moment of each frame of image, and the pose of the MR device is obtained, that is, the pose in the world coordinate system of SLAM. The SLAM algorithm itself will also build a point cloud map, but for the efficient operation of the MR device, only a sparse point cloud map is retained, and the geometric structure of the real training device cannot be displayed.

[0026] Therefore, in this step, based on multiple frames of environmental images collected by the binocular black-and-white camera, a real-time mapping algorithm is used to extract and match feature points (such as corner points, SIFT features, etc.) for each frame of image; after feature triangulation, a point cloud set in the camera coordinate system is restored; then, using the 6DOF pose of the MR device at the exposure moment of each frame of image and the external reference pose of the camera and the MR device, the point cloud set is transformed into the world coordinate system of SLAM. This process is repeated continuously in time to construct a real-time updated point cloud map, which is a dense point cloud map in the world coordinate system of SLAM and is used to reflect the geometric information of the real training device.

[0027] It should be noted that the number of points in the real-time updated point cloud map needs to reach a certain level before the next point cloud registration can be carried out. In order to obtain the point cloud map to be registered (i.e., the point cloud map that can be used for registration) more quickly and achieve virtual-real alignment quickly, in this embodiment, after the MR device is powered on, while performing the above operations to construct the real-time updated point cloud map, it is identified whether there is a historical point cloud map. If there is, the historical point cloud map is loaded, and the current real-time updated point cloud map is registered with the historical point cloud map. If the registration is successful, the historical point cloud map is fused into the current real-time updated point cloud map to obtain the point cloud map to be registered; otherwise, the historical point cloud map is discarded, and the current real-time updated point cloud map is continued to be updated to obtain the point cloud map to be registered.

[0028] Among them, the historical point cloud map is the latest real-time updated point cloud map saved when the MR device was powered off last time; moreover, the method of this embodiment further includes: calculating the most probable position of the MR device in the historical point cloud map.

[0029] Specifically, the most probable position of the MR device in the historical point cloud map is obtained by regularly collecting the position points of the MR device at non-static moments during the use of the MR device to form a position point set; dividing the space of the position point set into voxels, and calculating the average position of all position points in the top N largest voxels of the point set, where N > 1; the non-static moment is when both the position difference and the attitude difference in the poses of the MR device at adjacent moments are greater than the corresponding thresholds.

[0030] Exemplarily, the position points of the MR device are collected once every 3 seconds, the size of the position point set is always maintained at 1200 points, N is set to 3, and the top 3 largest voxels of the point set are taken.

[0031] Furthermore, registering the current real-time updated point cloud map with the historical point cloud map includes: ① Based on the most probable position of the MR device in the historical point cloud map and the current pose of the MR device, multiple first poses of the MR device in the historical point cloud map are constructed; based on each first pose and the current pose of the MR device, multiple second poses of the current real-time updated point cloud map relative to the historical point cloud map are obtained.

[0032] Specifically, based on the most probable position of the MR device in the historical point cloud map and the current pose of the MR device, multiple first poses of the MR device in the historical point cloud map are constructed, including: Taking the pitch angle and roll angle in the current pose of the MR device as the pitch angle and roll angle of the MR device in the historical point cloud map; and, setting the heading angle to increase from 0 at intervals of an angle, thereby obtaining multiple first rotation matrices; obtaining the first translation vector based on the most probable position of the MR device in the historical point cloud map; Combining multiple first rotation matrices with the first translation vector respectively to form multiple first transformation matrices to obtain multiple first poses.

[0033] Exemplarily, the heading angle is set to increase at intervals of 10 degrees within the range of [0, 360] degrees, and then combined with the pitch angle and roll angle to obtain 36 first rotation matrices.

[0034] Further, multiple second poses are calculated through the following formula: Formula (1), where, represents the k-th second pose, represents the k-th first pose; represents the current pose of the MR device.

[0035] ② Based on each second pose respectively, the current real-time updated point cloud map is registered with the historical point cloud map to obtain each third pose and the first registration score; if the maximum first registration score is not less than the score threshold, the registration is successful, otherwise, the registration fails.

[0036] It should be noted that when performing registration, the current real-time updated point cloud map is used as the source point cloud, each second pose is used as the initial pose of the source point cloud respectively, and the historical point cloud map is used as the target point cloud. Since there are multiple second poses, in this embodiment, multi-threading is used, and in each thread, the weighted ICP algorithm is used to iteratively optimize the corresponding second pose, so that the position of the source point cloud in the target point cloud reaches the optimal to obtain the corresponding third pose (i.e., the optimized second pose) and the first registration score, realizing the simultaneous correction of multiple initial poses and improving the calculation efficiency.

[0037] Considering that during the registration process, matching point pairs with consistent normal directions are more likely to belong to the same physical surface, and their corresponding relationships are more reliable; point pair matching is more stable in flat areas (i.e., low-curvature areas) and should be given higher weights; in high-curvature areas (such as edges), the weights should be reduced to reduce noise interference. Therefore, in this embodiment, different dynamic weights are assigned to point pairs according to the normals and curvatures of the point pairs, so that the registration accuracy of point pairs with consistent normals is higher, and the Huber error function is used to reduce the influence of outliers, thereby improving the overall registration effect. That is, the weighted ICP algorithm in this embodiment adopts a dynamic weight adjustment mechanism and combines the Huber error function to construct an objective function, takes the objective function as the optimization target, and minimizes the registration error between point clouds through iterative solution.

[0038] Specifically, set as the three-dimensional coordinates of the i-th point in the source point cloud, is the three-dimensional coordinates of the nearest point corresponding to in the target point cloud. The normal and curvature of the point pair are calculated through the following formula to obtain the dynamic weight of the point pair : Formula (2), where, represents the dynamic weight of the i-th point pair , represents the proportionality coefficient, represents the normal weight, represents the curvature weight, represents the normal angle of the point pair , represents the adjustment parameter; represents the curvature calculated according to the neighborhood point set of in the source point cloud, represents the proportionality coefficient.

[0039] Furthermore, the objective function obtained by combining the weight with the Huber error function is as follows: Formula (3), where, is the pose to be solved, represents the rotation matrix, represents the translation vector, represents the Huber error function, n represents the number of point pairs, represents the L2 norm.

[0040] Finally, according to the objective function of formula (3), the third pose after the registration of each source point cloud and the target point cloud is solved, and the first registration score is calculated. The better the registration effect, the higher the first registration score.

[0041] Exemplarily, the registration score is obtained by calculating the average value or sum of the registration probabilities of all point pairs, or by calculating the reciprocal of the mean squared error of all point pairs, etc.

[0042] If the maximum first registration score is not less than or equal to the score threshold, the registration is successful, and the point cloud map to be registered is quickly constructed using the historical point cloud map, that is: the historical point cloud map is transformed according to the third pose corresponding to the maximum first registration score, and the transformed historical point cloud map is matched with the currently real-time updated point cloud map using the point cloud registration algorithm. The unmatched historical point cloud is added to the currently real-time updated point cloud map, and after voxel filtering processing, the fused point cloud map is obtained as the point cloud map to be registered.

[0043] If the maximum first registration score is less than the score threshold, the registration fails, the historical point cloud map is no longer used, and the currently real-time updated point cloud map is continuously updated to obtain the point cloud map to be registered.

[0044] Specifically, the currently real-time updated point cloud map is used as the global point cloud map. After identifying the key frames for each subsequent real-time collected image, the feature points on the key frames are converted to the world coordinates to obtain a new point cloud. The new point cloud is matched with the point cloud map using the point cloud registration algorithm, the unmatched new point cloud is added to the global point cloud map, and voxel filtering processing is performed on the updated global point cloud map to ensure that only one point coordinate is retained for the same spatial point in a volume space of a certain scale.

[0045] Among them, identifying whether the current frame is a key frame is based on the pose data of the MR device in the current frame and the previous frame. The position difference in the pose data is compared with the distance threshold, and the attitude difference is compared with the attitude threshold. If both are greater than the corresponding thresholds, the current frame is a key frame.

[0046] It should be noted that to identify whether the attitude difference is greater than the attitude threshold, the rotation in the pose data is converted into the corresponding Lie algebra vector, and the modulus of the Lie algebra vector is calculated and compared with the attitude threshold.

[0047] Preferably, in order to make the real-time point cloud map more obviously reflect the geometric information of the real training device, the surface of the real training device is textured, so as to better extract the feature points and thus restore the 3D point cloud of the real training device.

[0048] With the addition of the point clouds of subsequent key frames, the real-time updated point cloud map is continuously updated and improved. When the number of point clouds in the real-time updated point cloud map is greater than the point cloud threshold, it is used as the point cloud map to be registered, and the next point cloud registration operation is performed. At the same time, the point cloud map is continuously updated in real time.

[0049] S2. Obtain the model point cloud according to the three-dimensional mesh model file of the real training device, register the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system.

[0050] It should be noted that the three-dimensional mesh model file of the real training device is imported into the MR device. The three-dimensional mesh model file is usually obtained by modeling and meshing the real training device using three-dimensional modeling software, including: vertex data, color data, and triangular mesh data of the model. Exemplarily, the three-dimensional modeling software includes: Maya and Blender.

[0051] Furthermore, using the PCL library (an open-source library dedicated to point cloud processing), the three-dimensional mesh model file is used as input, and the model point cloud is generated by sampling. Exemplarily, the pcl_mesh_sampling function in the PCL library is called to generate the model point cloud.

[0052] It should be noted that the vertex coordinates of the model are all set relative to the origin of the three-dimensional mesh model. Therefore, the origin of the coordinate system of the generated model point cloud is the same as the origin in the three-dimensional mesh model. Based on this origin, the model coordinate system where the model point cloud is located is constructed using the right-hand rule. Exemplarily, according to the orientation of the three-dimensional mesh model, the right, upper, and rear directions of the model are respectively the x, y, and z axis directions of the coordinate system.

[0053] Furthermore, considering that there may be multiple training devices in the real-time point cloud map, and if the initial pose of the registration is not good, it will lead to high registration error or non-convergence. Therefore, in this embodiment, the training device closest to the MR device is obtained by segmenting the point cloud map to be registered, and multiple initial poses are constructed to facilitate finding the best initial pose.

[0054] Specifically, registering the model point cloud with the point cloud map to be registered and obtaining the pose of the model point cloud in the world coordinate system includes: ① Segment the point cloud map to be registered, and obtain the sub-region with the minimum distance from the MR device as the target point cloud.

[0055] Segment the point cloud map to be registered according to the volume of the model point cloud and the minimum volume threshold, calculate the volume of each sub-region and compare it with the volume of the model point cloud, and retain the sub-regions within the error range of the volume of the model point cloud as candidate regions; calculate the distance between each candidate region and the current MR device, and select the candidate region corresponding to the minimum distance as the target point cloud.

[0056] It should be noted that the volume of the model point cloud is the volume of the space enclosed by the model point cloud, which is obtained by calculating the volume of the convex hull of the model point cloud. There are many methods for segmenting the point cloud map to be registered, and this embodiment does not limit the segmentation method. Preferably, the Euclidean clustering method is used and the neighborhood query is accelerated by constructing a KD-Tree or Octree to segment the point cloud map to be registered to obtain multiple sub-regions. The volume of each sub-region is obtained by obtaining the convex hull of each sub-region. If the volumes of adjacent sub-regions are both less than the minimum volume threshold, they are merged into a new sub-region and the volume is recalculated; if the volume of the sub-region is within the error range of the volume of the model point cloud, for example, within ±20% of the volume of the model point cloud, then this sub-region is used as a candidate region.

[0057] Furthermore, the position of each candidate region is the centroid position of each candidate region, which is obtained by calculating the average position of all the point clouds in the candidate region. The position of the current MR device is obtained according to the 6DOF pose of the MR device. Calculate the Euclidean distance between each candidate region and the current MR device, and select the candidate region corresponding to the minimum distance as the target point cloud, that is, the model point cloud needs to be registered with the target point cloud.

[0058] ② Use the model point cloud as the source point cloud and construct multiple initial poses for the source point cloud in the world coordinate system.

[0059] Based on the world coordinate system, incrementally set the heading angle from 0 at intervals of an angle, and set the pitch angle and roll angle to 0 degrees to obtain multiple second rotation matrices; obtain the second translation vector according to the position of the target point cloud; combine the multiple second rotation matrices with the second translation vector respectively to form multiple second transformation matrices to obtain multiple initial poses.

[0060] Specifically, for each heading angle, construct a 4×4 second transformation matrix containing the second rotation matrix and the second translation vector to obtain the initial pose.

[0061] Exemplarily, incrementally set the heading angle at intervals of 10 degrees in the range of [0, 360] degrees to obtain 36 rotation matrices around the axis perpendicular to the ground.

[0062] ③Based on each initial pose respectively, register the source point cloud with the target point cloud to obtain multiple optimized poses and second registration scores; if the maximum second registration score is not less than the score threshold, the corresponding optimized pose is the pose of the model point cloud in the world coordinate system, otherwise, update the point cloud map to be registered according to the latest real-time point cloud map, and register the model point cloud with the updated point cloud map to be registered again until the maximum second registration score is not less than the score threshold.

[0063] It should be noted that since there are multiple initial poses, the point cloud registration method in step S1 is adopted in this embodiment, that is, using multi-threading, and in each thread, the weighted ICP algorithm is used to iteratively optimize the corresponding initial pose, so that the position of the source point cloud in the target point cloud reaches the optimal, and the corresponding optimized pose (i.e., the optimized initial pose) and the second registration score are obtained.

[0064] When the maximum second registration score is not less than the score threshold, it indicates successful registration, and the corresponding optimized pose is the pose of the model point cloud in the world coordinate system of SLAM. At this time, obtain the coordinates of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score as the model registration position.

[0065] In order to avoid the problem of incorrect alignment in subsequent use, resulting in the user not being able to see the fused visual scene on the training device, regularly detect whether it is necessary to register the model point cloud with the target point cloud again according to the distance between the position of the MR device and the model registration position, and the minimum distance between the position of the MR device and the candidate area.

[0066] Specifically, if the distance between the position of the MR device and the model registration position is less than the first distance threshold, there is no need to perform point cloud registration again. If the distance between the position of the MR device and the model registration position is greater than the second distance threshold, and the minimum distance between the position of the MR device and the candidate area of the non-target point cloud is less than the first distance threshold, then update the candidate area corresponding to the minimum distance to the target point cloud, and register the model point cloud with the target point cloud again.

[0067] Preferably, it is detected once every 5 seconds, the first distance threshold is 1 meter, and the second distance threshold is 2 meters.

[0068] S3. Correct the poses of the MR device at each moment according to the pose of the model point cloud in the world coordinate system to obtain the poses of the MR device at each moment in the model coordinate system where the model point cloud is located, and then render the training application with virtual-real alignment.

[0069] It should be noted that the poses of the MR device at each moment in the model coordinate system where the model point cloud is located are calculated through the following formula: Formula (4), Among them, represents the pose of the MR device in the model coordinate system at time t, represents the pose of the MR device at time t, that is, the 6DOF pose in the world coordinate system of SLAM, represents the pose of the model point cloud in the world coordinate system of SLAM.

[0070] Rendering the training application with virtual-real alignment according to the pose of the MR device in the model coordinate system at each moment is obtained by superimposing multiple layers.

[0071] It should be noted that the order of the multiple layers from bottom to top is: virtual layer, perspective window layer, and real layer; among them, the virtual layer is an image rendered by the training application deployed in the MR device according to the pose of the MR device in the model coordinate system at each moment; the perspective window layer is the contour area of the real training device, and a region rendered according to the three-dimensional meshed model file of the real training device, which is used to cover the virtual training device in the virtual layer; the real layer is used to display the real image captured by the MR device in the perspective window. Virtual-real alignment is the effect produced by aligning the image of the real layer and the virtual image based on the alignment of the training devices with the same size and appearance in the two spaces.

[0072] Specifically, take as the view matrix for rendering the virtual training device, so that the relative pose of the MR device and the real training device in the real space is the same as the relative pose of the models of the MR device and the real training device in the virtual space. Through the conventional rendering process, the perspective window layer and the real layer are superimposed, so that only the real image in the contour coverage area of the real training device is retained. At this time, in the vision of the training personnel, the contours of the virtual training device and the real training device reach the overlapping effect. At the same time, transmit to the training application. After the training application renders the virtual image and combines it with the real image in the contour coverage area, the virtual-real alignment effect can be achieved, so that when the training personnel look at the training device through the MR device, they see the real image in the real space, and when they look outside the training device, they see the virtual training task image.

[0073] In this embodiment, steps S1 - S3 are a process of automatically performing virtual-real alignment, and usually no further adjustment by the user is required. However, considering the differences in human senses, when the user is not satisfied with the virtual-real alignment effect rendered in step S3, fine-tuning alignment is performed, including: adjusting through 3 translation positions (in meters) around the x, y, and z axes and 3 rotation Euler angles (in degrees) on the user operation panel, calculating the corresponding offset pose matrix to correct the pose of the MR device in the model coordinate system obtained by formula (4), and rendering the interface with the fine-tuned virtual-real alignment effect.

[0074] Compared with the prior art, a method for virtual-real fusion of an MR device based on point cloud registration provided in this embodiment rationally utilizes the historical point cloud map to supplement the global environmental information of the real-time updated point cloud map according to the registration result of the real-time updated point cloud map and the historical point cloud map, optimizes the process of real-time mapping, reduces the consumption of computing resources and time, facilitates quickly entering the registration process of virtual-real fusion, and improves the efficiency of virtual-real fusion. Through operations such as registration and coordinate system transformation of the point cloud of the three-dimensional mesh model of the real training device with the real-time point cloud map, the pose of the MR device is adjusted in real time, avoiding the alignment deviation caused by the movement of the device, realizing full-automatic and high-precision virtual-real fusion, reducing the operation steps and time, and enabling users to enter the immersive experience more quickly. By constructing initial poses in multiple initial orientations to cover various possible poses of the point cloud in space, the possibility of finding the correct match is increased, effectively improving the registration success rate, accuracy, and robustness, especially suitable for applications in complex scenarios and with high-precision requirements.

[0075] Those skilled in the art can understand that all or part of the processes for implementing the methods of the above embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium is a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0076] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A virtual-real fusion method for MR devices based on point cloud registration, characterized in that Including the following steps: Construct a real-time updated point cloud map based on the collected environmental data, and at the same time load the historical point cloud map; register the currently real-time updated point cloud map with the historical point cloud map. If the registration is successful, fuse the historical point cloud map in the currently real-time updated point cloud map to obtain the point cloud map to be registered; otherwise, continue to update the currently real-time updated point cloud map to obtain the point cloud map to be registered; Obtain the model point cloud according to the three-dimensional mesh model file of the real training device, register the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration; Correct the poses of the MR device at each moment according to the pose of the model point cloud in the world coordinate system, obtain the poses of the MR device at each moment in the model coordinate system where the model point cloud is located, and then render a training application with virtual-real alignment.

2. The method for virtual-real fusion of MR equipment based on point cloud registration according to claim 1, wherein, The historical point cloud map is the latest real-time updated point cloud map saved when the MR device was shut down last time; the method further includes: calculating the most probable position of the MR device in the historical point cloud map; the most probable position is obtained by regularly collecting the position points of the MR device at non-static moments during the use of the MR device to form a position point set; dividing the space of the position point set into voxels, and calculating the average position of all position points in the top N voxels with the largest point set, where N>1; the non-static moment is when both the position difference and the attitude difference in the poses of the MR device at adjacent moments are greater than the corresponding thresholds.

3. The method for virtual-real fusion of an MR device based on point cloud registration according to claim 2, wherein, The registration of the currently real-time updated point cloud map with the historical point cloud map includes: Construct multiple first poses of the MR device in the historical point cloud map according to the most probable position of the MR device in the historical point cloud map and the current pose of the MR device; obtain multiple second poses of the currently real-time updated point cloud map relative to the historical point cloud map according to each first pose and the current pose of the MR device; Based on each second pose respectively, register the currently real-time updated point cloud map with the historical point cloud map to obtain each third pose and the first registration score; if the largest first registration score is not less than the score threshold, the registration is successful, otherwise, the registration fails.

4. The method for virtual-real fusion of an MR device based on point cloud registration according to claim 3, wherein The construction of multiple first poses of the MR device in the historical point cloud map according to the most probable position of the MR device in the historical point cloud map and the current pose of the MR device includes: Take the pitch angle and roll angle in the current pose of the MR device as the pitch angle and roll angle of the MR device in the historical point cloud map; and set the heading angle to increase from 0 at interval angles to obtain multiple first rotation matrices; obtain the first translation vector according to the most probable position of the MR device in the historical point cloud map; Combine multiple first rotation matrices with the first translation vector respectively to form multiple first transformation matrices to obtain multiple first poses.

5. The method for virtual-real fusion of an MR device based on point cloud registration according to claim 3, wherein, The multiple second poses are calculated by the following formula: ， Among them, represents the k-th second pose, represents the k-th first pose; represents the current pose of the MR device.

6. The method for virtual-real fusion of an MR device based on point cloud registration according to claim 3, wherein The point cloud map to be registered is obtained by fusing the historical point cloud map with the currently updated real-time point cloud map. Specifically, the historical point cloud map is transformed according to the third pose corresponding to the maximum first registration score, and the transformed historical point cloud map is matched with the currently updated real-time point cloud map using a point cloud registration algorithm. The unmatched historical point clouds are added to the currently updated real-time point cloud map, and after voxel filtering, the fused point cloud map is obtained as the point cloud map to be registered.

7. The method for virtual-real fusion of an MR device based on point cloud registration according to any one of claims 1-6, characterized in that Registering the model point cloud with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system includes: Segmenting the point cloud map to be registered to obtain the sub-region with the minimum distance to the MR device as the target point cloud; Taking the model point cloud as the source point cloud and constructing multiple initial poses for the source point cloud in the world coordinate system; Based on each initial pose, registering the source point cloud with the target point cloud respectively to obtain each optimized pose and the second registration score; if the maximum second registration score is not less than the score threshold, the corresponding optimized pose is the pose of the model point cloud in the world coordinate system; otherwise, updating the point cloud map to be registered according to the latest real-time point cloud map, and registering the model point cloud with the updated point cloud map to be registered again until the maximum second registration score is not less than the score threshold.

8. The method for virtual-real fusion of an MR device based on point cloud registration according to claim 7, wherein, Segmenting the point cloud map to be registered to obtain the sub-region with the minimum distance to the MR device as the target point cloud includes: Segmenting the point cloud map to be registered according to the volume of the model point cloud and the minimum volume threshold, calculating the volume of each sub-region and comparing it with the volume of the model point cloud, and retaining the sub-regions within the error range of the volume of the model point cloud as candidate regions; calculating the distance between each candidate region and the current MR device, and selecting the candidate region corresponding to the minimum distance as the target point cloud.

9. The method for virtual-real fusion of an MR device based on point cloud registration according to claim 7, wherein, Constructing multiple initial poses for the source point cloud in the world coordinate system includes: incrementing the heading angle from 0 at intervals, and setting the pitch angle and roll angle to 0 degrees to obtain multiple second rotation matrices; obtaining the second translation vector according to the position of the target point cloud; combining the multiple second rotation matrices with the second translation vector respectively to form multiple second transformation matrices to obtain multiple initial poses.

10. The method for virtual-real fusion of an MR device based on point cloud registration according to claim 1, wherein Correcting the poses of the MR device at each moment according to the pose of the model point cloud in the world coordinate system to obtain the poses of the MR device at each moment in the model coordinate system where the model point cloud is located. The formula is as follows: ， Among them, represents the pose of the MR device in the model coordinate system at time t, represents the pose of the MR device at time t, represents the pose of the model point cloud in the world coordinate system.

Images