A virtual-reality fusion method for MR devices based on point cloud registration

Abstract

The present invention relates to a method for fusion of virtual and real MR devices based on point cloud registration, which belongs to the field of mixed reality technology and solves the problem that existing virtual and real fusion is time-consuming and has insufficient alignment accuracy. The method comprises: constructing a point cloud map that is updated in real time, and loading a historical point cloud map at the same time; aligning the current real-time updated point cloud map with the historical point cloud map, and if the registration is successful, fusing the historical point cloud map with the current real-time updated point cloud map to obtain a point cloud map to be registered; otherwise, continuing to update the current real-time updated point cloud map to obtain a point cloud map to be registered; obtaining a model point cloud based on a three-dimensional grid model file of a real training device, aligning the model point cloud with the point cloud map to be registered, and obtaining the pose of the model point cloud in the world coordinate system after successful registration; correcting the pose of the MR device at each moment according to the pose, and then rendering a training application with virtual and real alignment. Fast and efficient virtual and real fusion is achieved.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality technology, and in particular to a method for fusion of virtuality and reality of MR equipment based on point cloud registration. Background Art

[0002] Mixed Reality (MR) technology seamlessly blends the virtual and real worlds, providing users with a highly immersive experience. It not only overlays virtual objects onto real scenes but also enables real-time interaction between the virtual and real worlds.

[0003] Manual alignment is a common method for integrating virtual and real objects in MR devices. It involves manually adjusting the position and posture of virtual objects to align them with real-world objects. While this method is simple and straightforward, it is also difficult and inefficient, requiring a high level of operator experience. Furthermore, precise alignment is difficult in complex 3D scenes. Semi-automatic alignment methods also require manual intervention for final adjustments, making them cumbersome.

[0004] Moreover, existing methods either rely on historical point cloud maps or construct point cloud maps in real time, without considering the fusion of the two. They are easily affected by dynamic changes in the environment and cannot improve the efficiency of virtual-reality fusion. Summary of the Invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a virtual-reality fusion method for an MR device based on point cloud registration, so as to solve the problem that the existing virtual-reality fusion is time-consuming and has insufficient alignment accuracy.

[0006] An embodiment of the present invention provides a method for virtual-reality fusion of an MR device based on point cloud registration, comprising the following steps:

[0007] A real-time updated point cloud map is constructed based on the collected environmental data, and a historical point cloud map is loaded at the same time; the current real-time updated point cloud map is registered with the historical point cloud map. If the registration is successful, the historical point cloud map is integrated with the current real-time updated point cloud map to obtain the point cloud map to be registered; otherwise, the current real-time updated point cloud map is continuously updated to obtain the point cloud map to be registered;

[0008] Obtain the model point cloud based on the 3D mesh model file of the real training device, align the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration;

[0009] The pose of the MR device at each moment is corrected according to the pose of the model point cloud in the world coordinate system, and the pose of the MR device at each moment in the model coordinate system where the model point cloud is located is obtained, and then a virtual-real aligned training application is rendered.

[0010] Based on a further improvement of the above method, the historical point cloud map is the latest real-time updated point cloud map saved when the MR device was last turned off; the method also includes: calculating the maximum probability position of the MR device in the historical point cloud map; the maximum probability position is that during the use of the MR device, the position points of the MR device at non-stationary moments are regularly collected to form a position point set; the space of the position point set is divided into voxels, and the average position of all position points in the top N voxels with the largest point set is calculated, where N>1; the non-stationary moment is when the position difference and posture difference in the MR device posture at adjacent moments are both greater than the corresponding threshold.

[0011] Based on the further improvement of the above method, the current real-time updated point cloud map is aligned with the historical point cloud map, including:

[0012] Based on the maximum probability position of the MR device in the historical point cloud map and the current MR device pose, multiple first poses of the MR device in the historical point cloud map are constructed; based on each first pose and the current MR device pose, multiple second poses of the current real-time updated point cloud map relative to the historical point cloud map are obtained;

[0013] Based on each second pose, the current real-time updated point cloud map is registered with the historical point cloud map to obtain each third pose and the first registration score; if the maximum first registration score is not less than the score threshold, the registration is successful, otherwise, the registration fails.

[0014] Based on the further improvement of the above method, according to the maximum probability position of the MR device in the historical point cloud map and the current MR device pose, multiple first poses of the MR device in the historical point cloud map are constructed, including:

[0015] The pitch angle and roll angle in the current MR device posture are used as the pitch angle and roll angle of the MR device in the historical point cloud map; and the heading angle is set incrementally from 0 at intervals of angle, thereby obtaining multiple first rotation matrices; and a first translation vector is obtained according to the maximum probability position of the MR device in the historical point cloud map;

[0016] The plurality of first rotation matrices are respectively combined with the first translation vector to form a plurality of first transformation matrices to obtain a plurality of first poses.

[0017] Based on the further improvement of the above method, multiple second poses are calculated by the following formula:

[0018] ,

[0019] in, represents the k-th second pose, represents the kth first pose; Indicates the current MR device posture.

[0020] Based on the further improvement of the above method, the historical point cloud map is fused with the current real-time updated point cloud map to obtain the point cloud map to be registered. The historical point cloud map is transformed according to the third pose corresponding to the largest first registration score. The transformed historical point cloud map is matched with the current real-time updated point cloud map using the point cloud registration algorithm. The unmatched historical point cloud is added to the current real-time updated point cloud map. After voxel filtering, the fused point cloud map is obtained as the point cloud map to be registered.

[0021] Based on the further improvement of the above method, the model point cloud is registered with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system, including:

[0022] Segment the point cloud map to be registered and obtain the sub-region with the smallest distance to the MR device as the target point cloud;

[0023] Take the model point cloud as the source point cloud and construct multiple initial poses for the source point cloud in the world coordinate system;

[0024] Based on each initial pose, the source point cloud is registered with the target point cloud respectively to obtain each optimized pose and the second registration score; if the maximum second registration score is not less than the score threshold, the corresponding optimized pose is the pose of the model point cloud in the world coordinate system; otherwise, the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum second registration score is not less than the score threshold.

[0025] Based on the further improvement of the above method, the point cloud map to be registered is segmented, and the sub-region with the smallest distance to the MR device is obtained as the target point cloud, including:

[0026] According to the volume of the model point cloud and the minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-regions within the error range of the volume of the model point cloud are retained as candidate regions; the distance between each candidate region and the current MR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

[0027] Based on a further improvement of the above method, multiple initial poses are constructed for the source point cloud in the world coordinate system, including: setting the heading angle from 0 in increments according to the interval angle, and setting the pitch angle and roll angle to 0 degrees to obtain multiple second rotation matrices; obtaining the second translation vector according to the position of the target point cloud; and combining the multiple second rotation matrices with the second translation vector into multiple second transformation matrices to obtain multiple initial poses.

[0028] Based on the further improvement of the above method, the pose of the MR device at each moment is corrected according to the pose of the model point cloud in the world coordinate system, and the pose of the MR device in the model coordinate system where the model point cloud is located at each moment is obtained. The formula is as follows:

[0029] ,

[0030] in, represents the position of the MR device in the model coordinate system at time t, represents the posture of the MR device at time t, Represents the pose of the model point cloud in the world coordinate system.

[0031] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0032] 1. Based on the registration results of the real-time updated point cloud map and the historical point cloud map, the historical point cloud map is rationally used to supplement the global environmental information of the real-time updated point cloud map, which optimizes the real-time mapping process, reduces the consumption of computing resources and time, facilitates the rapid entry into the virtual-reality fusion registration process, and improves the efficiency of virtual-reality fusion.

[0033] 2. By aligning the point cloud of the 3D mesh model of the real training device with the real-time point cloud map and performing coordinate system transformation, the MR device posture is adjusted in real time to avoid alignment deviation caused by device movement, achieving fully automatic and high-precision virtual-real fusion, reducing operation steps and time, and allowing users to enter the immersive experience more quickly.

[0034] 3. By constructing the initial pose at multiple initial headings, covering multiple possible poses of the point cloud in space, the possibility of finding the correct match is increased, effectively improving the registration success rate, accuracy and robustness, which is especially suitable for complex scenes and applications with high precision requirements.

[0035] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Like reference symbols denote like components throughout the accompanying drawings.

[0037] Figure 1 This is a flow chart of a method for virtual-reality fusion of an MR device based on point cloud registration in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0039] A specific embodiment of the present invention discloses a method for fusion of virtual and real images of MR equipment based on point cloud registration, such as Figure 1 As shown, the following steps are included:

[0040] S1. Build a real-time updated point cloud map based on the collected environmental data, and load the historical point cloud map at the same time; align the current real-time updated point cloud map with the historical point cloud map. If the alignment is successful, fuse the historical point cloud map into the current real-time updated point cloud map to obtain the point cloud map to be aligned; otherwise, continue to update the current real-time updated point cloud map to obtain the point cloud map to be aligned.

[0041] It should be noted that the MR device in this embodiment is a head-mounted display device, including: 1 binocular color camera, 2 binocular black and white cameras and an IMU (Inertial Measurement Unit) sensor. The binocular color camera is used to capture real-scene images, and the binocular black and white camera and IMU are used to track the MR device in real time and calculate the position and posture of the MR device in the world coordinate system.

[0042] The MR device is equipped with a training application that provides users with a blended virtual and real world for simulated training. For example, the training application is a driver training application. A real cockpit training device is placed in the real world. The training application renders a virtual cockpit training device and the exterior of the cockpit in the virtual world. Once the virtual and real worlds are aligned, the MR device displays a fusion of the real cockpit and the exterior. The operator's actions in the real cockpit are reflected in the driver training application.

[0043] In this embodiment, users wear an MR device to collect data about the environment in which the real training device resides. A SLAM algorithm deployed in the MR device outputs six degrees of freedom (6DOF) pose data for each image frame exposure. This data is used to determine the MR device's pose, defined as its position in the SLAM world coordinate system. The SLAM algorithm itself also creates a point cloud map, but for efficient operation, the MR device only retains a sparse point cloud map, which cannot display the geometry of the real training device.

[0044] Therefore, this step uses a real-time mapping algorithm to extract and match feature points (such as corner points, SIFT features, etc.) for each frame of environmental images captured by a binocular black-and-white camera; restores the point cloud set in the camera coordinate system after feature triangulation; then uses the 6DOF posture of the MR device at the exposure moment of each frame of the image, as well as the external parameter posture of the camera and MR device, to convert the point cloud set to the world coordinate system of SLAM. This process is repeated in continuous time to construct a real-time updated point cloud map. This point cloud map is a dense point cloud map in the world coordinate system of SLAM, which is used to reflect the geometric information of the real training device.

[0045] It should be noted that the number of point cloud maps updated in real time must reach a certain level before the next step of point cloud registration can be carried out. In order to obtain the point cloud map to be registered more quickly (that is, the point cloud map that can be used for registration) and quickly realize virtual-real alignment, after the MR device is turned on, this embodiment performs the above operations to construct the real-time updated point cloud map while identifying whether there is a historical point cloud map. If so, the historical point cloud map is loaded, and the current real-time updated point cloud map is registered with the historical point cloud map. If the registration is successful, the historical point cloud map is integrated with the current real-time updated point cloud map to obtain the point cloud map to be registered; otherwise, the historical point cloud map is discarded, and the current real-time updated point cloud map is continued to be updated to obtain the point cloud map to be registered.

[0046] Among them, historical point cloud map It is the latest real-time updated point cloud map saved when the MR device was last shut down; moreover, the method of this embodiment further includes: calculating the maximum probability position of the MR device in the historical point cloud map.

[0047] Specifically, the maximum probability position of the MR device in the historical point cloud map is obtained by regularly collecting the position points of the MR device at non-stationary moments during the use of the MR device to form a position point set; the space of the position point set is divided into voxels, and the average position of all position points in the top N voxels with the largest point set is calculated, where N>1; the non-stationary moment is when the position difference and posture difference in the MR device posture at adjacent moments are both greater than the corresponding threshold.

[0048] For example, the position points of the MR device are collected every 3 seconds, the size of the position point set is always maintained at 1200 points, N is set to 3, and the first 3 voxels with the largest point set are selected.

[0049] Furthermore, the current real-time updated point cloud map is aligned with the historical point cloud map, including:

[0050] ① Based on the maximum probability position of the MR device in the historical point cloud map and the current MR device posture, multiple first postures of the MR device in the historical point cloud map are constructed; based on each first posture and the current MR device posture, multiple second postures of the current real-time updated point cloud map relative to the historical point cloud map are obtained.

[0051] Specifically, based on the maximum probability position of the MR device in the historical point cloud map and the current MR device pose, multiple first poses of the MR device in the historical point cloud map are constructed, including:

[0052] The pitch angle and roll angle in the current MR device posture are used as the pitch angle and roll angle of the MR device in the historical point cloud map; and the heading angle is set incrementally from 0 at intervals of angle, thereby obtaining multiple first rotation matrices; and a first translation vector is obtained according to the maximum probability position of the MR device in the historical point cloud map;

[0053] The plurality of first rotation matrices are respectively combined with the first translation vector to form a plurality of first transformation matrices to obtain a plurality of first poses.

[0054] For example, the heading angle is set in increments of 10 degrees within the range of [0, 360] degrees, and then combined with the pitch angle and the roll angle to obtain 36 first rotation matrices.

[0055] Furthermore, multiple second poses are calculated using the following formula:

[0056] Formula (1),

[0057] in, represents the k-th second pose, represents the kth first pose; Indicates the current MR device posture.

[0058] ② Based on each second pose, the current real-time updated point cloud map is registered with the historical point cloud map to obtain each third pose and the first registration score; if the maximum first registration score is not less than the score threshold, the registration is successful, otherwise, the registration fails.

[0059] It should be noted that when performing registration, the current real-time updated point cloud map is used as the source point cloud, each second pose is used as the initial pose of the source point cloud, and the historical point cloud map is used as the target point cloud. Since there are multiple second poses, this embodiment uses multi-threading, and at the same time uses the weighted ICP algorithm in each thread to iteratively optimize the corresponding second poses, so that the position of the source point cloud in the target point cloud is optimized and the corresponding third pose (i.e., the optimized second pose) and the first registration score are obtained, thereby realizing the simultaneous correction of multiple initial poses and improving the computational efficiency.

[0060] Considering that during the registration process, matching point pairs with consistent normal directions are more likely to belong to the same physical surface, and their correspondence is more reliable; point pair matching is more stable in flat areas (i.e., areas of low curvature) and should be assigned a higher weight; and weights should be lowered in areas of high curvature (such as edges) to reduce noise interference. Therefore, this embodiment assigns different dynamic weights to point pairs based on their normals and curvatures, resulting in higher registration accuracy for point pairs with consistent normals. The Huber error function is also used to reduce the impact of outliers, thereby improving the overall registration effect. Specifically, the weighted ICP algorithm of this embodiment employs a dynamic weight adjustment mechanism and combines it with the Huber error function to construct an objective function. This objective function is used as the optimization goal, and the registration error between point clouds is minimized through iterative solution.

[0061] Specifically, set is the 3D coordinate of the i-th point in the source point cloud, is the target point cloud with The three-dimensional coordinates of the corresponding nearest point are calculated using the following formula The normal and curvature of the point are obtained Dynamic weight of:

[0062] Formula (2),

[0063] in, Represents the i-th point pair The dynamic weight of represents the proportionality coefficient, represents the normal weight, represents the curvature weight, Indicates point pair The normal angle of represents the adjustment parameter; Indicates that according to the source point cloud The curvature calculated from the neighborhood point set of Represents the proportionality factor.

[0064] Furthermore, the objective function obtained by combining the weights with the Huber error function is As shown below:

[0065] Formula (3),

[0066] in, is the pose to be solved, represents the rotation matrix, represents the translation vector, represents the Huber error function, n represents the number of point pairs, represents the L2 norm.

[0067] Finally, the third pose of each source point cloud and target point cloud after registration is solved according to the objective function of formula (3), and the first registration score is calculated. The better the registration effect, the higher the first registration score.

[0068] For example, the registration score is obtained by calculating the average or sum of the registration probabilities of all point pairs, or by calculating the mean square error of all point pairs and then taking the inverse thereof.

[0069] If the maximum first registration score is not less than or equal to the score threshold, the registration is successful, and the historical point cloud map is used to quickly construct the point cloud map to be registered, that is: the historical point cloud map is transformed according to the third pose corresponding to the maximum first registration score, and the transformed historical point cloud map is matched with the current real-time updated point cloud map using the point cloud registration algorithm. The unmatched historical point cloud is added to the current real-time updated point cloud map, and the fused point cloud map is obtained after voxel filtering as the point cloud map to be registered.

[0070] If the maximum first registration score is less than the score threshold, the registration fails, the historical point cloud map is no longer used, and the current real-time updated point cloud map is continuously updated to obtain the point cloud map to be registered.

[0071] Specifically, the current real-time updated point cloud map is used as the global point cloud map. After identifying the key frame for each frame of image subsequently collected in real time, the feature points on the key frame are converted to world coordinates to obtain a new point cloud. The new point cloud is matched with the point cloud map using the point cloud registration algorithm, the unmatched new point cloud is added to the global point cloud map, and the updated global point cloud map is voxel filtered to ensure that only one point coordinate is retained for the same spatial point in a volume space of a certain scale.

[0072] Among them, identifying whether the current frame is a key frame is based on the posture data of the MR device in the current frame and the previous frame, comparing whether the position difference in the posture data is greater than the distance threshold, and whether the posture difference is greater than the posture threshold. If both are greater than the corresponding thresholds, the current frame is a key frame.

[0073] It should be noted that to identify whether the posture difference is greater than the posture threshold, the rotation matrix in the posture data is converted into the corresponding Lie algebraic vector, and the modulus of the Lie algebraic vector is calculated and compared with the posture threshold.

[0074] Preferably, in order to make the real-time point cloud map more obviously reflect the geometric information of the real training device, the surface of the real training device is textured so that feature points can be better extracted, thereby restoring the 3D point cloud of the real training device.

[0075] As the point clouds of subsequent keyframes are added, the real-time updated point cloud map is continuously updated and improved. When the number of point clouds in the real-time updated point cloud map exceeds the point cloud threshold, it is used as the point cloud map to be registered and the next step of point cloud registration is performed. At the same time, the real-time update of the point cloud map continues.

[0076] S2. Obtain a model point cloud based on the 3D mesh model file of the real training device, align the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system.

[0077] It should be noted that the 3D mesh model file of the real training device is imported into the MR device. The 3D mesh model file is typically obtained by modeling the real training device using 3D modeling software, meshing it, and then exporting it. The 3D mesh model file includes the model's vertex data, color data, and triangle mesh data. Exemplary 3D modeling software includes Maya and Blender.

[0078] Furthermore, the PCL library (an open source library specifically for point cloud processing) is used to take the 3D mesh model file as input and generate the model point cloud through sampling. For example, the pcl_mesh_sampling function in the PCL library is called to generate the model point cloud.

[0079] It should be noted that the vertex coordinates of the model are set relative to the origin of the three-dimensional mesh model. Therefore, the origin of the coordinate system of the generated model point cloud is consistent with the origin in the three-dimensional mesh model. Based on this origin, the model coordinate system where the model point cloud is located is constructed using the right-hand rule. For example, according to the orientation of the three-dimensional mesh model, the right, top and back of the model are the x, y and z axis directions of the coordinate system respectively.

[0080] Furthermore, considering that there may be multiple training devices in the real-time point cloud map, and that a poor initial registration pose can result in high registration errors or non-convergence, this embodiment segments the point cloud map to be registered to obtain the training device closest to the MR device, and constructs multiple initial poses to facilitate finding the optimal initial pose.

[0081] Specifically, the model point cloud is registered with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system, including:

[0082] ① Segment the point cloud map to be registered and obtain the sub-region with the smallest distance to the MR device as the target point cloud.

[0083] According to the volume of the model point cloud and the minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-regions within the error range of the volume of the model point cloud are retained as candidate regions; the distance between each candidate region and the current MR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

[0084] It should be noted that the volume of the model point cloud is the volume of the space enclosed by the model point cloud, which is obtained by calculating the convex hull volume of the model point cloud. There are many methods for segmenting the point cloud map to be registered, and this embodiment does not limit the segmentation method. Preferably, the Euclidean clustering method is adopted and the KD-Tree or Octree is constructed to accelerate the neighborhood query to segment the point cloud map to be registered to obtain multiple sub-regions. The volume of each sub-region is obtained by obtaining the convex hull of each sub-region. If the volumes of adjacent sub-regions are all less than the minimum volume threshold, they are merged into a new sub-region and the volume is recalculated; if the volume of the sub-region is within the error range of the volume of the model point cloud, such as within the range of ±20% of the volume of the model point cloud, the sub-region is used as a candidate region.

[0085] Furthermore, the position of each candidate area is the center of mass position of each candidate area, which is obtained by calculating the average position of all point clouds in the candidate area. The position of the current MR device is obtained according to the 6DOF posture of the MR device. The Euclidean distance between each candidate area and the current MR device is calculated, and the candidate area corresponding to the minimum distance is selected as the target point cloud, that is, the model point cloud needs to be aligned with the target point cloud.

[0086] ②Use the model point cloud as the source point cloud and construct multiple initial poses for the source point cloud in the world coordinate system.

[0087] Based on the world coordinate system, the heading angle is set incrementally from 0 at intervals, and the pitch angle and roll angle are both set to 0 degrees to obtain multiple second rotation matrices; a second translation vector is obtained according to the position of the target point cloud; and the multiple second rotation matrices are respectively combined with the second translation vector to form multiple second transformation matrices to obtain multiple initial poses.

[0088] Specifically, for each heading angle, a 4×4 second transformation matrix containing a second rotation matrix and a second translation vector is constructed to obtain the initial pose.

[0089] For example, the heading angle is set in increments of 10 degrees within the range of [0, 360] degrees, resulting in 36 rotation matrices around axes perpendicular to the ground.

[0090] ③ Based on each initial pose, the source point cloud is registered with the target point cloud respectively to obtain multiple optimized poses and second registration scores; if the maximum second registration score is not less than the score threshold, the corresponding optimized pose is the pose of the model point cloud in the world coordinate system; otherwise, the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum second registration score is not less than the score threshold.

[0091] It should be noted that, since there are multiple initial poses, this embodiment adopts the point cloud registration method in step S1, that is, it uses multi-threading, and uses the weighted ICP algorithm in each thread to iteratively optimize the corresponding initial poses, so that the position of the source point cloud in the target point cloud reaches the optimal position and obtains the corresponding optimized pose (that is, the optimized initial pose) and the second registration score.

[0092] When the maximum second registration score is not less than the score threshold, the registration is successful, and the corresponding optimized pose is the pose of the model point cloud in the SLAM world coordinate system. At this time, the coordinates of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score are obtained as the model registration position.

[0093] In order to avoid the problem of incorrect alignment in subsequent use, which may cause the user to not be able to see the fused visual scene on the training device, regular checks are performed to determine whether the model point cloud and the target point cloud need to be re-aligned based on the distance between the position of the MR device and the model registration position, as well as the minimum distance between the position of the MR device and the candidate area.

[0094] Specifically, if the distance between the position of the MR device and the model registration position is less than the first distance threshold, there is no need to perform point cloud registration again. If the distance between the position of the MR device and the model registration position is greater than the second distance threshold, and the minimum distance between the position of the MR device and the candidate area of the non-target point cloud is less than the first distance threshold, the candidate area corresponding to the minimum distance is updated to the target point cloud, and the model point cloud and the target point cloud are re-registered.

[0095] Preferably, detection is performed every 5 seconds, the first distance threshold is 1 meter, and the second distance threshold is 2 meters.

[0096] S3. Correct the MR device pose at each moment according to the pose of the model point cloud in the world coordinate system, obtain the pose of the MR device at each moment in the model coordinate system where the model point cloud is located, and then render a virtual-real aligned training application.

[0097] It should be noted that the position and posture of the MR device at each moment in the model coordinate system where the model point cloud is located is calculated using the following formula:

[0098] Formula (4),

[0099] in, represents the position of the MR device in the model coordinate system at time t, represents the pose of the MR device at time t, that is, the 6DOF pose in the world coordinate system of SLAM, Represents the pose of the model point cloud in the world coordinate system of SLAM.

[0100] According to the position of the MR device in the model coordinate system at each moment, the training application of rendering virtual and real alignment is obtained by superimposing multiple layers.

[0101] It should be noted that the order of the multiple layers from bottom to top is: virtual layer, perspective window layer, and real layer. The virtual layer is the image rendered by the training application deployed in the MR device based on the MR device's position in the model coordinate system at each moment. The perspective window layer is the outline area of the real training device, an area rendered based on the 3D mesh model file of the real training device, used to cover the virtual training device in the virtual layer. The real layer is used to display the real image captured by the MR device in the perspective window. Virtual-real alignment is the effect produced by aligning the image of the real layer and the virtual image based on two training devices of the same size and appearance in space.

[0102] Specifically, As the view matrix for rendering the virtual training device, the relative position of the MR device and the real training device in real space is kept consistent with the relative position of the MR device and the model of the real training device in virtual space. Through the conventional rendering process, the perspective window layer and the real layer are superimposed, so that only the real image in the area covered by the outline of the real training device is retained. At this time, from the trainee's perspective, the outline of the virtual training device and the outline of the real training device are overlapped. The training application renders the virtual image and combines it with the real image in the contour coverage area to achieve the effect of virtual-real alignment. This allows the trainees to see the real image of the real space when looking into the training device through the MR device, and to see the virtual training task image when looking outside the training device.

[0103] In this embodiment, steps S1-S3 are an automatic process of virtual-real alignment, which usually does not require further adjustment by the user. However, considering the differences in human sensory perception, when the user is not satisfied with the virtual-real alignment effect rendered in step S3, fine-tuning the alignment is performed, including: adjusting the three translation positions (in meters) and three rotation Euler angles (in degrees) around the x, y, and z axes on the user operation panel, calculating the corresponding offset pose matrix, and correcting the pose of the MR device in the model coordinate system obtained by formula (4), and rendering the interface of the fine-tuned virtual-real alignment effect.

[0104] Compared with the existing technology, the present embodiment provides a method for MR device virtual-reality fusion based on point cloud registration. Based on the registration results of the real-time updated point cloud map and the historical point cloud map, the historical point cloud map is rationally used to supplement the global environmental information of the real-time updated point cloud map. This optimizes the real-time mapping process, reduces the consumption and time of computing resources, facilitates the rapid entry into the virtual-reality fusion registration process, and improves the efficiency of virtual-reality fusion. By performing operations such as registration and coordinate system transformation of the point cloud of the three-dimensional mesh model of the real training device with the real-time point cloud map, the MR device posture is adjusted in real time to avoid alignment deviation caused by device movement, achieving fully automatic and high-precision virtual-reality fusion, reducing the number of operation steps and time, and enabling users to enter the immersive experience more quickly. By constructing the initial posture at multiple initial headings, covering multiple possible postures of the point cloud in space, the possibility of finding the correct match is increased, effectively improving the registration success rate, accuracy, and robustness. It is particularly suitable for applications with complex scenes and high precision requirements.

[0105] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0106] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for virtual-reality fusion of MR equipment based on point cloud registration, characterized in that: The following steps are involved: A real-time updated point cloud map is constructed based on the collected environmental data, and a historical point cloud map is loaded at the same time; the current real-time updated point cloud map is registered with the historical point cloud map. If the registration is successful, the historical point cloud map is integrated with the current real-time updated point cloud map to obtain the point cloud map to be registered; otherwise, the current real-time updated point cloud map is continuously updated to obtain the point cloud map to be registered; Obtain the model point cloud based on the 3D mesh model file of the real training device, align the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration; Correct the MR device pose at each moment based on the pose of the model point cloud in the world coordinate system, and obtain the pose of the MR device in the model coordinate system where the model point cloud is located at each moment, and then render a virtual-real aligned training application; The registering of the currently updated point cloud map in real time with the historical point cloud map includes: constructing a plurality of first poses of the MR device in the historical point cloud map based on the maximum probability position of the MR device in the historical point cloud map and the current MR device pose; obtaining a plurality of second poses of the currently updated point cloud map relative to the historical point cloud map based on each first pose and the current MR device pose; registering the currently updated point cloud map in real time with the historical point cloud map based on each second pose, respectively, to obtain each third pose and a first registration score; if the maximum first registration score is not less than a score threshold, the registration is successful; otherwise, the registration fails; The pose of the MR device at each moment is corrected according to the pose of the model point cloud in the world coordinate system to obtain the pose of the MR device at each moment in the model coordinate system where the model point cloud is located. The formula is as follows: ， in, represents the position of the MR device in the model coordinate system at time t, represents the posture of the MR device at time t, Represents the pose of the model point cloud in the world coordinate system.

2. The method for virtual-reality fusion of MR equipment based on point cloud registration according to claim 1, characterized in that: The historical point cloud map is the latest real-time updated point cloud map saved when the MR device was last shut down; the method also includes: calculating the maximum probability position of the MR device in the historical point cloud map; the maximum probability position is a position point set formed by regularly collecting the position points of the MR device at non-stationary moments during the use of the MR device; voxel-dividing the space of the position point set, and calculating the average position of all position points in the top N voxels with the largest point set, where N>1; the non-stationary moment is when the position difference and posture difference in the MR device posture at adjacent moments are both greater than the corresponding threshold.

3. The method for virtual-reality fusion of MR equipment based on point cloud registration according to claim 2, characterized in that: The method constructs multiple first poses of the MR device in the historical point cloud map based on the maximum probability position of the MR device in the historical point cloud map and the current MR device pose, including: The pitch angle and roll angle in the current MR device posture are used as the pitch angle and roll angle of the MR device in the historical point cloud map; and the heading angle is set incrementally from 0 at intervals of angle, thereby obtaining multiple first rotation matrices; and a first translation vector is obtained according to the maximum probability position of the MR device in the historical point cloud map; The plurality of first rotation matrices are respectively combined with the first translation vector to form a plurality of first transformation matrices to obtain a plurality of first poses.

4. The method for virtual-reality fusion of MR equipment based on point cloud registration according to claim 2, characterized in that: The multiple second postures are calculated by the following formula: ， in, represents the k-th second pose, represents the kth first pose; Indicates the current MR device posture.

5. The method for virtual-reality fusion of MR equipment based on point cloud registration according to claim 2, characterized in that: The point cloud map to be registered is obtained by fusing the historical point cloud map with the currently updated point cloud map in real time, wherein the historical point cloud map is transformed according to the third posture corresponding to the largest first registration score, and the transformed historical point cloud map is matched with the currently updated point cloud map in real time using the point cloud registration algorithm, and the unmatched historical point cloud is added to the currently updated point cloud map in real time. After voxel filtering, the fused point cloud map is obtained as the point cloud map to be registered.

6. The method for virtual-reality fusion of MR equipment based on point cloud registration according to any one of claims 1 to 5, characterized in that: The step of registering the model point cloud with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system includes: Segment the point cloud map to be registered and obtain the sub-region with the smallest distance to the MR device as the target point cloud; Take the model point cloud as the source point cloud and construct multiple initial poses for the source point cloud in the world coordinate system; Based on each initial pose, the source point cloud is registered with the target point cloud respectively to obtain each optimized pose and the second registration score; if the maximum second registration score is not less than the score threshold, the corresponding optimized pose is the pose of the model point cloud in the world coordinate system; otherwise, the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum second registration score is not less than the score threshold.

7. The method for virtual-reality fusion of MR equipment based on point cloud registration according to claim 6, characterized in that: The step of segmenting the point cloud map to be registered and obtaining a sub-region with the smallest distance from the MR device as a target point cloud includes: According to the volume of the model point cloud and the minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-regions within the error range of the volume of the model point cloud are retained as candidate regions; the distance between each candidate region and the current MR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

8. The method for virtual-reality fusion of MR equipment based on point cloud registration according to claim 6, characterized in that: The method constructs multiple initial poses for the source point cloud in the world coordinate system, including: setting the heading angle in increments from 0 at intervals, and setting the pitch angle and roll angle to 0 degrees to obtain multiple second rotation matrices; obtaining a second translation vector based on the position of the target point cloud; and combining the multiple second rotation matrices with the second translation vector to form multiple second transformation matrices to obtain multiple initial poses.

Images