Full-automatic XR equipment virtual and real alignment method and system

Abstract

The invention relates to a full-automatic XR equipment virtual-real alignment method and system, belongs to the technical field of mixed reality, and solves the problems of low efficiency and limited alignment precision caused by manual or semi-automatic alignment in the prior art. Comprising the following steps: constructing a real-time point cloud map under a world coordinate system according to environment data collected in real time, and obtaining a point cloud map to be registered when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold value; acquiring a model point cloud according to the three-dimensional grid model file of the real training equipment, registering the model point cloud with a to-be-registered point cloud map, and obtaining a pose of the model point cloud in a world coordinate system after successful registration; and correcting the pose of the XR equipment at each moment according to the pose of the model point cloud in the world coordinate system to obtain the pose of the XR equipment at each moment in the model coordinate system where the model point cloud is located, and further rendering training application of virtual-real alignment. And full-automatic and high-precision virtual and real alignment is realized.

Description

Technical Field

[0001] The present invention relates to the field of extended reality technology, and in particular to a fully automatic XR device virtual-real alignment method and system. Background Art

[0002] XR (Extended Reality) is a general term encompassing virtual reality (VR), augmented reality (AR), mixed reality (MR), and all technologies that enable the creation and experience of virtual or augmented reality. XR devices encompass a wide range of different types and have broad application prospects in fields such as gaming, education, healthcare, and architectural design. Aligning virtual and real experiences is key to the evolution of XR device technology.

[0003] Aligning virtual objects with real-world scenes on existing XR devices typically requires human intervention, which presents significant limitations. For example, some manual alignment methods are cumbersome, inefficient, and require a high level of operator experience. Other semi-automatic alignment methods require manual adjustment of offset positions and postures during the initial virtual-real alignment process, with subsequent automatic alignment enabled. However, these methods still require significant user intervention and struggle to achieve fully automated alignment.

[0004] Manual and semi-automatic alignment methods lack accuracy, efficiency, and adaptability to complex scenarios. In addition, when there are multiple real training devices in the real space, the XR device may be aligned to the wrong training device. Summary of the Invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a fully automatic XR device virtual-real alignment method and system to solve the problems of low efficiency and limited alignment accuracy caused by existing manual or semi-automatic alignment methods.

[0006] In one aspect, an embodiment of the present invention provides a fully automatic XR device virtual-real alignment method, comprising the following steps:

[0007] A real-time point cloud map in the world coordinate system is constructed based on the environmental data collected in real time. When the number of point clouds in the real-time point cloud map is greater than the point cloud threshold, a point cloud map to be registered is obtained.

[0008] Obtain the model point cloud based on the 3D mesh model file of the real training device, align the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration;

[0009] According to the pose of the model point cloud in the world coordinate system, the pose of the XR device at each moment is corrected to obtain the pose of the XR device in the model coordinate system where the model point cloud is located at each moment, and then a virtual-real aligned training application is rendered.

[0010] Based on the further improvement of the above method, the model point cloud is registered with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system, including:

[0011] Segment the point cloud map to be registered and obtain the sub-region with the smallest distance to the XR device as the target point cloud;

[0012] Use the model point cloud as the source point cloud and construct multiple initial poses for the source point cloud in the world coordinate system;

[0013] Based on each initial pose, the source point cloud is registered with the target point cloud respectively to obtain each optimized pose and registration score; if the maximum registration score is not less than the score threshold, the registration is successful, and the optimized pose corresponding to the maximum registration score is the pose of the model point cloud in the world coordinate system; otherwise, the registration fails, and the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum registration score is not less than the score threshold.

[0014] Based on the further improvement of the above method, the point cloud map to be registered is segmented, and the sub-region with the smallest distance to the XR device is obtained as the target point cloud, including:

[0015] According to the volume of the model point cloud and the minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-regions within the error range of the volume of the model point cloud are retained as candidate regions; the distance between each candidate region and the current XR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

[0016] Based on a further improvement of the above method, multiple initial poses are constructed for the source point cloud in the world coordinate system, including: setting the heading angle from 0 in increments according to the interval angle, and setting the pitch angle and roll angle to 0 degrees to obtain multiple second rotation matrices; obtaining the second translation vector according to the position of the target point cloud; and combining the multiple second rotation matrices with the second translation vector into multiple second transformation matrices to obtain multiple initial poses.

[0017] Based on the further improvement of the above method, after the registration is successful, it is regularly checked whether the model point cloud and the target point cloud need to be re-registered, including: obtaining the coordinates of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score as the model registration position, if the distance between the position of the XR device and the model registration position is less than the first distance threshold, then there is no need to re-register; if the distance between the position of the XR device and the model registration position is greater than the second distance threshold, and the minimum distance between the position of the XR device and the candidate area of ​​the non-target point cloud is less than the first distance threshold, then the candidate area corresponding to the minimum distance is updated to the target point cloud, and the model point cloud and the target point cloud are re-registered.

[0018] Based on the further improvement of the above method, the source point cloud is registered with the target point cloud based on each initial pose respectively. Multi-threading is used, and the weighted ICP algorithm is used in each thread to iteratively optimize the corresponding initial pose, so that the position of the source point cloud in the target point cloud reaches the optimal position and the corresponding optimized pose and registration score are obtained.

[0019] Based on the further improvement of the above method, the weighted ICP algorithm adopts a dynamic weight adjustment mechanism and combines it with the Huber error function to construct the objective function. The objective function is used as the optimization goal and the registration error between point clouds is minimized through iterative solution. The dynamic weight is obtained by calculating the normal and curvature of the point pair.

[0020] Based on the further improvement of the above method, the dynamic weight is calculated by the following formula:

[0021]

[0022] Among them, w i Represents the i-th point pair (p i ,q i ), α represents the proportional coefficient, w normal Represents the normal weight, w curvature represents the curvature weight, θ i Represents a point pair (p i ,q i ), σ represents the adjustment parameter; κ i Indicates that according to the source point cloud p i The curvature is calculated from the neighborhood point set of , and β represents the proportional coefficient.

[0023] Based on the further improvement of the above method, the XR device pose at each moment is corrected according to the pose of the model point cloud in the world coordinate system, and the pose of the XR device in the model coordinate system where the model point cloud is located at each moment is obtained. The formula is as follows:

[0024]

[0025] in, Indicates the pose of the XR device in the model coordinate system at time t, Indicates the XR device pose at time t, Represents the pose of the model point cloud in the world coordinate system.

[0026] On the other hand, an embodiment of the present invention provides a fully automatic XR device virtual-real alignment system, comprising:

[0027] The point cloud map construction module is used to construct a real-time point cloud map in the world coordinate system based on the environmental data collected in real time. When the number of point clouds in the real-time point cloud map is greater than the point cloud threshold, the point cloud map to be registered is obtained;

[0028] The point cloud map registration module is used to obtain the model point cloud based on the 3D mesh model file of the real training device, register the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration;

[0029] The device virtual-reality alignment module is used to correct the XR device pose at each moment based on the pose of the model point cloud in the world coordinate system, obtain the pose of the XR device at each moment in the model coordinate system where the model point cloud is located, and then render a virtual-reality aligned training application.

[0030] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0031] 1. By aligning the point cloud of the 3D mesh model of the real training device with the real-time point cloud map and performing coordinate system transformation, the XR device posture is adjusted in real time to avoid alignment deviation caused by device movement. This achieves fully automatic and high-precision virtual-real alignment, reduces operation steps and time, and enables users to enter the immersive experience more quickly.

[0032] 2. By constructing initial poses at multiple initial headings, covering multiple possible poses of the point cloud in space, the possibility of finding a correct match is increased, effectively improving the registration success rate, accuracy and robustness, which is especially suitable for complex scenes and applications with high precision requirements.

[0033] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0035] Figure 1 This is a flow chart of a fully automatic XR device virtual-real alignment method in Example 1 of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of a fully automatic XR device virtual-real alignment system in Example 2 of the present invention. DETAILED DESCRIPTION

[0037] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0038] Example 1

[0039] A specific embodiment of the present invention discloses a fully automatic XR device virtual-real alignment method, such as Figure 1 As shown, the following steps are included:

[0040] S1. Construct a real-time point cloud map in the world coordinate system based on the real-time collected environmental data. When the number of point clouds in the real-time point cloud map is greater than the point cloud threshold, obtain a point cloud map to be registered.

[0041] It should be noted that the XR device in this embodiment is a head-mounted display device, including: one binocular color camera, two binocular black-and-white cameras, and an IMU (Inertial Measurement Unit) sensor. The binocular color camera is used to capture real-world images, and the binocular black-and-white camera and IMU are used to track the XR device in real time and calculate the XR device's position in the world coordinate system. Exemplarily, the XR device is AR glasses or MR glasses.

[0042] XR devices are equipped with training applications that provide users with a blended virtual and real-world scenario for simulated training. For example, the training application is a driver training application. A real cockpit training device is placed in the real world. The training application renders a virtual cockpit training device and the exterior of the cockpit in the virtual world. After the virtual and real worlds are aligned, the real cockpit and the virtual exterior of the cockpit are combined, and the operator's actions in the real cockpit are reflected in the driver training application.

[0043] In this embodiment, the user wears an XR device to collect environmental data about the real training device. The SLAM algorithm deployed in the XR device outputs six degrees of freedom (6DOF) pose data at the moment of exposure for each frame of image. This data is used to determine the XR device pose, i.e., its pose in the SLAM world coordinate system. The SLAM algorithm itself also creates a point cloud map, but for efficient operation, the XR device only retains a sparse point cloud map, which cannot display the geometry of the real training device.

[0044] Therefore, this step uses a real-time mapping algorithm to extract and match feature points (such as corners and SIFT features) from multiple frames of environmental images captured by a binocular black-and-white camera. After triangulating these features, a point cloud is recovered in the camera coordinate system. The point cloud is then converted to the SLAM world coordinate system using the 6DOF pose of the XR device at the moment of exposure of each frame, along with the extrinsic poses of the camera and XR device. This process is repeated over time to construct a dense point cloud map in the world coordinate system, which reflects the geometric information of the actual training device.

[0045] Furthermore, in order to ensure the real-time performance of the point cloud map, the constructed point cloud map is used as the global point cloud map. After identifying the key frames for each frame of image subsequently collected in real time, the feature points on the key frames are converted to world coordinates to obtain a new point cloud. The new point cloud is matched with the point cloud map using the point cloud registration algorithm, the unmatched new point cloud is added to the global point cloud map, and the updated global point cloud map is voxel filtered to ensure that only one point coordinate is retained for the same spatial point in a volume space of a certain scale.

[0046] Among them, identifying whether the current frame is a key frame is based on the posture data of the XR device in the current frame and the previous frame, comparing whether the position difference in the posture data is greater than the distance threshold, and whether the posture difference is greater than the posture threshold. If both are greater than the corresponding thresholds, the current frame is a key frame.

[0047] It should be noted that to identify whether the posture difference is greater than the posture threshold, the rotation matrix in the posture data is converted into the corresponding Lie algebraic vector, and the modulus of the Lie algebraic vector is calculated and compared with the posture threshold.

[0048] Preferably, in order to make the real-time point cloud map more obviously reflect the geometric information of the real training device, the surface of the real training device is textured so that feature points can be better extracted, thereby restoring the 3D point cloud of the real training device.

[0049] As the point clouds of subsequent key frames are added, the real-time point cloud map is continuously updated and improved. When the number of point clouds in the real-time point cloud map exceeds the point cloud threshold, the point cloud map to be registered is obtained and the registration operation of step S2 is performed. At the same time, step S1 also continues to update the real-time point cloud map.

[0050] S2. Obtain a model point cloud based on the 3D mesh model file of the actual training device, align the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system.

[0051] It should be noted that the 3D mesh model file of the real training device is imported into the XR device. The 3D mesh model file is typically obtained by modeling the real training device using 3D modeling software, meshing it, and then exporting it. The file includes the model's vertex data, color data, and triangle mesh data. Exemplary 3D modeling software includes Maya and Blender.

[0052] Furthermore, the PCL library (an open source library specifically for point cloud processing) is used to take the 3D mesh model file as input and generate the model point cloud by sampling. For example, the pcl_mesh_sampling function in the PCL library is called to generate the model point cloud.

[0053] It should be noted that the vertex coordinates of the model are set relative to the origin of the three-dimensional mesh model. Therefore, the origin of the coordinate system of the generated model point cloud is consistent with the origin in the three-dimensional mesh model. Based on this origin, the model coordinate system where the model point cloud is located is constructed using the right-hand rule. For example, according to the orientation of the three-dimensional mesh model, the right, top and back of the model are the x, y and z axis directions of the coordinate system respectively.

[0054] Furthermore, considering that there may be multiple training devices in the real-time point cloud map, and that a poor initial registration pose can result in high registration errors or non-convergence, this embodiment segments the point cloud map to be registered to identify the training device closest to the XR device, and constructs multiple initial poses to facilitate finding the optimal initial pose.

[0055] Specifically, the model point cloud is registered with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system, including:

[0056] ① Segment the point cloud map to be registered and obtain the sub-area with the smallest distance to the XR device as the target point cloud.

[0057] According to the volume of the model point cloud and the minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-regions within the error range of the volume of the model point cloud are retained as candidate regions; the distance between each candidate region and the current XR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

[0058] It should be noted that the volume of the model point cloud is the volume of the space enclosed by the model point cloud, which is obtained by calculating the convex hull volume of the model point cloud. There are many methods for segmenting the point cloud map to be registered, and this embodiment does not limit the segmentation method. Preferably, the Euclidean clustering method is adopted and the KD-Tree or Octree is constructed to accelerate the neighborhood query to segment the point cloud map to be registered to obtain multiple sub-regions. The volume of each sub-region is obtained by obtaining the convex hull of each sub-region. If the volumes of adjacent sub-regions are all less than the minimum volume threshold, they are merged into a new sub-region and the volume is recalculated; if the volume of the sub-region is within the error range of the volume of the model point cloud, such as within the range of ±20% of the volume of the model point cloud, then the sub-region is used as a candidate region.

[0059] Furthermore, the position of each candidate area is the center of mass position of each candidate area, which is obtained by calculating the average position of all point clouds in the candidate area. The position of the current XR device is obtained according to the 6DOF posture of the XR device. The Euclidean distance between each candidate area and the current XR device is calculated, and the candidate area corresponding to the minimum distance is selected as the target point cloud, that is, the model point cloud needs to be aligned with the target point cloud.

[0060] ②Use the model point cloud as the source point cloud and construct multiple initial poses for the source point cloud in the world coordinate system.

[0061] Based on the world coordinate system, the heading angle is set incrementally from 0 at intervals, and the pitch angle and roll angle are both set to 0 degrees to obtain multiple second rotation matrices; a second translation vector is obtained according to the position of the target point cloud; and the multiple second rotation matrices are respectively combined with the second translation vector to form multiple second transformation matrices to obtain multiple initial poses.

[0062] Specifically, for each heading angle, a 4×4 second transformation matrix containing a second rotation matrix and a second translation vector is constructed to obtain the initial pose.

[0063] For example, the heading angle is set in increments of 10 degrees within the range of [0, 360] degrees, resulting in 36 rotation matrices around axes perpendicular to the ground.

[0064] ③ Based on each initial pose, the source point cloud is registered with the target point cloud respectively to obtain each optimized pose and registration score; if the maximum registration score is not less than the score threshold, the corresponding optimized pose is the pose of the model point cloud in the world coordinate system; otherwise, the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum registration score is not less than the score threshold.

[0065] It should be noted that since there are multiple initial poses, this embodiment uses multi-threading, and at the same time adopts the weighted ICP algorithm in each thread to iteratively optimize the corresponding initial poses, so that the position of the source point cloud in the target point cloud reaches the optimal position and the corresponding optimized pose and alignment score are obtained, thereby realizing the correction of multiple initial poses at the same time and improving the computational efficiency.

[0066] Considering that in the registration process, matching point pairs with consistent normal directions are more likely to belong to the same physical surface, and their correspondence is more reliable; point pair matching is more stable in flat areas (i.e., low curvature areas), and should be assigned higher weights; in high curvature areas (such as edges), the weights should be lowered to reduce noise interference. Therefore, this embodiment assigns different dynamic weights to point pairs according to their normals and curvatures, so that the registration accuracy of point pairs consistent with the normal is higher, and the Huber error function is used to reduce the influence of outliers, thereby improving the overall registration effect. That is, the weighted ICP algorithm of this embodiment adopts a dynamic weight adjustment mechanism and combines the Huber error function to construct an objective function, takes the objective function as the optimization target, and minimizes the registration error between point clouds through iterative solution.

[0067] Specifically, setting p i is the 3D coordinate of the i-th point in the source point cloud, q i is the target point cloud with p i The three-dimensional coordinates of the corresponding nearest point are calculated by the following formula (p i ,q i ) and the normal and curvature of the point pair (p i ,q i )’s dynamic weight:

[0068]

[0069] Among them, w i Represents the i-th point pair (p i ,q i ), α represents the proportional coefficient, w normal Represents the normal weight, w curvature represents the curvature weight, θ i Represents a point pair (p i ,q i), σ represents the adjustment parameter; κ i Indicates that according to the source point cloud p i The curvature is calculated from the neighborhood point set of , and β represents the proportional coefficient.

[0070] Furthermore, the objective function E(R′, t′) obtained by combining the weights with the Huber error function is as follows:

[0071]

[0072] Among them, (R′, t′) is the pose to be solved, R′ represents the rotation matrix, t′ represents the translation vector, and L σ (·) represents the Huber error function, n represents the number of point pairs, and ‖·‖ represents the L2 norm.

[0073] Finally, the optimal pose of each source point cloud and target point cloud after registration is solved according to the objective function of formula (2), and the registration score is calculated. The better the registration effect, the higher the registration score.

[0074] For example, the registration score is obtained by calculating the average or sum of the registration probabilities of all point pairs, or by calculating the mean square error of all point pairs and then taking the inverse thereof.

[0075] When the maximum registration score is not less than the score threshold, the registration is successful, and the corresponding optimized pose is the pose of the model point cloud in the SLAM world coordinate system. At this time, the coordinates of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score are obtained as the model registration position.

[0076] In order to avoid the problem of incorrect alignment in subsequent use, which may cause the user to not be able to see the fused visual scene on the training device, regular checks are performed to determine whether the model point cloud and the target point cloud need to be re-aligned based on the distance between the position of the XR device and the model registration position, as well as the minimum distance between the position of the XR device and the candidate area.

[0077] Specifically, if the distance between the position of the XR device and the model registration position is less than the first distance threshold, there is no need to re-register. If the distance between the position of the XR device and the model registration position is greater than the second distance threshold, and the minimum distance between the position of the XR device and the candidate area of ​​the non-target point cloud is less than the first distance threshold, the candidate area corresponding to the minimum distance is updated to the target point cloud, and the model point cloud and the target point cloud are re-registered.

[0078] Preferably, detection is performed every 5 seconds, the first distance threshold is 1 meter, and the second distance threshold is 2 meters.

[0079] S3. Correct the XR device pose at each moment based on the pose of the model point cloud in the world coordinate system, obtain the pose of the XR device at each moment in the model coordinate system where the model point cloud is located, and then render a virtual-real aligned training application.

[0080] It should be noted that the pose of the XR device at each moment in the model coordinate system where the model point cloud is located is calculated using the following formula:

[0081]

[0082] in, Indicates the pose of the XR device in the model coordinate system at time t, represents the XR device pose at time t, that is, the 6DOF pose in the SLAM world coordinate system, Represents the pose of the model point cloud in the world coordinate system of SLAM.

[0083] Based on the position of the XR device in the model coordinate system at each moment, the training application of rendering virtual and real alignment is obtained by superimposing multiple layers.

[0084] It should be noted that the order of the multiple layers from bottom to top is: virtual layer, perspective window layer, and real layer. The virtual layer is the image rendered by the training application deployed in the XR device based on the XR device's position in the model coordinate system at each moment. The perspective window layer is the outline area of ​​the real training device, an area rendered based on the 3D mesh model file of the real training device, used to cover the virtual training device in the virtual layer. The real layer is used to display the real image captured by the XR device in the perspective window. Virtual-real alignment is the effect produced by aligning the image of the real layer and the virtual image based on two training devices of the same size and appearance in space.

[0085] Specifically, As the view matrix for rendering the virtual training device, the relative position of the XR device and the real training device in the real space is kept consistent with the relative position of the XR device and the model of the real training device in the virtual space. Through the conventional rendering process, the perspective window layer and the real layer are superimposed so that only the real image in the area covered by the outline of the real training device is retained. At this time, from the perspective of the trainee, the outline of the virtual training device and the outline of the real training device are overlapped. At the same time, The training application then renders the virtual image and combines it with the real image in the outline coverage area to achieve the effect of virtual-real alignment. This allows the trainees to see the real image of the real space when looking into the training device through the XR device, and to see the virtual training task image when looking outside the training device.

[0086] In this embodiment, steps S1-S3 are a fully automatic process for virtual-real alignment, and usually no further adjustments are required by the user. However, considering the differences in human sensory perception, when the user is not satisfied with the virtual-real alignment effect rendered in step S3, fine-tuning the alignment is performed, including: adjusting the three translation positions (in meters) and three rotation Euler angles (in degrees) around the x, y, and z axes on the user operation panel, calculating the corresponding offset pose matrix, and correcting the pose of the XR device in the model coordinate system obtained by formula (3), and rendering the interface of the fine-tuned virtual-real alignment effect.

[0087] Compared to existing technologies, the fully automatic XR device virtual-real alignment method provided in this embodiment adjusts the XR device pose in real time by aligning the point cloud of the real-world training device's 3D mesh model with the real-time point cloud map and performing coordinate system transformations. This avoids alignment deviations caused by device movement, achieving fully automatic, high-precision virtual-real alignment, reducing operation steps and time, and enabling users to enter an immersive experience more quickly. By constructing initial poses at multiple initial headings, covering the various possible poses of the point cloud in space, the probability of finding a correct match is increased, effectively improving the registration success rate, accuracy, and robustness. This method is particularly suitable for complex scenarios and applications with high precision requirements.

[0088] Example 2

[0089] Another embodiment of the present invention discloses a fully automatic XR device virtual-real alignment system, thereby realizing the fully automatic XR device virtual-real alignment method in Example 1. The specific implementation of each module refers to the corresponding description in Example 1. Figure 2 As shown, the system includes:

[0090] The point cloud map construction module 101 is used to construct a real-time point cloud map in the world coordinate system based on the environmental data collected in real time. When the number of point clouds in the real-time point cloud map is greater than the point cloud threshold, a point cloud map to be registered is obtained;

[0091] The point cloud map registration module 102 is used to obtain a model point cloud based on the 3D mesh model file of the real training device, register the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration;

[0092] The device virtual-reality alignment module 103 is used to correct the XR device posture at each moment according to the posture of the model point cloud in the world coordinate system, obtain the posture of the XR device at each moment in the model coordinate system where the model point cloud is located, and then render a virtual-reality aligned training application.

[0093] Since the fully automatic XR device virtual-real alignment system of this embodiment and the previously described fully automatic XR device virtual-real alignment method share similarities and can be mutually referenced, they are redundantly described and will not be repeated here. Since the principles of this system embodiment and the aforementioned method embodiment are the same, this system embodiment also has the corresponding technical effects of the aforementioned method embodiment.

[0094] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0095] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A fully automatic XR device virtual-real alignment method, characterized in that: The following steps are involved: Constructing a real-time point cloud map in a world coordinate system based on the environmental data collected in real time, and obtaining a point cloud map to be registered when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold; Obtain the model point cloud based on the 3D mesh model file of the real training device, align the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration; The pose of the XR device at each moment is corrected according to the pose of the model point cloud in the world coordinate system, and the pose of the XR device at each moment in the model coordinate system where the model point cloud is located is obtained, and then a virtual-real aligned training application is rendered.

2. The fully automatic XR device virtual-real alignment method according to claim 1, characterized in that: The step of registering the model point cloud with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system includes: Segment the point cloud map to be registered and obtain the sub-region with the smallest distance to the XR device as the target point cloud; Take the model point cloud as the source point cloud and construct multiple initial poses for the source point cloud in the world coordinate system; Based on each initial pose, the source point cloud is registered with the target point cloud respectively to obtain each optimized pose and registration score; if the maximum registration score is not less than the score threshold, the registration is successful, and the optimized pose corresponding to the maximum registration score is the pose of the model point cloud in the world coordinate system; otherwise, the registration fails, and the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum registration score is not less than the score threshold.

3. The fully automatic XR device virtual-real alignment method according to claim 2, characterized in that: The step of segmenting the point cloud map to be registered and obtaining the sub-region with the smallest distance from the XR device as the target point cloud includes: According to the volume of the model point cloud and the minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-regions within the error range of the volume of the model point cloud are retained as candidate regions; the distance between each candidate region and the current XR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

4. The fully automatic XR device virtual-real alignment method according to claim 2, characterized in that: The method constructs multiple initial poses for the source point cloud in the world coordinate system, including: setting the heading angle in increments from 0 at intervals, and setting the pitch angle and roll angle to 0 degrees to obtain multiple second rotation matrices; obtaining a second translation vector based on the position of the target point cloud; and combining the multiple second rotation matrices with the second translation vector to form multiple second transformation matrices to obtain multiple initial poses.

5. The fully automatic XR device virtual-real alignment method according to claim 2, characterized in that: After successful registration, periodically check whether the model point cloud and the target point cloud need to be re-registered, including: obtaining the coordinates of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score as the model registration position; if the distance between the position of the XR device and the model registration position is less than the first distance threshold, no re-registration is required; if the distance between the position of the XR device and the model registration position is greater than the second distance threshold, and the minimum distance between the position of the XR device and the candidate area of ​​the non-target point cloud is less than the first distance threshold, then the candidate area corresponding to the minimum distance is updated as the target point cloud, and the model point cloud and the target point cloud are re-registered.

6. The fully automatic XR device virtual-real alignment method according to claim 2, characterized in that: The method of registering the source point cloud with the target point cloud based on each initial pose is to utilize multi-threading, and at the same time, use the weighted ICP algorithm in each thread to iteratively optimize the corresponding initial pose, so that the position of the source point cloud in the target point cloud reaches the optimal position and obtain the corresponding optimized pose and registration score.

7. The fully automatic XR device virtual-real alignment method according to claim 6, characterized in that: The weighted ICP algorithm adopts a dynamic weight adjustment mechanism and combines it with the Huber error function to construct an objective function. The objective function is used as the optimization goal and the registration error between point clouds is minimized through iterative solution. The dynamic weight is obtained by calculating the normal and curvature of the point pair.

8. The fully automatic XR device virtual-real alignment method according to claim 7, characterized in that: The dynamic weight is calculated using the following formula: Among them, w i Represents the i-th point pair (p i ,q i ), α represents the proportional coefficient, w normal Represents the normal weight, w curvature represents the curvature weight, θ i Represents a point pair (p i ,q i ), σ represents the adjustment parameter; κ i Indicates that according to the source point cloud p i The curvature is calculated from the neighborhood point set of , and β represents the proportional coefficient.

9. The fully automatic XR device virtual-real alignment method according to claim 1, characterized in that: The pose of the XR device at each moment is corrected according to the pose of the model point cloud in the world coordinate system to obtain the pose of the XR device at each moment in the model coordinate system where the model point cloud is located. The formula is as follows: in, Indicates the pose of the XR device in the model coordinate system at time t, Indicates the XR device pose at time t, Represents the pose of the model point cloud in the world coordinate system.

10. A fully automatic XR device virtual-real alignment system, characterized by: include: A point cloud map construction module is used to construct a real-time point cloud map in a world coordinate system based on the environmental data collected in real time. When the number of point clouds in the real-time point cloud map is greater than a point cloud threshold, a point cloud map to be registered is obtained; The point cloud map registration module is used to obtain the model point cloud based on the 3D mesh model file of the real training device, register the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration; The device virtual-reality alignment module is used to correct the XR device pose at each moment according to the pose of the model point cloud in the world coordinate system, obtain the pose of the XR device at each moment in the model coordinate system where the model point cloud is located, and then render a virtual-reality aligned training application.

Images