A full-automatic XR device virtual-real alignment method and system

Abstract

The present application relates to a kind of full-automatic XR equipment virtual-real alignment method and system, belong to mixed reality technical field, solve the problem of low efficiency and limited alignment accuracy caused by manual or semi-automatic alignment.The present application includes: according to the real-time point cloud map under the world coordinate system of real-time acquisition environment data, when the point cloud quantity in real-time point cloud map is greater than point cloud threshold, obtain the point cloud map to be registered;According to the model point cloud of the three-dimensional grid model file of real training equipment, the model point cloud is registered with the point cloud map to be registered, and the pose of model point cloud in world coordinate system is obtained after successful registration;According to the pose of model point cloud in world coordinate system, the pose of XR equipment at each time is corrected, and the pose of XR equipment at each time in the model coordinate system of model point cloud is obtained, and then the training application of virtual-real alignment is rendered.The present application realizes full-automatic and high-precision virtual-real alignment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of extended reality, and particularly relates to a full-automatic XR device virtual-real alignment method and system. BACKGROUND

[0002] XR (Extended Reality) is a collective term for virtual reality (VR), augmented reality (AR), mixed reality (MR) and all technologies that can create and experience virtual or augmented reality technologies. XR devices cover a variety of different types of devices, and have wide application prospects in many fields such as games, education, medical treatment, architectural design, etc., in which virtual-real alignment is a key to the evolution of XR device technology.

[0003] The virtual object on the existing XR device is usually aligned with the real scene with human intervention, which has great limitations. For example, some manual alignment methods are tedious and inefficient, and require high experience of the operator; some semi-automatic alignment methods need to manually adjust the offset position and posture for alignment at the first virtual-real alignment, and can realize automatic alignment in subsequent use. However, this method still requires a lot of user intervention, and it is difficult to achieve full-automatic alignment.

[0004] Manual and semi-automatic alignment methods are insufficient in precision, efficiency and adaptability in complex scenes. In addition, when there are multiple real training devices in the real space, the XR device may be aligned to the wrong training device. SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide a full-automatic XR device virtual-real alignment method and system to solve the problems of low efficiency and limited alignment accuracy caused by manual or semi-automatic alignment.

[0006] In one aspect, the embodiments of the present application provide a full-automatic XR device virtual-real alignment method, comprising the following steps:

[0007] Constructing a real-time point cloud map in a world coordinate system according to real-time collected environment data, when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold, obtaining a point cloud map to be registered;

[0008] Obtaining a model point cloud according to a three-dimensional mesh model file of a real training device, registering the model point cloud with the point cloud map to be registered, and obtaining a pose of the model point cloud in the world coordinate system after successful registration;

[0009] Correcting the pose of the XR device at each time according to the pose of the model point cloud in the world coordinate system, obtaining the pose of the XR device at each time in a model coordinate system in which the model point cloud is located, and rendering a training application of virtual-real alignment.

[0010] Based on the further improvement of the above method, the model point cloud is registered with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system, comprising:

[0011] The point cloud map to be registered is segmented to obtain a sub-region with the minimum distance from the XR device as a target point cloud;

[0012] The model point cloud is taken as a source point cloud, and a plurality of initial poses are constructed for the source point cloud in the world coordinate system;

[0013] Based on each initial pose respectively, the source point cloud is registered with the target point cloud respectively to obtain each optimized pose and a registration score; if the maximum registration score is not less than a score threshold, the registration is successful, and the optimized pose corresponding to the maximum registration score is the pose of the model point cloud in the world coordinate system; otherwise, the registration fails, the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum registration score is not less than the score threshold.

[0014] Based on the further improvement of the above method, the point cloud map to be registered is segmented to obtain a sub-region with the minimum distance from the XR device as a target point cloud, comprising:

[0015] According to the volume of the model point cloud and a minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-region within the error range of the volume of the model point cloud is reserved as a candidate region; the distance between each candidate region and the current XR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

[0016] Based on the further improvement of the above method, a plurality of initial poses are constructed for the source point cloud in the world coordinate system, comprising: setting a heading angle from 0 to increment, and setting a pitch angle and a roll angle to be 0 degrees to obtain a plurality of second rotation matrices; obtaining a second translation vector according to the position of the target point cloud; combining the plurality of second rotation matrices with the second translation vector into a plurality of second transformation matrices to obtain the plurality of initial poses.

[0017] Based on the further improvement of the above method, after the registration is successful, it is periodically detected whether the model point cloud needs to be registered with the target point cloud again, including: obtaining the coordinates of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score as the model registration position, if the distance between the position of the XR device and the model registration position is less than the first distance threshold, it is not necessary to register again; if the distance between the position of the XR device and the model registration position is greater than the second distance threshold, and the minimum distance between the position of the XR device and the candidate region of the non-target point cloud is less than the first distance threshold, the candidate region corresponding to the minimum distance is updated as the target point cloud, and the model point cloud is registered with the target point cloud again.

[0018] Based on the further improvement of the above method, the source point cloud is respectively registered with the target point cloud based on each initial pose, which is realized by using multi-threading, and in each thread, a weighted ICP algorithm is used to iteratively optimize the respective initial pose, so that the position of the source point cloud in the target point cloud is optimized to obtain the corresponding optimized pose and registration score.

[0019] Based on the further improvement of the above method, the weighted ICP algorithm adopts a dynamic weight adjustment mechanism and combines a Huber error function to construct a target function, takes the target function as an optimization target, and minimizes the registration error between the point clouds by iterative solving; the dynamic weight is obtained by calculating the normal and curvature of the point pair.

[0020] Based on the further improvement of the above method, the dynamic weight is calculated by the following formula:

[0021]

[0022] Wherein, w i represents the dynamic weight of the i-th point pair (p i , q i ), α represents a proportion coefficient, w normal represents a normal weight, w curvature represents a curvature weight, θ i represents the normal angle of the point pair (p i , q i ), and σ represents an adjustment parameter; κ i represents the curvature calculated according to the neighborhood point set of p i in the source point cloud, and β represents a proportion coefficient.

[0023] Based on the further improvement of the above method, the pose of the XR device at each time is corrected according to the pose of the model point cloud in the world coordinate system, to obtain the pose of the XR device in the model coordinate system in which the model point cloud is located, and the formula is as follows:

[0024]

[0025] wherein, represents the pose of the XR device in the model coordinate system at time t, represents the pose of the XR device at time t, represents the pose of the model point cloud in the world coordinate system.

[0026] On the other hand, the embodiment of the present application provides a full-automatic XR device virtual-real alignment system, comprising:

[0027] a point cloud map construction module, configured to construct a real-time point cloud map in a world coordinate system according to real-time collected environment data, and obtain a point cloud map to be registered when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold;

[0028] a point cloud map registration module, configured to obtain a model point cloud according to a three-dimensional grid model file of a real training device, register the model point cloud with the point cloud map to be registered, and obtain the pose of the model point cloud in the world coordinate system after successful registration;

[0029] a device virtual-real alignment module, configured to correct the pose of the XR device at each time according to the pose of the model point cloud in the world coordinate system, obtain the pose of the XR device in a model coordinate system in which the model point cloud is located at each time, and further render a training application in virtual-real alignment.

[0030] Compared with the prior art, the present application can at least achieve one of the following beneficial effects:

[0031] 1. By registering the point cloud of the three-dimensional grid model of the real training device with the real-time point cloud map and coordinate system transformation and other operations, the pose of the XR device is adjusted in real time, the alignment deviation caused by device movement is avoided, full-automatic and high-precision virtual-real alignment is achieved, the operation steps and time are reduced, and the user can enter the immersive experience more quickly.

[0032] 2. By constructing initial poses in multiple initial directions, multiple possible attitudes of the point cloud in space are covered, so as to improve the possibility of finding correct matching, effectively improve the registration success rate, precision and robustness, and it is especially suitable for complex scenes and high-precision applications.

[0033] In the present application, the above-mentioned technical solutions can be combined with each other to realize more preferred combination schemes. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification or be understood by implementing the present application. The purpose and other advantages of the present application can be achieved and obtained from the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are used to explain the principles of the application, but are not intended to limit the scope of the application.

[0035] Figure 1 A flow chart of a full-automatic XR device virtual-real alignment method in Embodiment 1 of the application;

[0036] Figure 2 A structural schematic diagram of a full-automatic XR device virtual-real alignment system in Embodiment 2 of the application. DETAILED DESCRIPTION

[0037] The preferred embodiments of the application will be described in detail with reference to the drawings, wherein the drawings form a part of the specification, illustrate embodiments of the application and are used to explain the principles of the application, but are not intended to limit the scope of the application.

[0038] Embodiment 1

[0039] In one specific embodiment of the application, a full-automatic XR device virtual-real alignment method is disclosed, as shown in the flow chart, comprising the following steps: Figure 1

[0040] S1, constructing a real-time point cloud map under a world coordinate system according to real-time collected environment data, and obtaining a point cloud map to be registered when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold.

[0041] It should be noted that the XR device in this embodiment is a head-mounted display device, which comprises one binocular color camera, two binocular monochrome cameras and an IMU (Inertial Measurement Unit) sensor. The binocular color camera is used to shoot real scene images, and the binocular monochrome camera and the IMU are used to track the XR device in real time and calculate the pose of the XR device in the world coordinate system. Exemplarily, the XR device is AR glasses or MR glasses.

[0042] A training application is deployed in the XR device, which is used to provide a scenario of virtual-real fusion for simulation training for a user. Exemplarily, the training application is a driver training application. There is a real cockpit training device in a real scene (real world), and the training application renders a virtual cockpit training device and a cockpit external scene in a virtual scene (virtual world). After virtual-real alignment, the real cockpit and the virtual cockpit external scene are fused together through the XR device, and various operations in the real cockpit can be reflected in the driver training application.

[0043] ​The embodiment collects environment data of the real training device by wearing an XR device. A SLAM algorithm is deployed in the XR device to output six degrees of freedom (6DOF) pose data at the exposure time of each frame of image, and the pose of the XR device is obtained, i.e., the pose in the world coordinate system of the SLAM. The SLAM algorithm itself also establishes a point cloud map, but the XR device only retains a sparse point cloud map for efficient operation, and cannot display the geometric structure of the real training device.

[0044] Therefore, in this step, a real-time mapping algorithm is used to extract and match feature points (such as corner points, SIFT features, etc.) from each frame of image based on multiple frames of environment images collected by a binocular black-and-white camera. After feature triangulation, a point cloud set in the camera coordinate system is recovered. Then, the 6DOF pose of the XR device at the exposure time of each frame of image and the extrinsic pose of the camera and the XR device are used to convert the point cloud set to the world coordinate system of the SLAM. This process is repeated at consecutive times to construct a dense point cloud map in the world coordinate system, which is convenient for reflecting the geometric information of the real training device.

[0045] Further, in order to ensure the real-time performance of the point cloud map, the constructed point cloud map is used as a global point cloud map. After identifying a key frame from each frame of image collected in real time, the feature points on the key frame are converted to the world coordinate to obtain a new point cloud. The new point cloud is matched with the point cloud map by using a point cloud registration algorithm. The unmatched new point cloud is added to the global point cloud map, and the updated global point cloud map is subjected to voxel filtering processing to ensure that only one point coordinate is retained for the same spatial point in a certain scale of volume space.

[0046] In the above process, whether the current frame is a key frame is identified based on the pose data of the XR device at the current frame and the previous frame. The position difference in the pose data is compared with a distance threshold, and the attitude difference is compared with an attitude threshold. If both the position difference and the attitude difference are greater than the corresponding threshold, the current frame is a key frame.

[0047] It should be noted that whether the attitude difference is greater than the attitude threshold is determined by converting the rotation in the pose data into a corresponding Lie algebra vector, and comparing the module length of the Lie algebra vector with the attitude threshold.

[0048] Preferably, in order to make the real-time point cloud map more obviously reflect the geometric information of the real training device, the surface of the real training device is textured to better extract feature points and recover the 3D point cloud of the real training device.

[0049] With the addition of the point cloud of each subsequent key frame, the real-time point cloud map is constantly updated and improved. When the number of point clouds in the real-time point cloud map is greater than the point cloud threshold, the point cloud map to be registered is obtained, and the registration operation of step S2 is performed. At the same time, step S1 also continues to update the real-time point cloud map.

[0050] S2, obtaining a model point cloud according to the three-dimensional mesh model file of the real training device, registering the model point cloud with the point cloud map to be registered, and obtaining the pose of the model point cloud in the world coordinate system.

[0051] It should be noted that the three-dimensional mesh model file of the real training device is imported in the XR device. The three-dimensional mesh model file is usually obtained by modeling and meshing the real training device using a three-dimensional modeling software, and includes vertex data, color data and triangular mesh data of the model. Exemplarily, the three-dimensional modeling software includes Maya and Blender.

[0052] Further, the PCL library (an open source library specifically for point cloud processing) is used to generate the model point cloud by sampling the three-dimensional mesh model file as input. Exemplarily, the pcl_mesh_sampling function in the PCL library is called to generate the model point cloud.

[0053] It should be noted that the vertex coordinates of the model are set relative to the origin of the three-dimensional mesh model. Therefore, the origin of the coordinate system of the generated model point cloud is consistent with the origin in the three-dimensional mesh model, and the model coordinate system in which the model point cloud is located is constructed based on the origin using the right-hand rule. Exemplarily, the right, top and back of the model are used as the x, y and z axis directions of the coordinate system, respectively, according to the orientation of the three-dimensional mesh model.

[0054] Further, considering that there may be multiple training devices in the real-time point cloud map, and if the initial pose of registration is not good, it will lead to high registration error or failure to converge. Therefore, the embodiment divides the point cloud map to be registered to obtain the training device closest to the XR device, and constructs multiple initial poses to facilitate finding the best initial pose.

[0055] Specifically, the model point cloud is registered with the point cloud map to be registered to obtain the pose of the model point cloud in the world coordinate system, including:

[0056] ①Divide the point cloud map to be registered to obtain a sub-region with the smallest distance from the XR device as the target point cloud.

[0057] According to the volume of the model point cloud and the minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-region within the error range of the volume of the model point cloud is reserved as a candidate region; the distance between each candidate region and the current XR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

[0058] It should be noted that the volume of the model point cloud is the volume of the space surrounded by the model point cloud, which is obtained by calculating the convex hull volume of the model point cloud. There are many methods for segmenting the point cloud map to be registered, and the embodiment does not limit the segmentation method. Preferably, the Euclidean clustering method is used and the KD-Tree or Octree is constructed to accelerate the neighborhood query to segment the point cloud map to be registered to obtain a plurality of sub-regions. The volume of each sub-region is obtained by obtaining the convex hull of each sub-region, if the volumes of adjacent sub-regions are all less than the minimum volume threshold, they are merged into a new sub-region and the volume is recalculated; if the volume of the sub-region is within the error range of the volume of the model point cloud, such as within ±20% of the volume of the model point cloud, the sub-region is taken as a candidate region.

[0059] Further, the position of each candidate region is the centroid position of each candidate region, which is obtained by calculating the average position of all point clouds in the candidate region, and the position of the current XR device is obtained according to the 6DOF pose of the XR device. The Euclidean distance between each candidate region and the current XR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud, i.e. the model point cloud needs to be registered with the target point cloud.

[0060] ②Take the model point cloud as the source point cloud, and construct a plurality of initial poses for the source point cloud in the world coordinate system.

[0061] Based on the world coordinate system, set the heading angle from 0 to increment by interval, and set the pitch angle and roll angle to 0 degrees to obtain a plurality of second rotation matrices; obtain a second translation vector according to the position of the target point cloud; combine the plurality of second rotation matrices with the second translation vector into a plurality of second transformation matrices to obtain a plurality of initial poses.

[0062] Specifically, for each heading angle, a 4x4 second transformation matrix containing the second rotation matrix and the second translation vector is constructed to obtain an initial pose.

[0063] Exemplarily, the heading angle is set to increment by 10 degrees in the range of [0, 360] degrees, and 36 rotation matrices around the axis perpendicular to the ground are obtained.

[0064] iii. registering the source point cloud and the target point cloud based on each initial pose respectively to obtain each optimized pose and a registration score; if the maximum registration score is not less than a score threshold, the optimized pose corresponding to the maximum registration score is the pose of the model point cloud in the world coordinate system, otherwise, updating the point cloud map to be registered according to the latest real-time point cloud map, and registering the model point cloud and the point cloud map to be registered again until the maximum registration score is not less than the score threshold.

[0065] It should be noted that since there are multiple initial poses, the embodiment utilizes multi-threading, and simultaneously in each thread, the weighted ICP algorithm is used to iteratively optimize the respective initial pose, so that the position of the source point cloud in the target point cloud is optimal to obtain the corresponding optimized pose and registration score, which realizes the correction of multiple initial poses at the same time and improves the calculation efficiency.

[0066] Considering that in the registration process, the matching point pairs with consistent normal directions are more likely to belong to the same physical surface, and their corresponding relationship is more reliable; in the flat area (i.e. low curvature area), point pair matching is more stable, and should be given higher weight; in the high curvature area (such as the edge), the weight should be reduced to reduce noise interference. Therefore, the embodiment gives different dynamic weights to the point pairs according to the normal and curvature of the point pairs, so that the point pair with consistent normal has higher registration accuracy, and uses the Huber error function to reduce the influence of outliers, thereby improving the overall registration effect. That is, the weighted ICP algorithm of the embodiment adopts a dynamic weight adjustment mechanism and combines a Huber error function to construct an objective function, takes the objective function as an optimization target, and minimizes the registration error between the point clouds through iterative solution.

[0067] Specifically, p i is the three-dimensional coordinates of the i th point in the source point cloud, q i is the three-dimensional coordinates of the nearest point corresponding to p i in the target point cloud, and the dynamic weight of the point pair (p i , q i ) is obtained by calculating the normal and curvature of the point pair (p i , q i ) through the following formula:

[0068]

[0069] wherein w i represents the dynamic weight of the i th point pair (p i , q i ), a represents a proportionality coefficient, w normal represents the normal weight, w curvature represents the curvature weight, and θ i represents the angle between the normal of the point pair (p i , q ithe normal included angle of the normal of the point p i represents the curvature calculated according to the neighborhood point set of p i in the source point cloud, and β represents a proportional coefficient.

[0070] Further, the target function E(R', t') obtained by combining the weight with the Huber error function is as follows:

[0071]

[0072] where (R', t') is the pose to be solved, R' represents a rotation matrix, t' represents a translation vector, L σ (·) represents the Huber error function, n represents the number of point pairs, and ‖·‖ represents the L2 norm.

[0073] Finally, the optimized pose after registration of each source point cloud and the target point cloud is solved according to the target function of formula (2), and the registration score is calculated. The better the registration effect is, the higher the registration score is.

[0074] Exemplarily, the registration score is obtained by calculating the average or sum of the registration probabilities of all point pairs, or by calculating the average square error of all point pairs and then taking the reciprocal.

[0075] When the maximum registration score is not less than the score threshold, it indicates that the registration is successful, and the corresponding optimized pose is the pose of the model point cloud in the world coordinate system of SLAM. At this time, the coordinate of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score is obtained as the model registration position.

[0076] In order to avoid the problem of incorrect alignment in subsequent use, causing the user to not be able to see the fused visual scene on the training device, according to the distance between the position of the XR device and the model registration position, and the minimum distance between the position of the XR device and the candidate region, it is periodically detected whether the model point cloud and the target point cloud need to be registered again.

[0077] Specifically, if the distance between the position of the XR device and the model registration position is less than a first distance threshold, it is not necessary to register again, if the distance between the position of the XR device and the model registration position is greater than a second distance threshold, and the minimum distance between the position of the XR device and the candidate region of the non-target point cloud is less than the first distance threshold, the candidate region corresponding to the minimum distance is updated as the target point cloud, and the model point cloud and the target point cloud are registered again.

[0078] Preferably, it is detected once every 5 seconds, the first distance threshold is 1 meter, and the second distance threshold is 2 meters.

[0079] S3, correct the XR device pose at each time according to the pose of the model point cloud in the world coordinate system, obtain the pose of the XR device at each time in the model coordinate system where the model point cloud is located, and then render the training application with virtual-real alignment.

[0080] It should be noted that the pose of the XR device at each time in the model coordinate system where the model point cloud is located is calculated by the following formula:

[0081]

[0082] wherein, represents the pose of the XR device at time t in the model coordinate system, represents the pose of the XR device at time t, i.e. the 6DOF pose in the world coordinate system of SLAM, represents the pose of the model point cloud in the world coordinate system of SLAM.

[0083] According to the pose of the XR device at each time in the model coordinate system, the training application with virtual-real alignment is obtained by superimposing multiple layers.

[0084] It should be noted that the order of the multiple layers from bottom to top is: virtual layer, perspective window layer and real layer; wherein the virtual layer is the image rendered by the training application deployed in the XR device according to the pose of the XR device at each time in the model coordinate system; the perspective window layer is the outline area of the real training device, which is a region rendered according to the three-dimensional gridding model file of the real training device, used to cover the virtual training device in the virtual layer; the real layer is used to display the real image taken by the XR device in the perspective window. Virtual-real alignment is the effect produced by aligning the real image and the virtual image based on two training devices with the same size and appearance in space.

[0085] Specifically, the view matrix of the virtual training device is taken as so that the relative pose of the XR device and the real training device in the real space is consistent with the relative pose of the model of the XR device and the real training device in the virtual space. By the conventional rendering process, the perspective window layer and the real layer are superimposed, so that only the real image in the outline coverage area of the real training device is retained. At this time, in the visual of the training personnel, the outline of the virtual training device and the outline of the real training device achieve the effect of coincidence. At the same time, is transmitted to the training application, and the training application renders the virtual image combined with the real image in the outline coverage area, so as to achieve the effect of virtual-real alignment, so that when the training personnel look at the training device through the XR device, they see the real image in the real space, and when they look at the outside of the training device, they see the virtual training task image.

[0086] Steps S1-S3 in this embodiment are a process of automatically aligning virtual and real, which is usually not required to be adjusted by personnel, but considering the difference of human senses, when the personnel are not satisfied with the effect of virtual and real alignment rendered in step S3, fine adjustment alignment is performed, including: adjusting through 3 translation positions (unit: meters) and 3 rotation Euler angles (unit: degrees) around x, y and z axes on the user operation panel, calculating the corresponding offset pose matrix, modifying the pose of the XR device in the model coordinate system obtained from formula (3), and then rendering the interface of the fine-adjusted virtual and real alignment effect.

[0087] Compared with the prior art, the full-automatic XR device virtual and real alignment method provided in the embodiment adjusts the XR device pose in real time through registration and coordinate system transformation of the point cloud of the three-dimensional grid model of the real training device and the real-time point cloud map, avoids alignment deviation caused by device movement, realizes full-automatic and high-precision virtual and real alignment, reduces operation steps and time, and enables users to enter immersive experience more quickly. By constructing an initial pose at multiple initial headings, the possible poses of the point cloud in space are covered, thereby improving the possibility of finding correct matching, effectively improving the registration success rate, precision and robustness, and being especially suitable for complex scenes and high-precision applications.

[0088] Embodiment 2

[0089] Another embodiment of the application discloses a full-automatic XR device virtual and real alignment system, so as to realize the full-automatic XR device virtual and real alignment method in embodiment 1. The specific implementation modes of the modules are referred to the corresponding descriptions in embodiment 1. As shown in Figure 2 The system comprises:

[0090] The point cloud map construction module 101 is configured to construct a real-time point cloud map in a world coordinate system according to real-time collected environment data, and obtain a point cloud map to be registered when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold;

[0091] The point cloud map registration module 102 is configured to obtain model point clouds according to a three-dimensional grid model file of a real training device, register the model point clouds with the point cloud map to be registered, and obtain a pose of the model point clouds in the world coordinate system after successful registration;

[0092] The device virtual and real alignment module 103 is configured to correct the pose of the XR device at each time according to the pose of the model point clouds in the world coordinate system, obtain the pose of the XR device in a model coordinate system in which the model point clouds are located at each time, and then render a training application of virtual and real alignment.

[0093] Since the automatic XR device virtual-real alignment system of the embodiment is related to the automatic XR device virtual-real alignment method described above, the two can be used as reference to each other, and thus, the description is not repeated here. Since the system embodiment has the same principle as the method embodiment described above, the system embodiment also has the corresponding technical effects of the method embodiment described above.

[0094] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0095] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A full-automatic XR device virtual-real alignment method, characterized in that, The method comprises the following steps: According to the real-time collected environmental data, a real-time point cloud map in a world coordinate system is constructed, and when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold, a point cloud map to be registered is obtained; According to a three-dimensional grid model file of a real training device, a model point cloud is obtained, the model point cloud is registered with the point cloud map to be registered, and after successful registration, a pose of the model point cloud in the world coordinate system is obtained, comprising: segmenting the point cloud map to be registered, obtaining a sub-region with the smallest distance from an XR device as a target point cloud; taking the model point cloud as a source point cloud, constructing a plurality of initial poses of the source point cloud in the world coordinate system; based on each initial pose, respectively registering the source point cloud with the target point cloud to obtain each optimized pose and a registration score; if the maximum registration score is not less than a score threshold, the registration is successful, and the optimized pose corresponding to the maximum registration score is the pose of the model point cloud in the world coordinate system; otherwise, the registration fails, the point cloud map to be registered is updated according to the latest real-time point cloud map, and the model point cloud is registered with the point cloud map to be registered again until the maximum registration score is not less than the score threshold; According to the pose of the model point cloud in the world coordinate system, the poses of the XR device at each time are corrected, the poses of the XR device at each time in a model coordinate system in which the model point cloud is located are obtained through the following formula, and a training application with virtual-real alignment is rendered: ， wherein, represents pose of the XR device in the model coordinate system at time instant X, represents pose of the XR device at time instant X, represents pose of the model point cloud in the world coordinate system.

2. The full-automatic XR device virtual-real alignment method of claim 1, wherein, The segmentation of the point cloud map to be registered to obtain a sub-region with the smallest distance from the XR device as a target point cloud comprises: According to the volume of the model point cloud and a minimum volume threshold, the point cloud map to be registered is segmented, the volume of each sub-region is calculated and compared with the volume of the model point cloud, and the sub-region within the error range of the volume of the model point cloud is reserved as a candidate region; the distance between each candidate region and the current XR device is calculated, and the candidate region corresponding to the minimum distance is selected as the target point cloud.

3. The full-automatic XR device virtual-real alignment method of claim 1, wherein, The construction of a plurality of initial poses of the source point cloud in the world coordinate system comprises: setting a heading angle from 0 to increment by an interval, and setting the pitch angle and the roll angle to be 0 degrees to obtain a plurality of second rotation matrices; obtaining a second translation vector according to the position of the target point cloud; combining the plurality of second rotation matrices with the second translation vector into a plurality of second transformation matrices to obtain the plurality of initial poses.

4. The full-automatic XR device virtual-real alignment method of claim 1, wherein, After successful registration, whether the model point cloud needs to be registered with the target point cloud again is periodically detected, comprising: obtaining the coordinates of the origin of the source point cloud in the target point cloud corresponding to the maximum registration score as a model registration position, if the distance between the position of the XR device and the model registration position is less than a first distance threshold, the registration does not need to be performed again; if the distance between the position of the XR device and the model registration position is greater than a second distance threshold, and the minimum distance between the position of the XR device and a candidate region other than the target point cloud is less than the first distance threshold, the candidate region corresponding to the minimum distance is updated as the target point cloud, and the model point cloud is registered with the target point cloud again.

5. The full-automatic XR device virtual-real alignment method of claim 1, wherein, The source point cloud is respectively registered with the target point cloud based on each initial pose respectively, which is achieved by using multi-threading, and in each thread, a weighted ICP algorithm is used to iteratively optimize the respective corresponding initial pose, so that the position of the source point cloud in the target point cloud is optimized to obtain the corresponding optimized pose and registration score.

6. The full-automatic XR device virtual-real alignment method of claim 5, wherein, The weighted ICP algorithm adopts a dynamic weight adjustment mechanism and combines a Huber error function to construct a target function, takes the target function as an optimization target, and minimizes the registration error between the point clouds by iterative solving; the dynamic weight is obtained by calculating the normal and curvature of the point pair.

7. The full-automatic XR device virtual-real alignment method of claim 6, wherein, The dynamic weight is calculated by the following formula: ， wherein, denotes the dynamic weight of the i-th point pair , denotes a proportionality coefficient, denotes a normal weight, denotes a curvature weight, denotes the normal included angle of the point pair , denotes an adjustment parameter; denotes the curvature calculated according to the neighborhood point set of in the source point cloud, denotes a proportionality coefficient.

8. A fully automatic XR device virtual-real alignment system, characterized by, It comprises: A point cloud map construction module, configured to construct a real-time point cloud map in a world coordinate system according to real-time collected environment data, and obtain a point cloud map to be registered when the number of point clouds in the real-time point cloud map is greater than a point cloud threshold; A point cloud map registration module, configured to obtain model point clouds from a three-dimensional grid model file of a real training device, register the model point clouds with the point cloud map to be registered, and obtain a pose of the model point clouds in the world coordinate system after successful registration, comprising: segmenting the point cloud map to be registered to obtain a sub-region with the smallest distance from the XR device as a target point cloud; constructing a plurality of initial poses for the source point cloud in the world coordinate system by taking the model point clouds as the source point cloud; registering the source point cloud with the target point cloud based on each initial pose respectively to obtain each optimized pose and registration score; if the maximum registration score is not less than a score threshold, the registration is successful, and the optimized pose corresponding to the maximum registration score is the pose of the model point clouds in the world coordinate system; otherwise, the registration fails, the point cloud map to be registered is updated according to the latest real-time point cloud map, the model point clouds are registered with the point cloud map to be registered again until the maximum registration score is not less than the score threshold; A virtual-real alignment module, configured to correct the pose of the XR device at each time according to the pose of the model point clouds in the world coordinate system, obtain the pose of the XR device in the model coordinate system at each time by the following formula, and render a virtual-real aligned training application: ， wherein, denotes pose of the XR device in the model coordinate system at time instant denotes pose of the XR device at time instant denotes pose of the model point cloud in the world coordinate system.

Images