An MR device virtual-real alignment method and system based on a point cloud map

Abstract

The present invention relates to a method and system for virtual-real alignment of an MR device based on a point cloud map, belonging to the field of mixed reality technology, and solves the problems of cumbersome existing virtual-real alignment operations and low accuracy. It includes collecting environmental data, constructing a point cloud map and importing it into the MR device; matching the images captured by the MR device with the point cloud map, taking the capture moment of the successfully matched image as the initial moment, and calculating the pose of the MR device in the point cloud map and the pose of the MR device in the real-world coordinate system at the initial moment; according to the external reference data of the point cloud map coordinate system relative to the virtual training device coordinate system, the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real-world coordinate system, modifying the pose of the MR device in the real-world coordinate system at each moment after the initial moment to obtain the pose of the MR device in the virtual training device coordinate system at each moment, and then rendering a virtual-real aligned training application. Simple and high-precision virtual-real alignment is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of mixed reality, and in particular, to a method and system for aligning virtual and real of an MR device based on a point cloud map. Background Art

[0002] Mixed Reality (MR) is a technology that deeply integrates the virtual world with the real environment. The core of this technology lies in the integration of virtual and real, that is, the seamless combination of virtual objects and real scenes, and the real-time interaction between users and virtual and real elements. Therefore, mixed reality applications are a specific implementation form of the integration of virtual and real. Through hardware devices (such as head-mounted displays) and software algorithms, virtual information is superimposed on the real environment and widely used in industries such as emergency, education, exhibition, and medical treatment.

[0003] In the MR application, the effect of aligning and integrating virtual and real can be directly visually perceived by the training personnel. In the prior art, calibration marks placed in advance are usually used for virtual-real alignment.

[0004] Since there are strict requirements for the placement position of the calibration marks, if there is an error in the position, it will affect the virtual-real alignment effect; at the same time, the alignment accuracy of a single calibration mark is limited. Setting multiple calibration marks can improve the accuracy but increases additional work. In addition, during the process of using the calibration marks for alignment, the users of the MR application are generally required to remain relatively stationary and maintain an appropriate distance and observation direction from the calibration marks, resulting in a lack of flexibility in the operation of virtual-real alignment. Summary of the Invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a method and system for aligning virtual and real of an MR device based on a point cloud map to solve the problems of cumbersome operation and low accuracy in existing virtual-real alignment.

[0006] On the one hand, the embodiments of the present invention provide a method for aligning virtual and real of an MR device based on a point cloud map, including the following steps:

[0007] Collect the environmental data of the real training device, construct a point cloud map and import it into the MR device;

[0008] Match the image captured by the MR device with the point cloud map, take the capture moment of the successfully matched image as the initial moment, calculate the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real-world coordinate system at the initial moment;

[0009] According to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real scene coordinate system at the initial moment, modify the pose of the MR device in the real scene coordinate system at each moment after the initial moment to obtain the pose of the MR device in the virtual training device coordinate system at each moment;

[0010] Render a training application with virtual and real alignment according to the pose of the MR device in the virtual training device coordinate system at each moment.

[0011] Based on a further improvement of the above method, the pose of the MR device in the virtual training device coordinate system at each moment is calculated by the following formula:

[0012] ,

[0013] where, represents the pose of the MR device in the virtual training device coordinate system at moment represents the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, represents the pose of the MR device in the point cloud map at the initial moment, represents the pose of the MR device in the real scene coordinate system at the initial moment; represents the pose of the MR device in the real scene coordinate system at moment

[0014] Based on a further improvement of the above method, the pose of the MR device in the point cloud map at the initial moment is calculated by the following formula:

[0015] ,

[0016] where, represents the pose of the MR device in the point cloud map at the initial moment, represents the external parameter data of the MR device relative to the camera, represents the pose of the camera in the point cloud map.

[0017] Based on a further improvement of the above method, the pose of the camera in the point cloud map is obtained by calling the PNP algorithm after obtaining multiple groups of 2D-3D matching point pairs between the 2D feature points in the successfully matched image and the 3D map points in the point cloud map, or by triangulating the 2D feature points in the successfully matched image to obtain 3D feature points, and then calling the ICP algorithm after obtaining multiple groups of 3D-3D matching point pairs between the 3D feature points and the 3D map points in the point cloud map.

[0018] Based on further improvements to the above method, the external reference data of the point cloud map coordinate system relative to the virtual training device coordinate system is initially obtained by adjusting the translation positions of the origin of the point cloud map coordinate system on the x, y, and z axes of the virtual training device coordinate system, as well as the Euler angles of the attitude of the origin of the point cloud map coordinate system relative to the x, y, and z axes of the virtual training device coordinate system, until the MR device completes the initial virtual-real alignment.

[0019] Based on further improvements to the above method, the virtual training device coordinate system is constructed by importing the three-dimensional meshed model file of the real training device, with the geometric center of the real training device as the origin, and the right, upper, and rear directions of the real training device as the x, y, and z axis directions respectively.

[0020] Based on further improvements to the above method, the rendered virtual-real aligned training application is obtained by overlaying multiple layers.

[0021] Based on further improvements to the above method, the order of the multiple layers from bottom to top is: virtual layer, perspective window, and real layer; the virtual layer is rendered by the training application deployed in the MR device according to the pose of the MR device in the virtual training device coordinate system at each moment; the perspective window is the contour area of the real training device, determined according to the imported three-dimensional meshed model file of the real training device, and is used to cover the virtual training device in the virtual layer; the real layer is used to display the real image captured by the MR device in the perspective window.

[0022] Based on further improvements to the above method, the pose of the MR device in the virtual training device coordinate system at each moment is periodically updated by re-matching the images captured by the camera on the MR device with the point cloud map, calculating the pose of the MR device in the point cloud map at the new initial moment, and the pose of the MR device in the real scene coordinate system at the new initial moment.

[0023] On the other hand, an embodiment of the present invention provides an MR device virtual-real alignment system based on a point cloud map, including:

[0024] A point cloud map offline construction module, configured to collect the environmental data of the real training device, construct a point cloud map and import it into the MR device;

[0025] A point cloud map matching module, configured to match the images captured by the MR device with the point cloud map, use the capture moment of the successfully matched image as the initial moment, calculate the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real scene coordinate system at the initial moment;

[0026] The MR device pose acquisition module is used to modify the pose of the MR device in the real-world coordinate system at each moment after the initial moment according to the external reference data of the point cloud map coordinate system relative to the virtual training device coordinate system, the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real-world coordinate system at the initial moment, so as to obtain the pose of the MR device in the virtual training device coordinate system at each moment;

[0027] The training application rendering module is used to render a training application with virtual-real alignment according to the pose of the MR device in the virtual training device coordinate system at each moment.

[0028] Compared with the prior art, the present invention can at least achieve one of the following beneficial effects:

[0029] 1. By constructing a point cloud map, rich three-dimensional space information is provided, which helps to improve the accuracy of pose estimation; by matching the image with the point cloud map and taking the shooting moment of the successfully matched image as the initial moment, the pose of the MR device in the point cloud map is more accurately determined, providing a reliable basis for subsequent pose modification.

[0030] 2. By combining the training device, the MR device and the alignment operation, the external reference data for achieving the initial virtual-real alignment is accurately obtained, simplifying the virtual-real alignment process; based on the pose calculated when the image and the point cloud map are successfully matched, the accurate pose of the MR device in the virtual training device coordinate system is calculated in real time, improving the virtual-real alignment accuracy and achieving a persistent virtual-real alignment effect, enhancing the user's perception and interaction with the virtual content.

[0031] In the present invention, the above technical solutions can also be combined with each other to implement more preferred combination schemes. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can be made obvious from the description, or understood by implementing the present invention. The purpose and other advantages of the present invention can be realized and obtained from the content specifically pointed out in the description and the drawings. Description of the Drawings

[0032] The drawings are only for the purpose of showing specific embodiments and are not considered as limiting the present invention. Throughout the drawings, the same reference signs represent the same components;

[0033] Figure 1 It is a flowchart of a method for virtual-real alignment of an MR device based on a point cloud map in Embodiment 1 of the present invention;

[0034] Figure 2 It is an example diagram of an operation panel for the alignment process of a cockpit training application in Embodiment 1 of the present invention. Detailed Embodiments

[0035] The preferred embodiments of the present invention will be specifically described below in conjunction with the accompanying drawings. The accompanying drawings form a part of this application and are used together with the embodiments of the present invention to explain the principles of the present invention, rather than to limit the scope of the present invention.

[0036] Embodiment 1

[0037] A specific embodiment of the present invention discloses a method for aligning virtual and real of an MR device based on a point cloud map. As Figure 1 shown, it includes the following steps:

[0038] S1. Collect the environmental data of the real training device, construct a point cloud map and import it into the MR device.

[0039] It should be noted that the MR device in this embodiment is a head-mounted display device, including: 1 binocular color camera, 2 binocular black-and-white cameras and an IMU (Inertial Measurement Unit) sensor. The binocular color camera is used to capture real-scene images, and the binocular black-and-white cameras and IMU are used to track the MR device in real time and calculate the pose of the MR device in the real-scene coordinate system.

[0040] A training application is deployed in the MR device to provide a virtual-real fusion scenario for the user to conduct simulation training. Exemplarily, the training application is a driver training application. In the real scene (real world), there is a real cockpit training device, and the training application renders a virtual cockpit training device and the external scene of the cockpit in the virtual scene (virtual world); after virtual-real alignment, through the MR device, a scene where the real cockpit is integrated with the virtual external scene of the cockpit can be seen, and various operations of the operator in the real cockpit can be reflected in the driver training application.

[0041] In this embodiment, the environmental data of the real training device is collected by using binocular cameras and an IMU sensor. It can be collected by wearing the MR device, or by using other devices including binocular cameras and an IMU sensor. The environment of the training device includes the external environment and the internal environment of the training device.

[0042] Exemplarily, in the cockpit training application scenario, when the collector wears the MR device and collects the external scene, taking the real cockpit as the center and keeping a distance of 1 meter from it, the camera on the MR device slowly and uniformly rotates around the cockpit for one week; when collecting the internal scene, the collector simulates the actions of the actual use scenario of the training personnel, including but not limited to: sitting on the seat to adjust the sitting posture and observing the instrument panel in the cockpit.

[0043] The collected environmental data includes: the grayscale images collected by the binocular cameras, and the motion data such as acceleration and angular velocity collected by the IMU sensor. Further, a point cloud map is constructed by using the SLAM algorithm.

[0044] It should be noted that the data obtained in the point cloud map and the construction process includes: the three-dimensional coordinates of the 3D map points, the descriptor vectors of the 3D map points, the historical image frame data, the historical key frame data selected from the historical image frame data, the correspondence between the feature points in the historical key frames and the 3D map points, and the bag-of-words vectors of the features of the historical image frames (including historical key frames).

[0045] Import the point cloud map file constructed offline into the MR device as a prerequisite for subsequent virtual-real alignment.

[0046] S2. Match the image captured by the MR device with the point cloud map, use the capture moment of the successfully matched image as the initial moment, calculate the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real-world coordinate system at the initial moment.

[0047] It should be noted that after the MR device is turned on, continuously collect the images captured by the camera on the MR device, and use the same descriptors as when constructing the point cloud map, such as: ORB (Oriented Fast and Rotated BRIEF) descriptor or SIFT (Scale-Invariant Feature Transform) descriptor, to extract 2D feature points and their descriptors from the real-time collected images, and obtain the bag-of-words vectors of the features of the real-time images.

[0048] Match the image captured by the MR device with the point cloud map, including: take the current frame from the real-time collected images and match it with the point cloud map, calculate the similarity according to the bag-of-words vectors of the features of the current frame and the historical key frames. If the similarity exceeds the similarity threshold, then obtain the 3D map points in the point cloud map associated with the feature points of the historical key frames, and match according to the descriptor vectors of the 2D feature points of the current frame and the 3D map points. After the matching is completed, multiple groups of 2D-3D matching point pairs are formed; or, triangulate the 2D feature points in the current frame to obtain 3D feature points, and match the 3D feature points with the 3D map points in the point cloud map. After the matching is completed, multiple groups of 3D-3D matching point pairs are formed.

[0049] When the number of multiple groups of 2D-3D matching point pairs meets the requirements of the PNP algorithm, the paired image is the successfully matched image, and the capture moment of this image is used as the initial moment. Call the PNP algorithm to calculate the pose of the camera in the point cloud map at the initial moment ; or, when the number of multiple groups of 3D-3D matching point pairs meets the requirements of the ICP algorithm, the paired image is the successfully matched image, and the capture moment of this image is used as the initial moment. Call the ICP algorithm to calculate the pose of the camera in the point cloud map at the initial moment ; Then, based on the extrinsic parameter data of the MR device relative to the camera , calculate the pose of the MR device in the point cloud map at the initial moment through the following formula :

[0050] Formula (1).

[0051] It should be noted that while matching the image captured by the MR device with the point cloud map, the position and orientation of the MR device are tracked according to the images collected by the MR device and the IMU sensor, and the six-degree-of-freedom data of the MR device are calculated in real time using the SLAM algorithm. This data is output at a high frequency (such as 1000Hz). In this embodiment, the moment when the image captured by the MR device is successfully matched with the point cloud map is used as the initial moment, and the corresponding six-degree-of-freedom data is taken as the pose of the MR device in the real-world coordinate system at the initial moment .

[0052] S3. Modify the pose of the MR device in the real-world coordinate system at each moment after the initial moment according to the extrinsic parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real-world coordinate system at the initial moment, to obtain the pose of the MR device in the virtual training device coordinate system at each moment.

[0053] It should be noted that the key to the virtual-real alignment in this embodiment is to modify the six-degree-of-freedom data of the MR device (i.e., the pose of the MR device in the real-world coordinate system), transfer the reference coordinate system of the six-degree-of-freedom data to the virtual training device coordinate system, that is, obtain the pose of the MR device in the virtual training device coordinate system, and then transmit it to the training application for rendering and use.

[0054] The virtual training device coordinate system is constructed by importing the three-dimensional meshed model file of the real training device, taking the geometric center of the real training device model as the origin, and taking the right, upper, and rear directions of the real training device as the x, y, and z axis directions respectively. The three-dimensional meshed model file is usually obtained by modeling and meshing the real training device using three-dimensional modeling software and then exporting it, including: vertex data, color data, and triangular mesh data of the model. Exemplarily, the three-dimensional modeling software includes: Maya and Blender.

[0055] This step is based on the pose of the MR device in the point cloud map at the initial moment obtained in step S2 , and the pose of the MR device in the real-world coordinate system at the initial moment , and then combines the extrinsic parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system , the pose of the MR device at each moment after the initial moment in the real-world coordinate system can be modified through the following formula to calculate the pose of the MR device in the virtual training device coordinate system:

[0056] Formula (2),

[0057] where, represents the pose of the MR device in the virtual training device coordinate system at the moment, and represents the pose of the MR device in the real-world coordinate system at the

[0058] S4. Render a training application with virtual and real alignment according to the pose of the MR device in the virtual training device coordinate system at each moment.

[0059] It should be noted that rendering a training application with virtual and real alignment according to the pose of the MR device in the virtual training device coordinate system at each moment is obtained by superimposing multiple layers.

[0060] Specifically, the order of the multiple layers from bottom to top is: virtual layer, perspective window, and real layer; among them, the virtual layer is rendered by the training application deployed in the MR device according to the pose of the MR device in the virtual training device coordinate system at each moment; the perspective window is the contour area of the real training device, determined according to the imported 3D meshed model file of the real training device, and is used to cover the virtual training device in the virtual layer; the real layer is used to display the real image captured by the MR device in the perspective window.

[0061] It should be noted that when the MR device is used for the first time, the translation vector in the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system is 0, and the rotation matrix is the identity matrix. That is to say, the translation position of the origin of the initial point cloud map coordinate system on the x, y, and z axes of the virtual training device coordinate system is 0, and the Euler angles of the pose of the origin of the point cloud map coordinate system relative to the x, y, and z axes of the virtual training device coordinate system (the rotation angles around the coordinate axes) are 0. At this time, it is equivalent that this external parameter data does not take effect. After calculating the pose of the MR device in the virtual training device coordinate system according to formula (2) and passing it to the training application, the interface rendered by the training application cannot achieve virtual and real alignment, and what the user sees in the MR device is the unaligned real training device and virtual training device.

[0062] Therefore, when the MR device is used for the first time, the user triggers the alignment button, first calculates the pose of the MR device in the point cloud map at the initial moment according to the method in step S2 , and After that, a virtual operation panel is displayed in the MR application. The user calculates the corresponding external parameter data by inputting the above 3 translation positions (in meters) and 3 attitude Euler angles (in degrees), and obtains the pose of the MR device in the virtual training device coordinate system according to formula (2). The training application renders an interface with different alignment effects according to the method in step S3. Taking the driving training application as an example, the example diagram of the displayed operation panel is as Figure 2 shown.

[0063] When the user adjusts the above 6 data to a satisfactory alignment effect, save the adjusted data to obtain the external parameter data for realizing the initial virtual-real alignment . When using the MR device subsequently, the external parameter data for realizing the initial virtual-real alignment is automatically loaded, and there is no need to adjust again. Calculate the pose of the MR device in the virtual training device coordinate system at each moment and transmit it to the training application, and then the virtual-real aligned training application can be rendered.

[0064] It should be noted that during the alignment process, it is not necessary for the user to maintain a fixed posture. The user can move normally according to the training operation process, including standing, sitting, moving left and right, forward and backward, and observing at different angles on the training device.

[0065] In this embodiment, the angle fluctuation of the attitude Euler angle can be ensured to be 0.4°, and the fluctuation of the translation position is 1.0 cm. This alignment accuracy ensures that the user can achieve a good alignment effect in each subsequent training, reducing the alignment operation process.

[0066] Preferably, step S2 is executed regularly to re-match the images captured by the camera on the MR device with the point cloud map, calculate the pose of the MR device in the point cloud map at the new initial moment, and the pose of the MR device in the real scene coordinate system at the new initial moment, which are used in formula (2). This effectively avoids the data drift problem of the six-degree-of-freedom data calculated by the SLAM algorithm during long-term operation, improving the stability and accuracy of pose estimation. Since the accuracy fluctuation of the external parameter data is small, the error caused by regular execution is not large, and the change of the rendered interface is not significant, which will not cause discomfort to the user, realizing the effect of persistent virtual-real alignment.

[0067] The interval time for regular execution is considered comprehensively according to the computing power and the usage environment. Exemplarily, it is set to 3 minutes.

[0068] Preferably, considering that the environmental data of the training device may change, which will affect the matching result of the images captured by the MR device and the point cloud map, after the MR device is started, run the point cloud map real-time update program to update the point cloud map according to the environmental data collected by the MR device in real time, avoiding the problem of point cloud map mismatch caused by environmental changes.

[0069] Compared with the prior art, a method for aligning the virtual and real of an MR device based on a point cloud map provided in this embodiment provides rich three-dimensional spatial information by constructing a point cloud map, which helps to improve the accuracy of pose estimation; by matching an image with the point cloud map and using the shooting moment of the successfully matched image as the initial moment, the pose of the MR device in the point cloud map is more accurately determined, providing a reliable basis for subsequent pose modification. Combining the training device, the MR device, and the alignment operation, the external parameter data for achieving the initial virtual-real alignment is accurately obtained, simplifying the virtual-real alignment process; then, based on the pose calculated when the image and the point cloud map are successfully matched, the accurate pose of the MR device in the coordinate system of the virtual training device is calculated in real time, improving the virtual-real alignment accuracy and achieving a persistent virtual-real alignment effect, enhancing the user's perception and interaction with virtual content.

[0070] Embodiment 2

[0071] Another embodiment of the present invention discloses a system for aligning the virtual and real of an MR device based on a point cloud map, so as to implement a method for aligning the virtual and real of an MR device based on a point cloud map in Embodiment 1. The specific implementation manners of each module refer to the corresponding descriptions in Embodiment 1. The system includes:

[0072] A point cloud map offline construction module, configured to collect environmental data of a real training device, construct a point cloud map, and import it into the MR device;

[0073] A point cloud map matching module, configured to match an image captured by the MR device with the point cloud map, use the shooting moment of the successfully matched image as the initial moment, calculate the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real scene coordinate system at the initial moment;

[0074] An MR device pose acquisition module, configured to modify the pose of the MR device in the real scene coordinate system at each moment after the initial moment according to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the pose of the MR device in the point cloud map at the initial moment, and the pose of the MR device in the real scene coordinate system at the initial moment, to obtain the pose of the MR device in the virtual training device coordinate system at each moment;

[0075] A training application rendering module, configured to render a virtual-real aligned training application according to the pose of the MR device in the virtual training device coordinate system at each moment.

[0076] Since the system for aligning the virtual and real of an MR device based on a point cloud map in this embodiment can be mutually referenced with the aforementioned method for aligning the virtual and real of an MR device based on a point cloud map, and the description here is repetitive, it will not be elaborated here. Since the principle of this system embodiment is the same as that of the above method embodiment, this system embodiment also has the corresponding technical effects of the above method embodiment.

[0077] Those skilled in the art can understand that all or part of the processes for implementing the methods of the above embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.

[0078] As mentioned above, only the preferred specific embodiments of the present invention are described, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for MR device virtual-real alignment based on point cloud map, characterized in that: The following steps are involved: Collect environmental data of real training equipment, build point cloud maps and import them into MR equipment; Matching the image captured by the MR device with the point cloud map, taking the shooting time of the successfully matched image as the initial time, and calculating the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real scene coordinate system at the initial time; According to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the position and posture of the MR device in the point cloud map at the initial moment, and the position and posture of the MR device in the real scene coordinate system at the initial moment, the position and posture of the MR device in the real scene coordinate system at each moment after the initial moment is modified to obtain the position and posture of the MR device in the virtual training device coordinate system at each moment; Rendering a virtual-real aligned training application based on the position of the MR device in the virtual training device coordinate system at each moment; The position and posture of the MR device in the virtual training device coordinate system at each moment is calculated by the following formula: ， in, express The position and posture of the MR device in the virtual training device coordinate system at this moment, Represents the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, Indicates the position of the MR device in the point cloud map at the initial moment, Indicates the position and posture of the MR device in the real-scene coordinate system at the initial moment; express The position and posture of the MR device in the real-scene coordinate system at that moment; The position and posture of the MR device in the point cloud map at the initial moment is calculated by the following formula: ， in, Represents the external parameter data of the MR device relative to the camera, Represents the position of the camera in the point cloud map.

2. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1, characterized in that: The position and posture of the camera in the point cloud map is obtained by calling the PNP algorithm after obtaining multiple groups of 2D-3D matching point pairs between 2D feature points in the successfully matched image and 3D map points in the point cloud map, or by triangulating the 2D feature points in the successfully matched image to obtain 3D feature points, obtaining multiple groups of 3D-3D matching point pairs between the 3D feature points and 3D map points in the point cloud map, and then calling the ICP algorithm.

3. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1, characterized in that: The external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system is initially obtained by adjusting the translation position of the origin of the point cloud map coordinate system in the x, y and z axes of the virtual training device coordinate system, and the posture Euler angles of the origin of the point cloud map coordinate system relative to the x, y and z axes of the virtual training device coordinate system until the MR device completes the initial virtual-real alignment.

4. The method for MR device virtual-real alignment based on point cloud map according to claim 1, characterized in that: The virtual training device coordinate system is constructed by importing a three-dimensional grid model file of a real training device, taking the geometric center of the real training device as the origin, and taking the right, top and back of the real training device as the x, y and z axis directions respectively.

5. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1, characterized in that: The training application for rendering virtual-real alignment is obtained by superimposing multiple layers.

6. The method for MR device virtual-real alignment based on point cloud map according to claim 5, characterized in that: The order of the multiple layers from bottom to top is: virtual layer, perspective window and real layer; the virtual layer is rendered by the training application deployed in the MR device according to the position of the MR device in the virtual training device coordinate system at each moment; the perspective window is the outline area of ​​the real training device, which is determined according to the imported three-dimensional grid model file of the real training device and is used to cover the virtual training device in the virtual layer; The real layer is used to display the real image captured by the MR device in the perspective window.

7. The method for MR device virtual-real alignment based on point cloud map according to claim 1, characterized in that: The position and posture of the MR device in the virtual training device coordinate system at each moment is regularly updated by regularly re-matching the image taken by the camera on the MR device with the point cloud map, calculating the new initial position and posture of the MR device in the point cloud map, and the new initial position and posture of the MR device in the real scene coordinate system.

8. A point cloud map-based MR device virtual-real alignment system, characterized in that: include: The offline point cloud map construction module is used to collect environmental data of real training equipment, build point cloud maps and import them into MR equipment; A point cloud map matching module is used to match the image captured by the MR device with the point cloud map, take the shooting time of the successfully matched image as the initial time, calculate the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real scene coordinate system at the initial time; The MR device posture acquisition module is used to modify the posture of the MR device in the real scene coordinate system at each moment after the initial moment according to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the posture of the MR device in the point cloud map at the initial moment, and the posture of the MR device in the real scene coordinate system at the initial moment, so as to obtain the posture of the MR device in the virtual training device coordinate system at each moment; A training application rendering module, used to render a virtual-real aligned training application according to the position of the MR device in the virtual training device coordinate system at each moment; The position and posture of the MR device in the virtual training device coordinate system at each moment is calculated by the following formula: ， in, express The position and posture of the MR device in the virtual training device coordinate system at this moment, Represents the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, Indicates the position of the MR device in the point cloud map at the initial moment, Indicates the position and posture of the MR device in the real-scene coordinate system at the initial moment; express The position and posture of the MR device in the real-scene coordinate system at that moment; The position and posture of the MR device in the point cloud map at the initial moment is calculated by the following formula: ， in, Represents the external parameter data of the MR device relative to the camera, Represents the position of the camera in the point cloud map.

Images