MR equipment virtual-real alignment method and system based on point cloud map

Abstract

The invention relates to an MR equipment virtual-real alignment method and system based on a point cloud map, belongs to the technical field of mixed reality, and solves the problems of tedious operation and low precision of existing virtual-real alignment. Comprising the steps of collecting environment data, constructing a point cloud map and importing the point cloud map into MR equipment; matching the image shot by the MR equipment with the point cloud map, taking the shooting moment of the successfully matched image as the initial moment, and calculating the pose of the MR equipment in the point cloud map and the pose of the MR equipment in the live-action coordinate system at the initial moment; according to the external parameter data of the coordinate system of the point cloud map relative to the coordinate system of the virtual training device, the pose of the MR device in the point cloud map at the initial moment and the pose of the MR device in the live-action coordinate system, the pose of the MR device in the live-action coordinate system at each moment after the initial moment is modified, and the pose of the MR device in the coordinate system of the virtual training device at each moment is obtained. And furthermore, a training application of virtual-real alignment is rendered. And simple and high-precision virtual and real alignment is realized.

Description

Technical Field

[0001] The present invention relates to the field of mixed reality technology, and in particular to a method and system for aligning virtual and real MR devices based on a point cloud map. Background Art

[0002] Mixed Reality (MR) is a technology that deeply integrates the virtual world with the real environment. The core of this technology lies in the fusion of virtual and real, that is, the seamless combination of virtual objects and real scenes, and the real-time interaction between users and virtual and real elements. Therefore, mixed reality applications are a specific form of virtual and real fusion. Through hardware devices (such as head-mounted displays) and software algorithms, virtual information is superimposed on the real environment, and it is widely used in industries such as emergency response, education, exhibitions, and medical care.

[0003] In MR applications, the effect of virtual-real alignment and fusion can be directly perceived by the trainees visually. In the prior art, pre-placed calibration markers are usually used for virtual-real alignment.

[0004] Since there are strict requirements for the placement of calibration marks, any position error will affect the virtual-real alignment effect; at the same time, the alignment accuracy of a single calibration mark is limited, and setting multiple calibration marks can improve the accuracy but also adds extra work. In addition, during the alignment process using calibration marks, MR application users are generally required to remain relatively still and maintain a suitable distance and observation direction from the calibration marks, resulting in a lack of flexibility in the virtual-real alignment operation. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method and system for virtual-reality alignment of an MR device based on a point cloud map, so as to solve the problem that the existing virtual-reality alignment operation is cumbersome and has low precision.

[0006] On the one hand, an embodiment of the present invention provides a method for MR device virtual-real alignment based on a point cloud map, comprising the following steps: Collect environmental data of real training equipment, build point cloud maps and import them into MR equipment; Match the image captured by the MR device with the point cloud map, take the shooting time of the successfully matched image as the initial time, calculate the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real scene coordinate system at the initial time; According to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the position and posture of the MR device in the point cloud map at the initial moment, and the position and posture of the MR device in the real scene coordinate system at the initial moment, the position and posture of the MR device in the real scene coordinate system at each moment after the initial moment is modified to obtain the position and posture of the MR device in the virtual training device coordinate system at each moment; According to the position and posture of the MR device in the virtual training device coordinate system at each moment, a virtual-real aligned training application is rendered.

[0007] Based on the further improvement of the above method, the position and posture of the MR device in the virtual training device coordinate system at each moment is calculated by the following formula: , in, express The position and posture of the MR device in the virtual training device coordinate system at this moment, Represents the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, Indicates the position of the MR device in the point cloud map at the initial moment, Indicates the position and posture of the MR device in the real-scene coordinate system at the initial moment; express The position and posture of the MR device in the real-scene coordinate system at this moment.

[0008] Based on the further improvement of the above method, the initial position of the MR device in the point cloud map is calculated by the following formula: , in, Indicates the position of the MR device in the point cloud map at the initial moment, Represents the external parameter data of the MR device relative to the camera, Represents the position of the camera in the point cloud map.

[0009] Based on the further improvement of the above method, the position and posture of the camera in the point cloud map is obtained by calling the PNP algorithm after obtaining multiple groups of 2D-3D matching point pairs between 2D feature points in the successfully matched image and 3D map points in the point cloud map, or by triangulating the 2D feature points in the successfully matched image to obtain 3D feature points, obtaining multiple groups of 3D-3D matching point pairs between the 3D feature points and the 3D map points in the point cloud map, and then calling the ICP algorithm.

[0010] Based on the further improvement of the above method, the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system is initially obtained by adjusting the translation position of the origin of the point cloud map coordinate system in the x, y and z axes of the virtual training device coordinate system, and the posture Euler angles of the origin of the point cloud map coordinate system relative to the x, y and z axes of the virtual training device coordinate system until the MR device completes the initial virtual-real alignment.

[0011] Based on the further improvement of the above method, the coordinate system of the virtual training device is constructed by importing the three-dimensional mesh model file of the real training device, taking the geometric center of the real training device as the origin, and the right, top and back of the real training device as the x, y and z axis directions respectively.

[0012] Based on the further improvement of the above method, rendering of virtual-real aligned training applications is obtained by superimposing multiple layers.

[0013] Based on further improvements of the above method, the order of multiple layers from bottom to top is: virtual layer, perspective window and real layer; the virtual layer is a training application deployed in the MR device and is rendered according to the position of the MR device in the virtual training device coordinate system at each moment; the perspective window is the outline area of ​​the real training device, which is determined according to the imported three-dimensional mesh model file of the real training device and is used to cover the virtual training device in the virtual layer; the real layer is used to display the real image taken by the MR device in the perspective window.

[0014] Based on the further improvement of the above method, the position and posture of the MR device in the virtual training device coordinate system at each moment is regularly updated by regularly re-matching the image taken by the camera on the MR device with the point cloud map, calculating the new initial position and posture of the MR device in the point cloud map, and the new initial position and posture of the MR device in the real scene coordinate system.

[0015] On the other hand, an embodiment of the present invention provides a virtual-reality alignment system for an MR device based on a point cloud map, comprising: The offline point cloud map construction module is used to collect environmental data of real training equipment, build point cloud maps and import them into MR equipment; The point cloud map matching module is used to match the image captured by the MR device with the point cloud map, take the shooting time of the successfully matched image as the initial time, calculate the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real scene coordinate system at the initial time; The MR device posture acquisition module is used to modify the posture of the MR device in the real scene coordinate system at each moment after the initial moment according to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the posture of the MR device in the point cloud map at the initial moment, and the posture of the MR device in the real scene coordinate system at the initial moment, so as to obtain the posture of the MR device in the virtual training device coordinate system at each moment; The training application rendering module is used to render a virtual-real aligned training application according to the position and posture of the MR device in the virtual training device coordinate system at each moment.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. By constructing a point cloud map, rich three-dimensional spatial information is provided, which helps to improve the accuracy of pose estimation; by matching the image with the point cloud map and taking the shooting time of the successfully matched image as the initial time, the pose of the MR device in the point cloud map is more accurately determined, providing a reliable basis for subsequent pose modification.

[0017] 2. By combining the training device, MR device and alignment operation, the external parameter data for initial virtual-reality alignment is accurately obtained, simplifying the virtual-reality alignment process; then based on the pose calculated when the image and point cloud map are successfully matched, the accurate pose of the MR device in the virtual training device coordinate system is calculated in real time, which improves the accuracy of virtual-reality alignment and achieves a lasting virtual-reality alignment effect, enhancing the user's perception and interaction with virtual content.

[0018] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components; Figure 1 This is a flow chart of a method for aligning the virtual and real parts of an MR device based on a point cloud map in Embodiment 1 of the present invention; Figure 2 This is an example diagram of the alignment process operation panel of the cockpit training application in Example 1 of the present invention. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0021] Example 1 A specific embodiment of the present invention discloses a method for aligning the virtual and real of an MR device based on a point cloud map, such as Figure 1 As shown, the following steps are included: S1. Collect environmental data of real training equipment, build a point cloud map and import it into the MR device.

[0022] It should be noted that the MR device of this embodiment is a head-mounted display device, including: 1 binocular color camera, 2 binocular black and white cameras and an IMU (Inertial Measurement Unit) sensor. The binocular color camera is used to capture real-scene images, and the binocular black and white camera and the IMU are used to track the MR device in real time and calculate the position and posture of the MR device in the real-scene coordinate system.

[0023] A training application is deployed in the MR device to provide users with a scene that integrates virtual and real scenes for simulation training. For example, the training application is a driver training application. There is a real cockpit training device in the real scene (real world), and the training application renders the virtual cockpit training device and the scene outside the cockpit in the virtual scene (virtual world); after the virtual and real scenes are aligned, the real cockpit and the virtual cockpit scene can be seen through the MR device. The various operations of the operator in the real cockpit can be reflected in the driver training application.

[0024] This embodiment uses binocular cameras and IMU sensors to collect environmental data of a real training device, which can be collected by wearing an MR device or by using other devices including binocular cameras and IMU sensors. The environment of the training device includes: the external environment and the internal environment of the training device.

[0025] For example, in a cockpit training application scenario, the collector wears an MR device. When collecting external scenes, the collector takes the real cockpit as the center and keeps a distance of 1 meter from it. The camera on the MR device slowly and evenly circles around the cockpit. When collecting internal scenes, the collector simulates the actions of the trainees in actual use of the scenario, including but not limited to: adjusting the sitting posture while sitting in the seat and observing the dashboard in the cockpit.

[0026] The collected environmental data includes: grayscale images collected by binocular cameras, and motion data such as acceleration and angular velocity collected by IMU sensors. Furthermore, the SLAM algorithm is used to construct a point cloud map.

[0027] It should be noted that the data obtained in the point cloud map and during the construction process include: three-dimensional coordinates of 3D map points, descriptor vectors of 3D map points, historical image frame data, historical key frame data selected from the historical image frame data, the correspondence between feature points in the historical key frames and 3D map points, and feature bag-of-words vectors of historical image frames (including historical key frames).

[0028] Import the offline constructed point cloud map file into the MR device as a prerequisite for subsequent virtual-real alignment.

[0029] S2. Match the image taken by the MR device with the point cloud map, take the shooting time of the successfully matched image as the initial time, calculate the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real-scene coordinate system at the initial time.

[0030] It should be noted that after the MR device is turned on, images taken by the camera on the MR device are continuously collected, and the same descriptors as those used to build the point cloud map are used, such as the ORB (Oriented Fast and Rotated BRIEF) descriptor or the SIFT (Scale-Invariant Feature Transform) descriptor. 2D feature points and their descriptors are extracted from the real-time collected images to obtain the feature bag-of-words vector of the real-time image.

[0031] Matching the image captured by the MR device with the point cloud map includes: taking out the current frame from the real-time acquired image and matching it with the point cloud map, calculating the similarity between the feature word bag vector of the current frame and the feature word bag vector of the historical key frame, and if the similarity exceeds the similarity threshold, obtaining the 3D map point in the point cloud map associated with the feature point of the historical key frame, matching the descriptor vector of the 2D feature point of the current frame with the descriptor vector of the 3D map point, and forming multiple groups of 2D-3D matching point pairs after the matching is completed; or triangulating the 2D feature points in the current frame to obtain 3D feature points, matching the 3D feature points with the 3D map points in the point cloud map, and forming multiple groups of 3D-3D matching point pairs after the matching is completed.

[0032] When the number of multiple sets of 2D-3D matching point pairs meets the requirements of the PNP algorithm, the paired image is a successfully matched image. The image shooting time is taken as the initial time, and the PNP algorithm is called to calculate the camera's position in the point cloud map at the initial time. Or, when the number of multiple sets of 3D-3D matching point pairs meets the requirements of the ICP algorithm, the paired image is a successfully matched image, the image shooting time is taken as the initial time, and the ICP algorithm is called to calculate the initial time The camera’s pose in the point cloud map ; Then according to the external parameter data of the MR device relative to the camera , the initial position of the MR device in the point cloud map is calculated by the following formula : Formula (1).

[0033] It should be noted that while matching the image captured by the MR device with the point cloud map, the position and posture of the MR device are tracked based on the image captured by the MR device and the IMU sensor, and the six-degree-of-freedom data of the MR device is calculated in real time using the SLAM algorithm. This data is output at a high frequency (such as 1000Hz). This embodiment takes the moment when the image captured by the MR device is successfully matched with the point cloud map as the initial moment, and takes out the corresponding six-degree-of-freedom data based on the initial moment as the position and posture of the MR device in the real-scene coordinate system at the initial moment. .

[0034] S3. According to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the position and posture of the MR device in the point cloud map at the initial moment, and the position and posture of the MR device in the real-scene coordinate system at the initial moment, the position and posture of the MR device in the real-scene coordinate system at each moment after the initial moment is modified to obtain the position and posture of the MR device in the virtual training device coordinate system at each moment.

[0035] It should be noted that the key to virtual-real alignment in this embodiment is to modify the six-degree-of-freedom data of the MR device (i.e., the position and posture of the MR device in the real-scene coordinate system), and transfer the reference coordinate system of the six-degree-of-freedom data to the virtual training device coordinate system, that is, to obtain the position and posture of the MR device in the virtual training device coordinate system, and then pass it to the training application for rendering.

[0036] The virtual training device coordinate system is constructed by importing the 3D mesh model file of the real training device, taking the geometric center of the real training device model as the origin, and taking the right, top and back of the real training device as the x, y and z axis directions respectively. The 3D mesh model file is usually obtained by modeling the real training device using 3D modeling software and meshing it before exporting, including: vertex data, color data and triangular mesh data of the model. Exemplarily, the 3D modeling software includes: Maya and Blender.

[0037] This step is based on step S2 to obtain the initial position and posture of the MR device in the point cloud map. , and the initial position of the MR device in the real scene coordinate system , combined with the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system , the position and posture of the MR device in the real scene coordinate system at each moment after the initial moment can be modified by the following formula, and the position and posture of the MR device in the virtual training device coordinate system at each moment can be calculated: Formula (2), in, express The position and posture of the MR device in the virtual training device coordinate system at this moment, Indicates the time after the initial moment The position and posture of the MR device in the real-scene coordinate system at this moment.

[0038] S4. Render a virtual-real aligned training application according to the position and posture of the MR device in the virtual training device coordinate system at each moment.

[0039] It should be noted that, according to the position and posture of the MR device in the virtual training device coordinate system at each moment, the rendering of the virtual-real alignment training application is obtained by superimposing multiple layers.

[0040] Specifically, the order of multiple layers from bottom to top is: virtual layer, perspective window and real layer; among them, the virtual layer is the training application deployed in the MR device, which is rendered according to the position of the MR device in the virtual training device coordinate system at each moment; the perspective window is the outline area of ​​the real training device, which is determined according to the imported three-dimensional mesh model file of the real training device and is used to cover the virtual training device in the virtual layer; the real layer is used to display the real image taken by the MR device in the perspective window.

[0041] It should be noted that when the MR device is used for the first time, the translation vector in the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system is 0, and the rotation matrix is ​​the unit matrix. That is to say, the translation position of the origin of the initial point cloud map coordinate system in the x, y and z axes of the virtual training device coordinate system is 0, and the attitude Euler angle (rotation angle around the coordinate axis) of the origin of the point cloud map coordinate system relative to the x, y and z axes of the virtual training device coordinate system is 0. At this time, it is equivalent to that the external parameter data is not effective. According to formula (2), the position and posture of the MR device in the virtual training device coordinate system is calculated and transmitted to the training application. The interface rendered by the training application cannot achieve virtual-real alignment. What the user sees in the MR device is the real training device and the virtual training device that are not aligned.

[0042] Therefore, when the MR device is used for the first time, the user triggers the alignment button and first matches the image with the point cloud map according to the method in step S2 to calculate the initial position and posture of the MR device in the point cloud map. , and the initial position of the MR device in the real scene coordinate system After that, a virtual operation panel is displayed in the MR application. The user enters the above three translation positions (in meters) and three posture Euler angles (in degrees) to calculate the corresponding external parameter data. According to formula (2), the position and posture of the MR device in the virtual training device coordinate system are obtained. The training application renders interfaces with different alignment effects according to the method in step S3. Taking the driving training application as an example, the displayed operation panel is shown in the figure below: Figure 2 shown.

[0043] When the user adjusts the above 6 data to a satisfactory alignment effect, the adjusted data is saved to obtain the external parameter data for initial virtual-real alignment. When the MR device is used subsequently, the external parameter data that realizes the initial virtual-real alignment is automatically loaded without further adjustment. The position and posture of the MR device in the virtual training device coordinate system at each moment is calculated and transmitted to the training application, and the virtual-real aligned training application can be rendered.

[0044] It should be noted that during the alignment process, the user does not need to maintain a fixed posture. The user can perform normal movements according to the training operation process, including standing, sitting, moving left and right, forward and backward, and observing at different angles on the training equipment.

[0045] This embodiment can ensure that the angle fluctuation of the posture Euler angle is 0.4° and the fluctuation of the translation position is 1.0 cm. This alignment accuracy ensures that the user can achieve a good alignment effect in each subsequent training and reduces the alignment operation process.

[0046] Preferably, step S2 is performed regularly to rematch the image captured by the camera on the MR device with the point cloud map, calculate the new initial position of the MR device in the point cloud map, and the new initial position of the MR device in the real scene coordinate system, and use them in formula (2), which effectively avoids the data drift problem of the six-degree-of-freedom data calculated by the SLAM algorithm during long-term operation, and improves the stability and accuracy of the position estimation. Since the accuracy fluctuation of the external parameter data is small, the error caused by regular execution is not large, the rendered interface does not change much, and will not cause discomfort to the user, achieving a lasting virtual-real alignment effect.

[0047] The interval of regular execution is determined based on the computing power and usage environment, and is set to 3 minutes by way of example.

[0048] Preferably, considering that the environmental data of the training device may change, which will affect the matching results between the images taken by the MR device and the point cloud map, after the MR device is started, the point cloud map real-time update program is run to update the point cloud map according to the environmental data collected by the MR device in real time, so as to avoid the problem of point cloud map mismatch caused by environmental changes.

[0049] Compared with the prior art, the virtual-reality alignment method of MR device based on point cloud map provided in this embodiment provides rich three-dimensional spatial information by constructing point cloud map, which helps to improve the accuracy of posture estimation; by matching the image with the point cloud map, the shooting time of the successfully matched image is used as the initial time, and the posture of the MR device in the point cloud map is more accurately determined, providing a reliable basis for subsequent posture modification. In combination with the training device, MR device and alignment operation, the external parameter data for initial virtual-reality alignment is accurately obtained, simplifying the virtual-reality alignment process; then based on the posture calculated when the image and point cloud map are successfully matched, the accurate posture of the MR device in the virtual training device coordinate system is calculated in real time, which improves the accuracy of virtual-reality alignment, achieves a lasting virtual-reality alignment effect, and enhances the user's perception and interaction with virtual content.

[0050] Example 2 Another embodiment of the present invention discloses a virtual-real alignment system of an MR device based on a point cloud map, thereby realizing a virtual-real alignment method of an MR device based on a point cloud map in Embodiment 1. The specific implementation of each module refers to the corresponding description in Embodiment 1. The system includes: The offline point cloud map construction module is used to collect environmental data of real training equipment, build point cloud maps and import them into MR equipment; The point cloud map matching module is used to match the image captured by the MR device with the point cloud map, take the shooting time of the successfully matched image as the initial time, calculate the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real scene coordinate system at the initial time; The MR device posture acquisition module is used to modify the posture of the MR device in the real scene coordinate system at each moment after the initial moment according to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the posture of the MR device in the point cloud map at the initial moment, and the posture of the MR device in the real scene coordinate system at the initial moment, so as to obtain the posture of the MR device in the virtual training device coordinate system at each moment; The training application rendering module is used to render a virtual-real aligned training application according to the position and posture of the MR device in the virtual training device coordinate system at each moment.

[0051] Since the virtual-real alignment system of an MR device based on a point cloud map in this embodiment and the virtual-real alignment method of an MR device based on a point cloud map in the above embodiment can be mutually referenced, it is a repeated description here, so it will not be repeated here. Since the principle of this system embodiment is the same as that of the above method embodiment, this system embodiment also has the corresponding technical effects of the above method embodiment.

[0052] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0053] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for MR device virtual-real alignment based on point cloud map, characterized in that: The following steps are involved: Collect environmental data of real training equipment, build point cloud maps and import them into MR equipment; Matching the image captured by the MR device with the point cloud map, taking the shooting time of the successfully matched image as the initial time, and calculating the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real scene coordinate system at the initial time; According to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the position and posture of the MR device in the point cloud map at the initial moment, and the position and posture of the MR device in the real scene coordinate system at the initial moment, the position and posture of the MR device in the real scene coordinate system at each moment after the initial moment is modified to obtain the position and posture of the MR device in the virtual training device coordinate system at each moment; According to the position and posture of the MR device in the virtual training device coordinate system at each moment, a virtual-real aligned training application is rendered.

2. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1, characterized in that: The position and posture of the MR device in the virtual training device coordinate system at each moment is calculated by the following formula: ， in, express The position and posture of the MR device in the virtual training device coordinate system at this moment, Represents the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, Indicates the position of the MR device in the point cloud map at the initial moment, Indicates the position and posture of the MR device in the real-scene coordinate system at the initial moment; express The position and posture of the MR device in the real-scene coordinate system at this moment.

3. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1 or 2, characterized in that: The position and posture of the MR device in the point cloud map at the initial moment is calculated by the following formula: ， in, Indicates the position of the MR device in the point cloud map at the initial moment, Represents the external parameter data of the MR device relative to the camera, Represents the position of the camera in the point cloud map.

4. The method for MR device virtual-real alignment based on point cloud map according to claim 3, characterized in that: The position and posture of the camera in the point cloud map is obtained by calling the PNP algorithm after obtaining multiple groups of 2D-3D matching point pairs between 2D feature points in the successfully matched image and 3D map points in the point cloud map, or by triangulating the 2D feature points in the successfully matched image to obtain 3D feature points, obtaining multiple groups of 3D-3D matching point pairs between the 3D feature points and 3D map points in the point cloud map, and then calling the ICP algorithm.

5. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1, characterized in that: The external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system is initially obtained by adjusting the translation position of the origin of the point cloud map coordinate system in the x, y and z axes of the virtual training device coordinate system, and the posture Euler angles of the origin of the point cloud map coordinate system relative to the x, y and z axes of the virtual training device coordinate system until the MR device completes the initial virtual-real alignment.

6. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1, characterized in that: The virtual training device coordinate system is constructed by importing a three-dimensional grid model file of a real training device, taking the geometric center of the real training device as the origin, and taking the right, top and back of the real training device as the x, y and z axis directions respectively.

7. The method for MR device virtual-real alignment based on point cloud map according to claim 1, characterized in that: The training application for rendering virtual-real alignment is obtained by superimposing multiple layers.

8. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 7, characterized in that: The order of the multiple layers from bottom to top is: virtual layer, perspective window and real layer; the virtual layer is rendered by the training application deployed in the MR device according to the position of the MR device in the virtual training device coordinate system at each moment; the perspective window is the outline area of ​​the real training device, which is determined according to the imported three-dimensional grid model file of the real training device and is used to cover the virtual training device in the virtual layer; The real layer is used to display the real image captured by the MR device in the perspective window.

9. The method for virtual-real alignment of MR equipment based on point cloud map according to claim 1, characterized in that: The position and posture of the MR device in the virtual training device coordinate system at each moment is regularly updated by regularly re-matching the image taken by the camera on the MR device with the point cloud map, calculating the new initial position and posture of the MR device in the point cloud map, and the new initial position and posture of the MR device in the real scene coordinate system.

10. A point cloud map-based MR device virtual-real alignment system, characterized in that: include: The offline point cloud map construction module is used to collect environmental data of real training equipment, build point cloud maps and import them into MR equipment; A point cloud map matching module is used to match the image captured by the MR device with the point cloud map, take the shooting time of the successfully matched image as the initial time, calculate the position and posture of the MR device in the point cloud map at the initial time, and the position and posture of the MR device in the real scene coordinate system at the initial time; The MR device posture acquisition module is used to modify the posture of the MR device in the real scene coordinate system at each moment after the initial moment according to the external parameter data of the point cloud map coordinate system relative to the virtual training device coordinate system, the posture of the MR device in the point cloud map at the initial moment, and the posture of the MR device in the real scene coordinate system at the initial moment, so as to obtain the posture of the MR device in the virtual training device coordinate system at each moment; The training application rendering module is used to render a virtual-real aligned training application according to the position and posture of the MR device in the virtual training device coordinate system at each moment.

Images