A virtual-real alignment method for MR equipment

Abstract

The present invention relates to a virtual-real alignment method for MR devices, belongs to the field of virtual reality technology, and solves the problem that it is difficult for existing MR devices to automatically achieve virtual-real alignment. The method includes: identifying calibration marks from each image taken by a camera, calculating the first pose of the calibration marks in each image relative to the camera, and calculating the second pose of the calibration marks in each image in the real world coordinate system according to the external parameters of the MR device, the pose and the first pose of the MR device; calculating the optimal second pose according to the second pose of the calibration marks in each image; the calibration marks correspond to virtual anchor points with the same position on the virtual reference object; correcting the pose of the MR device according to the optimal second pose of the calibration marks and the third pose of the virtual anchor points in the virtual world coordinate system; rendering multiple layers based on the corrected pose of the MR device, and if the multiple layers are not completely overlapped, re-executing the above steps until they are completely overlapped, and completing virtual-real alignment. Automatic virtual-real alignment is achieved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a virtual-reality alignment method for MR equipment. Background Art

[0002] MR (Mixed Reality) technology achieves a highly immersive experience by fusing the real world and virtual images in the user's field of view. Virtual-real alignment is a key step in MR technology, which ensures that the position, size, and orientation of virtual objects in the user's field of view match those in the real world, thereby improving the realism of the user experience.

[0003] MR devices usually use inside-out positioning technology to track the user's position and posture in real time. Through image recognition technology, MR devices can identify specific objects in the real world and superimpose virtual images on the real images of these objects.

[0004] When using MR equipment for human-computer interaction, the user can directly visually perceive the effect of virtual-real alignment. The current existing technology generally performs virtual-real alignment manually, but the manual method is not easy to align postures, which makes the user easily affected and unable to use it normally. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a virtual-real alignment method for an MR device, so as to solve the problem that it is difficult for existing MR devices to automatically achieve virtual-real alignment.

[0006] An embodiment of the present invention provides a virtual-real alignment method for an MR device, comprising the following steps:

[0007] S1, collecting multiple images taken by a camera on the MR device, identifying calibration marks on the physical reference object from each image, calculating the first pose of the calibration mark in each image relative to the camera, and, according to the external parameters of the MR device, the pose of the MR device and the first pose, calculating the second pose of the calibration mark in each image in the real world coordinate system;

[0008] S2, calculating the optimal second pose of the calibration marker according to the second pose of the calibration marker in each image;

[0009] S3, the calibration mark corresponds to a virtual anchor point at the same position on the virtual reference object; according to the optimal second posture of the calibration mark and the third posture of the virtual anchor point in the virtual world coordinate system, the posture of the MR device is corrected;

[0010] S4, rendering multiple layers based on the posture corrected by the MR device. If the multiple layers do not completely overlap, re-execute S1-S4 until the multiple layers completely overlap, completing the virtual-real alignment.

[0011] Based on the further improvement of the above method, there are one or more calibration marks on the physical reference object. When there are multiple calibration marks, one of them is taken as the main calibration mark, and there is a corresponding virtual anchor point on the virtual reference object with the same position as the main calibration mark.

[0012] Based on the further improvement of the above method, when there are multiple calibration marks, the calibration marks combined in pairs are verified respectively, and the second poses of the calibration marks that pass the verification are converted into the second pose of the main calibration mark, and the average is taken as the final second pose of the main calibration mark in each image.

[0013] Based on the further improvement of the above method, the calibration marks of the two combinations are verified respectively, including:

[0014] According to the second poses of the calibration markers in the combination of two pairs, the estimated relative pose between the two calibration markers in the combination is calculated, and the errors of the rotational part and the translational part between the estimated relative pose and the actual relative pose are calculated. If any error exceeds the corresponding second error threshold, both calibration markers fail the verification; otherwise, both calibration markers pass the verification.

[0015] Based on the further improvement of the above method, the optimal second pose of the calibration marker is calculated according to the second pose of the calibration marker in each image. It is obtained by removing abnormal calibration markers according to the pose of the physical reference object in the real world coordinate system, and then calculating the average of the second poses of the remaining calibration markers.

[0016] Based on the further improvement of the above method, the abnormal calibration marks are removed according to the pose of the physical reference object in the real world coordinate system, including:

[0017] According to the second pose of each calibration mark and the pose of each calibration mark relative to the physical reference object, the pose of the physical reference object in the real world coordinate system is calculated as a plurality of poses to be verified;

[0018] Calculate the mean of the rotation part and the translation part for multiple postures to be verified;

[0019] The errors between the rotation part and the translation part of each posture to be checked and the corresponding mean are checked in turn. If any error exceeds the corresponding first error threshold, the corresponding calibration mark is removed as an abnormal calibration mark.

[0020] Based on the further improvement of the above method, according to the external parameters of the MR device, the posture of the MR device and the first posture, the second posture of the calibration marker in each image in the real world coordinate system is calculated by the following formula:

[0021]

[0022] in, represents the second pose of the calibration marker m in the real world coordinate system wr, represents the position of the MR device, that is, the position of the MR device b in the real world coordinate system wr, Represents the relative position of camera c and device b on the MR device, which is a pre-calibrated external parameter. represents the first pose of the calibration marker m relative to the camera c.

[0023] Based on the further improvement of the above method, the posture of the MR device is corrected by the following formula:

[0024]

[0025] in, and They represent the posture of the MR device before and after correction, Represents the third pose of the virtual anchor point a in the virtual world coordinate system wv; Represents the optimal second pose of the calibration marker m in the real world coordinate system wr.

[0026] Based on the further improvement of the above method, the third pose of the virtual anchor point in the virtual world coordinate system is calculated by the following formula:

[0027]

[0028] in, Represents the position and posture of the virtual reference object dv in the virtual world coordinate system wv, Represents the pose of the virtual anchor point a relative to the virtual reference object dv, and the pose of the calibration marker m relative to the physical reference object dr same.

[0029] Based on the further improvement of the above method, the order of multiple layers from front to back is: perspective window, virtual layer and real layer; the perspective window is used to determine the shape of the perspective area, and the perspective area is composed of predetermined observation area vertices; the virtual layer is used to display a predetermined virtual image in the application scenario of the MR device; the real layer is used to display the real image taken by the camera on the MR device.

[0030] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0031] 1. According to the position of the calibration marker in the real world and the known position of the virtual anchor point corresponding to the calibration marker in the virtual world, the physical reference object and the virtual reference object are aligned, thereby automatically realizing virtual-real alignment, which is simple and fast.

[0032] 2. Improve recognition accuracy by setting multiple calibration marks, and perform verification based on the relative posture between the calibration marks to obtain calibration marks that have passed the verification; by taking multiple images, remove abnormal calibration marks based on the posture of the physical reference object in the real-world coordinate system, so that the posture of the calibration mark in the real world is more accurate, reducing the impact of inaccurate data caused by factors such as shooting angle, distance, and calculated posture data of the MR device.

[0033] 3. The posture of the MR device is corrected by using the posture of the calibration mark in the real world coordinate system and the posture of the virtual anchor point in the virtual world coordinate system. The correction method does not depend on the application scenario of the MR device, has high correction efficiency, and can quickly achieve virtual-real alignment.

[0034] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0036] Figure 1 This is a flow chart of a virtual-real alignment method for an MR device in an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of an MR device in an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the calibration mark in the embodiment of the present invention. DETAILED DESCRIPTION

[0039] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0040] A specific embodiment of the present invention discloses a virtual-real alignment method for an MR device, such as Figure 1 As shown, the following steps are included:

[0041] S1, collecting multiple images taken by a camera on the MR device, identifying the calibration mark on the physical reference object from each image, calculating the first pose of the calibration mark in each image relative to the camera, and, according to the external parameters of the MR device, the pose of the MR device and the first pose, calculating the second pose of the calibration mark in the real world coordinate system at each moment;

[0042] S2, calculating the optimal second pose of the calibration marker according to the second pose of the calibration marker in each image;

[0043] S3, the calibration mark corresponds to a virtual anchor point at the same position on the virtual reference object; according to the optimal second posture of the calibration mark and the third posture of the virtual anchor point in the virtual world coordinate system, the posture of the MR device is corrected;

[0044] S4, rendering multiple layers based on the posture corrected by the MR device. If the multiple layers do not completely overlap, re-execute S1-S4 until the multiple layers completely overlap, completing the virtual-real alignment.

[0045] It should be noted that the MR device is a head-mounted display device, including: 1 binocular color camera, 2 binocular black and white cameras and IMU (Inertial Measurement Unit). The binocular color camera is used to capture real-life images, and the binocular black and white camera and IMU are used to track the MR device and calculate the position and posture of the MR device. The relative position and posture between any camera on the MR device and the device is a pre-calibrated external parameter.

[0046] The MR device includes: IMU coordinate system, camera coordinate system and MR device coordinate system. The IMU is installed in the middle of the binocular color camera, the x-axis of the IMU coordinate system points to the right side of the MR device, the y-axis points to the top of the MR device, and the z-axis points to the front of the MR device; the origin of the MR device coordinate system is moved to the center of the head in the negative direction of the z-axis in the IMU coordinate system.

[0047] For example, Figure 2 As shown in the figure, the solid dots represent binocular color cameras, and the black hollow circles represent binocular black and white cameras. The cross in the middle of the solid dots represents the origin of the IMU coordinate system, and the other cross represents the origin of the MR device coordinate system.

[0048] Users wear MR devices, which create an interactive environment by integrating the real world (real scene) and the virtual world (virtual scene). They are widely used in education, games, medical care and other fields.

[0049] In this embodiment, in the application scenario of the MR device, there is a physical reference object in the real scene and a virtual reference object of the same size ratio in the virtual scene. For example, the MR device is used for cockpit simulation training, the physical reference object is the dashboard of the cockpit, and the virtual reference object is the virtual dashboard.

[0050] Furthermore, a calibration mark is provided on the physical reference object. For example, the calibration mark is as follows: Figure 3 As shown in Figure 2, since the position of the calibration marker is predetermined, the position and posture of the calibration marker m relative to the physical reference object dr is It can be pre-calculated based on the position coordinates of the calibration mark and the physical reference object. Correspondingly, a virtual calibration mark is also set at the same position on the virtual reference object, which is called a virtual anchor point. In this embodiment, the position and posture of the virtual anchor point a relative to the virtual reference object dv is The same as the position of the calibration mark relative to the physical reference object, that is,

[0051] It should be noted that the calibration mark can be captured by any binocular color camera or a binocular black and white camera on the MR device to determine the posture relationship between the user and the physical reference object. However, in order not to affect the use of the application scenario, the calibration mark is outside the user's observation area. The user's observation area is predetermined based on the shape parameters of the physical reference object, and the posture of the vertices of the observation area relative to the physical reference object is recorded. Ultimately, these vertices can generate a perspective area in which the image of the physical reference object is displayed.

[0052] During implementation, the calibration marks are identified by shooting images, and the posture of the MR device is corrected according to the posture of the calibration marks and virtual anchor points, so that the physical reference objects and virtual reference objects are quickly aligned, giving users a better perception; the physical reference objects in the real scene are replaced with virtual reference objects in the virtual scene, so that users can manipulate the real physical reference objects to perform corresponding operations in the scene and successfully complete various tasks in the application scenario.

[0053] Specifically, in step S1, the calibration marks on the physical reference objects are respectively identified from the multiple images taken by the camera using the existing technology, for example, the calibration marks are detected using the trained neural network model. Then, by determining the matching point pairs between the feature points in the calibration mark in the image and the corresponding points in the world coordinate system, the rotation matrix and translation vector of the calibration mark relative to the camera are solved according to at least 4 non-collinear matching point pairs to obtain the first position of the calibration mark m relative to the camera c.

[0054] It should be noted that, based on the image collected by the binocular black-and-white camera on the MR device and the position and posture of the MR device tracked by the IMU, the Inside-out positioning technology is used, and the SLAM algorithm is used to calculate the six-degree-of-freedom data of the MR device in real time. The data is output at a high frequency (such as 1000 Hz). This embodiment extracts the six-degree-of-freedom data corresponding to the image shooting moment (i.e., the image exposure moment) from the high-frequency output data as the posture of the MR device for subsequent calculations.

[0055] Furthermore, according to the external parameters of the MR device, the posture of the MR device and the first posture, the second posture of the calibration mark in each image in the real world coordinate system is calculated by the following formula:

[0056]

[0057] in, represents the second pose of the calibration marker m in the real world coordinate system wr, represents the position of the MR device, that is, the position of the MR device b in the real world coordinate system wr, Represents the relative position of camera c and device b on the MR device, which is a pre-calibrated external parameter. represents the first pose of the calibration marker m relative to the camera c.

[0058] In step S2, since the second pose of the calibration mark in the real world coordinate system is theoretically an absolute pose, the accuracy of the data is often affected by factors such as the shooting angle, distance, and pose data of the MR device. Therefore, this step obtains the second pose of the calibration mark of multiple images, removes abnormal calibration marks according to the pose of the physical reference object in the real world coordinate system, and obtains the optimal second pose by calculating the average of the second poses of the remaining calibration marks.

[0059] Specifically, according to the second pose of the calibration mark in each image and the pose of each calibration mark relative to the physical reference object, the pose of the physical reference object in the real world coordinate system, that is, the pose in the world coordinate system where the MR device is positioned, is calculated by the following formula as multiple poses to be verified:

[0060]

[0061] in, Indicates that according to the calibration mark m k The calculated pose of the entity reference object dr in the real world coordinate system wr, Indicates the calibration mark m k The pose relative to the physical reference, Indicates the calibration mark m k The second posture.

[0062] Furthermore, the mean of the rotation part and the translation part are calculated for each of the multiple postures to be verified.

[0063] It should be noted that the translation mean of the translation part in the posture to be verified is obtained by calculating the arithmetic mean of the translation vector in each dimension; the rotation part usually has a variety of representation methods. This embodiment uses quaternion to represent the rotation part. Each normalized quaternion is converted to logarithmic space, and then the average logarithm of all quaternions is calculated, the average logarithm is mapped back to the quaternion space, and finally normalized to obtain the mean quaternion of the rotation part.

[0064] Furthermore, the errors between the rotation part and the translation part of each posture to be verified and the corresponding mean are checked in turn. If any error exceeds the corresponding first error threshold, the corresponding calibration mark is removed.

[0065] Specifically, the rotation quaternion in each posture to be verified is multiplied by the inverse of the mean quaternion to obtain the error quaternion. The error quaternion is converted into a rotation matrix and the rank of the rotation matrix is ​​calculated. Since the rank of the unit rotation matrix is ​​3, the rank of the calculated rotation matrix is ​​compared with 3 to calculate the first error. If the first error exceeds the first error threshold corresponding to the rotation part, the calibration mark corresponding to the posture to be verified is removed.

[0066] The translation vector in each posture to be verified is compared with the translation mean in each dimension to obtain a three-dimensional error vector, and the modulus of the three-dimensional error vector is calculated. If it exceeds the error threshold corresponding to the translation part, the calibration mark corresponding to the posture to be verified is removed.

[0067] Finally, calculate the first pose of the remaining calibration markers The average of , get the optimal first pose; calculate the second pose of the remaining calibration markers in the calibration set The optimal second pose is obtained by taking the mean of

[0068] Preferably, in order to further improve the recognition accuracy of the calibration mark, multiple calibration marks are set on the physical reference object, one of which is used as the main calibration mark, and only one virtual anchor point corresponding to the main calibration mark is required on the virtual reference object.

[0069] At this time, before performing the above-mentioned removal of abnormal calibration mark points, multiple calibration marks on each image are first verified according to the relative postures between the calibration marks in pairs, and calibration marks that pass the verification are obtained.

[0070] Specifically, since multiple calibration markers are set on the physical reference object and the position of each calibration marker is known, the actual relative pose between two calibration markers is fixed and can be calculated in advance.

[0071] According to the second poses of the calibration markers in the pairwise combination, the estimated relative pose between the two calibration markers in the combination is calculated by the following formula:

[0072]

[0073] in, Indicates the calibration mark m j With calibration mark m i The estimated relative pose between them, i≠j; and Respectively represent the calibration mark m i and calibration mark m j The second posture.

[0074] Furthermore, the errors of the rotation part and the translation part between the estimated relative pose and the actual relative pose between the two calibration markers are calculated. If any error exceeds the corresponding second error threshold, both calibration markers fail the verification and are removed as abnormal calibration markers. Otherwise, both calibration markers pass the verification.

[0075] For example, if If the error between the rotation part or the translation part of the corresponding actual relative posture exceeds the corresponding second error threshold, the calibration mark m is removed. j With calibration mark m i .

[0076] For each image, the first and second poses of each calibration mark that has passed the verification, as well as the relative pose with the main calibration mark, are converted to the first and second poses of the main calibration mark, and then the average is taken as the final first and second poses of the main calibration mark on each image. Finally, according to the method when there is only one calibration mark, after removing the abnormal main calibration marks in multiple images, the optimal first and second poses are obtained according to the average of the final first and second poses of the remaining main calibration marks.

[0077] For example, the main calibration mark is m0, through Convert the second pose of the calibration marker m1 to the second pose of the main calibration marker

[0078] In step S3, the virtual anchor point is in the third position of the virtual world coordinate system It is based on the position of the virtual reference object dv in the virtual world coordinate system wv and the pose of the virtual anchor relative to the virtual reference The calculation formula is as follows:

[0079]

[0080] Furthermore, the posture of the MR device is corrected by the following formula:

[0081]

[0082] in, and They represent the postures of the MR device before and after correction, respectively; Represents the optimal second pose of the calibration marker m in the real world coordinate system wr.

[0083] In step S4, when rendering the fused image seen from the MR device, the order of multiple layers from front to back is: perspective window, virtual layer and real layer. Among them, the perspective window is used to determine the shape of the perspective area, and the perspective area is composed of the pre-determined observation area vertices; the virtual layer is used to display the pre-determined virtual image in the application scene of the MR device; the real layer is used to display the real image taken by the camera on the MR device. The above three layers are superimposed to obtain the fused image displayed from the MR device.

[0084] Since the positions of the vertices and calibration marks in the observation area are known, the pose of each vertex relative to the calibration mark can be calculated in advance. Therefore, after calculating the first pose of the calibration mark relative to the camera, the pose of each vertex relative to the camera can be calculated, and based on the position coordinates of each vertex relative to the physical reference object, the pixel coordinates of each vertex in the image taken by the camera can be calculated.

[0085] Furthermore, the perspective window is mapped according to the vertex pixel coordinates; based on the corrected posture of the MR device, the corresponding virtual image is obtained from the application scene of the MR device to render the virtual layer, and the real image is collected to render the real layer. If the images after the three layers are superimposed completely overlap, the virtual-real alignment is completed. At this time, the relative posture relationship between the user and the reference object is unified in the virtual and real world; otherwise, step S1 to step S4 are re-executed, and the posture of the MR device is corrected and multiple layers are rendered again by taking multiple images until the virtual-real alignment is completed.

[0086] That is to say, this embodiment corrects the posture data of the MR device according to the posture of the calibration mark in the real world and the known posture of the virtual anchor point corresponding to the calibration mark in the virtual world, and displays virtual images under different position perspectives to the user. At the same time, the rendering position perspective of the perspective window will also change, automatically realizing virtual-real alignment, and displaying the image of the physical reference object in the real world in the perspective window area, so as to achieve the effect of aligning the physical reference object with the virtual reference object. At this time, the relative posture relationship between the person and the reference object in the virtual world and the real world remains consistent.

[0087] Compared with the prior art, the virtual-real alignment method for MR devices provided in this embodiment aligns the physical reference object and the virtual reference object according to the position and posture of the calibration mark in the real world and the known position and posture of the virtual anchor point corresponding to the calibration mark in the virtual world, thereby automatically realizing virtual-real alignment, which is simple and fast. By setting multiple calibration marks, the recognition accuracy is improved, and cross-checking is performed according to the relative position and posture between multiple calibration marks, and abnormal calibration marks are removed according to the position and posture of the physical reference object in the real world coordinate system, so that the position and posture of the calibration mark in the real world is more accurate, and the influence of inaccurate data caused by factors such as shooting angle, distance and calculated position and posture data of the MR device is reduced. At the same time, the position and posture of the MR device are corrected by using the position and posture of the calibration mark in the real world coordinate system and the position and posture of the virtual anchor point in the virtual world coordinate system. The correction method does not depend on the application scenario of the MR device, the correction efficiency is high, and virtual-real alignment is realized quickly.

[0088] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0089] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A virtual-real alignment method for MR equipment, characterized in that: The following steps are involved: S1, collecting multiple images taken by a camera on the MR device, identifying calibration marks on the physical reference object from each image, calculating the first pose of the calibration mark in each image relative to the camera, and calculating the second pose of the calibration mark in each image in the real world coordinate system according to the external parameters of the MR device, the pose of the MR device and the first pose; S2, according to the second poses of the calibration marks in each image, calculating the optimal second pose of the calibration marks, specifically, by removing abnormal calibration marks according to the poses of the physical reference object in the real world coordinate system, and calculating the average of the second poses of the remaining calibration marks; S3, the calibration mark corresponds to a virtual anchor point at the same position on the virtual reference object; according to the optimal second posture of the calibration mark and the third posture of the virtual anchor point in the virtual world coordinate system, the posture of the MR device is corrected; The third position of the virtual anchor point in the virtual world coordinate system Calculated by the following formula: in, Represents the position and posture of the virtual reference object dv in the virtual world coordinate system wv, Represents the pose of the virtual anchor point a relative to the virtual reference object dv, and the pose of the calibration marker m relative to the physical reference object dr same; The posture of the MR device is corrected by the following formula: in, and They represent the posture of the MR device before and after correction, represents the optimal second pose of the calibration marker m in the real world coordinate system wr; S4, rendering multiple layers based on the posture corrected by the MR device. If the multiple layers do not completely overlap, re-execute S1-S4 until the multiple layers completely overlap, completing the virtual-real alignment.

2. The virtual-real alignment method for MR equipment according to claim 1, characterized in that: There are one or more calibration marks on the physical reference object. When there are multiple calibration marks, one of them is taken as the main calibration mark, and there is a corresponding virtual anchor point on the virtual reference object with the same position as the main calibration mark.

3. The virtual-real alignment method for MR equipment according to claim 2, characterized in that: When there are multiple calibration marks, the calibration marks combined in pairs are verified respectively, and the second poses of the calibration marks that pass the verification are converted into the second poses of the main calibration mark, and then the average is taken as the final second pose of the main calibration mark in each image.

4. The virtual-real alignment method for MR equipment according to claim 3, characterized in that: The step of respectively verifying the calibration marks in pairs comprises: According to the second poses of the calibration markers in the combination of two by two, the estimated relative pose between the two calibration markers in the combination is calculated, and the errors of the rotation part and the translation part between the estimated relative pose and the actual relative pose are calculated. If any error exceeds the corresponding second error threshold, the two calibration markers fail the verification; otherwise, the two calibration markers pass the verification.

5. The virtual-real alignment method for MR equipment according to claim 1, characterized in that: The removing of abnormal calibration marks according to the pose of the physical reference object in the real world coordinate system includes: According to the second pose of each calibration mark and the pose of each calibration mark relative to the physical reference object, the pose of the physical reference object in the real world coordinate system is calculated as a plurality of poses to be verified; Calculating the mean of the rotation part and the translation part for each of the plurality of postures to be verified; The errors between the rotation part and the translation part of each posture to be checked and the corresponding mean are checked in turn. If any error exceeds the corresponding first error threshold, the corresponding calibration mark is removed as an abnormal calibration mark.

6. The virtual-real alignment method for MR equipment according to claim 1, characterized in that: The second pose of the calibration mark in each image in the real world coordinate system is calculated according to the external parameters of the MR device, the pose of the MR device and the first pose by the following formula: in, represents the second pose of the calibration marker m in the real world coordinate system wr, represents the position of the MR device, that is, the position of the MR device b in the real world coordinate system wr, Represents the relative position of camera c and device b on the MR device, which is a pre-calibrated external parameter. represents the first pose of the calibration marker m relative to the camera c.

7. The virtual-real alignment method for MR equipment according to claim 1, characterized in that: The order of the multiple layers from front to back is: perspective window, virtual layer and real layer; the perspective window is used to determine the shape of the perspective area, and the perspective area is composed of predetermined observation area vertices; the virtual layer is used to display a predetermined virtual image in the application scenario of the MR device; the real layer is used to display the real image taken by the camera on the MR device.

Images