Augmented reality glasses high-precision automatic space registration method based on QR two-dimensional code

By calculating the spatial transformation between the reference frame and the QR code coordinate system, and combining it with the positioning of augmented reality glasses, the problem of insufficient spatial positioning accuracy between the real surgical scene and image guidance in the surgical navigation system was solved. This achieved a high-precision fusion of the virtual model and the real scene, improving the performance and clinical application effect of the surgical navigation system.

CN118967765APending Publication Date: 2024-11-15SHANGHAI FUDAN GRANDHORIZON INFORMATION TECH INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411253365.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing surgical navigation systems suffer from insufficient accuracy in spatial positioning and registration between the actual surgical scene and the image-guided space, which affects the quality of surgery.

Method used

By calibrating and calculating the spatial transformation between the reference frame coordinate system and the QR code coordinate system, and combining this with the positioning of the QR code by the augmented reality glasses, the spatial transformation between the reference frame coordinate system and the augmented reality glasses coordinate system is achieved, thus realizing high-precision automatic spatial registration.

Benefits of technology

It achieves high-precision overlay of virtual models and real scenes in augmented reality surgical navigation, improving the accuracy and practicality of surgery, and is suitable for clinical surgery and medical teaching.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967765A_ABST
    Figure CN118967765A_ABST
Patent Text Reader

Abstract

The invention provides an augmented reality glasses high-precision automatic space registration method based on a QR two-dimensional code, and belongs to the field of augmented reality technology and application. According to the method, firstly, space transformation between a reference frame coordinate system capable of being tracked by a space locator and a QR two-dimensional code coordinate system is obtained through calibration calculation, and then space transformation between the QR two-dimensional code coordinate system and a coordinate system established by augmented reality glasses is obtained through real-time tracking of the augmented reality glasses; and thus, spatial transformation between the reference frame coordinate system and the augmented reality glasses coordinate system is obtained. Through the transformation, the model in the reference frame coordinate system can be transformed into the coordinate system of the augmented reality glasses, so that the model is accurately registered with a real object in the reference frame coordinate system in spatial position when being observed through the glasses. The method is simple to implement and reliable in precision, and can be integrated in the existing surgical navigation system to realize augmented reality navigation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention introduces a high-precision automatic spatial registration method for augmented reality glasses based on QR two-dimensional codes, and belongs to the field of augmented reality technology and applications. Background Art

[0002] As a medical auxiliary positioning device, the surgical navigation system can provide doctors with real-time intraoperative information, effectively making up for the shortcomings of traditional surgical methods and improving the quality of surgery. From the clinical application, it can be seen that the core function of the surgical navigation system is to use images to locate and guide surgical instruments. Therefore, the positioning and registration accuracy of the real surgical scene and the image-guided space will directly affect the performance of the surgical navigation system.

[0003] In recent years, the rapid development of augmented reality glasses has made it easier for doctors to directly present more intuitive medical images. In addition, augmented reality glasses can track objects in real space, especially QR codes, which are the most accurate. This builds a bridge for doctors to combine the glasses space with the real world space. Summary of the invention

[0004] The purpose of the present invention is to provide a high-precision automatic spatial registration method for augmented reality glasses based on QR two-dimensional codes to obtain the position of the reference frame in the augmented reality glasses, so as to make the surgical navigation technology using augmented reality glasses more accurate, practical and convenient in clinical applications.

[0005] The technical solution of the present invention is: by system calibration, the spatial transformation between the reference frame coordinate system and the QR two-dimensional code coordinate system is calculated. Then, the spatial transformation between the QR two-dimensional code coordinate system and the augmented reality glasses coordinate system is obtained by positioning the QR two-dimensional code using augmented reality glasses. The spatial transformation between the reference frame coordinate system and the augmented reality glasses coordinate system is obtained by combining the two coordinate transformations.

[0006] The method of the present invention is achieved by the following steps:

[0007] 1. Using system calibration to obtain the spatial transformation between the reference frame coordinate system and the QR code coordinate system, the steps of the method are:

[0008] (1) Using the QR code to establish the QR code coordinate system, such as Figure 2 As shown, define the origin O, the x-axis and the y-axis, and take points A and B on the x-axis and y-axis respectively.

[0009] (2) The spatial locator establishes a spatial locator coordinate system, and the reference frame establishes a reference frame coordinate system. The reference frame can be tracked in real time by the spatial locator to obtain a transformation from the reference frame coordinate system to the spatial locator coordinate system, which is recorded as T reference2polaris.

[0010] (3) Fix the reference frame and the QR code together, keep the relative position between the two fixed, and use the probe to obtain points O, A, and B on the spatial locator O (x O , y O , z O )、A(x A , y A , z A )、B(x B , y B , z B ). Then, in the QR code coordinate system, the unit vectors u and v in the x-axis and y-axis directions can be obtained through the coordinates of points O, A, and B in the spatial locator coordinate system:

[0011] (1)

[0012] (2)

[0013] According to the right-hand rule, the unit vector n=u×v in the z-axis direction.

[0014] The coordinates of point O in the QR code coordinate system (0, 0, 0) and the coordinates of point O in the spatial locator coordinate system (x O , y O , z O ), we can get the translation matrix T as:

[0015] (3)

[0016] Use unit vectors u, v, n to represent the rotation matrix R:

[0017] (4)

[0018] Therefore, we can obtain the transformation from the spatial locator coordinates to the QR code coordinate system:

[0019] (5)

[0020] in:

[0021] (6)

[0022] (7)

[0023] (8)

[0024] Then, the transformation from the reference frame coordinate system to the QR code coordinate system can be obtained:

[0025] T reference2QR =T reference2polaris ·T polaris2QR (9)

[0026] 2. Use the augmented reality glasses to track the QR code and obtain the transformation from the QR code coordinate system to the augmented reality glasses coordinate system.

[0027] The idea of ​​this method is that the augmented reality glasses automatically establish the augmented reality glasses coordinate system. Through the augmented reality glasses recognizing the QR two-dimensional code, a spatial transformation relationship T between the augmented reality glasses coordinate system and the QR two-dimensional code coordinate system is established. QR2AR .

[0028] 3. Use T obtained in step 2 and step 3 reference2QR and T QR2AR , the spatial transformation T between the augmented reality glasses coordinate system and the reference frame coordinate system can be obtained reference2AR = T reference2QR ·T QR2AR , achieving spatial registration for augmented reality.

[0029] The present invention provides a technical solution for high-precision automatic spatial registration of augmented reality glasses based on QR two-dimensional codes, which has the following advantages over the prior art:

[0030] This method can use augmented reality glasses to superimpose virtual three-dimensional models onto real scenes through augmented reality technology. It can be applied in clinical surgery to superimpose the three-dimensional models established by CT / MR and the constructed surgical path on real patients to guide doctors in surgery and realize augmented reality navigation. It can also be applied to medical teaching to help medical students better understand the relationship between various tissues and organs of the human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation of the present invention, the drawings required for use in the specific implementation are briefly introduced below.

[0032] Figure 1 Flow chart of the method of the present invention

[0033] Figure 2 Schematic diagram of the QR code coordinate system

[0034] Figure 3 Schematic diagram of coordinate transformation of the method of the present invention DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] In order to make the description of the present disclosure more detailed and complete, the following provides an illustrative description of the implementation and specific embodiments of the present invention; however, this is not the only form of implementing or using the specific embodiments of the present invention. The implementation covers the features of multiple specific embodiments and the method steps and sequences used to construct and operate these specific embodiments. However, other specific embodiments can also be used to achieve the same or equivalent functions and step sequences.

[0037] like Figure 1 and Figure 3 As shown, a high-precision automatic spatial registration method for augmented reality glasses based on QR two-dimensional codes includes the following steps:

[0038] (1) Fix the reference frame and the QR code, and keep their relative position relationship unchanged. At the same time, establish the reference frame coordinate system and the QR code coordinate system (such as Figure 2 shown);

[0039] (2) Calibrate the established QR code coordinate system under the tracking of the spatial locator to obtain the transformation relationship T between the QR code coordinate system and the spatial locator coordinate system polaris2QR The specific calibration process is as follows:

[0040] Use the probe to obtain points O, A, and B in the space locator O (x O , y O , z O )、A(x A , y A , z A )、B(x B , y B ,z B ). Then in the QR code coordinate system, the unit vectors u and v in the x-axis and y-axis directions can be obtained through the coordinates of points O, A, and B in the spatial locator coordinate system:

[0041]

[0042]

[0043] According to the right-hand rule, the unit vector n=u×v in the z-axis direction.

[0044] The coordinates of point O in the QR code coordinate system (0, 0, 0) and the coordinates of point O in the spatial locator coordinate system (xO , y O , z O ), we can get the translation matrix T as:

[0045]

[0046] Use unit vectors u, v, n to represent the rotation matrix R:

[0047]

[0048] Therefore, we can get the transformation from the spatial locator coordinates to the QR code coordinate system:

[0049]

[0050] in:

[0051]

[0052]

[0053]

[0054] (3) By tracking the reference frame with the spatial positioner, the transformation relationship between the spatial positioner coordinate system and the reference frame coordinate system is obtained: reference2polaris , combined with the transformation relationship T obtained in step (2) polaris2QR , we can get the transformation relationship T between the reference frame coordinate system and the QR code coordinate system reference2QR =T reference2polaris ·T polaris2QR ;

[0055] (4) Establishing the coordinate system of augmented reality glasses;

[0056] (5) By tracking the QR code with the augmented reality glasses, the position of the QR code in the field of view of the augmented reality glasses is obtained, thereby obtaining the transformation relationship T between the QR code coordinate system and the augmented reality glasses coordinate system QR2AR ;

[0057] (6) Combining steps (3) and (5), the transformation relationship between the augmented reality glasses coordinate system and the reference frame coordinate system can be obtained: reference2AR = T reference2QR ·T QR2AR In this way, the virtual objects in the reference coordinate system can be transformed and displayed in the corresponding positions in the augmented reality glasses, completing the precise fusion and registration of the augmented reality.

[0058] This method can be used in augmented reality surgical navigation, with the following specific applications:

[0059] (1) Through virtual 3D reconstruction of real patients’ CT / MR, a virtual model is obtained. At the same time, the surgical plan can be made and the surgical approach can be determined in the virtual model;

[0060] (2) Before surgery, the patient space is established through the reference frame, and the image space established by CT / MR is aligned with the real patient through the tracking of the reference frame by the spatial positioner. In this way, the image space established by CT / MR, the virtual model obtained through 3D reconstruction, the surgical plan, the determined surgical approach and other information can be transformed into the reference frame coordinate system;

[0061] (3) The transformation relationship between the reference frame coordinate system and the augmented reality glasses coordinate system can be obtained by the method of the present invention, so that all information in the reference frame coordinate system can be transformed into the augmented reality glasses coordinate system;

[0062] (4) During the operation, the doctor wears augmented reality glasses and can see the virtual model, surgical approach, and other effects superimposed on the real patient in the field of view of the glasses, helping the doctor to perform the operation better.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-precision automatic spatial registration method for augmented reality glasses based on QR codes, characterized in that: include: Establish the QR code coordinate system, reference frame coordinate system, spatial locator coordinate system, and augmented reality glasses coordinate system; The spatial transformation between the QR code coordinate system and the reference frame coordinate system is obtained through system calibration; Obtain the spatial transformation between the augmented reality glasses coordinate system and the QR code coordinate system; Through two coordinate transformations, the spatial transformation between the reference frame coordinate system and the augmented reality glasses coordinate system is obtained.

2. According to claim 1, it is characterized in that The established coordinate systems are all Cartesian coordinate systems.

3. According to claims 1 and 2, it is characterized in that: The spatial transformation between the QR code coordinate system and the reference frame coordinate system is obtained through system calibration. While keeping the QR code and the reference frame relatively fixed, the probe is used to track the spatial locator to obtain the coordinates of three specified points on the QR code in the spatial locator coordinate system. Simultaneously, the coordinates of these three points in the QR code coordinate system are obtained. The transformation relationship between the QR code coordinate system and the spatial locator coordinate system is calculated using these two sets of coordinates. The transformation relationship between the spatial locator coordinate system and the reference frame coordinate system is obtained by tracking the reference frame using the spatial locator. The transformation relationship between the reference frame coordinate system and the QR code coordinate system is obtained through these two transformation relationships.

4. According to claims 1 and 2, it is characterized in that: The QR code is tracked by the augmented reality glasses to obtain the position of the QR code in the augmented reality glasses space, and the transformation relationship between the augmented reality glasses coordinate system and the QR code coordinate system is obtained.

5. According to claims 1, 2, 3, and 4, it is characterized in that: The transformation relationship between the reference frame coordinate system and the QR code coordinate system is obtained through the spatial transformation relationship between the reference frame coordinate system and the QR code coordinate system, and the spatial transformation relationship between the augmented reality glasses coordinate system and the QR code coordinate system.