An endoscope augmented reality method based on QR code positioning technology

Abstract

The present invention provides an endoscope augmented reality method based on QR two-dimensional code positioning technology, which belongs to the field of augmented reality technology and application. The present invention first fixes the QR two-dimensional code at a specified position of the endoscope display, and establishes a world coordinate system through the QR two-dimensional code, and then superimposes and displays the model with the endoscope image displayed by the endoscope display according to the position of the model in the world coordinate system. Through real-time positioning by a spatial locator, as the endoscope image changes, the model in the augmented reality glasses changes accordingly, and always maintains real-time and accurate superposition display with the endoscope image. The method is simple to implement and has reliable accuracy. It can be integrated with the existing endoscope navigation system to realize endoscope augmented reality navigation.

Description

Technical Field

[0001] The invention introduces an endoscope augmented reality method based on QR two-dimensional code positioning technology, belonging to the field of augmented reality technology and applications. Background Art

[0002] As a medical auxiliary positioning device, the surgical navigation system can provide doctors with real-time intraoperative information, effectively making up for the shortcomings of traditional surgical methods and improving the quality of surgery. From the clinical application, it can be seen that the core function of the surgical navigation system is to use images to locate and guide surgical instruments. Therefore, the positioning and registration accuracy of the real surgical scene and the image-guided space will directly affect the performance of the surgical navigation system.

[0003] Endoscopic technology plays an important role in surgical operations. It achieves accurate diagnosis and treatment through minimally invasive methods, significantly reducing surgical trauma and postoperative recovery time. It is also widely used in gastroenterology, gynecology, urology and other fields. With the continuous development of high-definition imaging, 3D technology and robot-assisted surgery, endoscopy has further improved the accuracy and safety of surgery, greatly improving patients' diagnosis and treatment experience and quality of life.

[0004] In recent years, the rapid development of augmented reality technology has provided doctors with more intuitive medical images and models. In addition, augmented reality glasses can track objects in real space, especially QR codes, which are the most accurate. This builds a bridge for doctors to combine the glasses space with the real world space.

[0005] At present, the traditional surgical navigation system transmits the position of the endoscope to the navigation system in real time through the endoscope adapter, and can guide the operation through the endoscope. However, due to the physical structure of the endoscope, its display range is very limited. In addition, because the application scenarios of endoscopes are mostly minimally invasive surgeries, its activity space can be adjusted in a very small range, which limits the development of minimally invasive surgery. Summary of the invention

[0006] The purpose of the present invention is to provide an endoscope augmented reality method based on QR two-dimensional code positioning technology, which combines endoscope, spatial positioning technology and augmented reality technology to make it more accurate, practical and convenient in clinical applications.

[0007] The technical solution of the present invention is: a world coordinate system is established through a QR two-dimensional code, the QR two-dimensional code is fixed at a specified position of an endoscope display, and the model is superimposed and displayed with the endoscope image displayed on the endoscope display using augmented reality technology according to the position of the model in the world coordinate system. By positioning the endoscope with a spatial locator, if the endoscope image changes, the model in the augmented reality glasses changes accordingly, and always maintains a real-time and accurate superimposed display with the endoscope image.

[0008] The coordinate systems included in the present invention include: a world coordinate system, a spatial positioner coordinate system, an augmented reality glasses coordinate system, an endoscope adapter coordinate system, an endoscope image coordinate system, and an endoscope display coordinate system;

[0009] The world coordinate system is established through a QR code;

[0010] The spatial locator coordinate system is established by the spatial locator;

[0011] The augmented reality glasses coordinate system is established by the augmented reality glasses internal coordinate system;

[0012] The endoscope adapter coordinate system is established by an endoscope adapter fixed to the endoscope;

[0013] The endoscopic image coordinate system is established by the endoscopic imaging plane;

[0014] The endoscope display coordinate system is established by the endoscope display.

[0015] The method of the present invention is achieved by the following steps:

[0016] 1. Transformation relationship T between the spatial locator coordinate system and the endoscope image coordinate system tracker2image

[0017] (1) The endoscope adapter is fixed at a designated position of the endoscope. The endoscope adapter can be tracked in real time by the spatial positioner to obtain a transformation relationship T between the spatial positioner coordinate system and the endoscope adapter coordinate system. tracker2adapter ;

[0018] (2) Transformation relationship T between the endoscope adapter coordinate system and the endoscope image coordinate system adapter2image , obtained through system calibration, the specific method is as follows:

[0019] ① Take a chessboard image through the endoscope, define the origin O, the x-axis and the y-axis, and take points A and B on the x-axis and the y-axis respectively.

[0020] ② Use a probe that can be tracked by the spatial locator to obtain points O, A, and B in the endoscope adapter coordinate system O (x O , y O , z O )、A(x A , y A , z A )、B(x B , y B , z B ). Then, in the endoscope image coordinate system, the unit vectors u and v in the x-axis and y-axis directions can be obtained respectively through the coordinates of points O, A, and B in the endoscope adapter coordinate system:

[0021]

[0022]

[0023] According to the right-hand rule, the unit vector n=u×v in the z-axis direction.

[0024] The coordinates of point O in the endoscope image coordinate system (0, 0, 0) and the coordinates of point O in the endoscope adapter coordinate system (x O , y O , z O ), we can get the translation matrix T as:

[0025]

[0026] Use unit vectors u, v, n to represent the rotation matrix R:

[0027]

[0028] Then, we can obtain the transformation from the endoscope adapter coordinates to the endoscope image coordinate system:

[0029]

[0030] in:

[0031]

[0032] (3) Transformation relationship between the spatial positioning instrument coordinate system and the endoscope image coordinate system T tracker2image =T tracker2adapter *T adapter2image .

[0033] 2. Transformation relationship between the endoscope image coordinate system and the world coordinate system T image2QR

[0034] (1) The QR code is fixed at a specified position on the endoscope display. The transformation relationship between the world coordinate system and the endoscope display coordinate system is T display2QR , always remains unchanged;

[0035] (2) The image presented by the endoscope imaging plane is directly displayed in a fixed area of ​​the endoscope display. The transformation relationship between the endoscope image coordinate system and the endoscope display coordinate system is T image2display , always remains unchanged;

[0036] (3) Transformation relationship between the endoscope image coordinate system and the world coordinate system T image2QR= T image2display* T display2QR。

[0037] 3. Transformation relationship T between the world coordinate system and the augmented reality glasses coordinate system QR2AR

[0038] The augmented reality glasses automatically establish the augmented reality glasses coordinate system. The augmented reality glasses recognize the QR code to obtain the spatial transformation relationship T between the augmented reality glasses coordinate system and the world coordinate system. QR2AR .

[0039] 4. Transformation relationship between the spatial locator coordinate system and the augmented reality glasses coordinate system T tracker2AR

[0040] Through steps 1, 2, and 3, the transformation relationship T between the spatial locator coordinate system and the augmented reality glasses coordinate system can be obtained. tracker2AR =T tracker2image *T image2QR *T QR2AR .

[0041] The present invention provides a technical solution for endoscope augmented reality based on QR two-dimensional code positioning technology, which has the following advantages over the prior art:

[0042] This method can use augmented reality glasses to superimpose virtual three-dimensional models on endoscopic images through augmented reality technology, and can follow the model in real time according to the changes in the endoscopic image to maintain a real-time and accurate superposition state. This method can be applied to clinical surgery, superimposing the three-dimensional model established by CT / MR on the endoscope display, superimposing it with the endoscope image in real time to guide doctors in surgery; it can also be applied to medical teaching to help medical students better understand the relationship between various tissues and organs of the human body through endoscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation of the present invention, the drawings required for use in the specific implementation are briefly introduced below.

[0044] Figure 1 The figure is a flow chart of the method of the present invention.

[0045] Figure 2 Schematic diagram of the chessboard for system calibration.

[0046] Figure 3 Schematic diagram of coordinate transformation of the method of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] In order to make the description of the present disclosure more detailed and complete, the following provides an illustrative description of the implementation and specific embodiments of the present invention; however, this is not the only form of implementing or using the specific embodiments of the present invention. The implementation covers the features of multiple specific embodiments and the method steps and sequences used to construct and operate these specific embodiments. However, other specific embodiments can also be used to achieve the same or equivalent functions and step sequences.

[0049] like Figure 1 and Figure 3 As shown, the endoscope augmented reality method based on QR two-dimensional code positioning technology includes the following steps:

[0050] 1. Establish the world coordinate system, spatial locator coordinate system, augmented reality glasses coordinate system, endoscope adapter coordinate system, endoscope image coordinate system, and endoscope display coordinate system respectively;

[0051] 2. Fix the endoscope adapter at the specified position of the endoscope. The endoscope adapter can be tracked in real time by the spatial locator to obtain the transformation relationship T between the spatial locator coordinate system and the endoscope adapter coordinate system tracker2adapter ;

[0052] 3. Transformation relationship between the endoscope adapter coordinate system and the endoscope image coordinate system T adapter2image , obtained through system calibration, the specific method is as follows:

[0053] ① If Figure 2 As shown, a chessboard image is captured through an endoscope, the origin O, the x-axis and the y-axis are defined, and points A and B are taken on the x-axis and the y-axis respectively.

[0054] ② Use a probe that can be tracked by the spatial locator to obtain the coordinates of points O, A, and B in the endoscope adapter coordinate system O (x O , y O , z O )、A(x A , y A , z A )、B(x B , y B , z B ). Then, in the endoscope image coordinate system, the unit vectors u and v in the x-axis and y-axis directions can be obtained through the coordinates of points O, A, and B in the endoscope adapter coordinate system:

[0055]

[0056]

[0057] According to the right-hand rule, the unit vector n=u×v in the z-axis direction.

[0058] The coordinates of point O in the endoscope image coordinate system (0, 0, 0) and the coordinates of point O in the endoscope adapter coordinate system (x O , y O , z O ), we can get the translation matrix T as:

[0059]

[0060] Use unit vectors u, v, n to represent the rotation matrix R:

[0061]

[0062] Thus, we can obtain the transformation from the endoscope adapter coordinates to the endoscope image coordinate system:

[0063]

[0064] in:

[0065]

[0066] 4. Combining steps 2 and 3, the transformation relationship between the spatial locator coordinate system and the endoscope image coordinate system is T tracker2image =T tracker2adapter *T adapter2image .

[0067] 5. Fix the QR code at the specified position of the endoscope display. The transformation relationship between the world coordinate system and the endoscope display coordinate system is T display2QR , always remains unchanged;

[0068] 6. The image presented by the endoscope imaging plane is directly displayed in a fixed area of ​​the endoscope display. The transformation relationship between the endoscope image coordinate system and the endoscope display coordinate system is T image2display , always remains unchanged;

[0069] 7. Combining steps 5 and 6, the transformation relationship between the endoscope image coordinate system and the world coordinate system is T image2QR= T image2display* T display2QR。

[0070] 8. The augmented reality glasses automatically establish the augmented reality glasses coordinate system. By recognizing the QR code with the augmented reality glasses, the spatial transformation relationship between the augmented reality glasses coordinate system and the world coordinate system is obtained. QR2AR .

[0071] 9. Combining steps 4, 7 and 8, we can get the transformation relationship T between the spatial locator coordinate system and the augmented reality glasses coordinate system tracker2AR =T tracker2image *T image2QR *T QR2AR .

[0072] This method can be used in endoscopic augmented reality surgical navigation, with the following specific applications:

[0073] (1) Perform three-dimensional reconstruction using CT / MR data to obtain a virtual model;

[0074] (2) Before surgery, the image space created by CT / MR is registered with the real patient through tracking by the spatial localizer. In this way, the image space created by CT / MR and the virtual model obtained by three-dimensional reconstruction can be transformed into the spatial localizer coordinate system;

[0075] (3) When performing minimally invasive surgery through a calibrated endoscope with an endoscope adapter, the endoscopic image is displayed in real time on the endoscope display, and the surgical navigation can also locate the position of the endoscope in real time through real-time tracking of the endoscope adapter by the spatial positioner;

[0076] (4) The method of the present invention can obtain the transformation relationship between the spatial locator coordinate system and the augmented reality glasses coordinate system, so that all information in the spatial locator coordinate system can be transformed into the augmented reality glasses coordinate system;

[0077] (5) During the operation, the doctor wears augmented reality glasses and can see the effect of superimposing the virtual model and the endoscopic image on the endoscope display in the field of view of the glasses, helping the doctor to perform the operation better.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An endoscope augmented reality method based on QR two-dimensional code positioning technology, characterized in that: include: World coordinate system, spatial locator coordinate system, augmented reality glasses coordinate system, endoscope adapter coordinate system, endoscope image coordinate system, endoscope display coordinate system; The world coordinate system is established through a QR code; The spatial locator coordinate system is established by the spatial locator; The augmented reality glasses coordinate system is established by the augmented reality glasses internal coordinate system; The endoscope adapter coordinate system is established by an endoscope adapter fixed to the endoscope; The endoscopic image coordinate system is established by the endoscopic imaging plane; The endoscope display coordinate system is established by the endoscope display.

2. According to claim 1, it is characterized in that: The established coordinate systems are all Cartesian coordinate systems.

3. According to claims 1 and 2, it is characterized in that: The endoscope adapter is fixed at a designated position of the endoscope, and the endoscope adapter can be tracked in real time by the spatial positioner to obtain a transformation relationship T between the spatial positioner coordinate system and the endoscope adapter coordinate system. tracker2adapter ; Transformation relationship T between the endoscope adapter coordinate system and the endoscope image coordinate system adapter2image , obtained by system calibration; the transformation relationship between the spatial locator coordinate system and the endoscope image coordinate system T tracker2image =T tracker2adapter *T adapter2image .

4. According to claims 1 and 2, it is characterized in that: The QR code is fixed at a specified position on the endoscope display. The transformation relationship between the world coordinate system and the endoscope display coordinate system is T display2QR , always remains unchanged; the image presented by the endoscope imaging plane is directly displayed in a fixed area of ​​the endoscope display, and the transformation relationship between the endoscope image coordinate system and the endoscope display coordinate system is T image2display , remains unchanged; the transformation relationship between the endoscope image coordinate system and the world coordinate system T image2QR= T image2display* T display2QR。 5. According to claims 1 and 2, it is characterized in that: The QR code is tracked by the augmented reality glasses to obtain the position of the QR code in the augmented reality glasses space, and the transformation relationship T between the world coordinate system and the augmented reality glasses coordinate system is obtained. QR2AR .

6. According to claims 1, 2, 3, 4, and 5, it is characterized in that: The transformation relationship between the spatial locator coordinate system and the augmented reality glasses coordinate system T tracker2AR =T tracker2image *T image2QR *T QR2AR .

Images