Endoscope projection method and system based on augmented reality

Abstract

The present invention provides an endoscope projection method and system based on augmented reality, comprising the following steps: performing a preoperative CT or MR scan on a patient to obtain medical scan data of a target part; reconstructing the scan data in three dimensions and inputting the data into a graph convolutional network to segment the target surgical part; establishing a surgical scene coordinate system, and using an optical tracking system to align the three-dimensional model with the patient and the position and posture of the endoscope end; tracking the position and posture of the endoscope in real time through the optical tracking system and the built-in three-axis accelerometer of the endoscope and aligning it with the three-dimensional model in real time; based on the coordinate transformation and camera projection model of the real-time registration, superimposing the endoscope image on the three-dimensional model to obtain an augmented reality image of the part shown by the endoscope, and displaying it on a display device. The present invention enables doctors to simultaneously obtain the precise structural information of the three-dimensional image and the rich real detail information of the endoscope image, thereby improving the accuracy and safety of the operation.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and medical technology, and in particular relates to an endoscope projection method and system based on augmented reality. Background Art

[0002] In Image Guided Surgery (IGS), doctors use surgical navigation technology to fuse the position and images of instruments during surgery with the patient's anatomical structure images. The fused endoscopic images and preoperative three-dimensional scanning information such as MR / CT can help doctors improve the accuracy and safety of surgery and enhance doctors' perception of patient information.

[0003] Augmented Reality (AR) is a technology that can cleverly integrate virtual information with the real world. The endoscopic projection method based on augmented reality generates an endoscopic projection view in a virtual preoperative 3D model through 3D reconstruction and spatial coordinate transformation, and supplements the real detail information of the endoscopic image with the accurate structural information of the 3D model, providing doctors with more intuitive and realistic surgical information. Summary of the invention

[0004] To solve the above technical problems, the present invention proposes an endoscopic projection method and system based on augmented reality, which enables doctors to simultaneously obtain the precise structural information of three-dimensional images and the rich real detail information of endoscopic images, thereby improving the accuracy and safety of surgery.

[0005] On the one hand, to achieve the above-mentioned purpose, the present invention provides an endoscope projection method based on augmented reality, comprising the following steps:

[0006] Acquiring three-dimensional medical image data of a target surgical site, and preprocessing the three-dimensional medical image data;

[0007] Performing three-dimensional volume reconstruction on the preprocessed three-dimensional medical image data to obtain a volume-reconstructed three-dimensional model, and segmenting the three-dimensional model using a graph convolutional network;

[0008] Establishing a surgical scene coordinate system, and using an optical tracking system to register the segmented three-dimensional model with the intraoperative image;

[0009] Acquire the position and posture of the endoscope in real time, and perform real-time registration of the position and posture of the endoscope with the segmented three-dimensional model;

[0010] An endoscopic image is obtained, and based on the real-time registered coordinate transformation and camera projection model, the endoscopic image is superimposed and displayed on the segmented three-dimensional model using a spatial projection method to obtain an augmented reality image of the endoscope.

[0011] Optionally, the method for preprocessing the three-dimensional medical image data is:

[0012] Based on the image intensity, the three-dimensional medical image data is subjected to threshold segmentation.

[0013] Optionally, the method for obtaining the reconstructed three-dimensional model is:

[0014] The image information of each voxel in the preprocessed three-dimensional medical image data is used to perform three-dimensional volume reconstruction on the patient's body data to obtain a reconstructed three-dimensional model.

[0015] Optionally, the method of segmenting the three-dimensional model using a graph convolutional network is:

[0016] Establishing a mapping weight matrix, merging the data features of the three-dimensional model, and obtaining a node connection graph;

[0017] A graph convolutional network is used to extract interaction features between nodes in the node connection graph to obtain a segmentation result of the three-dimensional model.

[0018] Optionally, the method for obtaining the position and posture of the endoscope in real time is:

[0019] The optical tracking system and the built-in three-axis accelerometer of the endoscope are used to obtain the position and posture of the endoscope in real time.

[0020] Optionally, the method for obtaining an augmented reality image of an endoscope is:

[0021] Mapping the points on the segmented three-dimensional model and the endoscope camera parameters to the endoscope image pixel coordinate system through a registration relationship to obtain a one-to-one correspondence between the segmented three-dimensional model and the pixel coordinates;

[0022] The endoscopic image is superimposed on the corresponding three-dimensional model point to obtain an endoscopic projection based on augmented reality.

[0023] On the other hand, to achieve the above-mentioned purpose, the present invention provides an endoscopic projection system based on augmented reality, comprising an acquisition module, a volume reconstruction module, a first registration module, a second registration module and an enhancement implementation module;

[0024] The acquisition module is used to acquire three-dimensional medical image data of the target surgical site and pre-process the three-dimensional medical image data;

[0025] The volume reconstruction module is used to perform three-dimensional volume reconstruction on the preprocessed three-dimensional medical image data to obtain a volume-reconstructed three-dimensional model, and segment the three-dimensional model using a graph convolutional network;

[0026] The first registration module is used to establish a surgical scene coordinate system, and to register the segmented three-dimensional model with the intraoperative image using an optical tracking system;

[0027] The second registration module is used to obtain the position and posture of the endoscope in real time, and to perform real-time registration between the position and posture of the endoscope and the segmented three-dimensional model;

[0028] The enhancement realization module is used to obtain the endoscopic image, and based on the real-time registration coordinate transformation and camera projection model, the endoscopic image is superimposed and displayed on the segmented three-dimensional model using the space projection method to obtain an augmented reality image of the endoscope.

[0029] Optionally, the optical tracking system comprises: an optical tracker, a rigid bracket, an optical locator, a navigator host and a navigation image display;

[0030] The optical tracker is used to reflect the infrared light emitted by the optical locator;

[0031] The rigid bracket is used to install the optical tracker;

[0032] The optical locator is used to receive the infrared light reflected by the optical tracker and perform three-dimensional spatial positioning on the optical tracker;

[0033] The navigator host is used to provide computing resources for real-time spatial positioning, navigation image registration and endoscopic image projection;

[0034] The navigation image display is used to display the endoscope projection result of augmented reality.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] The present invention proposes an endoscopic projection method and system based on augmented reality, which reconstructs preoperative 3D images such as CT / MR, segments the surgical target area using a graph convolutional network, establishes a surgical scene coordinate system, tracks the endoscope posture in real time through an optical tracking system and an endoscope three-axis accelerometer, and aligns it with the patient's CT model, projects the endoscopic image onto its corresponding 3D model area, and implements augmented reality endoscopic projection on a display device, so that doctors can simultaneously obtain the precise structural information of the 3D image and the rich real detail information of the endoscopic image, thereby improving the accuracy and safety of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 The figure is a flow chart of an endoscopic projection method based on augmented reality according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Embodiment 1

[0042] like Figure 1 As shown, the present invention provides an endoscope projection method based on augmented reality, characterized in that it includes the following steps:

[0043] Perform CT or MR scans on the patient's surgical target area before surgery to obtain three-dimensional medical imaging data of the target area;

[0044] The 3D image data is preprocessed by threshold segmentation based on the image intensity, and the patient's body data is reconstructed in 3D using the image information of each voxel in the image. The reconstructed 3D model is input into the trained graph convolutional network for deep learning, and each part in the 3D model is refined and segmented to accurately locate the surgical target.

[0045] Establish the surgical scene coordinate system and use the optical tracking system to align the 3D model with the patient and the position and posture of the endoscope tip;

[0046] The position and posture of the endoscope are tracked in real time through the optical tracking system and the built-in three-axis accelerometer of the endoscope and aligned with the three-dimensional model in real time;

[0047] Based on the real-time registration of coordinate transformation and camera projection model, the endoscopic image is superimposed on the three-dimensional model through the spatial projection method to obtain an augmented reality image of the part shown by the endoscope and display it on the display device.

[0048] Furthermore, volume rendering is a method for directly visualizing volume data such as 3D medical images, using information such as the opacity and color of each voxel of the 3D image to visualize the volume data in 3D. Starting from the viewpoint, the image is tracked along the line of sight, and the output image is calculated by accumulating the opacity and grayscale values ​​of each sample point on the line of sight.

[0049] Furthermore, the graph convolutional network includes the following establishment process:

[0050] Obtain relevant medical imaging data and expert annotations as training sets;

[0051] Establish a trainable mapping weight matrix to merge similar features in the data and form a node connection graph;

[0052] Use graph convolutional networks to extract interaction features between nodes and obtain predicted segmentation results;

[0053] Calculate the loss between the predicted segmentation result and the expert annotation result, adjust the network parameters, and repeat the above steps to optimize the network performance.

[0054] The specific implementation scheme of the graph convolutional network used in this embodiment is as follows:

[0055] For the tomographic image I∈R input to the network 2 , use convolutional neural network for feature extraction to obtain features at different positions of the image Where L = W × H represents the position of the point in the image, and C represents the feature dimension. In order to further establish the connection between features, a learnable weight matrix B = [b1,…,b N ], according to the following formula:

[0056]

[0057] Map the obtained features to the interaction space In the process, a connection graph is constructed and graph convolution feature extraction is performed. For each node feature v i , through the learnable weight matrix B, features with similar characteristics are automatically aggregated into one node;

[0058] The interactive features of the constructed connection graph are extracted according to the following formula:

[0059] Z=GVW g =((IA g )V)W g

[0060] After Laplace smoothing of the features, the learnable adjacency matrix W is used g After learning the interaction relationship between nodes, the output features are mapped back to the original space according to the following formula, and network segmentation prediction is performed:

[0061]

[0062] In order to reduce network parameters,

[0063] Further, the optical tracking system comprises an optical tracker, a rigid bracket, an optical positioner, a navigator host and a navigation image display;

[0064] The optical tracker is a reflective ball that can reflect infrared light. It is used to reflect the infrared light emitted by the optical locator. The optical locator receives the infrared light reflected by the optical tracker and performs three-dimensional spatial positioning of the optical tracker. The optical tracker is installed on a rigid bracket, and the rigid bracket is fixedly mounted on the endoscope and the patient's body to achieve alignment in the surgical scene coordinate system. The navigator host provides computing resources for real-time spatial positioning, navigation image alignment, endoscopic image projection and other operations. The navigation image display is used to display the endoscopic projection results of augmented reality.

[0065] Furthermore, the endoscope is provided with a three-axis accelerometer for obtaining the rotation angle of the endoscope tip relative to the gravity field and for use in the registration algorithm.

[0066] The specific implementation scheme of the registration used in this embodiment is as follows:

[0067] A rigid bracket is fixed somewhere on the patient's body and on the endoscope. The optical positioner tracks the position of the optical tracker on the bracket, and the position T of the patient's body bracket and the endoscope bracket in the surgical scene coordinate system can be obtained. p1 and T e1 ;

[0068] After obtaining the positions of the patient's body support and the endoscope support, the three-dimensional model and the patient's body support, as well as the inherent coordinate transformation relationship T between the endoscope end and the endoscope support are used. p2 and T e2 , we can obtain the three-dimensional model coordinates T in the surgical scene coordinate system p =T p1 ·T p2 and the endoscope tip coordinate T e =T e1 ·T e2 Therefore, the transformation relationship between the three-dimensional model coordinates and the endoscope end coordinates can be obtained. pe =T p ·T pe .

[0069] During the operation, the endoscope needs to move and rotate according to the needs of the operation. During the rotation process, the optical locator may fail to locate the optical tracking ball. In order to prevent such a situation from happening, the endoscope of the present invention is provided with a three-axis accelerometer, which can calculate the rotation angle T of the endoscope end. ω , using this rotation angle to update the endoscope tip coordinate T in real time e =T e1 ·T e2 ·Tω , to ensure that the coordinates of the endoscope at any position and posture can be obtained.

[0070] Furthermore, the camera module of the endoscope used is equipped with a high-precision three-axis accelerometer, which can measure the rolling angle of the endoscope relative to the gravity field in all directions. The endoscope inevitably needs to be rotated during the operation, so the reflective ball of the optical tracking system will be blocked at certain angles. Therefore, using a three-axis accelerometer to estimate the posture of the endoscope can improve the accuracy of the alignment and reduce restrictions on the scope of use for doctors.

[0071] Furthermore, the points on the three-dimensional model are mapped to the image pixel coordinate system through the registration relationship and the internal parameters of the endoscope camera. After obtaining a one-to-one correspondence between the three-dimensional model and the pixel coordinates, the endoscopic image is superimposed on the model points corresponding to the image to realize endoscopic projection based on augmented reality.

[0072] The specific implementation scheme of the projection method used in this scheme is as follows:

[0073] For any point V = (x, y, z) on the three-dimensional model, the coordinate transformation relationship T between the three-dimensional model and the endoscope end is used. pe , as well as the relationship between the camera model and its projection matrix, map V to the image coordinate system:

[0074]

[0075] where w v Indicates the zoom ratio based on the camera focal length;

[0076] Using the camera's intrinsic parameter matrix P, we can get the transformation relationship between the image coordinate system and the pixel coordinate system, so we can get the position of V' in the imaging pixel coordinate system:

[0077]

[0078] In summary, it is possible to determine whether point V on the three-dimensional model is visible in the endoscopic image, and through the corresponding relationship, the endoscopic image is superimposed on the visible point of the three-dimensional model to achieve augmented reality endoscopic projection.

[0079] Embodiment 2

[0080] The present invention discloses an endoscope projection system based on augmented reality, comprising an acquisition module, a volume reconstruction module, a first registration module, a second registration module and an enhancement realization module;

[0081] The acquisition module is used to acquire three-dimensional medical image data of the target surgical site and pre-process the three-dimensional medical image data;

[0082] The volume reconstruction module is used to perform three-dimensional volume reconstruction on the preprocessed three-dimensional medical image data to obtain a volume-reconstructed three-dimensional model, and segment the three-dimensional model using a graph convolutional network;

[0083] The first registration module is used to establish a surgical scene coordinate system, and to register the segmented three-dimensional model with the intraoperative image using an optical tracking system;

[0084] The second registration module is used to obtain the position and posture of the endoscope in real time, and to perform real-time registration between the position and posture of the endoscope and the segmented three-dimensional model;

[0085] The enhancement realization module is used to obtain the endoscopic image, and based on the real-time registration coordinate transformation and camera projection model, the endoscopic image is superimposed and displayed on the segmented three-dimensional model using the space projection method to obtain an augmented reality image of the endoscope.

[0086] The optical tracking system comprises: an optical tracker, a rigid bracket, an optical locator, a navigator host and a navigation image display;

[0087] The optical tracker is used to reflect the infrared light emitted by the optical locator;

[0088] The rigid bracket is used to install the optical tracker;

[0089] The optical locator is used to receive the infrared light reflected by the optical tracker and perform three-dimensional spatial positioning on the optical tracker;

[0090] The navigator host is used to provide computing resources for real-time spatial positioning, navigation image registration and endoscopic image projection;

[0091] The navigation image display is used to display the endoscope projection result of augmented reality.

[0092] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An endoscope projection system based on augmented reality, characterized in that: It includes an acquisition module, a volume reconstruction module, a first registration module, a second registration module and an enhancement implementation module; The acquisition module is used to acquire three-dimensional medical image data of the target surgical site and pre-process the three-dimensional medical image data; The volume reconstruction module is used to perform three-dimensional volume reconstruction on the preprocessed three-dimensional medical image data to obtain a volume-reconstructed three-dimensional model, and segment the three-dimensional model using a graph convolutional network; The first registration module is used to establish a surgical scene coordinate system, and to register the segmented three-dimensional model with the intraoperative image using an optical tracking system; The second registration module is used to obtain the position and posture of the endoscope in real time, and to perform real-time registration between the position and posture of the endoscope and the segmented three-dimensional model; The enhancement realization module is used to obtain an endoscopic image, and based on the real-time registration coordinate transformation and camera projection model, the endoscopic image is superimposed and displayed on the segmented three-dimensional model using a spatial projection method to obtain an augmented reality image of the endoscope; The optical tracking system comprises: an optical tracker, a rigid bracket, an optical locator, a navigator host and a navigation image display; The optical tracker is used to reflect the infrared light emitted by the optical locator; The rigid bracket is used to install the optical tracker; The optical locator is used to receive the infrared light reflected by the optical tracker and perform three-dimensional spatial positioning on the optical tracker; The navigator host is used to provide computing resources for real-time spatial positioning, navigation image registration and endoscopic image projection; The navigation image display is used to display the endoscope projection result of augmented reality; The graph convolutional network includes the following establishment process: Obtain relevant medical imaging data and expert annotations as training sets; Establish a trainable mapping weight matrix to merge similar features in the data and form a node connection graph; Use graph convolutional networks to extract interaction features between nodes and obtain predicted segmentation results; Calculate the loss between the predicted segmentation result and the expert annotation result, adjust the network parameters, and repeat the steps to optimize the network performance; The specific implementation scheme of the graph convolutional network used is as follows: For the tomographic image I∈R input to the network 2 , use convolutional neural network for feature extraction to obtain features at different positions of the image Where L = W × H represents the position of the point in the image, and C represents the feature dimension. In order to further establish the connection between features, a learnable weight matrix B = [b1,…,b N ], according to the following formula: Map the obtained features to the interaction space In the process, a connection graph is constructed and graph convolution feature extraction is performed. For each node feature v i , through the learnable weight matrix B, features with similar characteristics are automatically aggregated into one node; The interactive features of the constructed connection graph are extracted according to the following formula: Z=GVW g =((IA g )V)W g After Laplace smoothing of the features, the learnable adjacency matrix W is used g After learning the interaction relationship between nodes, the output features are mapped back to the original space according to the following formula, and network segmentation prediction is performed: In order to reduce network parameters,

Images