An image fusion system and method for endoscopic three-dimensional navigation

Abstract

The application discloses an image fusion system and method for endoscope three-dimensional navigation. The system comprises a graphic workstation, an optical positioning marker, an optical positioning system and an endoscope system, wherein the graphic workstation is used for importing pre-examination three-dimensional images, reconstructing and post-processing, extracting image point cloud data and optical point cloud data of a lesion and surrounding tissue structure, matching the image point cloud data and the optical point cloud data based on a point cloud registration algorithm, calculating a space transformation from a coordinate system of the optical positioning system to a coordinate system of the pre-examination three-dimensional images; calculating a space transformation from an endoscope visual angle coordinate system to the coordinate system of the pre-examination three-dimensional images; based on a real-time space position of the endoscope, real-time displaying a current field of view corresponding to a three-dimensional image of an endoscope image, and superimposedly and fusionally displaying the endoscope image. By using the application, the endoscope visual field two-dimensional image is combined with the pre-examination three-dimensional image, so that abundant image information is provided for doctors to perform endoscope examination and is used for auxiliary diagnosis.

Description

Technical Field

[0001] This invention relates to the technical field of endoscopic images, and more particularly to an image fusion system and method for three-dimensional navigation of endoscopes. Background Technology

[0002] CT / MR and other three-dimensional imaging data provide rich anatomical information about the patient's body tissues, including the spatial relationship between lesions and surrounding structures such as blood vessels. In medical image-guided interventional treatments, doctors use these three-dimensional images to determine the relative spatial relationship between the lesion and surrounding tissues and blood vessels, avoiding damage to important anatomical structures during examination and treatment.

[0003] An endoscope is a diagnostic instrument that integrates traditional optics, ergonomics, precision mechanics, modern electronics, mathematics, and software. It can be inserted into the stomach through the mouth or other natural orifices. Therefore, it can see lesions that cannot be detected by X-rays. With the help of an endoscope, doctors can observe ulcers or tumors in the stomach and formulate the best treatment plan for the patient.

[0004] However, the field of view of an endoscope is limited, making it difficult to see the whole lesion clearly. At the same time, it is impossible to accurately distinguish the spatial relationship between the lesion and the surrounding tissues and blood vessels. In addition, since the endoscope is inside the patient's body, the doctor cannot accurately determine its current specific location in the patient's body, which may introduce errors in the localization of the lesion. Summary of the Invention

[0005] To address the aforementioned technical problems in existing technologies, this invention provides an image fusion system for endoscopic three-dimensional navigation, which can fuse and display pre-examination 3D images such as CT / MR images with the current endoscopic field of view image in real time. By installing optical positioning markers on the endoscope to locate its spatial position in real time, the doctor uses the endoscope to scan and view the lesion and surrounding tissue structures from multiple angles. The software reconstructs the spatial point cloud of the lesion and surrounding tissue structures using multiple frames of endoscopic images and registers it with the pre-segmented and reconstructed point cloud in 3D images such as CT / MR images, completing the registration between the human body space and the pre-operative 3D image space. Then, by moving the endoscope in real time, the software provides a 3D image of the current endoscopic field of view and overlays it with the endoscopic image for display.

[0006] This invention provides an image fusion system for endoscopic three-dimensional navigation, including a graphics workstation, optical positioning markers, an optical positioning system, and an endoscope system. The graphics workstation imports and reconstructs pre-examination three-dimensional images, extracts image point cloud data of the lesion and surrounding tissue structures, extracts optical point cloud data of the lesion and surrounding tissue structures in the optical positioning space, matches the image point cloud data and optical point cloud data based on a point cloud registration algorithm, calculates the spatial transformation from the optical positioning system coordinate system to the pre-examination three-dimensional image coordinate system, calculates the spatial transformation from the endoscope's perspective coordinate system to the pre-examination three-dimensional image coordinate system, and displays the current field of view of a three-dimensional image corresponding to the endoscope image in real time based on the endoscope's real-time spatial position, overlaying and fusing it with the endoscope image. Optical positioning markers are installed on the endoscope. The optical positioning system tracks the spatial position of the optical positioning markers in real time. The endoscope system is used for endoscopic examination.

[0007] This invention also provides an image fusion method for endoscopic three-dimensional navigation, comprising: importing a pre-examination three-dimensional image into a graphics workstation; segmenting the lesion and its surrounding tissue structure on the graphics workstation, outputting a two-dimensional image of the segmentation result, converting the two-dimensional image into a triangular patch with only outlines, and further extracting image point cloud data of the lesion and its surrounding tissue structure; installing optical positioning markers on the endoscope, and setting up an optical positioning system for real-time tracking of the spatial position of the optical positioning markers; capturing multiple frames of two-dimensional visual field images of the lesion and its surrounding tissue structure using the endoscope; and based on the multiple frames of two-dimensional visual field images, the graphics workstation fused the multiple frames of two-dimensional visual field images into a three-dimensional image fusion method. The field image is reconstructed into three-dimensional point cloud data of the optical positioning space associated with the optical positioning markers. Optical point cloud data of the lesion and its surrounding tissue structures in the optical positioning space are extracted. The graphics workstation matches the image point cloud data and the optical point cloud data based on the point cloud registration algorithm and calculates the spatial transformation from the coordinate system of the optical positioning system to the coordinate system of the three-dimensional image before examination. The graphics workstation calculates the spatial transformation from the coordinate system of the endoscope viewpoint to the coordinate system of the three-dimensional image before examination. The endoscope is moved, and based on the real-time spatial position of the endoscope, the graphics workstation displays the current field of view of the three-dimensional image corresponding to the endoscope image in real time, and overlays and fuses it with the endoscope image for display.

[0008] In some embodiments, the optical positioning marker is a reflective ball or a black and white checkerboard pattern used for optical positioning.

[0009] In some embodiments, the spatial transformation from the endoscope perspective coordinate system to the pre-examination three-dimensional image coordinate system is obtained based on the spatial transformation from the endoscope perspective coordinate system to the optical positioning mark coordinate system, the spatial transformation from the optical positioning mark coordinate system to the optical positioning system coordinate system, and the spatial transformation from the optical positioning system coordinate system to the pre-examination three-dimensional image coordinate system.

[0010] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows:

[0011] 1. By combining two-dimensional images of the endoscopic field of view with three-dimensional images before the examination, doctors can obtain rich image information to assist in diagnosis during endoscopic examinations;

[0012] 2. The three-dimensional images before the examination can provide a clear and complete 3D map of the lesion and surrounding tissue structures, which makes it convenient for doctors to observe the relative position of the lesion and surrounding tissue structures in three-dimensional space from multiple angles in real time during endoscopic examinations.

[0013] 3. Based on the point cloud registration algorithm, the human body space is automatically registered with the preoperative 3D image space, saving manual registration time. Attached Figure Description

[0014] Figure 1 This is a structural block diagram of an image fusion system for endoscopic three-dimensional navigation according to an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of spatial transformations for each coordinate system.

[0016] Figure 3 This is a flowchart of a method for reconstructing a three-dimensional scene based on endoscopic images, according to an embodiment of the present invention.

[0017] Figure 4 This is a schematic diagram showing the overlay of an endoscopic image and a three-dimensional image of the current field of view. Detailed Implementation

[0018] The following detailed description of specific embodiments of the present invention, taken in conjunction with the accompanying drawings, is not intended to limit the scope of the invention. These and other features of the invention will become apparent from the description of preferred forms of the given non-limiting examples with reference to the accompanying drawings.

[0019] This invention provides an image fusion system for three-dimensional navigation of endoscopes.

[0020] Figure 1 This is a structural block diagram of an image fusion system for endoscopic three-dimensional navigation according to an embodiment of the present invention.

[0021] An image fusion system for endoscopic 3D navigation includes a graphics workstation, optical positioning markers, an optical positioning system, and an endoscope system.

[0022] The graphics workstation is used to import pre-examination 3D images for reconstruction and post-processing, extract image point cloud data of lesions and surrounding tissue structures, extract optical point cloud data of lesions and surrounding tissue structures in optical positioning space, match image point cloud data and optical point cloud data based on point cloud registration algorithm, calculate the spatial transformation from the optical positioning system coordinate system to the pre-examination 3D image coordinate system, calculate the spatial transformation from the endoscope viewpoint coordinate system to the pre-examination 3D image coordinate system, and display the current field of view of the 3D image corresponding to the endoscope image in real time based on the real-time spatial position of the endoscope, and overlay and fuse it with the endoscope image for display.

[0023] Optical positioning markers are installed on the endoscope; for example, reflective balls or black and white checkerboard patterns can be installed at the end of a ureteroscope for optical positioning.

[0024] Optical positioning systems are used to track the spatial position of optical positioning markers in real time. For example, a binocular camera can be used to track the spatial position of optical positioning markers, such as reflective balls or black and white checkerboard patterns, mounted on an endoscope in real time based on the principle of binocular positioning.

[0025] Endoscopic systems are used in routine endoscopic examinations. Since endoscopic systems are already standard, they will not be described further.

[0026] Figure 3 This is a flowchart of an image fusion method for endoscopic three-dimensional navigation according to an embodiment of the present invention. The steps of the image fusion method for endoscopic three-dimensional navigation of the present invention are as follows:

[0027] Step S100: Import the 3D image before inspection.

[0028] In this step, the patient's pre-examination three-dimensional image data is imported into the graphics workstation. The three-dimensional image data includes, for example, CT images, MR images, three-dimensional ultrasound images, PET / CT images, etc.

[0029] In step S200, the graphics workstation can automatically segment the lesion and its surrounding tissue structure based on algorithms such as threshold segmentation, active contour, sparse field level set, and deep learning, and output a two-dimensional image of the segmentation result. The two-dimensional image is then converted into a triangular patch with only contours using algorithms such as isosurface extraction, and the image point cloud data of the lesion and its surrounding tissue structure is further extracted.

[0030] In this step, the graphics workstation segments the lesion and its surrounding tissue structure, and extracts the image point cloud data D. I .

[0031] Step S300: Install optical positioning markers on the endoscope and set up an optical positioning system to track the spatial position of the optical positioning markers in real time.

[0032] Optical positioning marks can be reflective spheres coated with a special material or traditional flat black and white checkerboard patterns. Optical positioning marks can be installed at the end of the endoscope. It is necessary to ensure that the relative spatial position relationship between the optical positioning marks and the endoscope remains unchanged during the endoscopy, and that the optical positioning marks can always be tracked and positioned by the optical positioning system.

[0033] Step S400: Use an endoscope to capture multiple frames of two-dimensional field images of the lesion and its surrounding tissue structures.

[0034] In this step, the doctor uses an endoscope to scan the lesion and its surrounding tissue structure from multiple angles. This step is routine and will not be described in detail here.

[0035] Step S500: Based on multiple frames of two-dimensional visual field images, the graphics workstation reconstructs the multiple frames of two-dimensional visual field images into three-dimensional point cloud data of the optical positioning space associated with the optical positioning markers, and extracts the optical point cloud data of the lesion and its surrounding tissue structures in the optical positioning space.

[0036] Based on multi-frame two-dimensional field-of-view images, the graphics workstation can automatically reconstruct three-dimensional point cloud data of the optical positioning space using weighted iterative feature algorithms. Based on prior information such as shape contour constraints, it can automatically segment lesions and surrounding tissue structures within the entire point cloud data, extracting the optical point cloud data (D) of the lesions and surrounding tissue structures in the optical positioning space. O ;

[0037] The difference between optical point cloud data and image point cloud data is that the latter is generated based on the three-dimensional image before the examination, which is the coordinate position information of the lesion and its surrounding tissue structure in the three-dimensional image space before the examination, while the former is generated from multiple frames of endoscopic two-dimensional field of view images in the optical positioning space, which is the coordinate position information of the lesion and its surrounding tissue structure in the optical positioning space.

[0038] In step S600, the graphics workstation automatically matches the image point cloud data and optical point cloud data based on point cloud registration algorithms such as initial registration with sampling consistency and iterative nearest point, and calculates the spatial transformation from the coordinate system of the optical positioning system to the coordinate system of the three-dimensional image before inspection.

[0039] In this step, the image point cloud data D is automatically completed based on the point cloud registration algorithm. I With optical point cloud data D O Registration is performed to unify the optical positioning space and the three-dimensional image space, and the spatial transformation from the optical positioning system coordinate system to the three-dimensional image coordinate system before inspection is calculated.

[0040]

[0041] Where R is a 3x3 rotation matrix, representing the pose transformation between the two coordinate systems, and is an orthogonal identity matrix; t is a 3x1 translation vector, representing the translation transformation between the two coordinate systems. A point (x...) in the optical positioning space... o ,y o ,z o ) T Its homogeneous coordinates (x o ,y o ,z o ,1) T Through left multiplication space transformation Its coordinates in the three-dimensional image space (x, y) can then be determined. I ,y I ,z I ,1) T .

[0042] Step S700: The graphics workstation calculates the spatial transformation from the endoscope viewpoint coordinate system to the pre-examination three-dimensional image coordinate system.

[0043] The spatial transformations involved in this system are as follows: Figure 2 As shown, the symbol T denotes the homogeneous matrix of the spatial transformation. Wherein, The spatial transformation from the endoscope viewpoint coordinate system to the optical positioning mark coordinate system is obtained through mechanical design dimensions and is known. This represents the spatial transformation from the optical positioning mark coordinate system to the optical positioning system coordinate system, which is obtained through the positioning information of the optical positioning system and is known. The spatial transformation from the optical positioning system coordinate system to the pre-examination three-dimensional image coordinate system is obtained by step S600; therefore, the spatial transformation from the endoscope viewpoint coordinate system to the pre-examination three-dimensional image coordinate system... The calculation formula is:

[0044]

[0045] Step 800: Move the endoscope. Based on the real-time spatial position of the endoscope, the graphics workstation displays the current field of view of a three-dimensional image corresponding to the endoscope image in real time, and overlays and fuses it with the endoscope image for display.

[0046] In this step, the endoscope is moved, and the graphics workstation displays the current field of view in real time a three-dimensional image corresponding to the endoscope image, which is then overlaid and fused with the endoscope image.

[0047] Thus, by using the system and method of the present invention, two-dimensional images of the endoscopic field of view can be combined with three-dimensional images before the examination, providing doctors with rich image information for endoscopic examination to assist in diagnosis; the three-dimensional images before the examination can provide a clear and complete 3D map of the lesion and surrounding tissue structures, which facilitates doctors to observe the relative positional relationship of the lesion and surrounding tissue structures in three-dimensional space from multiple angles in real time during endoscopic examination; and the registration of the human body space and the preoperative three-dimensional image space is automatically completed based on the point cloud registration algorithm, saving manual registration time.

[0048] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments can be used by those skilled in the art when reading the above description. Furthermore, in the above detailed description, various features may be grouped together to simplify the invention. This should not be construed as an intention that a disclosed feature, which is not claimed, is necessary for any claim. Rather, the subject matter of the invention may be less than all the features of the particular disclosed embodiments. Thus, the following claims are incorporated herein by reference as examples or embodiments, wherein each claim is independently considered as a separate embodiment, and these embodiments are contemplated as being able to be combined with each other in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

[0049] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. An image fusion system for three-dimensional navigation of an endoscope, characterized in that, Includes graphics workstations, optical positioning markers, optical positioning systems, and endoscope systems; The graphics workstation is used to import the pre-examination 3D image and perform reconstruction and post-processing, extract the image point cloud data of the lesion and its surrounding tissue structure, extract the optical point cloud data of the lesion and its surrounding tissue structure in the optical positioning space, match the image point cloud data and optical point cloud data based on the point cloud registration algorithm, and calculate the spatial transformation from the optical positioning system coordinate system to the pre-examination 3D image coordinate system. Calculate the spatial transformation from the endoscope viewpoint coordinate system to the pre-examination 3D image coordinate system; based on the real-time spatial position of the endoscope, display the current field of view of the 3D image corresponding to the endoscope image in real time, and overlay and fuse it with the endoscope image for display; Optical positioning markers are installed on the endoscope; Optical positioning systems are used to track the spatial position of optical positioning markers in real time; Endoscopic systems are used for endoscopic examinations; The optical positioning marker is a reflective ball or a black and white checkerboard pattern used for optical positioning; The spatial transformation from the endoscope perspective coordinate system to the pre-examination three-dimensional image coordinate system is obtained based on the spatial transformation from the endoscope perspective coordinate system to the optical positioning mark coordinate system, the spatial transformation from the optical positioning mark coordinate system to the optical positioning system coordinate system, and the spatial transformation from the optical positioning system coordinate system to the pre-examination three-dimensional image coordinate system.

Images