A system and method for reconstructing 3D scenes based on endoscopic images

Abstract

This invention discloses a system and method for reconstructing a three-dimensional scene based on endoscopic images. The system includes a graphics workstation, optical positioning markers, an optical positioning system, and an endoscope system. The graphics workstation constructs a three-dimensional point cloud of optical positioning space from multiple frames of two-dimensional visual field images, obtaining a region of interest (ROI) scene point cloud; it then performs three-dimensional reconstruction of the ROI scene, thereby reconstructing the lesion and its surrounding tissue structures; and it displays the relative spatial relationship between the endoscope position and the lesion and its surrounding tissue structures in real time. This invention displays the relative positional relationship between the endoscope and the lesion and its surrounding tissue structures in a three-dimensional scene in real time. Compared to a single endoscopic image, this allows doctors to more clearly observe the current diagnostic scene; and it can automatically complete the three-dimensional reconstruction of the scene based on multiple frames of two-dimensional endoscopic images, without relying on CT / MR or other three-dimensional image data, saving time spent registering endoscopic images with three-dimensional image data.

Description

Technical Field

[0001] This invention relates to the technical field of endoscopic images, and more particularly to a system and method for reconstructing three-dimensional scenes based on endoscopic images. Background Technology

[0002] An endoscope is a diagnostic instrument that integrates traditional optics, ergonomics, precision mechanics, modern electronics, mathematics, and software. It can be inserted into the stomach through the mouth or other natural orifices. Therefore, it can see lesions that cannot be detected by X-rays. With the help of an endoscope, doctors can observe ulcers or tumors in the stomach and formulate the best treatment plan for the patient.

[0003] However, the field of view of an endoscope is limited, making it difficult to see the whole lesion clearly. At the same time, it is impossible to accurately distinguish the spatial relationship between the lesion and the surrounding tissues and blood vessels. In addition, since the endoscope is inside the patient's body, the doctor cannot accurately determine its current specific location in the patient's body, which may introduce errors in the localization of the lesion. Summary of the Invention

[0004] To address the aforementioned technical problems in existing technologies, this invention provides a system for reconstructing three-dimensional scenes based on endoscopic images. By installing optical positioning markers on the endoscope to locate its spatial position in real time, doctors use the endoscope to scan and view lesions and surrounding tissue structures from multiple angles. The software uses multiple frames of endoscopic images to reconstruct a spatial point cloud of the lesion and surrounding tissue structures, i.e., a region of interest scene point cloud. Shape constraints are used to segment and reconstruct the lesion and surrounding tissues such as blood vessels in the point cloud in three dimensions. Then, the endoscope is moved in real time, and the software provides a real-time display of the three-dimensional spatial relationship between the current endoscope position and the lesion and surrounding tissue structures.

[0005] This invention provides a system for reconstructing a three-dimensional scene based on endoscopic images, including a graphics workstation, optical positioning markers, an optical positioning system, and an endoscope system. The graphics workstation is used to construct three-dimensional point cloud data of the optical positioning space from multiple frames of two-dimensional visual field images obtained through the endoscope system, obtaining a region of interest (ROI) scene point cloud; segmenting the ROI scene point cloud data, performing three-dimensional reconstruction of the ROI scene, thereby reconstructing the lesion and its surrounding tissue structures; based on the transformation from the endoscope's orientation coordinate system to the ROI scene coordinate system, displaying the relative spatial relationship between the endoscope's position and the lesion and its surrounding tissue structures in real time; optical positioning markers are installed on the endoscope; the optical positioning system is used to track the spatial position of the optical positioning markers in real time; and the endoscope system is used for endoscopic examination.

[0006] In some embodiments, the optical positioning marker is a reflective ball or a black and white checkerboard pattern used for optical positioning.

[0007] In some embodiments, the spatial transformation from the endoscope orientation coordinate system to the optical positioning mark coordinate system is obtained based on the spatial transformation from the optical positioning mark coordinate system to the optical positioning system coordinate system, the spatial transformation from the optical positioning mark coordinate system to the optical positioning system coordinate system, and the spatial transformation from the optical positioning system coordinate system to the region of interest scene coordinate system.

[0008] This invention also provides an image fusion method for endoscopic three-dimensional navigation, comprising: installing optical positioning markers on the endoscope and setting up an optical positioning system to track the spatial position of the optical positioning markers in real time; using the endoscope to scan and view the lesion and its surrounding tissue structure from multiple angles to obtain multiple frames of two-dimensional visual field images; a graphics workstation constructing three-dimensional point cloud data of the optical positioning space from the multiple frames of two-dimensional visual field images to obtain the region of interest scene point cloud; the graphics workstation segmenting the region of interest scene point cloud data and performing three-dimensional reconstruction of the region of interest scene to reconstruct the lesion and its surrounding tissue structure; moving the endoscope, and based on the transformation from the endoscope orientation coordinate system to the region of interest scene coordinate system, the graphics workstation displaying the relative spatial relationship between the endoscope position and the lesion and its surrounding tissue structure in real time.

[0009] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows:

[0010] The relative positional relationship between the endoscope and the lesion and its surrounding tissue structures is displayed in real time in a three-dimensional scene, which makes it easier for doctors to observe the current diagnostic scene more clearly compared to a single endoscopic image;

[0011] Based on multi-frame endoscopic 2D images, the scene can be automatically reconstructed into 3D without relying on CT / MR or other 3D image data, saving time for registering endoscopic images with 3D image data. Attached Figure Description

[0012] Figure 1 This is a structural block diagram of a system for reconstructing three-dimensional scenes based on endoscopic images, according to an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the spatial transformation from the endoscope orientation coordinate system to the scene coordinate system of the region of interest.

[0014] Figure 3 This is a flowchart of a method for reconstructing a three-dimensional scene based on endoscopic images, according to an embodiment of the present invention. Detailed Implementation

[0015] The following detailed description of specific embodiments of the present invention, taken in conjunction with the accompanying drawings, is not intended to limit the scope of the invention. These and other features of the invention will become apparent from the description of preferred forms of the given non-limiting examples with reference to the accompanying drawings.

[0016] This invention provides a system for reconstructing three-dimensional scenes based on endoscopic images. Figure 1 This is a structural block diagram of a system for reconstructing three-dimensional scenes based on endoscopic images, according to an embodiment of the present invention.

[0017] A system for reconstructing 3D scenes based on endoscopic images includes a graphics workstation, optical positioning markers, an optical positioning system, and an endoscope system.

[0018] The graphics workstation is used to construct three-dimensional point cloud data of optical positioning space from multiple frames of two-dimensional field images obtained through the endoscope system, and obtain the region of interest scene point cloud; segment the region of interest scene point cloud data, perform three-dimensional reconstruction of the region of interest scene, thereby reconstructing the lesion and its surrounding tissue structure; based on the transformation from the endoscope orientation coordinate system to the region of interest scene coordinate system, the relative spatial relationship between the endoscope position and the lesion and its surrounding tissue structure is displayed in real time.

[0019] Optical positioning markers are installed on the endoscope; for example, reflective balls or black and white checkerboard patterns can be installed at the end of a ureteroscope for optical positioning.

[0020] Optical positioning systems are used to track the spatial position of optical positioning markers in real time. For example, a binocular camera can be used to track the spatial position of optical positioning markers, such as reflective balls or black and white checkerboard patterns, mounted on an endoscope in real time based on the principle of binocular positioning.

[0021] Endoscopic systems are used in routine endoscopic examinations. Since endoscopic systems are already standard, they will not be described further.

[0022] Figure 3 This is a flowchart illustrating a method for reconstructing a 3D scene based on endoscopic images according to an embodiment of the present invention. The steps of the method for reconstructing a 3D scene based on endoscopic images according to the present invention are as follows:

[0023] Step S1: Install optical positioning markers on the endoscope.

[0024] Optical positioning marks can be reflective spheres coated with a special material or traditional flat black and white checkerboard patterns. Optical positioning marks can be installed at the end of the endoscope. It is necessary to ensure that the relative spatial position relationship between the optical positioning marks and the endoscope remains unchanged during the endoscopy, and that the optical positioning marks can always be tracked and positioned by the optical positioning system.

[0025] Step S2: Use an endoscope to scan from multiple angles to view the lesion and its surrounding tissue structures, and obtain multiple frames of two-dimensional visual images.

[0026] In this step, the doctor uses an endoscope to scan the lesion and its surrounding tissue structure from multiple angles. This step is routine and will not be described in detail here.

[0027] Step S3: The graphics workstation constructs three-dimensional point cloud data of the optical positioning space from multiple frames of two-dimensional field-of-view images to obtain the scene point cloud data of the region of interest.

[0028] In this step, the graphics workstation can automatically construct three-dimensional point cloud data of the optical positioning space from multiple frames of two-dimensional field-of-view images based on weighted iterative feature algorithms, i.e., point cloud data of the region of interest scene.

[0029] Step S4: The graphics workstation segments the point cloud data of the region of interest scene and performs three-dimensional reconstruction of the region of interest scene to reconstruct the lesion and its surrounding tissue structure.

[0030] In this step, the graphics workstation segments the lesion and surrounding tissues (such as blood vessels) in the point cloud data of the entire region of interest scene based on prior information such as shape contour constraints, outputs a binary image of the segmentation result, and uses algorithms such as isosurface extraction to convert the two-dimensional image into triangular patches with only contours. Based on the surface drawing algorithm, the region of interest scene is reconstructed in three dimensions.

[0031] Step S5: Move the endoscope to display the relative spatial relationship between the endoscope position and the lesion and its surrounding tissue structures in real time;

[0032] In this step, the endoscope is moved in real time, and the graphics workstation provides a real-time display of the three-dimensional spatial relationship between the current position of the endoscope and the lesion and its surrounding tissue structure. The endoscope can be drawn based on CAD and exported as an STL model. Then, the STL model is imported into the graphics workstation and rendered and displayed together with the segmented and reconstructed lesion and its surrounding tissue structure.

[0033] The spatial transformations involved in this system are as follows: Figure 2 As shown, the symbol T denotes the homogeneous matrix of the spatial transformation. Wherein, The spatial transformation from the endoscope orientation coordinate system to the optical positioning mark coordinate system is obtained through mechanical design dimensions and is known. This represents the spatial transformation from the optical positioning mark coordinate system to the optical positioning system coordinate system, obtained through the positioning information acquired by the optical positioning system, and is known. This represents the spatial transformation from the optical positioning system coordinate system to the scene coordinate system of the region of interest. Since both are in the same world coordinate system, this matrix is ​​an identity matrix. The spatial transformation from the endoscope orientation coordinate system to the scene coordinate system of the region of interest is then... The calculation formula is:

[0034]

[0035] Spatial transformation The specific form of expression is:

[0036]

[0037] Where R is a 3x3 rotation matrix, representing the attitude transformation between the two coordinate systems, and is an orthogonal identity matrix; t is a 3x1 translation vector, representing the translation transformation between the two coordinate systems. A point (x...) in the endoscope's orientation coordinate system... E ,y E ,z E ) T Its homogeneous coordinates (x E ,y E ,z E ,1) T Through left multiplication space transformation Then its coordinates (x, y) in the scene coordinate system of the region of interest can be calculated. S ,y S ,z S ,1) T .

[0038] Doctors can manipulate the scene view to obtain the best scene view of the area of ​​interest.

[0039] In this step, operations on the software scene perspective include zooming, panning, rotating, and cross-sectional display. It is also possible to hide a component in the scene, such as a lesion, blood vessel, or endoscope, to obtain the best observation effect.

[0040] Thus, this invention displays the relative positional relationship between the endoscope and the lesion and its surrounding tissue structures in a three-dimensional scene in real time. Compared with a single endoscope image, it makes it easier for doctors to observe the current diagnostic scene more clearly. Based on multiple frames of two-dimensional endoscope images, the three-dimensional reconstruction of the scene can be completed automatically without relying on CT / MR or other three-dimensional image data, saving time for registering endoscope images and three-dimensional image data.

[0041] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments can be used by those skilled in the art when reading the above description. Furthermore, in the above detailed description, various features may be grouped together to simplify the invention. This should not be construed as an intention that a disclosed feature, which is not claimed, is necessary for any claim. Rather, the subject matter of the invention may be less than all the features of the particular disclosed embodiments. Thus, the following claims are incorporated herein by reference as examples or embodiments, wherein each claim is independently considered as a separate embodiment, and these embodiments are contemplated as being able to be combined with each other in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

[0042] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A system for reconstructing a three-dimensional scene based on endoscopic images, characterized in that, The system comprises a graphic workstation, an optical positioning marker, an optical positioning system, and an endoscope system, wherein, The graphic workstation is used to construct multi-frame two-dimensional field of view images obtained by the endoscope system into three-dimensional point cloud data of the optical positioning space, to obtain scene point cloud of a region of interest, to segment the scene point cloud data of the region of interest, to perform three-dimensional reconstruction on the region of interest scene, and to thereby reconstruct a lesion and surrounding tissue structure; based on transformation from the endoscope orientation coordinate system to the region of interest scene coordinate system, the relative spatial relationship between the endoscope position and the lesion and surrounding tissue structure is displayed in real time; The optical positioning marker is installed on the endoscope; The optical positioning system is used to track the spatial position of the optical positioning marker in real time; The endoscope system is used for endoscopy; The optical positioning marker is a reflective bead or a black-and-white chessboard for optical positioning; Based on spatial transformation from the endoscope orientation coordinate system to the optical positioning marker coordinate system, spatial transformation from the optical positioning marker coordinate system to the optical positioning system coordinate system, and spatial transformation from the optical positioning system coordinate system to the region of interest scene coordinate system, the spatial transformation from the endoscope orientation coordinate system to the region of interest scene coordinate system is obtained.

Images