Method for guiding endoscopic surgery, computer-readable storage medium, control apparatus, computer program product, electronic device, navigation system, and robotic system

Abstract

The present invention relates to a method for guiding an endoscopic surgery, a corresponding computer-readable storage medium, a control apparatus, a computer program product, and an electronic device for navigation of an endoscopic surgery, a navigation system, and a robotic system. The method comprises: a step of endoscopic image acquisition; a step of enhancement information acquisition: acquiring at least two of the following three types of enhancement information: a) a spliced image obtained by splicing a plurality of images of an endoscope; b) a three-dimensional image of a physiological structure of a patient; and c) a marker on the three-dimensional image or a two-dimensional image of the physiological structure of the patient; a step of endoscopic enhanced image fusion: fusing an acquired current image of the endoscope with at least two types of enhancement information; and a step of displaying a view including at least a portion of an endoscopic enhanced image. According to the present invention, the current image of the endoscope is fused with at least two of the three types of enhancement information, such that the endoscopic image is enhanced in multiple dimensions, thereby achieving omnibearing multi-dimensional guidance for an operator.

Description

Method for guiding endoscopic surgery, computer-readable storage medium, control device and computer program product, electronic device, navigation system and robotic system Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a surgical navigation system, and more specifically to a method, electronic equipment, navigation system, and robotic system for guiding endoscopic surgery. Background Art

[0002] In traditional endoscopic surgery, the endoscope is used solely for surgical observation, tool manipulation, and control. However, the surgeon's field of vision is limited to the narrow scope of the endoscope, and the surgeon cannot see areas outside the scope of the endoscope. This can result in the endoscope not reaching the affected area correctly or inadequate manipulation, leading to failure to achieve the intended surgical goal.

[0003] In particular, endoscopic spinal surgery, due to its minimally invasive nature, is an effective means of treating nerve compression symptoms without pyramidal structural instability and has gained widespread clinical recognition and promotion in recent years. However, due to the physiological structure of the intervertebral foramen and the unique configuration of the spinal endoscope, doctors must face the following challenges in their journey to familiarize themselves with and ultimately master endoscopic spinal surgery techniques:

[0004] First, the field of view under the microscope is limited. The spinal endoscope is used in a working channel sleeve, which causes the doctor's field of view to be blocked by the working channel sleeve during use. When the doctor pulls up the spinal endoscope and moves it away from the target point of observation, hoping to observe a larger range of physiological structures according to common thinking, the first thing he sees is the wall of the working channel sleeve. If the doctor also pulls up the working channel sleeve away from the target point, the soft tissue on the outside of the working channel sleeve will shrink inward and invade the channel. What the doctor sees will be the soft tissue on the outside of the working channel sleeve, which blocks the structure of the target point that the doctor wants to observe. Therefore, he still cannot achieve the purpose of observing a wider range of physiological structures. "Seeing the trees but not the forest" is a figurative metaphor for the difficulty of limited field of view under spinal endoscopes by spinal endoscope surgeons.

[0005] Second, it is easy to get lost. To overcome the challenge of limited visual field under the microscope, the distal end of the spinal endoscope is designed with an angled bevel. The lens of the imaging device is mounted on the angled bevel and arranged eccentrically relative to the axis of the endoscope, so that a wider range can be observed by rotating the endoscope around the axis.

[0006] However, it is precisely because of this configuration that beginners who are not familiar with the soft tissue structure under the endoscope will often get lost due to rotating the endoscope.

[0007] Third, hand-eye coordination is difficult. The configuration of the spinal endoscope is that the optical rigid endoscope module of the endoscope (also called the imaging device of the endoscope) and the channel for the surgical tool are integrated into the same endoscope insertion tube. When the doctor axially rotates the spinal endoscope to observe the target position, the surgical tool also rotates with the spinal endoscope. Because the actual position of the optical rigid endoscope module and the surgical tool cannot undergo any relative change due to the special configuration of the spinal endoscope, the intuitive result is that the doctor observes that the position of the surgical tool on the screen has not changed, but the actual situation at this time is that due to the axial rotation of the spinal endoscope, the position of the surgical tool relative to the physiological structure under the mirror has changed. However, the doctor cannot "see through" the human body and intuitively see the change in the position of the tool relative to the surrounding physiological structure, but can only observe through the screen displaying the spinal endoscope image. Therefore, doctors are very likely to have difficulty in "hand-eye coordination". After each rotation of the endoscope, they need to readjust to the movement direction of the tool on the screen. Their thinking is always in a state of operation, which easily leads to fatigue.

[0008] Fourth, the "learning curve" is steep. Because of the difficulties encountered during these three endoscopic procedures, spinal endoscopists require superior anatomical and physiological knowledge, excellent spatial visualization, and a large caseload to overcome this so-called "learning curve." Typically, doctors need 30 to 50 surgeries, or even more, to master spinal endoscopy. This makes training a qualified spinal endoscopy surgeon often time-consuming and costly, which also limits the promotion and widespread adoption of spinal endoscopy.

[0009] Furthermore, traditional navigation technology is primarily used to guide the placement of pedicle screws during spinal fixation surgery, enabling real-time tracking and positioning of tools and implants relative to the patient's anatomy. This helps doctors achieve the clinical goals of "precision, safety, and minimally invasive procedures." In recent years, navigation technology, particularly electromagnetic navigation, has begun to be applied to endoscopic spinal surgery. This approach involves using an electromagnetic tracker within the navigation system to track the real-time spatial position of the endoscopic tracer. The navigation system's processor calculates the endoscope's real-time position and field of view, projecting this information onto a spatial 3D image model or 2D fluoroscopic image model that has been aligned with the patient's actual anatomy through a navigation registration process. This relative positional relationship is then displayed on the navigation system's display. This solution presents a problem: On the electromagnetic navigation interface, the doctor's simulated viewing angle corresponds to the position of the X-ray source when acquiring the 3D or 2D fluoroscopic image, located somewhere outside the patient's body. However, on the display showing the endoscopic image of the spinal endoscope, the actual viewing angle is located behind the lens of the rigid endoscope module, within the patient's body. Therefore, because the navigation view and the spinal endoscope image view have different viewing angles, each time the doctor moves or rotates the spinal endoscope to change the position or orientation of the lens of the spinal endoscope's optical rigid lens module, the doctor must combine his or her understanding of the anatomical structure and think through the different viewing angles to transform and unify them, so as to "mentally" determine the exact position and orientation of the physiological structures under the endoscope within the patient's body. As a result, the doctor's mind is constantly in a state of motion, which makes him or her more prone to fatigue.

[0010] Therefore, this navigation solution, which tracks the endoscope like a standard surgical tool and visualizes its spatial position and orientation, simply and directly applies navigation technology to endoscopic surgery. It does not effectively address clinical needs such as limited visual field and difficulty with hand-eye coordination. Even with regard to the "positioning" requirement of "easy disorientation under the endoscope," current electromagnetic navigation technology has not substantially resolved a problem that has long plagued spinal endoscopic surgeons.

[0011] How to leverage the advantages of the navigation system to address the challenges of "limited visual field under the microscope", "easy to get lost", "difficult hand-eye coordination" and the steep "learning curve" encountered in current endoscopic surgeries such as spinal endoscopic surgery, so that beginners can also complete surgical operations accurately and efficiently with the assistance of the navigation system, is an urgent problem to be solved in this field.

[0012] Summary of the Invention

[0013] The object of the present invention is to solve at least one of the above problems and defects in the prior art as well as other technical problems.

[0014] In one aspect, the present invention provides a method for guiding endoscopic surgery, the method comprising the following steps:

[0015] Endoscopic image acquisition steps: acquiring an endoscopic image;

[0016] Enhanced information acquisition step: acquiring at least two of the following three types of enhanced information:

[0017] a) a stitched image obtained by stitching multiple images of the endoscope;

[0018] b) 3D images of the patient’s physiological structures;

[0019] c) markings on a three-dimensional image or a two-dimensional image of a patient's physiological structure;

[0020] Endoscopic enhanced image fusion step: fusing the acquired current endoscopic image with the at least two types of enhanced information acquired in the enhanced information acquisition step to obtain an endoscopic enhanced image; and

[0021] Displaying step: displaying a view including at least a portion of the endoscopically enhanced image.

[0022] In this example, by fusing the current image of the endoscope with at least two of the three types of enhanced information, the endoscopic image is enhanced in multiple dimensions (at least two) by including soft tissue within a larger field of view around the endoscope tip, bony structures invisible to the naked eye, and planned orientation information. Because the enhanced endoscopic image incorporates a larger mosaic of images from around the endoscope tip, it provides the operator with a wider field of view; because it incorporates three-dimensional images of bony structures, the operator can see previously invisible bony structures and thus grasp the overall orientation; and because it incorporates guidance instructions related to markers, it facilitates the operator to quickly locate the target structure and determine its direction and orientation as well as the orientation of surgical tools. In the prior art, even if endoscopic images are enhanced, they are only enhanced in one limited manner, without simultaneously and multi-dimensionally enhancing the endoscopic image from at least two aspects. In the present invention, the operator can intuitively observe the current image within the endoscopic field of view and the surrounding soft tissue, bony structures, target location, and orientation information, achieving comprehensive and multi-dimensional guidance for the operator. This solution perfectly solves the problems encountered in endoscopic surgery, such as "limited visual field under the microscope", "easy to get lost", "difficult hand-eye coordination" and "steep learning curve".

[0023] According to an example, the endoscopic enhanced image fusion step is performed with the aid of a navigation system, wherein the method further comprises, before the endoscopic enhanced image fusion step:

[0024] Imaging orientation acquisition step: acquiring the orientation of the imaging device of the endoscope under the navigation system; and

[0025] Enhanced information position acquisition step: obtaining the position of the required fused spliced ​​image, three-dimensional image or marker under the navigation system;

[0026] In the endoscopic enhanced image fusion step, fusion is performed according to the orientation of the imaging device and the orientation of the spliced ​​image, three-dimensional image or marker to be fused.

[0027] In this example, with the help of the navigation system, the images of the imaging devices to be fused and the various enhancement information are unified into the navigation coordinate system. This fusion method has higher fusion accuracy of the images and has higher navigation accuracy and navigation effect.

[0028] According to one example, the method further includes: a global enhanced image fusion step, wherein in the global enhanced image fusion step, at least two of the stitched image, the three-dimensional image and the guidance indication associated with the marker are fused to obtain a global enhanced image, wherein the current field of view position of the imaging device and / or the position of the view of the endoscopic enhanced image are indicated on the global enhanced image; and a step of displaying the global enhanced image, wherein the edges of the current field of view position of the imaging device and the edges of the view of the endoscopic enhanced image are indicated by lines of different colors and / or line types on the global enhanced image.

[0029] In this example, the endoscopic enhanced view is displayed simultaneously with the global enhanced view as a part of the global enhanced image, and the correlation information between the two is displayed in the global enhanced image, that is, the edge of the current field of view of the imaging device and the edge of the view of the endoscopic enhanced image are prominently indicated by lines of different colors and / or line types on the global enhanced image, so that the operator can know the position of the endoscopic real-time image in the global image and the information relative to the surrounding soft tissue, bony structure, target position and direction while watching the larger endoscopic real-time image, thereby realizing all-round guidance for the operator.

[0030] According to one example, the type of enhancement information acquired and fused is selected based on operator input, which enables the surgeon to flexibly select the enhancement information for the endoscopic image according to his or her needs and preferences.

[0031] According to one example, the range of the endoscopically enhanced image displayed in the endoscopically enhanced image view can be determined based on operator input. In this example, the range of the endoscopically enhanced image displayed in the endoscopically enhanced image view (corresponding to the size of the current endoscope image in the endoscopically enhanced view) can be determined based on operator input. In other words, the operator can change the size ratio of the current endoscope image in the endoscopically enhanced view as needed by interacting with the system, for example, by zooming in or out. When a physician needs to observe a detail in the endoscopic field of view in greater detail, the real-time endoscopic image can be zoomed in (correspondingly, the range of the entire endoscopically enhanced image that can be displayed in the view window becomes smaller), while the content displayed in the surrounding enhanced information remains proportional to the real-time endoscopic image. Enhanced information that extends beyond the edge of the "endoscopically enhanced view" is no longer displayed in the "endoscopically enhanced view."

[0032] According to one example, when the fused enhancement information includes a stitched image, the enhancement information acquisition step includes a stitching step, in which a plurality of endoscopic images are stitched together based on the orientation of an imaging device corresponding to at least one endoscopic image to obtain a stitched image. In this solution, because the images are stitched together based on the orientation of the imaging device corresponding to the at least one image, less computational processing is required compared to existing stitching methods (e.g., algorithm-based methods), significantly improving stitching speed. This is particularly important for real-time observation of endoscopic images during surgery.

[0033] According to one example, when the fused enhanced information includes a stitched image, the enhanced information acquisition step further includes: a distortion calibration step prior to the stitching step, wherein the distortion of the endoscope images to be stitched is calibrated; and a processing step subsequent to the stitching step, wherein a planar image is generated based on the images obtained in the stitching step, wherein the planar image is fused with the current endoscope image as the stitched image in the endoscope enhanced image fusion step. In this example, the distortion calibration prior to stitching removes the distortion effects of the endoscope, making the stitched image closer to the real image. Furthermore, the processing step after stitching eliminates or reduces visual distortion in the stitched image caused by different viewing angles. The two distortion processing methods are combined to make the stitched image closer to the real surrounding scene, making it easier for the operator to observe.

[0034] According to one example, when the fused enhanced information includes a marker, the enhanced information orientation acquisition step includes a marker orientation acquisition step, in which the orientation of the marker under the navigation system is acquired with the aid of the alignment of the three-dimensional image or the two-dimensional image under the navigation system; wherein in the endoscope enhanced image fusion step, the guidance instructions related to the marker are fused with the current image of the endoscope according to the orientation of the marker under the navigation system.

[0035] According to one example, the method enables the operator to select one or more of a directional marker and a marker indicating a physiological structure. This enables the operator to make a selection based on their preferences or actual scenarios. According to one example, the directional marker includes one or more of a directional marker toward the dorsal, ventral, cranial, and caudal sides of the patient. According to one example, the patient's physiological structure is a spine, and the marker indicating the physiological structure includes an image model of a marked target (e.g., one or more of a protruding or free intervertebral disc, osteophytes, and ossified yellow ligament). The image model can be removed or separated from the image of the patient's physiological structure, in which case the image model is the marker indicating the physiological structure.

[0036] According to one example, the marker indicating the physiological structure includes a physiological structure marker point; preferably, the patient's physiological structure is a spine, and the physiological structure marker point is a marker point indicating one or more of a protruding or free intervertebral disc, osteophytes and ossified yellow ligament, or a physiological structure marker point that does not displace during endoscopic surgery, such as one or more of the ventral side of the articular process, the pedicle of the anterior vertebra, the pedicle of the posterior vertebra, and the intervertebral disc; preferably, there are multiple physiological structure marker points.

[0037] According to one example, when the fused enhancement information includes a three-dimensional image of the patient's physiological structure, the enhancement information orientation acquisition step includes registering the three-dimensional image of the patient's physiological structure with the navigation system to acquire the orientation of the three-dimensional image. Preferably, the three-dimensional image of the patient's physiological structure includes a preoperative three-dimensional image or an intraoperative three-dimensional image.

[0038] According to one example, the endoscope is a spinal endoscope.

[0039] The navigation method of the present invention is particularly beneficial for spinal endoscopes (such as foraminal endoscopes). As introduced in the background technology, the spinal endoscope is used in the working channel sleeve, which further limits the doctor's field of view under the microscope. The imaging device of the endoscope and the channel of the surgical tool are integrated in the same endoscope insertion tube, which makes it easy for the doctor to have difficulty in hand-eye coordination. How to prevent the doctor from being affected by the special use of the spinal endoscope (the endoscope is used in the working channel sleeve) and the limited viewing angle of the endoscope optical rigid endoscope module during spinal endoscopic surgery, reduce the "hand-eye coordination difficulty" and the problem of disorientation caused by the rotation of the endoscope, has not yet had a convenient and reliable solution. The method of the present invention solves the above problems by providing multi-dimensional enhanced information for the endoscopic image, which can largely avoid the doctor's misoperation and increase the reliability of spinal surgery.

[0040] According to an example, the imaging position acquisition step is performed by acquiring the position of a tracer having a fixed positional relationship relative to the endoscope from a tracking device of a navigation system. This method conveniently implements position tracking of the imaging device of the endoscope by means of the navigation system.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the methods in the above examples are executed.

[0042] According to another aspect of the present invention, a control device is provided. The control device includes a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the methods in the above examples are executed when the processor runs the program.

[0043] According to yet another aspect of the present invention, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps of the methods in the above examples are implemented.

[0044] According to yet another aspect of the present invention, an electronic device for endoscopic surgery navigation is provided, characterized in that the electronic device includes a display device and a processor, wherein the processor has a data interface, wherein the data interface is connectable to an endoscope, so that the processor can acquire an image from the endoscope; and wherein the processor is configured to acquire one or more of a three-dimensional image of a patient's physiological structure and a marker on the three-dimensional image or a two-dimensional image of the patient's physiological structure; wherein the processor is configured to display a view of at least a portion of an endoscope-enhanced image on the display device for at least a period of time when the processor is running, wherein the endoscope-enhanced image is an image that fuses a current image of the endoscope with at least two of the following three types of enhancement information:

[0045] a) a stitched image obtained by stitching multiple images of the endoscope;

[0046] b) 3D images of the patient’s physiological structures;

[0047] c) Guidance instructions related to the marking.

[0048] According to another aspect of the present invention, a navigation system for endoscopic surgery is also provided, which includes a tracking device suitable for tracking a tracer having a fixed positional relationship relative to an endoscope, a display device and a processor, wherein the processor is suitable for being connected to the tracking device and the endoscope, wherein the methods in the above examples are executed when the processor is running, and the display in the method is realized through the display device.

[0049] Because the navigation system integrates a larger range of spliced ​​images around the endoscope, it provides the operator with a wider field of view; it integrates the three-dimensional image of the bone structure, allowing the operator to see the bone structure that was originally invisible and grasp the overall orientation; it integrates direction marks or marks indicating physiological structures or their guiding instructions to facilitate the operator to quickly locate the target position and grasp the orientation of surgical tools, thus having better navigation effect and navigation accuracy.

[0050] According to another aspect of the present invention, a robot system is provided, which includes a robot arm and the above-mentioned navigation system. In other words, the concept of the present invention can also be implemented in the navigation system of the robot system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention is described in detail below by way of exemplary embodiments with reference to the accompanying drawings.

[0052] FIG1 exemplarily shows a flow chart of a method for guiding endoscopic surgery according to the present invention.

[0053] FIG2 is a schematic diagram showing a principle of a navigation system for spinal endoscopic surgery according to an exemplary embodiment of the present invention.

[0054] FIG3 exemplarily shows an endoscopic enhanced image obtained by fusing the stitched image, the three-dimensional image of the patient's physiological structure, and the current image of the endoscope.

[0055] FIG. 4 exemplarily shows a view in which the endoscopy enhanced image of FIG. 3 is displayed on a window of a display device.

[0056] Figure 5 exemplarily shows a view of an endoscopically enhanced image displayed on a window of a display device according to another exemplary embodiment, wherein the endoscopically enhanced image fuses a stitched image, a three-dimensional image of the patient's physiological structure, physiological structure markers and a current image of the endoscope.

[0057] FIG6 exemplarily shows a display interface of a navigation system having an endoscopic enhanced image view and a global enhanced image.

[0058] FIG7 exemplarily shows a global enhanced image that is a fusion of a stitched image, a three-dimensional image of a patient's physiological structure, and a direction mark, in which the current field of view position of the imaging device and the position of the view of the endoscopic enhanced image are indicated.

[0059] FIG8 exemplarily shows a global enhanced image that fuses the stitched image, the three-dimensional image of the patient's physiological structure, and the physiological structure markers, in which the current field of view position of the imaging device and the position of the view of the endoscopic enhanced image are indicated.

[0060] FIG9 exemplarily shows a flow chart of an enhancement information acquisition step including a stitching step to acquire a stitched image.

[0061] FIG10 exemplarily shows a schematic diagram of the principle of moving the endoscope to obtain multiple endoscopic images of a larger range.

[0062] FIG11 is a schematic diagram showing the principle of rotating an endoscope to obtain a stitched image.

[0063] FIG. 12 shows an example of an operator marking a direction on a preoperative 3D image or an intraoperative 3D image.

[0064] FIG13 shows an example of an operator marking a direction on an image obtained by performing quasi-three-dimensional fitting on a two-dimensional perspective image.

[0065] FIG14 shows an example of an operator completing physiological structure marking on a preoperative 3D image or an intraoperative 3D image.

[0066] FIG15 shows an example of an operator completing physiological structure marking on an intraoperative anteroposterior image.

[0067] FIG16 shows an example of an operator completing physiological structure marking on an intraoperative lateral image.

[0068] It should be noted that the drawings are schematic only. They illustrate only those components or steps necessary to illustrate the present invention, and other components or steps may be omitted or only briefly mentioned. In addition to the components or steps shown in the drawings, the present invention may also include other components or steps. DETAILED DESCRIPTION

[0069] The following examples and accompanying drawings further illustrate the technical solution of the present invention. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall concept of the present invention and should not be construed as limiting the present invention.

[0070] The following describes, as a specific embodiment, the specific steps of the method of the present invention for image-guided endoscopic surgery using an endoscope (e.g., a spinal endoscope, a neuroendoscope, etc.) under a navigation system, as well as the electronic equipment, navigation system, and robotic system (or positioning and navigation system) involved. In the detailed description below, many specific details and steps are set forth in a highly specific and detailed manner to provide a comprehensive understanding of the embodiments. However, it should be understood that one or more other embodiments may be implemented without these specific details and steps.

[0071] It should be noted that while this document uses the terms "imaging device" and "image" of an endoscope, those skilled in the art will understand that "imaging device" is a broad concept encompassing functions such as video recording, video capture, and image capture, and "image" is a broad concept encompassing video, dynamic continuous images, and static images. In this document, the "imaging device" of an endoscope may be an endoscope module used for video imaging of the endoscope. The image acquisition step of the present invention acquires a framed image of the endoscope's image.

[0072] In a specific embodiment of the present invention, the navigation system used, as shown in FIG2 , includes a tracking device 1, a control device 2, and a display device 3. The tracking device 1 can be an optical tracking device (e.g., an NDI navigator), and a corresponding tracer 4 can be provided on the endoscope 5. As a specific example, the control device 2 can be a general-purpose computer, a dedicated computer, an embedded processor, or any other suitable programmable data processing device, such as a single-chip microcomputer or chip. The control device 2 can include a processor and a memory for storing programs, or it can include only a processor, in which case the processor can be attached to the memory storing the programs. In other words, the control device includes at least a processor. The control device 2 (or processor) and the display device 3 can be integrated or provided separately. The control device or processor has a data interface, which can include a data interface that can be connected to the endoscope, allowing the control device / processor to obtain images of the endoscope in real time. The control device or processor also includes a data interface that can be connected to the tracking device 1 of the navigation system, so that the position and orientation of the tracked target, such as the tracer 4 on the endoscope, can be obtained from the tracking device 1 in real time. As an example, the endoscope 5 and / or the tracer 4 can also be considered part of the navigation system of the present invention.

[0073] Typically, the distal end of the endoscope enters the patient's tissue structure or bony structure to observe and / or operate on it, and the proximal end of the endoscope (the end closest to the operator, i.e., the end opposite to the distal end of the endoscope's insertion tube) is located outside the patient's body for manipulation by the operator. In the navigation system, by providing a tracer 4 suitable for being tracked by a tracking device 1 at the proximal end of the endoscope located outside the patient's body, the navigation system can obtain real-time information about the position and orientation of the tracer 4 on the endoscope 5 in the navigation coordinate system. As an example of the present invention, the imaging device of the endoscope, i.e., the distal lens of the optical hard lens module, is provided at the distal end of the insertion tube of the endoscope. By calibrating the relative positional relationship of the distal end of the insertion tube of the endoscope with respect to the tracer 4 on the endoscope 5, the relative positional relationship of the imaging device with respect to the tracer 4 can be obtained, thereby enabling the tracking device 1 in the navigation system to determine the orientation of the tracer 4 and, therefore, the position and orientation of the imaging device in the navigation coordinate system. This calibration is typically performed before performing endoscopic surgery and its navigation, and is also referred to as calibration of the external parameters of the endoscope.

[0074] Specifically, before performing endoscopic surgery and its navigation, the relative position relationship of the imaging device of the endoscope, i.e., the hard mirror module, relative to the tracer 4 can be calibrated first. Preferably, in this embodiment, the calibration is performed by a calibration tool (not shown in the figure) without the need to obtain images with the help of an endoscope, which reduces the workload of image processing. The navigation system of the present invention optionally includes the calibration tool. A plurality of calibration holes can be formed on the calibration tool, and these calibration holes have bottom surfaces with different inclination angles and / or different apertures to calibrate endoscopes with different end bevel inclination angles and / or barrel diameters. Specifically, the calibration tool also includes another tracer fixed on its bracket, and the navigation system can know the position of the tracer on the calibration tool in the navigation coordinate system. Moreover, the positional relationship of each calibration hole relative to the tracer on the calibration tool can be known based on the design size of the calibration tool, and thus the positions of these calibration holes in the navigation system are known. The endoscope is positioned by inserting the distal end of the insertion tube into a matching calibration hole. The inclined surface of the distal end of the insertion tube aligns with the inclined bottom surface of the calibration hole. Because the navigation system also knows the position of the endoscope's tracer 4 within the navigation coordinate system, it can determine the relative position of the distal end of the endoscope's insertion tube relative to the tracer 4, completing the calibration process.

[0075] During surgery, the position (location and orientation) of the tracer 4 on the endoscope is acquired in real time by the tracking device 1, thereby indirectly acquiring the position and orientation of the endoscope tip, i.e., the orientation of the imaging device. The imaging orientation acquisition step in the method according to the present invention is performed in this manner.

[0076] In this specific embodiment of the present invention, prior to performing endoscopic surgery and navigation, the imaging parameters of the endoscope, also known as internal parameters, can be calibrated. For example, a calibration plate can be used to calibrate the internal parameters. For example, the endoscope can be used to photograph the calibration plate at different azimuth angles, saving the images and recording the calibration plate specifications. The internal parameters of the endoscope are then calibrated, including the distortion matrix and / or rotation and displacement vectors. These calibrated internal parameters will be used in the distortion calibration process described later.

[0077] The following, in conjunction with FIG1 , describes in detail the steps of a specific embodiment of the method for guiding endoscopic surgery of the present invention. Before the surgery begins, the internal and external parameters of the endoscope can be calibrated as described above. Furthermore, before the surgery and navigation begin, the navigation system is aligned, i.e., the navigation coordinate system is determined. The method of the present invention can be used in scenarios where two-dimensional images are used for navigation, as well as in scenarios where three-dimensional images are used for navigation. The alignment method for the navigation system is well known and will not be described in detail here.

[0078] During endoscopic surgery, a control device automatically acquires a real-time image of the endoscope. Furthermore, the type of enhancement information used to enhance the real-time image of the endoscope is determined based on input from an operator. The operator's input can be selected based on their needs. For example, the operator can input the instruction via various input components, such as a keyboard, mouse, handle, or foot switch. The present invention provides the operator with a solution capable of fusing at least two of the following three types of enhancement information onto the endoscopic image: a spliced ​​image obtained by splicing multiple images around the end of the endoscope, a three-dimensional image of the patient's physiological structure, and, for example, directional markings or markings indicating physiological structures made by the operator on the three-dimensional or two-dimensional image of the patient's physiological structure during surgical planning.

[0079] Among the three types of enhanced information mentioned above, the stitched image (indicated by reference numeral 6 in Figures 3 and 4) can also be called a "complete three-dimensional map" or a "panoramic image". However, it should be noted that the "complete" or "panoramic" here only refers to an image of a larger range relative to the limited endoscopic field of view of the endoscope, and does not necessarily mean a 360-degree panoramic image. This is obtained, for example, by the doctor according to his observation needs during the operation, by flexibly translating (for example, as shown in Figure 10) and / or rotating (for example, as shown in Figure 11) the endoscope within a larger range that needs to be observed. Figures 10 and 11 also exemplarily show the conical field of view 51 of the endoscope 5 at three positions. The conical field of view 51 represents the field of view that the imaging device of the endoscope can see.

[0080] This stitched image around the endoscope tip allows the physician to observe a wider field of view (or global field of view), better displaying the soft tissue surrounding the endoscope tip, such as nerves, dura mater, blood vessels, and intervertebral discs. This allows the physician to quickly determine the endoscope's orientation and the spatial position of key physiological structures relative to the current endoscope image, eliminating the limitation of the endoscope's field of view. The acquisition and fusion of stitched images will be further described below.

[0081] The three-dimensional image of the patient's physiological structure displays the three-dimensional bone structure from the CT or CBCT image. Therefore, when it is fused with the endoscopic image, the doctor can observe the three-dimensional bone structure that is covered by soft tissue and cannot be observed under the endoscopic field of view on the endoscopic enhanced image (the dotted part pointed to by the figure mark 7 in Figure 4 represents the three-dimensional image of the three-dimensional bone structure), thereby obtaining the global orientation information of the endoscope tip and the surgical tool tip, for example, in the entire spinal structure.

[0082] For the above-mentioned marks on the three-dimensional image or two-dimensional image of the patient's physiological structure, they can be marked by the operator in the surgical plan, for example, including the direction, physiological structure landmarks, and image models of marked targets (such as protruding or free intervertebral discs, osteophytes, or ossified yellow ligaments) marked in the surgical plan, and / or target information. When the guidance instructions related to the mark (which may include the mark itself, such as the direction mark, physiological structure landmarks, image models of physiological structure targets, and indicators pointing to the mark such as arrows, which are collectively referred to as guidance instructions related to the mark) are fused to the endoscopic image, the displayed endoscopic image has orientation guidance instructions, such as direction marks, physiological structure marks, target image models, etc. Therefore, the doctor can quickly determine the orientation of the physiological structure under the endoscope at this moment, thereby guiding the operation of the endoscopic tool. The making of the mark and the acquisition and fusion of its orientation will be further described below.

[0083] In summary, in this solution of the present invention, by fusing the current image of the endoscope (a framed image of the real-time image from the endoscope) with at least two of the three types of enhanced information mentioned above, the endoscopic image is enhanced from multiple angles with respect to the bony structure, soft tissue, and / or planning marker orientation indication information. The fusion of a larger range of spliced ​​images around the endoscope provides the operator with a wider field of view; the fusion of the three-dimensional image of the bony structure allows the operator to see bony structures that would otherwise be invisible and grasp the overall orientation; the fusion of directional markers or markers indicating physiological structures or their guiding instructions facilitates the operator to quickly locate the target position and determine the direction. This perfectly solves the problems encountered in endoscopic surgery, such as "limited visual field under the microscope", "easy to get lost", "difficult hand-eye coordination", and "steep learning curve".

[0084] Specifically, as shown in FIG1 , the method of this specific embodiment of the present invention includes an endoscopic image acquisition step, where the endoscopic image can be a frame of the endoscopic image acquired in real time by the control device. At the same time, the control device acquires and records the position and direction of the imaging device of the endoscope corresponding to at least one image of the endoscope (for example, corresponding to each image), that is, the imaging orientation acquisition step in FIG1 . In this imaging orientation acquisition step, the processor acquires the position of the tracer 4 located outside the patient's body and disposed on the distal end of the endoscope 5 from the tracking device 1 of the navigation system, and indirectly acquires the position and direction of the imaging device in combination with the external parameters calibrated above. The endoscopic image acquisition step here includes the acquisition of the current image of the endoscope fused in the endoscopic enhanced image fusion.

[0085] As shown in FIG1 , the method of the present invention further includes an enhancement information acquisition step, in which the type of enhancement information to be acquired is determined based on the input of the operator. The input component for the operator to input may include, for example, a selection box, a dialog box, etc. displayed on a display device. The operator may, for example, select two types of enhancement information: a stitched image and a three-dimensional image of the patient's physiological structure; or may select two types of enhancement information: a stitched image and a marker; or may select two types of enhancement information: a three-dimensional image of the patient's physiological structure and a marker; or three types of enhancement information: a stitched image, a three-dimensional image of the patient's physiological structure, and a marker. The present invention can acquire corresponding enhancement information based on the above selections, and correspondingly acquire the orientation information of these enhancement information, so as to be used for fusing the endoscopic enhanced image in the fusion step based on the orientation of the imaging device and the orientation of the stitched image, three-dimensional image, or marker to be fused.

[0086] Specifically, when the enhanced information to be fused includes a stitched image, the enhanced information acquisition step includes stitching multiple endoscope images. Figure 9 illustrates the enhanced information acquisition step including the stitching step. In the image acquisition step shown in Figure 9 , the control device automatically acquires multiple images around the endoscope's distal end. Simultaneously, the control device automatically acquires and records the position and orientation of the endoscope's imaging device corresponding to at least one image (e.g., each image) of the endoscope, i.e., the imaging orientation acquisition step in Figure 9 . The above steps are repeated at multiple positions and orientations of the endoscope, thereby obtaining images of the imaging device at multiple orientations, for example, after translating (as shown in Figure 10 ) and / or rotating (as shown in Figure 11 ) the endoscope over a larger range of observation. Simultaneously or subsequently, distortion calibration can be performed on the multiple images of the endoscope, combining the calibrated imaging device internal parameters described above, to remove distortion effects of the imaging device, i.e., the distortion calibration step in Figure 9 . Subsequently, the images are stitched together based on the position and orientation (i.e., orientation) of the endoscope corresponding to the at least one image, thereby obtaining a stitched image.

[0087] In this approach, because the images are stitched together based on the orientation of the imaging device corresponding to at least one image to produce a stitched image, compared to existing stitching methods (e.g., algorithm-based), fewer computations are required, significantly improving stitching speed. This is particularly important for real-time observation during surgery. Furthermore, this approach offers higher stitching accuracy and requires less processor power. It should be noted that the imaging orientation acquisition step acquires the position and / or orientation of the imaging device under the navigation system corresponding to at least one of the stitched images. For example, the orientation of the imaging device under the navigation system can be acquired for each acquired image, but it is also possible to acquire the position and / or orientation of the imaging device for only a portion of the images, or even just one image. For example, in some cases, it is only necessary to acquire the position and / or orientation of the imaging device corresponding to certain images at intervals in time, or only to acquire the position and / or orientation of the imaging device at the start of acquisition of the images to be stitched.

[0088] As shown in FIG9 , the enhanced information acquisition step may further include a processing step after the stitching step, in which a planar image is generated based on the image obtained in the stitching step to reduce visual distortion. This planar image is then used as a stitched image and fused with the current endoscope image in the endoscopic enhanced image fusion step (e.g., placed around the current endoscope image). This method eliminates or reduces distortion in the stitched image caused by different viewing angles.

[0089] Because the stitching step involves stitching the images together based on the orientation of the imaging device corresponding to at least one image to produce a stitched image, the orientation of the stitched image as viewed by the navigation system is known. Therefore, the orientation of the stitched image as viewed by the navigation system can be obtained during the enhancement information orientation acquisition step, allowing for the subsequent endoscopic enhanced image fusion step to fuse the stitched image with the current endoscopic image and other enhancement information based on this orientation.

[0090] When the enhanced information to be fused includes the markers mentioned above, the enhanced information orientation acquisition step includes a marker orientation acquisition step, in which the orientation of the marker under the navigation system is acquired by means of the registration of the three-dimensional image or the two-dimensional image under the navigation system; wherein in the endoscope enhanced image fusion step, the guidance instructions related to the marker are fused with the current image of the endoscope and other enhanced information according to the orientation of the marker under the navigation system.

[0091] Specifically, in this context, the image on which the operator makes markings can be a preoperative 3D image, such as a preoperative CT image, or an image obtained during surgery, such as an intraoperative CBCT image or an intraoperative 2D fluoroscopic image. When the navigation system uses a preoperative CT image for navigation, the operator makes markings on the preoperative 3D image. When the navigation system uses an intraoperative CBCT image or an intraoperative 2D fluoroscopic image for navigation, the operator makes markings on the image after it is acquired during surgery.

[0092] In the present invention, the operator can choose to make various marks, such as directional marks or marks indicating physiological structures, thereby providing the operator with a variety of options and possibilities. In the example where the patient's physiological structure is the spine, the mark indicating the physiological structure can, for example, include an image of the target tissue or target bony structure (such as a protruding or free intervertebral disc, ossified yellow ligament or osteophyte generated by degeneration) removed by the operator during surgical planning, or it can be a physiological structure marker point. For the operator, if the target tissue or bony structure can be distinguished in the three-dimensional image used in the surgical planning (for example, osteophytes can be distinguished in CT images, or protruding or free intervertebral discs can be distinguished in MRI images), then its image model is directly formed in the surgical planning. The image model is used as the marker to enhance information and is fused into the endoscopic enhanced image. If the target tissue or target bony structure cannot be displayed in the three-dimensional image used for surgical planning, for example, if a CT image is used and the intervertebral disc cannot be displayed, the doctor can refer to the MRI image to mark the points indicating the target tissue or target bony structure in the surgical planning (for example, marking points indicating one or more of a protruding or free intervertebral disc, osteophytes and ossified yellow ligament), that is, physiological structure marking points.

[0093] Physiological structure markers may also include physiological structure markers that do not shift during endoscopic surgery. As shown in FIG14 , the operator may also mark (for example, using preoperative planning software) certain physiological structure markers within or around the intervertebral foramen that do not shift during the entire spinal endoscopic surgery on the three-dimensional image, including but not limited to physiological structure points such as the ventral side of the articular process, the pedicles of the anterior vertebral body, the pedicles of the posterior vertebral body, and the intervertebral disc. As shown in FIG15 and FIG16 , when using intraoperative two-dimensional fluoroscopic images, the doctor uses the intraoperative planning software to select physiological structure markers on the intraoperative anteroposterior image ( FIG15 ) and lateral image ( FIG16 ), including but not limited to physiological structure points such as the ventral side of the articular process, the pedicles of the anterior vertebral body, the pedicles of the posterior vertebral body, and the intervertebral disc.

[0094] As shown in Figures 12 and 13, the operator can make direction marks on the preoperative or intraoperative three-dimensional image (Figure 12), such as arrow marks toward the dorsal, ventral, cephalic, and caudal sides of the patient. The operator can also mark the direction on the intraoperative two-dimensional image. Preferably, in order to facilitate the operator's marking, the intraoperative two-dimensional image (for example, the intraoperative anteroposterior and lateral images) can also be fitted with a quasi-three-dimensional fitting, and the operator marks the direction on the fitted image (as shown in Figure 13). Compared with the method of marking directly on the intraoperative two-dimensional image, such as the anteroposterior image and the lateral image, this method can solve the problem of easily marking the head and tail in reverse when marking on the two-dimensional image.

[0095] Those skilled in the art will appreciate that, although some specific examples of direction markers and physiological structure markers are described herein, the selection of direction markers or physiological structure markers can be freely selected based on the operator's preferences and needs, and is not limited to these examples. Furthermore, although four direction markers or physiological structure markers are shown in the embodiments shown in Figures 12-16, the number of direction markers or physiological structure markers can be one, two, three, or more than three as needed.

[0096] For the enhancement method of placing markers on the preoperative 3D image, the processor first obtains the preoperative 3D image and simultaneously acquires the markers (e.g., directional markers or markers indicating physiological structures) placed on the preoperative 3D image by the operator as described above. The preoperative 3D image is then registered with the navigation system, and the coordinates of each marker in the navigation system are determined based on the registration relationship established in the registration step, thereby completing the marker position acquisition step.

[0097] For enhancement methods involving marking on intraoperative 3D or 2D images, registration of the image with the navigation system is typically completed simultaneously with acquisition. Therefore, in this case, the operator marks the registered 3D or 2D image. Since the coordinates of the registered image under the navigation system are known, the coordinates, i.e., the position, of the mark placed on the image can be determined accordingly, thus completing the step of obtaining the mark's position.

[0098] As described above, when using intraoperative 2D images, a quasi-3D fitting can be performed on at least two 2D images from different body positions to obtain a quasi-3D image to facilitate operator marking. The markings subsequently captured by the processor are the markings made by the operator on the fitted image. Those skilled in the art will appreciate that the images used for the quasi-3D fitting can be intraoperative anteroposterior and lateral images, or images from other body positions, depending on the operator's needs and the actual location of the patient undergoing surgery.

[0099] The operator can input or select the various forms of markings described above in various ways. This can be input through the interface of the display device, for example, by using the touch interface of the display device and using preoperative planning software or intraoperative planning software to select the cephalad, caudal, ventral, and dorsal direction markings, or the selection of physiological structure markings, as shown in Figures 12 and 13. Input can also be performed using the keyboard and / or mouse of an electronic device, as exemplarily shown in Figure 14.

[0100] It should be noted that, although the description herein uses the method of an operator making marks on an image of a patient's physiological structure to form the marks, it is also understood that in other embodiments, the marks may not be made by the operator, for example, the marks may be automatically formed when the image is formed. For example, a marker may be pre-placed on a certain part of the patient's body or on a location such as an operating table so that the marks are generated when the image is acquired.

[0101] When the fused enhanced information includes a three-dimensional image of the patient's physiological structure, the enhanced information acquisition step acquires the three-dimensional image. Furthermore, since the three-dimensional image used for navigation has determined position information under the navigation system through registration, the position of the three-dimensional image under the navigation system can be easily acquired.

[0102] After obtaining the enhanced information (stitched images, three-dimensional images, or markers) required for fusion and their orientation under the navigation system as described above, fusion of the endoscopic enhanced image can be performed in conjunction with the acquired orientation information of the endoscopic imaging device. Because the enhanced information required for fusion and the imaging device have a defined orientation relationship under the navigation system, and thus, the enhanced information required for fusion and the image captured by the imaging device also have a defined orientation relationship under the navigation system, fusion of the enhanced information and the current endoscopic image can be performed based on these orientation relationships.

[0103] It should be understood that although the above steps are listed and described sequentially in the flowchart of FIG1 , the claims, and the description herein, this does not imply a specific sequential relationship between the steps. For example, the enhanced information acquisition step and the enhanced information orientation acquisition step can be performed before, after, or simultaneously with the endoscopic image acquisition step and / or the imaging orientation acquisition step. The global enhanced image fusion step (described below) can also be performed before, after, or simultaneously with the endoscopic enhanced image fusion step.

[0104] FIG3 shows an example of the obtained endoscopic enhanced image, in which a stitched image 6, a three-dimensional image 7 of the patient's physiological structure and a current image 10 of the endoscope are fused. FIG4 exemplarily shows a view (also referred to as an endoscopic enhanced view) in which the endoscopic enhanced image in FIG3 is displayed on a window of a display device, wherein the current image of the endoscope (i.e., the real-time image) is located in the central area of ​​the window. The endoscopic image displayed in this view is larger and shows the surrounding enhancement information. Exemplarily, the range of the endoscopic enhanced image displayed in the view of the endoscopic enhanced image (corresponding to the proportion of the current image of the endoscope in the endoscopic enhanced view) can be determined based on the input of the operator. In other words, the operator can change the proportion of the current image of the endoscope in the endoscopic enhanced view by interacting with the system, for example, by zooming in or out the current image of the endoscope. When the doctor needs to observe a detail in the endoscopic field of view in more detail, the real-time endoscopic image can be enlarged (correspondingly, the portion of the endoscopic enhanced image displayed in the view window becomes smaller), while the proportion of the content displayed in the surrounding enhanced information to the real-time endoscopic image remains unchanged, and the enhanced information beyond the edge of the "endoscopic enhanced view" is no longer displayed on the "endoscopic enhanced view". The operator's input can be carried out in the form of a zoom icon input component on the display device. However, those skilled in the art will understand that the operator's input method for determining the size of the real-time endoscopic image in the view is not limited to the zoom icon method, and can also be carried out in other ways, such as providing the operator with several different size options, or determining it by the operator entering a numerical value. The input component displayed on the display interface can be, for example, a tab, or a dialog box for the operator to enter a numerical value.

[0105] FIG5 exemplarily shows a view of an endoscopic enhanced image displayed on a window of a display device according to another exemplary embodiment, wherein the endoscopic enhanced image fuses the current image 10 of the endoscope and three types of enhanced information, namely, the stitched image 6, the three-dimensional image 7 of the patient's physiological structure and the physiological structure markers shown in the figure. In this example, the ventral side of the articular process, the pedicles and intervertebral discs of the anterior vertebra are shown, and the pedicles of the posterior vertebra are not displayed because they are located outside the edge of the view. Guidance instructions related to the physiological structure markers can also be fused to the endoscopic enhanced image and displayed. For example, when the physiological structure marker is outside the view range of the endoscopic enhanced image, the guidance instruction can be an arrow pointing to the physiological structure marker (for example, an arrow pointing to the pedicles of the posterior vertebra that are not displayed in FIG5 ).

[0106] Because the current endoscopic image, or real-time image, in the endoscopic enhanced view is desired to be as large as possible for easier observation by the operator, the enhanced information that can be displayed is very localized and limited. To provide the operator with better global guidance, the present invention may also include a global enhanced image. Specifically, at least two of the mosaic image, the three-dimensional image, and the guidance instructions associated with the markers are fused to generate and display a global enhanced image. The global enhanced image indicates the current field of view of the imaging device and the position of the endoscopic enhanced image. As shown in FIG6 , the endoscopic enhanced image is shown on the left side of the display device, and the global enhanced image is shown on the lower right. Specifically, FIG7 exemplarily illustrates a global enhanced image fused with the mosaic image 6, the three-dimensional image 7 of the patient's anatomy, and directional markers pointing to the dorsal, ventral, caudal, and cephalad directions. FIG8 exemplarily illustrates a global enhanced image fused with the mosaic image 6, the three-dimensional image 7 of the patient's anatomy, and markers of physiological structures such as the intervertebral disc. The fusion method for the global enhanced image is the same as the fusion method for the enhanced information described above: each acquired enhanced information is fused based on its orientation information as determined by the navigation system. The detailed process will not be elaborated upon.

[0107] In the global enhanced images shown in Figures 7 and 8, the edge of the imaging device's current field of view (i.e., the position of the current endoscope image in the global enhanced image) is indicated by a dotted line 8, and the edge of the view corresponding to the endoscopic enhanced image is framed by a dotted line 9. Specifically, the method of the present invention also includes a step of indicating the current imaging device's field of view in the global enhanced image. In this step, the region corresponding to the current imaging device's field of view is determined in the global enhanced image using the imaging device's current position and orientation, and a marker 8 indicating the edge of this region is generated and displayed in the display step. It will be appreciated by those skilled in the art that other forms other than dotted lines may be used to indicate the edge of the current field of view and the edge of the view of the endoscopic enhanced image, and both may be represented by lines of different colors and / or line types.

[0108] In the above scheme, the endoscopic enhanced view is displayed simultaneously with the global enhanced view as a part of the global enhanced image, and the correlation information between the two is displayed in the global enhanced image, that is, the position of the edge of the real-time endoscopic image and the edge of the endoscopic enhanced view in the global enhanced image is displayed, so that the operator can know the position of the endoscopic field of view in the entire world and the surrounding soft tissue, bone structure, target position and direction information while watching the larger real-time endoscopic image, thereby realizing all-round guidance for the operator.

[0109] It should also be noted that the "step of displaying the endoscopic enhanced image" or "step of displaying the global enhanced image" described herein does not necessarily mean that the image is displayed at all times. For example, the endoscopic enhanced image or the global enhanced image may be displayed only for a certain period of time based on the operator's needs. For example, when the operator desires to observe the enhanced image, the operator may trigger the display of the corresponding enhanced image using, for example, a foot switch.

[0110] The present invention also provides an electronic device for endoscopic surgical navigation, comprising a display device 3 and a processor as described above. In the specific example shown in FIG2 , the processor is included in the control device 2, i.e., the illustrated host. Those skilled in the art will appreciate that the processor and display device may be integrated or separate. The control device 2 or processor has a data interface. The control device or processor is electrically connected to the tracking device 1 and the endoscope 5 of the navigation system via the data interface to obtain the position and orientation of the imaging device of the endoscope and to obtain an image of the endoscope in the corresponding position and orientation. The processor's data interface also enables the processor to obtain one or more of a three-dimensional image of a patient's physiological structure and a marker on the three-dimensional or two-dimensional image of the patient's physiological structure. When the processor is running, it executes a computer program (which may be stored in a memory included in the control device or in another memory) to perform the method of the present invention and display a view of at least a portion of an endoscopically enhanced image on the display device 3 for at least a period of time, wherein the endoscopically enhanced image is an image that fuses the current image of the endoscope with at least two of the following three types of enhancement information:

[0111] a) a stitched image obtained by stitching multiple images of the endoscope;

[0112] b) 3D images of the patient’s physiological structures;

[0113] c) Guidance instructions related to the marking.

[0114] As an example, a global enhanced image is also displayed on the display interface of the display device 3. The global enhanced image is an image obtained by fusing at least two of the stitched image 6, the three-dimensional image 7 and the guidance indication, wherein the global enhanced image indicates the current field of view position 8 of the imaging device and / or the view position 9 of the endoscopic enhanced image.

[0115] As an example, the display interface of the display device 3 may also display one or more of the following images or views: a view on a fitted two-dimensional perspective plane, a sagittal plane view, a coronal plane view, an axial plane view, a real-time endoscopic image, etc. This allows the operator to conveniently obtain more comprehensive navigation information. Those skilled in the art will appreciate that other views of other planes, orientations, or any other suitable view may also be displayed as needed.

[0116] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, is capable of performing the steps of the above-mentioned method of the present invention. Furthermore, the present invention also provides a control device, which may include a memory, a processor, and a program stored on the memory and executable by the processor, wherein the steps of the above-mentioned method of the present invention are performed when the program is executed by the processor. The present invention also provides a computer program product, including the computer program, wherein the computer program, when executed by the processor, implements the steps of the above-mentioned method of the present invention.

[0117] Those skilled in the art will appreciate that the steps of the methods or algorithms described herein can be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0118] In the above embodiments, the method can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0119] Those skilled in the art will appreciate that the memory of the control device of the present invention may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device separate from the processor.

[0120] The processor of the control device can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0121] Although certain embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the general inventive concept, the scope of which is defined in the claims and their equivalents.

Claims

1. A method for guiding endoscopic surgery, characterized in that The method comprises the following steps: Endoscopic image acquisition steps: acquiring an endoscopic image; Enhanced information acquisition step: acquiring at least two of the following three types of enhanced information: a) a stitched image obtained by stitching multiple images of the endoscope; b) 3D images of the patient’s physiological structures; c) markings on a three-dimensional image or a two-dimensional image of a patient's physiological structure; Endoscopic enhanced image fusion step: fusing the acquired current endoscopic image with the at least two types of enhanced information acquired in the enhanced information acquisition step to obtain an endoscopic enhanced image; as well as Displaying step: displaying a view including at least a portion of the endoscopically enhanced image.

2. The method according to claim 1, characterized in that The endoscopic enhanced image fusion step is performed with the aid of a navigation system, wherein the method further comprises, before the endoscopic enhanced image fusion step: Imaging orientation acquisition step: acquiring the orientation of the imaging device of the endoscope under the navigation system; and Enhanced information position acquisition step: acquiring the position of the desired fused spliced ​​image, the three-dimensional image or the marker under the navigation system; In the endoscopic enhanced image fusion step, the fusion is performed according to the orientation of the imaging device and the orientation of the spliced ​​image, three-dimensional image or marker to be fused.

3. The method according to claim 1 or 2, characterized in that The following steps are also included: a global enhanced image fusion step: fusing at least two of the stitched image, the three-dimensional image, and the guidance indication associated with the marker to obtain a global enhanced image, wherein the global enhanced image indicates a current field of view position of an imaging device and / or a position of the view of the endoscopic enhanced image; as well as a step of displaying the globally enhanced image; Wherein, on the global enhanced image, the edge of the current field of view position of the imaging device and the edge of the view of the endoscopic enhanced image are represented by lines of different colors and / or line types.

4. The method according to claim 1 or 2, characterized in that The type of enhancement information acquired and fused is selected based on operator input.

5. The method according to claim 1 or 2, characterized in that The range of the endoscopy enhanced image displayed in the endoscopy enhanced image view can be determined according to an operator's input.

6. The method according to claim 2, characterized in that In the case where the fused enhancement information includes a stitched image, the enhancement information acquisition step includes: A stitching step, in which a plurality of the endoscopic images are stitched together according to the orientation of the imaging device corresponding to at least one endoscopic image to obtain the stitched image.

7. The method according to claim 6, characterized in that In the case where the fused enhancement information includes a spliced ​​image, the enhancement information acquisition step further includes: a distortion calibration step before the stitching step, wherein the distortion of the endoscope images to be stitched is calibrated in the distortion calibration step; and A processing step follows the stitching step, in which a planar image is generated based on the image obtained in the stitching step, wherein the planar image is fused with the current image of the endoscope as the stitched image in the endoscopic enhanced image fusion step.

8. The method according to claim 2, characterized in that In the case where the fused enhanced information includes the marker, the enhanced information position acquisition step includes a marker position acquisition step, in which the position of the marker under the navigation system is acquired by means of registration of the three-dimensional image or the two-dimensional image under the navigation system; In the endoscopic enhanced image fusion step, according to the marker under the navigation system The guidance instructions associated with the marker are fused with the current image of the endoscope.

9. The method according to claim 8, characterized in that The method enables an operator to select one or more of a direction mark and a mark indicating a physiological structure.

10. The method according to claim 9, characterized in that The patient's physiological structure is a spine, and the markers indicating the physiological structure include an image model of one or more of a herniated or free intervertebral disc, osteophytes, and ossified ligamentum flavum.

11. The method according to claim 9, characterized in that The marks indicating physiological structures include physiological structure marking points; preferably, the patient's physiological structure is the spine, and the physiological structure marking points are marking points indicating one or more of a protruding or free intervertebral disc, osteophytes and ossified yellow ligaments, or are physiological structure marking points that do not displace during endoscopic surgery, such as one or more of the ventral side of the articular process, the pedicle of the anterior vertebra, the pedicle of the posterior vertebra, and the intervertebral disc; preferably, there are multiple physiological structure marking points.

12. The method according to claim 9, characterized in that The direction markings include one or more of direction markings toward the dorsal side, the ventral side, the cephalad side, and the caudal side of the patient.

13. The method according to claim 2, characterized in that In the case where the fused enhancement information includes a three-dimensional image of the patient's physiological structure, the enhancement information position acquisition step includes the step of registering the three-dimensional image of the patient's physiological structure with a navigation system to acquire the position of the three-dimensional image.

14. The method according to claim 13, characterized in that The three-dimensional image of the patient's physiological structure includes a pre-operative three-dimensional image or an intra-operative three-dimensional image.

15. The method according to any one of claims 1 to 14, characterized in that The endoscope is a spinal endoscope.

16. The method according to claim 2, characterized in that The imaging position acquisition step is performed by acquiring the position of a tracer (4) having a fixed positional relationship relative to the endoscope from a tracking device (1) of a navigation system.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are performed.

18. A control device, wherein the control device comprises a memory, a processor, and a program stored in the memory and capable of running on the processor, characterized in that: When the processor runs the program, the steps of the method according to any one of claims 1 to 16 are performed.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

20. An electronic device for endoscopic surgery navigation, characterized in that: The electronic device comprises a display device (3) and a processor, wherein the processor has a data interface. wherein the data interface is connectable to an endoscope (5) so that the processor can acquire images of the endoscope; and wherein the processor is configured to acquire one or more of a three-dimensional image of the patient's physiological structure and a marking on the three-dimensional image or the two-dimensional image of the patient's physiological structure; The processor is configured to display a view of at least a portion of an endoscopically enhanced image on the display device (3) for at least a period of time when the processor is running, wherein the endoscopically enhanced image is an image obtained by fusing a current image of the endoscope with at least two of the following three types of enhancement information: a) a stitched image obtained by stitching multiple images of the endoscope; b) 3D images of the patient’s physiological structures; c) Guidance instructions associated with the marking.

21. The electronic device according to claim 20, characterized in that The data interface is connectable to a tracking device (1) of a navigation system.

22. The electronic device according to claim 20, wherein: A global enhanced image is also displayed on the display device (3), wherein the global enhanced image is an image obtained by fusing at least two of the stitched image, the three-dimensional image and the guidance indication, and lines of different colors and / or line types are used on the global enhanced image to indicate the edge of the current field of view position of the imaging device and the edge of the view of the endoscopic enhanced image.

23. The electronic device according to any one of claims 20 to 22, characterized in that: The display interface of the display device also displays one or more of the following items: a view on a fitted two-dimensional perspective section, a sagittal section view, a coronal section view, an axial section view, and a real-time image of the endoscope.

24. The electronic device according to any one of claims 20 to 22, characterized in that: The electronic device further includes an input component for an operator to input an instruction to select the type of enhanced information to be fused.

25. The electronic device according to any one of claims 20 to 22, characterized in that: The electronic device further includes an input component for an operator to input an instruction to determine a range of the endoscopy enhanced image displayed in the endoscopy enhanced image view.

26. The electronic device according to any one of claims 20 to 22, characterized in that: The mark includes one or more of a direction mark and a mark indicating a physiological structure; Preferably, the patient's physiological structure is a spine, and the marker indicating the physiological structure includes an image model of one or more of a protruding or free intervertebral disc, osteophytes, and ossified yellow ligaments; or, the marker indicating the physiological structure includes physiological structure marker points, which are marker points indicating one or more of a protruding or free intervertebral disc, osteophytes, and ossified yellow ligaments, or physiological structure marker points that do not displace during endoscopic surgery, such as one or more of the ventral side of the articular process, the pedicle of the anterior vertebral body, the pedicle of the posterior vertebral body, and the intervertebral disc; Preferably, the direction mark includes one or more of direction marks toward the dorsal side, ventral side, cranial side and caudal side of the patient.

27. A navigation system for endoscopic surgery, characterized in that: The navigation system includes: A tracking device (1) adapted to track a tracer (4) having a fixed positional relationship relative to the endoscope (5); and A display device (3) and a processor, wherein the processor is suitable for being connected to the tracking device (1) and the endoscope (5), wherein the method according to any one of claims 1 to 16 is executed when the processor is running, and the display in the method is realized by the display device (3).

28. A navigation system for endoscopic surgery, characterized in that: The navigation system includes: A tracking device (1) adapted to track a tracer (4) having a fixed positional relationship relative to the endoscope (5); and An electronic device according to any one of claims 20-26, wherein a processor in the electronic device is adapted to be connected to the tracking device (1) and the endoscope (5).

29. A robot system, characterized in that: The apparatus comprises a robotic arm and a navigation system according to any one of claims 27-28.

Images