Navigation method for endoscopic spine surgery, electronic device, and navigation system

Abstract

A navigation method and navigation system for endoscopic spine surgery. The method comprises the following steps: a three-dimensional image acquisition step: acquiring a three-dimensional image of at least a part of a spine; an image acquisition step: acquiring an image of an endoscope in real time; an orientation acquisition step: acquiring the position and direction of the tip of the endoscope in real time; an image segmentation step: segmenting the three-dimensional image on the basis of the position and direction of the tip of the endoscope; a fusion step: fusing the segmented three-dimensional image and the image of the endoscope; and a display step: displaying a fused image. According to the method, in an endoscopic spine surgery scenario, augmented reality fusion of a real-time endoscopic field of view and the three-dimensional image around the endoscope is achieved, providing, in spine surgery, doctors with more visual orientation information of the tip of the endoscope and of the tip of a surgical tool in a spine structure, thereby avoiding misoperation to a great extent and increasing the reliability of the spine surgery.

Description

Navigation method, electronic device and navigation system for spinal endoscopic surgery Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a surgical navigation system, and more specifically to a navigation method and a navigation system for spinal endoscopic surgery based on image fusion. Background Art

[0002] Endoscopic spinal surgery is popular with patients and minimally invasive spinal surgeons due to its clinical advantages, including minimal invasiveness, minimal bleeding, and shortened hospital stays. However, traditional endoscopic spinal surgery often has a very steep learning curve, requiring a high level of anatomical and physiological knowledge and spatial visualization, which has greatly limited its clinical application.

[0003] In traditional endoscopic surgery, surgical observation, tool manipulation, and control are performed solely through the endoscope. However, with only the endoscope in view, the surgeon lacks bony structures for reference, making it extremely difficult to identify soft tissue structures in the two-dimensional image. This can lead to a loss of overall vision and sense of direction, resulting in difficulty in eye-hand coordination. This can lead to the endoscope failing to reach the affected area or insufficient decompression, resulting in failure to achieve the intended surgical goal.

[0004] In spinal endoscopic surgery, particularly transforaminal endoscopic lumbar discectomy, these shortcomings are particularly pronounced due to the unique structure of spinal endoscopes. The camera and tool channels of these endoscopes are integrated into the same endoscope insertion tube. When the surgeon rotates the endoscope to adjust the camera and view the desired area, the surgical tool also rotates with it, changing its orientation relative to the spine. This makes it more difficult for the surgeon to accurately grasp the tool's overall position relative to the spine, making misoperation more likely.

[0005] Furthermore, the tools used in minimally invasive spinal endoscopic surgery include, for example, tools with angled distal ends, tools with flexible handles, and tools with expandable distal ends. Therefore, doctors cannot determine the actual position of the distal end (working end) of the tool by relying solely on the two-dimensional image from the endoscope, which can easily lead to misoperation and surgical errors.

[0006] Traditional navigation technology is primarily used to guide pedicle screw placement during spinal fixation surgery, enabling real-time tracking and positioning of tools and implants relative to the patient's anatomy. This helps doctors achieve the clinical goals of "precision, safety, and minimally invasive procedures." Optical navigation, with its high positioning accuracy, low susceptibility to interference, and low cost, is gaining increasing clinical attention in spinal surgery.

[0007] Summary of the Invention

[0008] The object of the present invention is to solve at least one of the above problems and defects in the prior art as well as other technical problems.

[0009] On the one hand, the present invention provides a navigation method for spinal endoscopic surgery, which includes the following steps: a three-dimensional image acquisition step: acquiring a three-dimensional image of at least a portion of the spine; an image acquisition step: acquiring an endoscopic image in real time; an orientation acquisition step: acquiring the position and direction of the end of the endoscope in real time; a three-dimensional image cutting step: cutting the three-dimensional image according to the position and direction of the endoscope end; a fusion step: fusing the cut three-dimensional image with the image; and a display step: displaying the fused image.

[0010] In this navigation method for spinal endoscopic surgery, the real-time image of the endoscope is combined with the global three-dimensional image through augmented reality, which is particularly beneficial for the navigation of spinal endoscopic surgery. As described in the background art, due to the special structure of the spinal endoscope, the surgical tool will rotate synchronously with the endoscopic camera, and due to the particularity of the surgical tool used, the orientation of the end of the surgical tool is variable and difficult to track. The surgeon is more likely to get lost when observing the endoscopic image, and it is even more difficult to grasp the position and orientation of the end of the surgical tool. The present invention realizes the augmented reality fusion of the real-time endoscopic field of view (the end of the surgical tool is visible in the real-time endoscopic field of view) and the three-dimensional image around the endoscope in the scene of spinal endoscopic surgery through the navigation system, providing doctors with more intuitive information on the orientation of the end of the endoscope and the end of the surgical tool in the spinal structure during spinal surgery, which can largely avoid misoperation and increase the reliability of spinal surgery.

[0011] According to one example, in the three-dimensional image cutting step, a cylindrical shape is cut out of the three-dimensional image, wherein the diameter of the cylinder is determined according to the operator's input, thereby determining the range of cutting the three-dimensional image to obtain a three-dimensional image of the range that the operator needs to display.

[0012] This exemplary solution allows the operator to select the range of the observed 3D image as needed. For example, by operating the zoom icon described below, the diameter of the cylinder can be expanded or reduced, enabling free scaling of the observed 3D image range. This is particularly beneficial for spinal endoscopic surgery. Because the endoscope's field of view is very small, even experienced spinal surgeons may find it difficult to identify bony structures within the limited range covered by the endoscope, or the bony structures may be at an unfamiliar viewing angle. Surgeons sometimes need to see a larger or global 3D image to determine the precise position and lens orientation of the endoscope relative to the patient's anatomy and make appropriate adjustments. Sometimes, they need to see a magnified 3D image (with a correspondingly reduced display range) to observe the details of the patient's anatomy. These two operations sometimes require frequent switching. In this exemplary solution, the size of the 3D image cut range can be controlled through operator interaction, meeting the operator's real-time observation needs. In particular, the provided zoom function overcomes the limitations of the endoscope's narrow field of view and greatly facilitates observation during spinal endoscopic surgery.

[0013] Furthermore, as mentioned above, due to the unique structure of spinal endoscopes and the specificity of the surgical tools used, surgeons are more likely to become disoriented when observing endoscopic images. This solution not only integrates the real-time endoscopic field of view (in which the surgical tool tip is visible) with augmented reality of the three-dimensional image surrounding the endoscope, but also allows for adjustment or selection of the display range of the three-dimensional image as needed, enabling better observation of the orientation relationship between the endoscope tip and the surgical tool tip and the surrounding three-dimensional image (corresponding to the patient's bony structure), thereby significantly preventing misoperation and increasing the reliability of spinal surgery.

[0014] According to an example, the fusion step further includes scaling the image size of the endoscope to correspond to the three-dimensional images of different cutting ranges before performing the fusion.

[0015] This exemplary solution ensures that the endoscopic image has an appropriate proportion within the entire fused image. For example, when the 3D image's cutting range is small, and its display ratio is correspondingly large, the doctor can observe the orthopedic structures beneath the soft tissue in the endoscopic field of view. When the 3D image's cutting range is increased, and its display ratio is correspondingly reduced, the endoscopic image is reduced accordingly, allowing the doctor to observe the surrounding bony structures outside the endoscopic field of view, making it easier for the doctor to determine the position and orientation of the endoscope and adjust the operation accordingly.

[0016] According to one example, the endoscope tip is oriented along the optical axis of the endoscope's camera, and the cut begins at the focal plane of the endoscope's camera and extends along the optical axis throughout the entire depth of the three-dimensional image. Cutting through the entire depth of the three-dimensional image allows for a more complete presentation of the bony structure of the spine within and surrounding the endoscope's field of view.

[0017] According to one example, the distal end of the endoscope forms an inclined surface that is inclined relative to the longitudinal axis of the endoscope, and the optical axis of the endoscope is perpendicular to the inclined surface.

[0018] According to one example, in the image acquisition step, a preoperative three-dimensional image of at least a portion of the spine is registered with a patient posture in a navigation coordinate system.

[0019] According to one example, a distortion processing step is also included before the fusion step, wherein in the distortion processing step, a selection can be made from the following three methods according to the operator's input: performing distortion calibration on the acquired endoscopic image to at least partially remove the distortion effect of the endoscopic camera, so that the distortion-calibrated endoscopic image matches the cut-out three-dimensional image; or, performing distortion processing on the cut-out three-dimensional image according to the parameters of the endoscopic camera, so that the distortion-processed three-dimensional image matches the endoscopic image acquired in real time; or, performing distortion calibration on the endoscopic image and distortion processing on the three-dimensional image at the same time, so that the distortion degrees of the two match, and then, corresponding to the distortion processing step, in the fusion step, the distortion-calibrated endoscopic image is fused with the cut-out three-dimensional image, or the real-time acquired endoscopic image is fused with the distortion-processed three-dimensional image, or the distortion-calibrated endoscopic image is fused with the distortion-processed three-dimensional image.

[0020] This example provides the surgeon with multiple options, allowing them to flexibly choose based on their preferences. During distortion processing, the endoscope's camera distortion parameters are used to calibrate the endoscopic image to remove the camera's distortion effects, producing a view identical to a true perspective scene. This results in an image with a normal field of view that visually matches the undistorted three-dimensional image, making it easier for the surgeon to view. The endoscope's camera parameters can also be used to distort the cut three-dimensional image so that it has the same distortion effects as the endoscopic image and visually matches the distorted actual endoscope image, meaning both are distorted. This approach is more convenient for surgeons accustomed to viewing distorted endoscopic images. Furthermore, it is possible to both calibrate the endoscopic image and perform distortion processing on the three-dimensional image so that the degree of distortion matches.

[0021] According to an example, in the fusion step, it is also possible to select from the following two fusion methods according to the operator's input or provide images obtained by the following two fusion methods at the same time:

[0022] The fusion is performed in a manner that the cut three-dimensional image is placed below the real-time endoscope image; and

[0023] The fusion is performed by placing the endoscopic image acquired in real time into the three-dimensional image acquired in the image acquisition step in a manner corresponding to the position and orientation of the endoscope tip.

[0024] This example provides the surgeon with two fusion options: the first is more convenient for surgeons accustomed to the endoscopic field of view, while the second allows the surgeon to see the entire image. This allows the surgeon to flexibly choose according to their preference.

[0025] According to one example, in the display step, the three-dimensional image located behind the end of the endoscope is blurred and displayed.

[0026] This example provides the surgeon with a more intuitive image, enabling him to better understand the direction of the endoscope.

[0027] According to one example, the navigation method further includes an endoscope tip calibration step for calibrating a positional relationship of the endoscope tip relative to a first tracker on the endoscope, wherein the endoscope tip calibration step is performed using a calibration tool rather than acquiring images through the endoscope. This reduces the workload of image processing.

[0028] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the methods in the above examples are executed.

[0029] According to another aspect of the present invention, a control device is also provided, wherein the control device includes a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the methods in the above examples are executed when the processor runs the program.

[0030] According to another aspect of the present invention, there is also provided an electronic device for navigation of spinal endoscopic surgery, the electronic device comprising a display device and a processor, wherein the processor has a data interface.

[0031] wherein the data interface is connectable to an endoscope, so that the processor can acquire an image of the endoscope and the position and direction of the endoscope tip in real time; and

[0032] The data interface also enables the processor to obtain a three-dimensional image of at least a portion of the spine; and when the processor is running, a fused image is displayed on the display device, wherein the fused image is an image obtained by fusing the three-dimensional image corresponding to the position and orientation of the endoscope tip with the image.

[0033] According to another aspect of the present invention, a navigation system for spinal endoscopic surgery is also provided, which includes a tracking device, a display device and a processor; wherein the tracking device is used to track a first tracker set on the endoscope and a second tracker set at the target part of the patient; when the processor is running, the steps of the methods in the above examples are executed, and the display in the navigation method is realized through the display device.

[0034] According to yet another aspect of the present invention, a navigation system for spinal endoscopic surgery is provided. The navigation system includes a tracking device and the electronic device described in each example.

[0035] According to an example, the navigation system further includes one or more of the endoscope, the first tracker, and the second tracker.

[0036] According to one example, the navigation system also includes a calibration tool for calibrating the position of the end of the endoscope relative to the first tracker, wherein a plurality of calibration holes are formed on the calibration tool, and the plurality of calibration holes have bottom surfaces with different inclination angles and / or different apertures to respectively calibrate endoscopes with different end bevel inclination angles and / or barrel diameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention is described in detail below by way of exemplary embodiments with reference to the accompanying drawings.

[0038] FIG1 shows a flow chart of a navigation method for spinal endoscopic surgery according to an exemplary embodiment of the present invention.

[0039] FIG2 is a schematic diagram showing a principle of a navigation system for spinal endoscopic surgery according to an exemplary embodiment of the present invention.

[0040] Figures 3a-3d are schematic diagrams showing the principles of cutting a three-dimensional image and fusing the cut three-dimensional image with an image according to the operator's scaling requirements, wherein Figure 3a is a schematic diagram showing the principles of cutting a three-dimensional image in a smaller range, Figure 3b is a diagram showing the image after the three-dimensional image cut according to Figure 3a is fused with the endoscopic image, Figure 3c is a schematic diagram showing the principles of cutting a three-dimensional image in a larger range, and Figure 3d is a diagram showing the image after the three-dimensional image cut according to Figure 3c is fused with the endoscopic image.

[0041] 4a and 4b schematically illustrate images obtained by two fusion methods.

[0042] FIG5 shows a schematic diagram of an exemplary display device.

[0043] It should be noted that the drawings are schematic only. They illustrate only those parts or steps necessary to illustrate the present invention, and other parts or steps may be omitted or only briefly mentioned. In addition to the parts or steps shown in the drawings, the present invention may also include other parts or steps. DETAILED DESCRIPTION

[0044] The following examples and accompanying drawings further illustrate the technical solution of the present invention. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall concept of the present invention and should not be construed as limiting the present invention.

[0045] The following describes, as an example, the specific steps of a navigation method for spinal endoscopic surgery and a navigation system for spinal endoscopic surgery according to the present invention. In the following detailed description, many specific details and steps are set forth in a highly specific and detailed manner to provide a comprehensive understanding of the embodiments. However, it should be understood that one or more other embodiments may be implemented without these specific details and steps.

[0046] FIG1 shows a flowchart of an exemplary method for endoscopic spinal surgery according to the present invention. The method includes an image acquisition step for acquiring a three-dimensional image of at least a portion of the spine, an image acquisition step for acquiring an endoscopic image in real time, an orientation acquisition step for acquiring the direction and position of the endoscope tip of the spine in real time, a three-dimensional image cutting step for cutting the three-dimensional image according to the position and direction of the endoscope tip, a fusion step for fusing the cut three-dimensional image with the image, and a display step for displaying the fused image. It should be understood that although the steps are listed and described sequentially in the flowchart of FIG1 , the claims, and the description herein, this does not imply a specific order relationship between the steps. For example, the image acquisition step may be performed before, after, or simultaneously with the image acquisition step and / or the orientation acquisition step. The image acquisition step may also be performed before, after, or simultaneously with the orientation acquisition step.

[0047] As shown in Figure 1, in the navigation method provided by the present invention, a three-dimensional image is acquired along with an endoscopic image in real time. The position and orientation of the endoscope tip are then acquired in real time using a navigation system. The three-dimensional image is then fused with the real-life image acquired by the endoscope in real time, corresponding in position and orientation to the endoscope tip, and displayed accordingly. This combination of real-time, real-life imagery and global, augmented reality of the three-dimensional image is particularly beneficial for navigation during spinal endoscopic surgery. As described in the background art, due to the unique structure of spinal endoscopes and the specificity of the surgical tools used, surgeons are more likely to become disoriented when observing endoscopic images and have difficulty determining the position and orientation of the surgical tool tip. The present invention implements an augmented reality surgical plan combining three-dimensional images with a real-time endoscopic field of view (with the surgical tool tip visible in the real-time endoscopic field of view) in the context of spinal endoscopic surgery. This integration of the navigation system provides the surgeon with more intuitive and effective information during surgery, significantly increasing the reliability of the surgeon's surgery.

[0048] In a specific embodiment, the navigation system for spinal endoscopic surgery includes a tracking device 1, a control device 2 and a display device 3 as shown in Figure 2. The tracking device 1 can be an optical tracking device (such as an NDI navigator), and accordingly, a first optical tracker 4 can be set on the endoscope and a second optical tracker 5 can be set at the target part of the patient. It will be understood by those skilled in the art that the tracking device 1 can also be of other types, such as an electromagnetic tracking device. As a specific example, the control device 2 can be a general-purpose computer, a special-purpose computer, an embedded processor, or any other appropriate programmable data processing device such as a single-chip microcomputer or a chip. The control device 2 may include a processor and a memory for storing programs, but may also include only a processor, in which case the processor may be attached to a memory storing programs. In other words, the control device includes at least a processor. The control device 2 (or processor) and the display device 3 can be integrated into one or provided separately. The control device or processor has a data interface. This data interface may include a data interface that can be connected to an endoscope, allowing the control device / processor to obtain images from the endoscope and the position and orientation of the endoscope's distal end in real time. The data interface may also include an interface that connects to, for example, a CT device, allowing the processor to obtain a three-dimensional image of at least a portion of the spine via the data interface. As an example, one or more of the endoscope, the first tracker, and the second tracker may also be considered part of the navigation system of the present invention.

[0049] An endoscope 8 is schematically shown in FIG2 . The endoscope may be, for example, a perforaminal endoscope. Typically, the distal end of the endoscope enters the patient's tissue structure or bony structure to observe and / or operate the tissue structure, and the proximal end of the endoscope (the end close to the operator and the end opposite to the distal end) is located outside the patient's body for manipulation by the operator. In the navigation system, a first tracker suitable for being tracked by the tracking device 1 is typically provided at the proximal end of the endoscope located outside the patient's body, that is, the navigation system can obtain the position of the first tracker 4 on the endoscope 8 in the navigation coordinate system in real time. In the present invention, the camera of the endoscope is provided at the distal end of the insertion barrel of the endoscope. By calibrating the relative positional relationship of the distal end of the insertion barrel of the endoscope with respect to the first tracker 4 on the endoscope, the position of the camera of the endoscope in the navigation coordinate system can be further obtained.

[0050] Specifically, before performing spinal endoscopic surgery and its navigation, the relative position of the endoscope's camera relative to the first tracker can be calibrated. Preferably, in this embodiment, this calibration is performed using a calibration tool (not shown) without the need for endoscope image acquisition. This reduces the workload of image processing. The navigation system of the present invention optionally includes this calibration tool. The calibration tool is formed with multiple calibration holes, each with a bottom surface having different tilt angles and / or different apertures, to calibrate endoscopes with different end bevel angles and / or barrel diameters. Specifically, the calibration tool also includes a third tracker fixed to its bracket, and the navigation system can determine the position of this third tracker in the navigation coordinate system. Furthermore, based on the design dimensions of the calibration tool, the positional relationship of each calibration hole relative to the third tracker is known, and thus the positions of these calibration holes in the navigation coordinate system are known. The end of the insertion barrel of an endoscope to be calibrated is inserted into a matching calibration hole, with the inclined surface of the end of the insertion barrel aligned with the inclined bottom surface of the calibration hole, thereby achieving positioning of the endoscope. Since the navigation system can also obtain the position of the first tracker on the endoscope in the navigation coordinate system, it can obtain the relative position relationship of the end of the endoscope insertion tube relative to the first tracker on the endoscope, completing the calibration process.

[0051] During surgery, the position of the first tracker on the endoscope is acquired in real time by the tracking device 1, thereby indirectly acquiring the position and orientation of the endoscope tip, i.e., the position and orientation of the endoscope camera, in real time. This is how the orientation acquisition step in the navigation method of the present invention is performed.

[0052] In this specific embodiment of the present invention, the camera parameters of the endoscope can also be calibrated before performing spinal endoscopic surgery and its navigation operation. Exemplarily, the calibration of the camera parameters can use a calibration plate, for example, using an endoscope to shoot the calibration plate at different azimuth angles, save the image and record the specifications of the calibration plate. Then, the camera intrinsic parameters, distortion matrix, and / or rotation and displacement vectors of the endoscope camera are calibrated. This can be achieved exemplarily through the calibration function calibrateCamera provided by OpenCV (a cross-platform computer vision library used for camera-related image processing). The camera parameters of the endoscope calibrated here will be used in the distortion calibration process to be introduced later.

[0053] According to the navigation method of the present invention, in the 3D image acquisition step, a preoperative 3D image of the spine (at least a portion thereof), such as a preoperative CT image, is acquired and registered to an intraoperative navigation coordinate system. Intraoperative 3D images, such as those acquired by CBCT, may also be used.

[0054] Next, as shown in Figures 3a-3d, the acquired three-dimensional image is cut based on the position and orientation of the endoscope tip to fuse the endoscope's real-time image with the three-dimensional image of the surrounding area, thereby performing the three-dimensional image cutting and fusion steps described above in the navigation method according to the present invention. In the example shown in Figures 3a and 3c, the end of the endoscope, i.e., the end of the insertion tube, forms an inclined plane tilted relative to the longitudinal axis of the endoscope. The optical axis 6 of the endoscope camera is perpendicular to this inclined plane and has a certain angle with the axial direction of the insertion tube. The direction and position of the cut three-dimensional image vary depending on the position of the endoscope tip and the direction of the optical axis. As shown in Figures 3a and 3c, the direction of the viewing angle of the three-dimensional image (the direction of the arrow in Figures 3a and 3c) is coaxial with the optical axis, i.e., coaxial with the viewing angle of the endoscope. In other words, in the fused image, the three-dimensional image is presented from the viewing angle of the endoscope. In Figures 3a and 3c, the end surface of the cut cylinder is perpendicular to the optical axis 6.

[0055] Preferably, an operator, such as a surgeon, can select the size of the range of the 3D image to be fused as needed. In other words, the range of the cut 3D image can be changed based on the operator's real-time input. For example, a zoom icon 9 (illustrated schematically in Figures 3b, 3d, and 5) can be provided on the display interface of the display device. The operator can operate this zoom icon using an input device, depending on the range of the 3D image they wish to view. Input devices include, but are not limited to, a mouse, keyboard, touch screen, etc. When the operator wishes to view a 3D image of the spinal structure within a larger range around the endoscope (which corresponds to a smaller display scale, thus operating the zoom icon in the zoom direction), the diameter of the cut 3D image volume (preferably a cylinder in the illustrated embodiment) increases, as shown in Figure 3c. The resulting fused image, shown in Figure 3d, reveals more of the 3D image of the spinal structure. When the operator wishes to zoom in to view details of the spinal structure (which corresponds to an increased display scale, thus operating the zoom icon in the zoom direction), the diameter of the cut cylinder decreases, as shown in Figure 3a. The resulting fused image, shown in Figure 3b, allows for a clearer view of the details of the 3D spinal structure. This solution allows the operator to freely zoom in and out of the observed 3D image as needed. This is particularly beneficial for spinal surgery. Surgeons sometimes need to see a larger or more comprehensive 3D image, making it easier for them to determine the endoscope's position and orientation and adjust their procedures. Furthermore, during specific procedures, they need to observe the details of the 3D image beneath the soft tissue in the endoscope's field of view. The zoom function provided by the present invention, which changes the cutting range of the 3D image based on the operator's real-time input, greatly facilitates spinal endoscopic surgery. Those skilled in the art will appreciate that the operator input method for determining the cutting range of the 3D image is not limited to a zoom icon; other methods are also possible, such as providing the operator with several different ranges to choose from or requiring the operator to enter a numerical value to determine the cutting range. This can be achieved, for example, through input components displayed on the display interface, such as the aforementioned zoom icon, a tab for the operator to select different 3D image display ranges, or a dialog box for the operator to enter a numerical value.

[0056] Preferably, the cutting of the three-dimensional image runs through the entire depth of the three-dimensional image, which is particularly suitable for spinal application scenarios because the three-dimensional image of the spine obtained only involves bony structures without interference from other soft tissues. Cutting through the entire depth of the three-dimensional image can more completely present the structure of the relevant spine. In this specific embodiment, as shown in Figures 3a and 3c, the direction of the endoscope end is the optical axis direction 6 of the endoscope camera, and the cutting of the three-dimensional image starts from the position of the focal plane 7 of the endoscope camera, with a diameter d, extending along the optical axis direction 6 and running through the entire depth of the three-dimensional image to cut out a cylinder.

[0057] Preferably, in the fusion step, the size of the endoscope image is scaled along with the three-dimensional image before fusion. Comparing Figures 3a and 3b with Figures 3c and 3d, when the cut range shown in Figure 3c is larger than that in Figure 3a, the range of the fused and displayed three-dimensional image is larger than that in Figure 3b, as shown in Figure 3d. Correspondingly, the display ratio of the three-dimensional image shown in Figure 3d is smaller, and the endoscope image is correspondingly reduced so that the endoscope image has an appropriate proportion in the entire fused image (see the corresponding fused image shown in Figure 3d).

[0058] Because the endoscopic image exhibits distortion within the endoscopic camera's field of view, to better match the endoscopic image with the three-dimensional image and facilitate viewing for the operator, the endoscopic image can be calibrated for distortion, or the three-dimensional image can be distorted to achieve a distortion consistent with the endoscopic image, or both the endoscopic image and the three-dimensional image can be calibrated for distortion to achieve a matching degree of distortion. In this embodiment of the present invention, the distortion calibration step can be performed using any of the three methods described above, depending on operator input, allowing the surgeon to flexibly select based on their preference. In the distortion processing step, the endoscopic image is calibrated using the endoscopic camera parameters calibrated above to remove the camera's distortion. Endoscopic image acquisition can be performed, for example, using a high-definition image capture card, which converts the endoscopic image into a video stream. The calibrated endoscopic image can then be obtained through frame-by-frame distortion calibration. Alternatively, the surgeon may choose not to calibrate the endoscopic image, but instead use the calibrated endoscope camera parameters to distort the cut 3D image so that it has the same distortion effect as the endoscopic image. This approach is more convenient for surgeons accustomed to observing distorted endoscopic images. Alternatively, another approach can be chosen, namely, simultaneously calibrating the endoscopic image and performing distortion processing on the 3D image so that the distortion levels of the two images match.

[0059] Specific embodiments of the navigation method of the present invention also provide the surgeon with two image fusion options. Specifically, during the fusion step, the method of the present invention can select from the following two fusion options based on operator input: The first fusion option involves placing the cut 3D image below the real-time endoscopic image. Figure 4b shows an example of an image fused in this manner. The second fusion option involves placing the real-time endoscopic image onto the 3D image acquired during the image acquisition step in a manner corresponding to the position and orientation of the endoscope tip. Figure 4a shows an example of an image fused in this manner. The first fusion option, also known as endoscopic field of view fusion, maintains the endoscope's native field of view while simultaneously overlaying a 3D image field of view underneath, preserving the surgeon's endoscope usage habits while providing a macroscopic view. The second fusion option, also known as 3D field of view fusion, places the real-time endoscopic image at the endoscope tip's position in the 3D image space, in the endoscope's field of view, based on the endoscope tip's position and viewing angle (i.e., the direction of the endoscope's optical axis) acquired by the navigation system. This second fusion option facilitates the surgeon's overall view. Those skilled in the art will appreciate that images obtained by these two fusion methods may also be provided on a display device simultaneously.

[0060] During the display step, the three-dimensional image located behind the end of the endoscope (i.e., closer to the proximal side relative to the end of the endoscope) can also be blurred and displayed according to the surgeon's choice. This method provides the surgeon with a more intuitive image so that he or she can better understand the direction of the endoscope.

[0061] The present invention also provides an electronic device comprising a display device 3 and a processor. In the specific embodiment shown in FIG2 , the processor is included in the control device 3, i.e., the illustrated host. Those skilled in the art will appreciate that the processor and display device may be integrated or separate. The control device 3 or processor has a data interface. The control device is electrically connected to the endoscope via the data interface to acquire endoscopic images and the position and orientation of the endoscope's distal end in real time. The control device may also include, for example, a data interface for connection to a CT device (e.g., a CBCT device, but those skilled in the art will appreciate that this is not limited to CT devices and may also be connected to a storage device storing three-dimensional image data) to acquire a three-dimensional image of at least a portion of the spine via the data interface. When the processor is running, it executes a computer program (which may be stored in a memory included in the control device or in another memory) to display a fused image (the image on the left in FIG5 ) on the display device 3. This fused image is obtained by fusing the three-dimensional image corresponding to the position and orientation of the endoscope's distal end with the image. The fused three-dimensional image can be an image obtained by cutting the three-dimensional image according to the position and direction of the endoscope tip in the method described above (preferably, the cutting range of the three-dimensional image is expanded or reduced according to the operator's real-time operation of the zoom icon).

[0062] The display interface may have an input component, wherein the range of the three-dimensional image displayed in the fused image is determined based on the input of the operator through the input component. The input component is, for example, a zoom icon, and the range of the three-dimensional image displayed in the fused image is expanded or reduced in real time in response to the operator's operation of the zoom icon. Exemplarily, the operator's input for determining the range of the displayed three-dimensional image can also be implemented, for example, through a component (such as a tab) on the display interface of the display device for the operator to select different three-dimensional image display ranges. It can also be implemented through other components, such as a dialog box for the operator to input a numerical value, such as the diameter of a cylinder.

[0063] As an example, the display interface of the display device 3 may also display a three-dimensional navigation view, for example, a view showing the spatial positional relationship between the endoscope and its field of view relative to the patient's physiological structures on various three-dimensional sections, such as fitted two-dimensional perspective or sagittal, coronal, axial, and sections along the tool's axis. To this end, during the three-dimensional image acquisition step, preoperative CT images or intraoperative CBCT three-dimensional images are acquired for navigation. FIG5 exemplifies some of these views. Each view in the display interface is updated in real time as the position and orientation of the endoscope tip change. In this way, the operator can obtain more comprehensive navigation information. Those skilled in the art will appreciate that the views that can be displayed on the display interface are not limited to the views described herein and the views exemplified in FIG5 ; other sections, orientations, or any other suitable views may be displayed as needed.

[0064] For example, the display interface of the display device also includes a component (not shown) for the operator to select the distortion calibration or distortion processing. This component can take various forms, such as a radio button or a dialog box. This allows the operator to select whether to perform distortion calibration on the endoscopic image, to perform distortion processing on the cut 3D image using the calibrated endoscopic camera parameters described above, or to simultaneously perform distortion calibration on the endoscopic image and distortion processing on the 3D image, so that the distortion levels of the two images match. The display interface of the display device also includes a component (not shown) for the operator to select the image fusion method. This component can take various forms, such as a radio button, a checkbox, or a dialog box. This allows the operator to flexibly select the first image fusion method or the second image fusion method described above, or to simultaneously display images obtained by both fusion methods. Those skilled in the art will appreciate that the steps of the methods or algorithms described herein can be implemented directly using hardware, a software module executed by a processor, or a combination of both. The software module may be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the technical field.

[0065] In the above embodiments, the method can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0066] As described above, in this specific embodiment, the navigation system includes a control device 2 and a display device 3. The control device 2 includes a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the navigation method described above is executed when the processor runs the program, and the display step in the navigation method is executed through the display device 3. It will be understood by those skilled in the art that the memory of the control device of the present invention may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device separate from the processor.

[0067] The processor of the control device can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0068] Although certain embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the general inventive concept, the scope of which is defined in the claims and their equivalents.

Claims

1. A navigation method for spinal endoscopic surgery, characterized in that: The method comprises the following steps: A three-dimensional image acquisition step: acquiring a three-dimensional image of at least a portion of the spine; Image acquisition step: acquiring the image of the endoscope in real time; Position acquisition step: acquiring the position and direction of the endoscope tip in real time; Three-dimensional image cutting step: cutting the three-dimensional image according to the position and direction of the endoscope end; Fusion step: fusing the cut three-dimensional image with the image; and Display step: display the fused image.

2. The method according to claim 1, characterized in that In the three-dimensional image cutting step, a cylindrical shape is cut out of the three-dimensional image, wherein the diameter of the cylinder is determined according to the operator's input, thereby determining the range of cutting the three-dimensional image to obtain a cut three-dimensional image of the range that the operator needs to display.

3. The method according to claim 2, characterized in that The fusion step also includes scaling the size of the endoscopic image corresponding to the three-dimensional images of different cutting ranges before performing the fusion.

4. The method according to claim 1, characterized in that: The direction of the endoscope tip is the optical axis direction of the endoscope camera, wherein the cutting starts from the position of the focal plane of the endoscope camera, extends along the optical axis direction and runs through the entire depth of the three-dimensional image.

5. The method according to any one of claims 1 to 4, characterized in that In the three-dimensional image acquisition step, the preoperative three-dimensional image of at least a portion of the spine is registered with the patient's posture in the navigation coordinate system.

6. The method according to any one of claims 1 to 5, characterized in that Before the fusion step, a distortion processing step is also included, wherein in the distortion processing step, a selection can be made from the following three methods according to the input of the operator: Performing distortion calibration on the acquired endoscopic image to at least partially remove the distortion effect of the endoscopic camera, so that the endoscopic image after distortion calibration matches the cut 3D image; or, The cut three-dimensional image is distorted according to the parameters of the endoscope camera so that the distorted three-dimensional image matches the image acquired by the endoscope in real time; or At the same time, the endoscope image is calibrated for distortion and the three-dimensional image is processed for distortion so that the degree of distortion of the two images matches. And, corresponding to the distortion processing step, in the fusion step: fusing the distortion-calibrated endoscope image with the cut-out three-dimensional image, or, The endoscopic image acquired in real time is fused with the three-dimensional image after distortion processing, or the endoscopic image after distortion calibration is fused with the three-dimensional image after distortion processing.

7. The method according to any one of claims 1 to 6, characterized in that In the fusion step, it is also possible to select from the following two fusion methods according to the operator's input or provide images obtained by the following two fusion methods at the same time: The fusion is performed in such a way that the cut three-dimensional image is placed below the real-time acquisition endoscope image; and The fusion is performed by placing the endoscopic image acquired in real time into the three-dimensional image acquired in the image acquisition step in a manner corresponding to the position and orientation of the endoscope tip.

8. The method according to any one of claims 1 to 7, characterized in that In the display step, the three-dimensional image located behind the end of the endoscope is blurred and displayed.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises an endoscope tip calibration step for calibrating the positional relationship of the endoscope tip relative to a first tracker on the endoscope, wherein the endoscope tip calibration step comprises obtaining an image through a calibration tool instead of through the endoscope.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are executed.

11. A control device, wherein the control device comprises a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1 to 9 are executed.

12. An electronic device for navigation of spinal endoscopic surgery, characterized in that: The electronic device comprises a display device (3) and a processor, wherein the processor has a data interface. wherein the data interface can be connected to an endoscope, so that the processor can acquire the image of the endoscope and the position and direction of the endoscope tip in real time; and, wherein the data interface further enables the processor to acquire a three-dimensional image of at least a portion of the spine; and When the processor is running, a fused image is displayed on the display device (3), wherein the fused image is an image obtained by fusing a three-dimensional image corresponding to the position and direction of the endoscope end with the endoscope image acquired in real time.

13. The electronic device according to claim 12, characterized in that: An input component is provided on the display interface of the display device (3), wherein the range of the three-dimensional image displayed in the fused image is determined according to the input of the operator through the input component.

14. The electronic device according to claim 13, characterized in that: The input component includes a zoom icon for the range of the three-dimensional image, or a component for the operator to select a different three-dimensional image display range, or a component for the operator to input a numerical value related to the three-dimensional image display range.

15. The electronic device according to claim 13 or 14, characterized in that: The size of the endoscope image in the fused image displayed on the display device (3) is scaled in accordance with the range of the displayed three-dimensional image.

16. The electronic device according to any one of claims 12 to 15, characterized in that: The endoscope image in the fused image displayed on the display device (3) is a distortion-calibrated image, wherein the distortion calibration at least partially removes the distortion effect of the endoscope camera so that the distortion-calibrated endoscope image matches the three-dimensional image; or The three-dimensional image in the fused image displayed on the display device (3) is a three-dimensional image after distortion processing, wherein the distortion processing is performed according to the parameters of the endoscope camera so that the three-dimensional image after distortion processing matches the endoscope image acquired in real time; or, The endoscopic image in the fused image displayed on the display device (3) is a distortion-calibrated image and the three-dimensional image is a distortion-processed three-dimensional image, and the distortion degrees of the two images match.

17. The electronic device according to claim 16, characterized in that: The display device (3) is also provided with a component for an operator to operate so as to select to perform the distortion calibration or the distortion processing or both at the same time.

18. The electronic device according to any one of claims 12 to 17, characterized in that: In the fused image displayed on the display device (3), the three-dimensional image located behind the end of the endoscope is blurred.

19. The electronic device according to any one of claims 12 to 18, characterized in that: The display interface of the display device also displays one or more of the views on the fitted two-dimensional perspective section, the sagittal section view, the coronal section view, and the axial section view obtained based on the acquired three-dimensional image, in which the spatial position relationship of the endoscope and its field of view angle relative to the patient's physiological structure is displayed.

20. A navigation system for spinal endoscopic surgery, characterized in that: The navigation system comprises: A tracking device (1), the tracking device (1) being used to track a first tracker (4) arranged on an endoscope and a second tracker (5) arranged at a target site of a patient; A display device (3) and a processor, wherein the processor is connected to the tracking device (1) and the endoscope, wherein the method of any one of claims 1 to 9 is executed when the processor is running, and the display in the method is realized through the display device (3).

21. The navigation system according to claim 20, characterized in that The navigation system also includes one or more of the endoscope, the first tracker, and the second tracker.

22. The navigation system according to claim 20 or 21, characterized in that: It also includes a calibration tool for calibrating the position of the end of the endoscope relative to the first tracker, wherein a plurality of calibration holes are formed on the calibration tool, and the plurality of calibration holes have bottom surfaces with different inclination angles and / or different apertures to respectively calibrate endoscopes with different end bevel inclination angles and / or barrel diameters.

23. A navigation system for spinal endoscopic surgery, characterized in that: The navigation system comprises: A tracking device (1), the tracking device (1) being used to track a first tracker (4) disposed on an endoscope and a second tracker (5) disposed at a target site of a patient; and An electronic device according to any one of claims 12-19, wherein a processor in the electronic device is connected to the tracking device (1).

24. The navigation system according to claim 23, characterized in that The navigation system also includes one or more of the endoscope, the first tracker, and the second tracker.

25. The navigation system according to claim 23 or 24, characterized in that: It also includes a calibration tool for calibrating the position of the end of the endoscope relative to the first tracker, wherein a plurality of calibration holes are formed on the calibration tool, and the plurality of calibration holes have bottom surfaces with different inclination angles and / or different apertures to respectively calibrate endoscopes with different end bevel inclination angles and / or barrel diameters.

Images