Navigation method for optical rigid endoscope surgery, electronic device, navigation system, and robot system

Abstract

The present invention relates to a navigation method for an optical rigid endoscope surgery, a corresponding computer-readable storage medium, a control apparatus, a computer program product, and an electronic device, navigation system, and robot system for navigation of the optical rigid endoscope surgery. The navigation method comprises: an image acquisition step of acquiring a plurality of images of an optical rigid endoscope; an imaging orientation acquisition step of acquiring a position and a direction of an imaging apparatus of the optical rigid endoscope corresponding to each image of the optical rigid endoscope; a splicing step of splicing the plurality of images according to the orientation of the imaging apparatus corresponding to each image to obtain a spliced image; and a display step of displaying the spliced image. According to the present invention, by forming the spliced image, a global visual field is provided for an operator, thereby solving the problem of the limited visual field of the operator. The present invention is particularly beneficial to the optical rigid endoscope surgery in the endoscopic surgery, the calculation required by the splicing mode is less, the splicing speed and the splicing precision are higher, and the requirement for the operation capability of a processor is lower.

Description

Navigation method, electronic device, navigation system and robotic system for optical endoscopic surgery Technical Field

[0001] The present invention relates to the field of medical equipment technology, and in particular to a surgical navigation system, and more specifically to a navigation method, a navigation system, and a robotic system for optical endoscopic surgery. Background Art

[0002] In traditional endoscopic surgeries, such as optical rigid endoscope surgeries, the surgeon's field of view is solely used for surgical observation, tool manipulation, and control. However, the surgeon's field of view is limited to the narrow field of view of the endoscope, and the surgeon cannot see areas outside the field of view. This can result in the endoscope not reaching the affected area correctly or insufficient decompression, leading to failure to achieve the intended surgical goal.

[0003] Especially for spinal endoscopes, due to their special way of use, that is, the spinal endoscope is used in a working channel sleeve, the doctor's field of view under the microscope is further limited. This can be shown in Figures 1A and 1B, where the working channel sleeve 51 opens a channel for the spinal endoscope 52, but it also blocks the field of view of the spinal endoscope 52. When the doctor pulls up the spinal endoscope according to common thinking, away from the observation target point, hoping to observe a larger range of physiological structures, the first thing he sees is the tube wall of the working channel sleeve 51. If the doctor also pulls up the working channel sleeve 51, the soft tissue on the outside of the working channel sleeve will shrink inward and invade the channel. What the doctor sees will be the soft tissue on the outside of the working channel sleeve 51, blocking the structure of the target point that the doctor wants to observe, so the purpose of observing a larger range of physiological structures is still not achieved.

[0004] To overcome this clinical challenge, the distal end of some endoscopes is designed to have an angled bevel (for example, an inclination angle of 15 degrees, 30 degrees, 45 degrees, etc.), and the lens of the imaging device is mounted on the angled bevel and arranged eccentrically relative to the axis of the endoscope so that a larger range can be observed by rotating the endoscope around the axis.

[0005] During actual surgical operations, doctors need to frequently rotate the endoscope around its axis in order to observe various areas, which significantly reduces the efficiency of the surgical operation. However, this still means that doctors can only see local physiological structures in a certain direction at a given moment.

[0006] In this case, endoscopic surgeons, especially spinal endoscopists, need superior physiological and anatomical knowledge, excellent spatial imagination, memory ability and more surgical case accumulation to overcome the so-called "learning curve" and master endoscopic technology, which also limits the promotion and popularization of endoscopic surgery.

[0007] Summary of the Invention

[0008] The object of the present invention is to solve at least one of the above problems and defects in the prior art as well as other technical problems.

[0009] In one aspect, the present invention provides a navigation method for optical hard endoscope surgery, the method comprising the following steps: an image acquisition step: acquiring a plurality of images of the optical hard endoscope;

[0010] Imaging orientation acquisition step: acquiring a position and / or orientation of an imaging device of the optical hard mirror corresponding to at least one of the plurality of images of the optical hard mirror under a navigation system;

[0011] a stitching step of stitching a plurality of images of the optical hard mirror according to the position and / or direction of the imaging device corresponding to at least one image to obtain a stitched image; and

[0012] Displaying step: displaying at least a portion of the stitched image.

[0013] The stitched images formed in this example provide the operator with a wider field of view (e.g., a global field of view) around the end of the optical rigid scope, successfully resolving the problem of the operator's limited field of view. The navigation method of the present invention is particularly beneficial for endoscope surgery using optical rigid scopes, because, compared to soft scopes whose ends can be bent appropriately, optical rigid scopes are generally limited in their observation range due to the inflexibility of their insertion tubes. This solution addresses this problem, as well as the resulting loss of sense of direction and reduced surgical efficiency caused by the need to frequently rotate the optical rigid scope, thereby improving the operational accuracy and efficiency of the optical rigid scope and reducing its learning difficulty. Furthermore, in this solution, since the orientation of the imaging device is obtained while the image is being acquired and the images are stitched together based on the orientation of the imaging device corresponding to at least one image to obtain a stitched image, compared to various existing stitching methods (e.g., algorithm-based), less computation is required, greatly improving the stitching speed, which is particularly important for real-time observation during surgery. In addition, this solution has higher stitching accuracy and lower requirements on the processor's computing power. It should be noted that in the imaging orientation acquisition step, the position and / or orientation of the imaging device under the navigation system corresponding to at least one of the stitched images is acquired. For example, the orientation of the imaging device under the navigation system may be acquired for each image, but it is also possible to acquire the position and / or orientation of the imaging device for only a portion of the images, or even only one image. For example, in some cases, it is only necessary to acquire the position and / or orientation of the imaging device corresponding to certain images at intervals in time, or only to acquire the position and / or orientation of the imaging device at the beginning of acquisition of the images to be stitched.

[0014] According to one example, the navigation method further includes an image fusion step between the stitching step and the display step, wherein in the image fusion step, the stitched image is fused with the current image of the optical hard lens according to the current position and / or orientation of the imaging device, and the fused image is displayed in the display step.

[0015] The solution in this example enables the operator to more conveniently observe the current image and the stitched image at the same time.

[0016] According to one example, the navigation method further includes a processing step between the stitching step and the image fusion step, wherein a planar image is generated based on the images obtained in the stitching step, and the planar image is fused with the current image of the optical hard lens as the stitched image in the image fusion step. The processing step in this example eliminates or reduces visual distortion of the stitched image caused by different viewing angles.

[0017] According to one example, the navigation method further includes: a marker orientation acquisition step, in which the position and / or orientation of the marker on the image of the patient's physiological structure under the navigation system is acquired; and a marker fusion step between the stitching step and the display step, wherein the marker and / or guidance instructions related to the marker are fused to the stitched image based on the position and / or orientation of the marker under the navigation system.

[0018] According to one example, the navigation method further includes: a marker orientation acquisition step, in which the position and / or orientation of the marker on the image of the patient's physiological structure under the navigation system is acquired; and a marker fusion step between the image fusion step and the display step, wherein the marker and / or guidance instructions related to the marker are fused to the fused image obtained in the image fusion step based on the position and / or orientation of the marker under the navigation system.

[0019] According to one example, the mark on the image of the patient's physiological structure includes one or more of a direction mark and a physiological structure mark. In this example, by setting the direction mark, it can be observed on the display window that the direction mark on the global image fused with the direction mark always indicates a predetermined direction, such as pointing to the patient's head, tail, ventral or dorsal side, providing an orientation indication for the operator. The direction mark may include one or more of the direction marks facing the patient's dorsal, ventral, head and tail sides. Preferably, the direction mark is multiple, for example, four direction marks pointing to the patient's dorsal, ventral, head and tail sides, respectively.

[0020] According to the above example, the markers may include physiological structure markers. Since the markers in this example are set at the patient's physiological structure, during the surgical operation, these marker points displayed on the spliced ​​image can help the doctor identify specific position points of the patient's physiological structure even when covered by soft tissue, such as the pedicles of the anterior vertebra, etc., to better help the doctor determine the endoscopic position. The physiological structure markers include physiological structure markers that do not shift during optical hard endoscope surgery. This method can ensure the accuracy and reliability of directional guidance. For example, the patient's physiological structure is a spine, and the physiological structure markers include one or more of the ventral side of the articular process, the pedicles of the anterior vertebra, the pedicles of the posterior vertebra, and the intervertebral disc. Preferably, the guidance indication is multiple, for example, four indicator symbols pointing to the ventral side of the articular process, the pedicles of the anterior vertebra, the pedicles of the posterior vertebra, and the intervertebral disc respectively.

[0021] Since the marking may include one or both of the direction marking and the physiological structure marking, this provides the operator with a variety of marking options, allowing the operator to flexibly select according to needs.

[0022] According to one example, the image of the patient's physiological structure includes a preoperative three-dimensional image, an intraoperative three-dimensional image, or an intraoperative two-dimensional image.

[0023] According to one example, the imaging position acquisition step is performed by acquiring the position of a tracer located outside the patient's body and having a fixed positional relationship relative to the optical hard mirror from a tracking device of a navigation system.

[0024] The method conveniently realizes the position tracking of the imaging device of the optical hard mirror by means of the surgical navigation system.

[0025] According to an example, the start of the navigation method can be triggered by an operator, and then the steps included in the navigation method are automatically performed.

[0026] This solution enables the operator to start displaying the stitched image at any time according to his / her preference or need, making his / her operation more convenient.

[0027] According to an example, the navigation method is automatically started and the steps included in the navigation method are performed in real time.

[0028] According to an example, the implementation of the navigation method can be stopped after receiving a stop input from an operator.

[0029] According to one example, the navigation method further includes a step of indicating the current imaging device field of view position in the stitched image. In this step, an area corresponding to the current imaging device field of view position in the stitched image is determined with the aid of the position and orientation of the imaging device, and a mark indicating the area is generated and displayed in the display step.

[0030] According to one example, the optical rigid endoscope is a spinal endoscope.

[0031] The navigation method of the present invention is particularly beneficial for spinal endoscopes (such as interlaminar foraminal endoscopes). As introduced in the background technology, the spinal endoscope is used in the working channel sleeve, which further limits the doctor's field of view under the microscope. How to allow the doctor to not be affected by the special use of the spinal endoscope (the endoscope is used in the working channel sleeve) and the limited viewing angle of the optical hard endoscope module during spinal endoscopic surgery, and to quickly obtain a "complete map" of the target physiological structure and its surrounding soft tissues, blood vessels and nerves under the microscope and guide the efficient operation of the endoscopic tools only by relying on the spinal endoscope image, has not yet been a convenient and reliable solution. The navigation method of the present invention obtains the spliced ​​image of the end of the spinal endoscope, which solves the problem of its limited field of view, can largely avoid the doctor's misoperation and increase the reliability of spinal surgery. At the same time, because of the existence of a larger range of spliced ​​images (which can also be panoramic images), the frequency of the doctor frequently rotating the endoscope around the axis to observe various directions is reduced, which also improves the efficiency of spinal endoscopic surgery to a certain extent.

[0032] According to one example, the method further includes a calibration step for calibrating the positional relationship of the imaging device of the optical hard scope relative to the tracer, wherein the calibration step is performed by acquiring images using a calibration tool rather than the optical hard scope. This calibration method can avoid image processing.

[0033] According to one example, a distortion processing step is included before the stitching step, wherein the distortion of the optical rigid lens images to be stitched is calibrated in the distortion processing step. This example removes the distortion effect of the optical rigid lens through distortion calibration, making the stitched image closer to the real scene and easier for the operator to observe.

[0034] According to one example, the navigation method can be combined with both 3D image-based navigation and 2D image-based navigation. For both, the reconstruction of the stitched image and the fusion display effect with the current image are the same, which also makes the navigation method of the present invention have a wider range of applications.

[0035] According to an example, on the fused image, the current image is located in the central area of ​​the stitched image, which is more convenient for the operator to observe the current image.

[0036] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the navigation method in the above examples are executed.

[0037] According to another aspect of the present invention, a control device is provided. The control device includes a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the methods in the above examples are executed when the processor runs the program.

[0038] According to yet another aspect of the present invention, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps of the methods in the above examples are implemented.

[0039] According to another aspect of the present invention, there is provided an electronic device for navigation of optical endoscopic surgery, characterized in that the electronic device comprises a display device and a processor, the processor having a data interface, wherein the data interface is connectable to a tracking device of a navigation system and an optical endoscopic lens, so that the processor can obtain multiple images of the optical endoscopic lens and can obtain the position and / or direction of the imaging device of the optical endoscopic lens corresponding to at least one of the multiple images of the optical endoscopic lens under the navigation system; and wherein the processor is configured to display at least a portion of a stitched image on the display device (3) for at least a period of time when the processor is running, the stitched image being stitched together from multiple images of the optical endoscopic lens.

[0040] According to another aspect of the present invention, a navigation system for optical rigid endoscope surgery is provided. The navigation system includes a tracking device, a display device, and a processor. The tracking device is adapted to track a tracer disposed on an optical rigid endoscope. The processor is adapted to be connected to the tracking device and the optical rigid endoscope. When the processor is executed, the navigation method described in each of the above examples is executed, and the display of the navigation method is achieved via the display device.

[0041] According to another aspect of the present invention, a navigation system for optical rigid endoscope surgery is provided. The navigation system includes a tracking device and the electronic device described above. The tracking device is adapted to track a tracer disposed on an optical rigid endoscope; and a processor in the electronic device is adapted to be connected to the tracking device and the optical rigid endoscope.

[0042] According to another aspect of the present invention, a robot system is provided, which includes a robot arm and the above-mentioned navigation system. In other words, the concept of the present invention can also be implemented in the navigation system of the robot system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention is described in detail below by way of exemplary embodiments with reference to the accompanying drawings.

[0044] Figure 1A is a schematic diagram showing the field of view of a spinal endoscope in one orientation, and Figure 1B is a schematic diagram showing the field of view of the spinal endoscope shown in Figure 1A in another orientation. Both Figures 1A and 1B show a working channel sleeve.

[0045] FIG2 exemplarily shows a flow chart of a navigation method for optical endoscopic surgery according to the present invention.

[0046] FIG3 shows a schematic diagram of the principle of a navigation system for spinal endoscopic surgery according to an exemplary embodiment of the present invention.

[0047] FIG4 exemplarily shows a schematic diagram of moving an optical hard mirror to obtain multiple images of a larger range.

[0048] FIG5 and FIG6 are schematic diagrams showing the principle of rotating the optical hard mirror to obtain a stitched image for an optical hard mirror with an end bevel angle of 45 degrees and 15 degrees, respectively.

[0049] FIG. 7 schematically shows a schematic diagram of fusing a real-time image of an endoscope, ie, a current image, into a stitched image.

[0050] FIG. 8 is a schematic diagram showing the image of FIG. 7 being displayed on a window of a display device.

[0051] FIG9 shows a schematic diagram of the fusion of direction markers on the stitched image.

[0052] FIG10 shows a schematic diagram of integrating physiological structure markers into a stitched image.

[0053] FIG. 11 shows an example of an operator marking a direction on a preoperative 3D image or an intraoperative 3D image.

[0054] It should be noted that the drawings are schematic only. They illustrate only those components or steps necessary to illustrate the present invention, and other components or steps may be omitted or only briefly mentioned. In addition to the components or steps shown in the drawings, the present invention may also include other components or steps. DETAILED DESCRIPTION

[0055] The following examples and accompanying drawings further illustrate the technical solution of the present invention. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall concept of the present invention and should not be construed as limiting the present invention.

[0056] As a specific example, the following describes the specific steps of the method of the present invention for navigating an optical rigid endoscope (e.g., a spinal endoscope, a neuroendoscope, or a nasal endoscope) using images from an optical rigid endoscope under a navigation system, as well as the navigation system, electronic equipment, and robotic system (or positioning and navigation system) involved. In the detailed description that follows, many specific details and steps are set forth in a highly specific and detailed manner to provide a comprehensive understanding of the embodiments. However, it should be understood that one or more other embodiments may be implemented without these specific details and steps.

[0057] It should be noted that while this document uses the terms "imaging device" and "image" of an optical rigid scope, those skilled in the art will understand that "imaging device" is a broad concept encompassing functions such as video recording, video capture, and image capture, and "image" is a broad concept encompassing video, dynamic continuous images, and static images. In this document, the "imaging device" of an optical rigid scope may be the optical rigid scope module used for video imaging of the optical rigid scope. The image acquisition step of the present invention captures a framed image of the image from the optical rigid scope.

[0058] In a specific embodiment of the present invention, the navigation system used, as shown in FIG3 , includes a tracking device 1, a control device 2, and a display device 3. The tracking device 1 can be an optical tracking device (e.g., an NDI navigator), and accordingly, a tracer 4 can be provided on an endoscope 5. As a specific example, the control device 2 can be a general-purpose computer, a dedicated computer, an embedded processor, or any other suitable programmable data processing device, such as a single-chip microcomputer or a chip. The control device 2 can include a processor and a memory for storing programs, but it can also include only a processor, in which case the processor can be attached to the memory storing the programs. In other words, the control device includes at least a processor. The control device 2 (or processor) and the display device 3 can be integrated or provided separately. The control device or processor has a data interface, which can include a data interface that can be connected to an optical hard mirror, allowing the control device / processor to obtain images of the optical hard mirror in real time. The control device or processor also includes a data interface that can be connected to the tracking device 1 of the navigation system, so that the position and orientation of the tracked target, such as the tracer 4 on the optical hard mirror, can be obtained from the tracking device 1 in real time. As an example, the endoscope 5 and / or the tracer 4 can also be considered as part of the navigation system of the present invention.

[0059] The optical hard endoscope herein may refer to an endoscope whose insertion tube is inflexible (for example, a tube made of metal). Usually, the end of the optical hard endoscope enters the patient's tissue structure or bone structure to observe and / or operate it, and the proximal end of the optical hard endoscope (the end close to the operator, that is, the end opposite to the end of the insertion tube of the optical hard endoscope) is located outside the patient's body for the operator to manipulate. In the navigation system, by providing a tracer suitable for being tracked by the tracking device 1 at the proximal end of the optical hard endoscope located outside the patient's body, the navigation system can obtain the position and direction of the tracer 4 on the optical hard endoscope 5 in the navigation coordinate system in real time. In the present invention, the imaging device of the optical hard endoscope, that is, the distal lens of the hard endoscope module, is provided at the end of the insertion tube of the optical hard endoscope. By calibrating the relative position of the end of the scope's insertion tube relative to the tracer 4 on the optical scope 5, the relative position of the scope's module relative to the tracer 4 can be determined. This allows the tracking device 1 in the navigation system to determine the orientation of the tracer 4 and, consequently, the position and orientation of the scope's module within the navigation coordinate system. This calibration, also known as extrinsic calibration of the optical scope, is typically performed prior to performing optical scope surgery and navigation.

[0060] Specifically, before performing optical hard endoscope surgery and its navigation, the relative position relationship of the imaging device of the optical hard endoscope, i.e., the hard endoscope module, relative to the tracer 4 can be calibrated first. Preferably, in this embodiment, the calibration is performed by a calibration tool (not shown in the figure) without the need to obtain images with the help of an optical hard endoscope, which reduces the workload of image processing. The navigation system of the present invention optionally includes the calibration tool. A plurality of calibration holes can be formed on the calibration tool, and these calibration holes have bottom surfaces with different inclination angles and / or different apertures to calibrate optical hard endoscopes with different end bevel angles and / or barrel diameters. Specifically, the calibration tool also includes another tracer fixed on its bracket, and the navigation system can know the position of the tracer on the calibration tool in the navigation coordinate system. Moreover, the positional relationship of each calibration hole relative to the tracer on the calibration tool can be known based on the design size of the calibration tool, and thus the positions of these calibration holes in the navigation system are known. The end of the insertion tube of the optical rigid scope to be calibrated is inserted into the matching calibration hole. The inclined surface of the end of the insertion tube aligns with the inclined bottom surface of the calibration hole, thereby achieving the positioning of the optical rigid scope. Because the navigation system also knows the position of the tracer 4 on the optical rigid scope in the navigation coordinate system, it can also determine the relative position of the end of the endoscope's insertion tube with respect to the tracer 4 on the endoscope, completing the calibration process.

[0061] By using the tracking device 1 to obtain the position of the tracer 4 on the optical hard endoscope in real time during surgery, the position and orientation of the optical hard endoscope tip, that is, the position and orientation of the hard endoscope module, can be indirectly obtained in real time. The imaging orientation acquisition step in the navigation method according to the present invention is performed in this manner.

[0062] In this specific embodiment of the present invention, prior to performing optical endoscopic surgery and navigation, the imaging parameters of the endoscopic scope, also known as internal parameters, can be calibrated. For example, a calibration plate can be used to calibrate the internal parameters. For example, the endoscopic scope can be used to photograph the calibration plate at different azimuth angles, saving the images and recording the plate specifications. The internal parameters of the endoscopic scope can then be calibrated, including the distortion matrix and / or rotation and displacement vectors. These calibrated internal parameters will be used in the distortion calibration process described later.

[0063] The following will describe in detail the steps of an embodiment of the navigation method for optical endoscopic surgery of the present invention in conjunction with Figure 2. Before the operation begins, the internal and external parameters of the optical endoscopic surgery can be calibrated as described above. And before the operation and navigation begin, the navigation system is aligned, that is, the navigation coordinate system is determined. The navigation method of the present invention can be used in scenarios where two-dimensional images are used for navigation, and can also be used in scenarios where three-dimensional images are used for navigation. The alignment method of the navigation system is a known method and will not be described in detail here.

[0064] The activation of the navigation method of the present invention can be triggered by the operator, so that the operator (doctor) can choose whether to use the navigation function provided by the navigation method according to his or her preferences or needs, or choose when to use the function, for example, when the operator wants to observe the global image around the end of the endoscope, the method is activated. The operator's triggering instruction can be input through various input components such as a keyboard, a mouse, a handle, a foot switch, etc. The navigation method can also be performed automatically. For example, during the surgical operation, the processor executes the various steps in the navigation method in real time and displays the acquired images in real time. Compared with the automatic activation method, the method of manually triggering the activation of the navigation method by the operator processes a smaller amount of data and requires less information processing capability of the processor.

[0065] After the navigation method is manually initiated by an operator, for example, the operator can rotate and / or move the endoscope to acquire images around the distal end of the optical hard endoscope. Specifically, during this process, the control device automatically acquires multiple images of the optical hard endoscope, i.e., the image acquisition step in FIG2 ; simultaneously, the control device automatically acquires and records the position and orientation of the imaging device of the optical hard endoscope corresponding to at least one image of the optical hard endoscope (e.g., corresponding to each image), i.e., the imaging orientation acquisition step in FIG2 . In this step, the processor acquires the position of the tracer 4 located outside the patient's body and disposed on the distal end of the optical hard endoscope 5 from the tracking device 1 of the navigation system, and indirectly acquires the position and orientation of the imaging device, i.e., the hard endoscope module, in combination with the external parameters calibrated above. The above steps are repeated at multiple positions and orientations of the optical rigid scope. For example, after translating the endoscope over a larger range of observation (as shown in FIG. 4 ) and / or rotating it (as shown in FIG. 5 and FIG. 6 ), images of the imaging device at multiple orientations are obtained. Simultaneously or subsequently, distortion correction can be performed on the multiple optical rigid scope images, incorporating the previously calibrated imaging device internal parameters, to remove the distortion effects of the imaging device. Subsequently, the images are stitched together based on the position and orientation (i.e., orientation) of the optical rigid scope corresponding to at least one image, thereby obtaining a stitched image of the patient's physiological structure. This stitched image can then be displayed on a display device.

[0066] The spliced ​​image can also be called a "complete three-dimensional map" or a "panoramic image", but it should be noted that the "complete" or "panoramic" here only refers to an image of a larger range relative to the limited field of view of the optical rigid endoscope, and does not necessarily mean a 360-degree panoramic image. This can be obtained, for example, by the doctor's flexible choice of translating (for example, as shown in FIG4 ) and / or rotating (for example, as shown in FIG5 and FIG6 ) the optical rigid endoscope within a larger range that needs to be observed according to his observation needs during the operation. The spliced ​​image around the end of the optical rigid endoscope enables the doctor to observe a larger field of view (which can also be a global field of view), thereby guiding the doctor to operate the optical rigid endoscope, quickly determine the direction of the endoscope and the spatial position relationship of key physiological structures relative to the endoscopic image at the moment, and eliminate the defect of the limited field of view of the optical rigid endoscope.

[0067] For the example of rotating the optical hard lens 360 degrees to obtain a stitched image, the overall shape of the stitched image can be a shape similar to a spherical cap with the focal length of the optical hard lens module as the radius. FIG5 exemplarily shows a schematic diagram of three exemplary positions of an optical hard lens with an inclined surface angle of 45 degrees during the rotation process, wherein the conical field of view 53 of the optical hard lens at the three positions is shown. FIG6 exemplarily shows a schematic diagram of three exemplary positions of an optical hard lens with an inclined surface angle of 15 degrees during the rotation process, wherein the conical field of view 53 of the optical hard lens at the three positions is shown. In FIG5 and FIG6, the field of view angle of the exemplarily shown conical field of view is 120 degrees. In the example of FIG5, the shape of the spherical cap formed by the rotation of the optical hard lens is a spherical cap that is larger than a hemisphere, and in the example of FIG6, the shape of the spherical cap formed by the rotation of the optical hard lens is a spherical cap that is smaller than a hemisphere.

[0068] Preferably, the current image of the optical hard lens can also be fused with the stitched image for display, i.e., an image fusion step. For example, as shown in FIG7 , the current image 6 is fused to the center area of ​​the stitched image 7. In this case, the displayed image is the fused image, so that the operator can simultaneously view the current image and the surrounding stitched image. FIG8 shows a schematic diagram of displaying the fused image in FIG7 on a window of a display device. A portion of the fused image in FIG7 is displayed in the window (surrounding square frame), with the current real-time image 6 located in the center area.

[0069] When fusing the stitched image with the current real-time image from the optical rigid lens, a processing step may be included between the stitching step and the image fusion step. In this processing step, a planar image is generated based on the image obtained in the stitching step to reduce visual distortion. This planar image is then used as the stitched image and fused with the current image from the optical rigid lens in the image fusion step (e.g., placed around the current image). This method eliminates or reduces distortion in the stitched image caused by different viewing angles.

[0070] Preferably, as exemplarily shown in Figures 9 and 10, markers (e.g., direction markers or physiological structure markers) can also be fused onto the stitched image, thereby providing orientation guidance indicators, such as direction guidance indicators, physiological structure point guidance indicators, etc., on the displayed stitched image (or the image after the stitched image is fused with the current image). Therefore, the operator can rely on the displayed image to quickly obtain the global orientation and determine the orientation of the physiological structure at the moment. For example, Figure 9 shows a schematic diagram of a stitched image fused with direction markers, and Figure 10 shows a schematic diagram of a stitched image fused with physiological structure marker points.

[0071] Specifically, the navigation method of the present invention may further include a marker orientation acquisition step, in which the position and / or orientation of the marker on the image of the patient's physiological structure under the navigation system is acquired. The image of the patient's physiological structure described herein may be a preoperative three-dimensional image, such as a preoperative CT image, or an image obtained during surgery, such as an intraoperative CBCT image or an intraoperative two-dimensional fluoroscopic image. When the navigation system uses the preoperative CT image for navigation, the operator makes a mark on the preoperative three-dimensional image. When the navigation system uses the intraoperative CBCT image or the intraoperative two-dimensional fluoroscopic image for navigation, the operator makes a mark on the image after acquiring it during surgery.

[0072] In an example of the present invention, the operator can choose to make various markings, such as directional markings or physiological structure point markings, thereby providing the operator with a variety of options and possibilities. The operator can mark directions on the preoperative or intraoperative 3D image, such as arrows pointing toward the patient's dorsal, ventral, cranial, and caudal sides. As an example only, Figure 11 shows a schematic diagram of the operator marking directions on a preoperative 3D image or an intraoperative 3D image. The operator can also mark directions on an intraoperative 2D image.

[0073] The operator can also mark on the three-dimensional image or two-dimensional image (for example, using preoperative planning software) certain physiological structure marker points within or around the intervertebral foramen that do not undergo any displacement during the entire spinal endoscopic surgery, including but not limited to the ventral side of the articular process, the pedicle of the anterior vertebra, the pedicle of the posterior vertebra, the intervertebral disc and other physiological structure points.

[0074] The marker position acquisition step described above can be performed, for example, as follows. If the operator has made marks on the preoperative 3D image, the processor first acquires the preoperative 3D image and simultaneously acquires the marks (e.g., point marks or direction marks) made by the operator on the preoperative 3D image, i.e., the image and marker acquisition step. This is followed by an image registration step, where the preoperative 3D image is registered with the navigation system. The coordinates of each mark in the navigation system are then determined based on the registration relationship determined in this registration step, thereby completing the marker position acquisition step described above.

[0075] When the operator marks the intraoperative three-dimensional image or the intraoperative two-dimensional image, the three-dimensional image or the two-dimensional image used for navigation is generated during the operation. Usually, the registration of the intraoperative three-dimensional image or the intraoperative two-dimensional image to the navigation system is completed at the same time as the intraoperative three-dimensional image or the intraoperative two-dimensional image is acquired. Therefore, in this case, the mark acquisition step is performed after the image acquisition and registration steps, and the operator marks the intraoperative three-dimensional image or the intraoperative two-dimensional image after the registration is completed. Since the coordinates of the registered image under the navigation system are known, the coordinates or positions of the marks made by the operator in the image under the navigation system can be obtained accordingly, thereby completing the above-mentioned mark orientation acquisition step.

[0076] It should be noted that, although the description herein uses the method of an operator making marks on an image of a patient's physiological structure to form the marks, it is also understood that in some other embodiments, the marks may not be made by the operator, for example, the marks may be automatically formed when the image of the patient's physiological structure is formed. This can be achieved, for example, by pre-positioning a marker on a certain part of the patient's body or on a location such as an operating table so that the marks are generated when the image is acquired.

[0077] After obtaining the position and / or orientation of the marker on the image of the patient's physiological structure under the navigation system, i.e., the marker orientation acquisition step, based on the obtained position and / or orientation of the marker under the navigation system, and because any point in the stitched image has a determined position and / or orientation under the navigation system, in other words, the relative position of the marker and the stitched image can be determined under the navigation system, the marker 10 and / or a guide indicator related to the marker (e.g., an arrow pointing in a certain direction or to a certain marked point) can be fused to the stitched image described above. Alternatively, the marker and / or the guide indicator related to the marker can also be fused to the fused image obtained in the above-mentioned image fusion step.

[0078] Those skilled in the art will appreciate that the above-mentioned marker position acquisition step may be performed simultaneously with or before the image acquisition step, imaging position acquisition step, and stitching step (and possible image fusion step) described above.

[0079] Figures 9 and 10 also exemplarily illustrate the display of a marker 9 on the stitched image, indicating the area corresponding to the current imaging device's field of view. Specifically, the navigation method of the present invention further includes a step of indicating the current imaging device's field of view in the stitched image. In this step, the area corresponding to the current imaging device's field of view in the stitched image is determined using the imaging device's current position and orientation, and a marker 9 indicating this area is generated and displayed in the display step. The dashed line 11 in Figures 9 and 10 represents the area displayed in the display device's window.

[0080] It should be noted that, as mentioned above, the navigation method and the navigation function it provides can be started without being triggered by the operator, but the steps in the navigation method can be automatically and in real time executed by the control device. That is, during the operation, the control device automatically records all real-time optical hard mirror images and the corresponding positions and orientations of the optical hard mirror in the navigation system, performs distortion calibration on the acquired images, performs real-time stitching and reconstruction, and updates the stitched image (or the image after the stitched image is fused with the current image and / or the marker).

[0081] Preferably, whether the navigation method of the present invention is initiated by an operator or automatically executed by the system, the implementation of the navigation method and the display of the navigation image provided by the navigation method can be stopped upon receiving a stop instruction from the operator, thereby allowing the operator to choose when to stop the navigation function provided by the navigation method according to their preference. The stop instruction can be provided using the input means described above, or another input means can be used.

[0082] It should also be noted that the steps of "displaying the stitched image" or "displaying the fused image" described herein do not necessarily mean that the image is always displayed. For example, the stitched image / fused image can be displayed only for a certain period of time based on the operator's needs. For example, when the operator desires to observe the entire area around the end of the optical microscope, the operator can trigger the display of the stitched image using a foot switch, for example.

[0083] The present invention also provides an electronic device for navigation of optical endoscopic surgery, which includes a display device 3 and a processor as described above. In the specific example shown in Figure 3, the processor is included in the control device 2, i.e., the host shown in the figure. It will be understood by those skilled in the art that the processor and the display device can be integrated or split. The control device 2 or the processor has a data interface. The control device or the processor is electrically connected to the tracking device 1 and the optical endoscopic 5 of the navigation system through the data interface to obtain the position and direction of the imaging device of the optical endoscopic and to obtain an image of the optical endoscopic in the corresponding position and direction. The data interface of the processor also enables the processor to obtain images of the patient's physiological structure, and the processor is configured to be able to obtain marks made by the operator on the images of the patient's physiological structure. When the processor is running, a computer program is executed (the computer program can be stored in a memory included in the control device or in other memories) to execute the navigation method of the present invention and display a stitched image formed by stitching images of the optical hard lens obtained at multiple positions and directions on the display device 3 for at least a period of time, or according to the preferred scheme described above, display an image after the stitched image is fused with the current image of the optical hard lens, and / or the stitched image described above that is fused with direction marks / physiological structure marks or guidance instructions related to these marks, or an image after the plane image processed by the stitched image is fused with the current image of the optical hard lens.

[0084] As an example, one or more of the following images or views can be displayed on the display interface of the display device 3: a stitched image, an image fused with the stitched image and the current image of the optical hard endoscope, an image fused with the plane image corresponding to the stitched image and the current image of the optical hard endoscope, an image obtained by fusing the direction mark with the stitched image, an image obtained by fusing the physiological structure mark with the stitched image, navigation views such as the view on the fitted two-dimensional perspective section, the sagittal section view, the coronal section view, and the axial section view, and real-time endoscopic images. For example, the stitched image and the real-time endoscopic image can be displayed simultaneously with any one or more of the navigation views such as the fitted two-dimensional perspective view, the sagittal view, the coronal view, and the axial view; the image obtained by fusing the stitched image and the current image of the optical hard endoscope can also be displayed simultaneously with any one or more of the navigation views such as the fitted two-dimensional perspective view, the sagittal view, the coronal view, and the axial view, as well as the real-time endoscopic image; the image obtained by fusing the direction mark or physiological structure mark with the stitched image can also be displayed simultaneously with any one or more of the navigation views such as the fitted two-dimensional perspective view, the sagittal view, the coronal view, and the axial view, as well as the real-time endoscopic image, etc., etc. This allows the operator to conveniently obtain more comprehensive navigation information. It will be understood by those skilled in the art that other views of sections, orientations, or any suitable views can also be displayed as needed.

[0085] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is capable of performing the steps of the navigation method of the present invention. Furthermore, the present invention also provides a control device, which may include a memory, a processor, and a program stored on the memory and executable by the processor, wherein the steps of the navigation method of the present invention are performed when the processor executes the program. The present invention also provides a computer program product, including the computer program, which, when executed by the processor, implements the steps of the navigation method of the present invention.

[0086] As described above, the present invention also provides a navigation system. Because the navigation system utilizes an optical hard lens to form a spliced ​​image of a larger area around the user, such as a panoramic image, thereby providing a wider field of view for the operator, and incorporates directional markers or physiological structure markers or other guiding indicators to facilitate the operator's grasp of the overall orientation, the navigation system has improved navigation effectiveness and accuracy.

[0087] According to another aspect of the present invention, a robotic system (also referred to as a positioning and navigation system) is provided, which includes a robotic arm and the navigation system of each example of the present invention. In other words, the concept of the present invention can also be implemented under the navigation system in the robotic system.

[0088] Those skilled in the art will appreciate that the steps of the methods or algorithms described herein can be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0089] In the above embodiments, the method can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0090] Those skilled in the art will appreciate that the memory of the control device of the present invention may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device separate from the processor.

[0091] The processor of the control device can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0092] Although certain embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the general inventive concept, the scope of which is defined in the claims and their equivalents.

Claims

1. A navigation method for optical endoscopic surgery, characterized in that: The method comprises the following steps: Image acquisition step: acquiring multiple images of the optical hard mirror; Imaging orientation acquisition step: acquiring a position and / or orientation of an imaging device of the optical hard mirror corresponding to at least one of the plurality of images of the optical hard mirror under a navigation system; a stitching step of stitching a plurality of images of the optical hard mirror according to the position and / or direction of the imaging device corresponding to at least one image to obtain a stitched image; and Displaying step: displaying at least a portion of the stitched image.

2. The navigation method according to claim 1, wherein: The navigation method also includes an image fusion step between the stitching step and the display step, wherein in the image fusion step, the stitched image is fused with the current image of the optical hard lens according to the current position and / or orientation of the imaging device, and the fused image is displayed in the display step.

3. The navigation method according to claim 2, wherein: The navigation method also includes a processing step between the stitching step and the image fusion step, in which a planar image is generated based on the image obtained in the stitching step, and in the image fusion step, the planar image is fused as the stitched image with the current image of the optical hard lens.

4. The navigation method according to claim 1, wherein: The navigation method further comprises: a marker position acquisition step, in which the position and / or orientation of the marker on the image of the patient's physiological structure under the navigation system is acquired; and A marker fusion step is provided between the stitching step and the displaying step, wherein the marker and / or guidance instructions associated with the marker are fused onto the stitched image according to the position and / or orientation of the marker under a navigation system.

5. The navigation method according to claim 2 or 3, characterized in that: The navigation method further comprises: a marker position acquisition step, in which the position and / or orientation of the marker on the image of the patient's physiological structure under the navigation system is acquired; and A marking fusion step is performed between the image fusion step and the display step, wherein the marking fusion step is performed according to the The position and / or direction of the mark under the navigation system is marked, and the mark and / or the guidance indication related to the mark are fused onto the fused image obtained in the image fusion step.

6. The navigation method according to any one of claims 1 to 4, characterized in that: The imaging position acquisition step is performed by acquiring the position of a tracer (4) located outside the patient's body and having a fixed positional relationship relative to the optical hard mirror from a tracking device (1) of a navigation system.

7. The navigation method according to any one of claims 1 to 4, characterized in that: The start of the navigation method can be triggered by an operator, and then the steps included in the navigation method are automatically performed.

8. The navigation method according to any one of claims 1 to 4, characterized in that: The navigation method is automatically started and each step included in the navigation method is performed in real time.

9. The navigation method according to claim 7 or 8, characterized in that: The execution of the navigation method can be stopped after receiving a stop input from the operator.

10. The navigation method according to claim 4, characterized in that: The method further includes the step of indicating the current imaging device field of view position in the stitched image. In this step, an area corresponding to the current imaging device field of view position in the stitched image is determined with the help of the position and orientation of the imaging device, a mark indicating the area is generated, and the mark is displayed in the display step.

11. The navigation method according to claim 4 or 5, characterized in that: The markings on the image of the patient's physiological structure include one or more of direction markings and physiological structure markings.

12. The navigation method according to claim 11, characterized in that: The direction mark includes one or more direction marks facing the dorsal, ventral, cephalic and caudal sides of the patient. Preferably, the direction mark is multiple, for example, four direction marks pointing to the dorsal, ventral, cephalic and caudal sides of the patient respectively.

13. The navigation method according to claim 11, characterized in that: The physiological structure marker includes a physiological structure marker point that does not displace during optical endoscopic surgery. Preferably, the patient's physiological structure is a spine, and the physiological structure marker points include one or more of the ventral side of the articular process, the pedicle of the anterior vertebra, the pedicle of the posterior vertebra and the intervertebral disc. Preferably, there are multiple physiological structure marker points, and preferably, there are multiple guidance indicators, for example, four indicator symbols pointing to the ventral side of the articular process, the pedicle of the anterior vertebra, the pedicle of the posterior vertebra and the intervertebral disc respectively.

14. The navigation method according to claim 11, characterized in that: The image of the patient's physiological structure includes a preoperative three-dimensional image, an intraoperative three-dimensional image, or an intraoperative two-dimensional image.

15. The navigation method according to any one of claims 1 to 14, characterized in that: The optical hard endoscope is a spinal endoscope.

16. The navigation method according to claim 6, characterized in that: The method further comprises a calibration step of calibrating the positional relationship of the imaging device of the optical hard mirror relative to the tracer (4), wherein the calibration step comprises acquiring an image through a calibration tool instead of through the optical hard mirror.

17. The navigation method according to any one of claims 1 to 16, characterized in that: The method further includes a distortion processing step before the stitching step, wherein in the distortion processing step, the image of the optical hard mirror to be stitched is subjected to distortion calibration.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the navigation method according to any one of claims 1 to 17 are executed.

19. A control device, wherein the control device comprises a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein when the processor runs the program, the steps of the navigation method according to any one of claims 1 to 17 are executed.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the navigation method according to any one of claims 1 to 17 are implemented.

21. An electronic device for navigation of optical endoscopic surgery, characterized in that: The electronic device comprises a display device (3) and a processor, wherein the processor has a data interface. wherein the data interface is connectable to a tracking device and an optical hard mirror of a navigation system, so that the processor is capable of acquiring a plurality of images of the optical hard mirror and acquiring a position and / or orientation of an imaging device of the optical hard mirror corresponding to at least one of the plurality of images of the optical hard mirror under the navigation system; and The processor is configured to display at least a portion of a stitched image on the display device (3) for at least a period of time when the processor is running, wherein the stitched image is stitched together from a plurality of images of the optical hard lens.

22. The electronic device according to claim 21, wherein: The image displayed on the display device (3) is an image obtained by fusing the spliced image with the current image of the optical hard lens.

23. The electronic device according to claim 22, wherein: The image displayed on the display device (3) is an image obtained by fusing the plane image corresponding to the spliced image with the current image of the optical hard lens.

24. The electronic device according to claim 21, wherein The processor is also configured to acquire a mark on an image of a patient's physiological structure, and the image displayed on the display device (3) is an image obtained by fusing the mark or a guidance instruction related to the mark onto the stitched image.

25. The electronic device according to claim 22, wherein: The processor is also configured to be able to obtain a mark on the image of the patient's physiological structure, and the image displayed on the display device (3) is an image obtained by fusing the mark or the guidance instructions related to the mark, the stitched image and the current image of the optical hard mirror.

26. The electronic device according to claim 24 or 25, characterized in that: The mark includes one or more of a direction mark and a physiological structure mark; Preferably, the direction mark comprises one or more of direction marks toward the dorsal side, ventral side, head side and tail side of the patient; and / or Preferably, the physiological structure marker includes a physiological structure marker point that does not displace during optical endoscopic surgery. For example, the patient's physiological structure is a spine, and the physiological structure marker point includes one or more of the ventral side of the articular process, the pedicle of the anterior vertebra, the pedicle of the posterior vertebra, and the intervertebral disc.

27. The electronic device according to claim 21, wherein The electronic device further includes an input component for an operator to input instructions to trigger display of the stitched image.

28. The electronic device according to claim 21, wherein A mark indicating an area corresponding to the current imaging device field of view position in the stitched image is also displayed on the displayed stitched image.

29. The electronic device according to claim 26, wherein: On the display interface of the display device, one or more of the following images or views are displayed: a stitched image, an image fused with the stitched image and the current image of the optical hard lens, an image fused with the plane image corresponding to the stitched image and the current image of the optical hard lens, an image obtained by fusing the direction mark with the stitched image, an image obtained by fusing the physiological structure mark with the stitched image, a view on a fitted two-dimensional perspective section, a sagittal section view, a coronal section view, an axial section view, and a real-time image of the optical hard lens.

30. A navigation system for optical endoscopic surgery, characterized in that: The navigation system includes: A tracking device (1) adapted to track a tracer (4) disposed on an optical hard mirror (5); and A display device (3) and a processor, wherein the processor is suitable for being connected to the tracking device (1) and the optical hard mirror (5), wherein the navigation method described in any one of claims 1 to 17 is executed when the processor is running, and the display in the navigation method is realized through the display device (3).

31. The navigation system according to claim 30, characterized in that The navigation system further includes the optical hard scope and / or the tracer.

32. A navigation system for optical endoscopic surgery, characterized in that: The navigation system includes: A tracking device (1) adapted for tracking a tracer (4) disposed on an optical hard mirror (5); and An electronic device according to any one of claims 21-29, wherein the processor in the electronic device is suitable for connecting to the tracking device (1) and the optical hard mirror (5).

33. A robot system, characterized in that: The apparatus comprises a robotic arm and a navigation system according to any one of claims 30-32.

Images