Computerized tomographic image correction using azimuth and direction tracking assistance

Abstract

A system (11) includes a medical probe (36) for insertion into a cavity of an organ, the medical probe including an orientation and orientation sensor (60) and a camera (45), both operating in a sensor coordinate system (62). The system further comprises a processor (44) configured to: receive a three-dimensional image of a cavity comprising an open space and tissue from an imaging system (21) operating in an image coordinate system (28); receiving, from the medical probe, a signal indicative of an orientation and a corresponding orientation of the medical probe within the lumen; receiving a corresponding visual position within the cavity from the camera; the image coordinate system is registered with the sensor coordinate system to identify one or more voxels in the image at the visualization location, and when the identified voxels have density values in the received image that do not correspond to the open space, the density values of the identified voxels are updated to correspond to the open space.

Description

[0001] This patent application is a divisional application of PCT patent application PCT / IB2019 / 058348 (international application date October 1, 2019, priority date October 4, 2018, Chinese national application number 201980065108.X, invention title "Computerized Computed Tomography (CT) Image Correction Using Aspect and Direction (P&D) Tracking Assistance for Optical Visualization"), which entered the Chinese national phase on April 1, 2021.

[0002] Cross-references to related applications

[0003] This patent application claims the benefit of U.S. Provisional Patent Application 62 / 741,403, filed October 4, 2018, the disclosure of which is incorporated herein by reference. Technical Field

[0004] The present invention relates generally to medical imaging, and more specifically to updating outdated medical images during the performance of invasive medical procedures. Background Technology

[0005] During medical procedures, real-time representations of anatomical structures detected by medical devices (such as probes tracked by orientation tracking systems) can be overlaid on pre-acquired medical images to improve the representation of anatomical structures. For example, U.S. Patent Application Publication 2018 / 0146883 describes a method, apparatus, and computer program product that includes receiving a three-dimensional image of a body cavity, including open spaces and body tissues, from an imaging system operating in an image coordinate system, and receiving a signal from a medical probe having a position sensor and inserted into the body cavity, indicating the position of the distal end of the medical probe in a sensor coordinate system. The image coordinate system is registered with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the indicated position, and when the identified voxels have density values ​​in the received three-dimensional image that do not correspond to open spaces, the density values ​​of the identified voxels are updated to correspond to open spaces.

[0006] For example, U.S. Patent Application Publication 2014 / 0147027 describes an imaging correction system that includes a tracking ultrasound imaging probe configured to generate imaging volumes of regions of interest from different orientations. An image compensation module is configured to process image signals from an ultrasound imaging device associated with the probe and compare one or more ultrasound image volumes with a reference such as a CT image to determine distortions in the ultrasound imaging and generate corrected ultrasound images for display based on compensation for the distortions.

[0007] Image-guided surgery (IGS), such as the surgery described above, is a technique in which a computer is used to obtain the real-time correlation between the position of instruments already inserted into the patient and a set of images obtained before surgery (e.g., CT or MRI scans, 3D mapping, etc.), allowing the computer system to overlay the current position of the instruments onto the images obtained before surgery. An example of an electromagnetic IGS navigation system that can be used in IGS surgery is provided by Biosense-Webster, Inc. (Irvine, California). 3. System.

[0008] In some IGS procedures, a digital tomographic scan of the surgical area (e.g., CT or MRI, 3D mapping, etc.) is obtained prior to surgery. The digital tomographic data is then converted into a digital mapping using a specially programmed computer. During the surgery, specialized instruments equipped with orientation sensors (e.g., electromagnetic coils emitting electromagnetic fields and / or responding to externally generated electromagnetic fields) are used to execute the procedure, while the sensors send data to a computer indicating the current orientation of each surgical instrument. This computer correlates the data received from the sensors with the digital map generated from the preoperative tomographic scan. The tomographic images, along with indicators (e.g., crosshairs or illuminated dots, etc.), are displayed on a video monitor, showing the real-time orientation of each surgical instrument relative to the anatomical structures shown in the scan images. Even if the surgeon cannot directly visually view the instruments at their current positions within the body, they can understand the exact orientation of each sensor-equipped instrument by viewing the video monitor.

[0009] While IGS navigation systems provide useful views and information during surgical procedures, surgeons may also desire real-time photographs and videos of the anatomical structures being manipulated. In such cases, the endoscope can be deployed to the surgical site using IGS navigation to acquire images of the anatomical structures; it can also be paired with or otherwise deployed with other surgical instruments such as cutting instruments, ablation instruments, and dilation catheters. The photographic and video images acquired in this way may be more useful than the IGS navigation images, which may only provide general and analog images. Alternatively, photographic and video images acquired in this way can provide a useful supplement to the IGS navigation images.

[0010] Compared to direct visual observation by surgeons, images acquired by conventional endoscopy can be limited to a two-dimensional (2D) representation of anatomical structures. Procedures such as cutting, ablation, and dilation can be performed using a combination of 2D endoscopy and IGS navigation, neither of which provides the depth required for true three-dimensional (3D) observation. Manipulating and operating surgical instruments without 3D observation increases the time spent on instrument positioning and the likelihood of errors. While some 3D endoscopic cameras exist, they are highly complex and expensive, and may have rigid and non-flexible components that make them difficult or impossible to use in some procedures. Implementations with both flexible and 3D features may require expensive and fragile fiber optic components to transmit acquired images from the distal camera to the proximal observer. Therefore, conventional options for 3D endoscopy can be expensive, have limited features, and are not suitable for high-volume use and disposability.

[0011] Although several systems and methods have been developed and used in surgical procedures, it is believed that no one has previously developed or used the invention described in the appended claims. Summary of the Invention

[0012] An embodiment of the present invention provides a system comprising a medical probe, an orientation and direction sensor, a camera, and a processor. The medical probe is configured to be inserted into a cavity of a patient's organ. The orientation and direction sensor is located within the medical probe and operates in a sensor coordinate system. The camera is located at the distal edge of the medical probe and operates in the sensor coordinate system. The processor is configured to: (a) receive a three-dimensional image of the cavity, including an open space and organ tissue, from an imaging system operating in an image coordinate system; (b) receive from the medical probe a signal indicating the orientation and corresponding direction of the distal edge of the medical probe within the cavity; (c) receive a corresponding visualized location within the cavity from the camera of the probe; (d) register the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualized location; and (e) update the density value of the identified voxel to correspond to the open space when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space.

[0013] In some implementations, the imaging system includes a computed tomography scanner.

[0014] In some implementations, the orientation and direction sensors include magnetic field sensors.

[0015] In one embodiment, the processor is configured to form a correspondence between the density value and a visual effect, wherein a given visual effect corresponds to a given density value indicating the open space. In another embodiment, the visual effect is selected from a group consisting of colors, shadows, and patterns.

[0016] In some embodiments, the processor is configured to render the 3D image on a display using the visual effect. In other embodiments, the given visual effect includes a first given visual effect, and wherein, before updating the density value, the processor is configured to render the 3D image by using a second given visual effect different from the first given visual effect to render the one or more identified voxels.

[0017] In one embodiment, when updating the density value, the processor is configured to render the 3D image by presenting the one or more identified voxels using the first given visual effect. In another embodiment, the processor is configured to use multi-view... Figure 3 The angular segmentation model extracts the distance from the location to the camera from the visual signal.

[0018] According to an embodiment of the invention, a method is also provided, the method comprising receiving a three-dimensional image of a cavity of a patient's organ, comprising open space and organ tissue, from an imaging system operating in an image coordinate system. Receiving signals from a medical probe having orientation and orientation sensors and a camera, indicating the orientation and corresponding direction of the distal edge of the medical probe within the cavity, wherein the probe operates in a sensor coordinate system and is inserted into the cavity. Registering the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualization location. When an identified voxel has a density value in the received three-dimensional image that does not correspond to the open space, updating the density value of the identified voxel to correspond to the open space.

[0019] According to an embodiment of the invention, a computer software product is also provided that operates in conjunction with a probe configured for insertion into a cavity of a patient's organ and includes an orientation and direction sensor operating in a sensor coordinate system and a camera operating in the sensor coordinate system at the distal edge of the medical probe. The product includes a non-transitory computer-readable medium storing program instructions that, when read by a computer, cause the computer to: (a) receive a three-dimensional image of the cavity, including an open space and organ tissue, from an imaging system operating in an image coordinate system; (b) receive a signal from the medical probe indicating the orientation and corresponding direction of the distal edge of the medical probe within the cavity; (c) receive a corresponding visualized location of the wall of the cavity; (d) register the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualized location; and (e) update the density value of the identified voxel to correspond to the open space when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space.

[0020] According to an embodiment of the present invention, a three-dimensional (3D) imaging system including an endoscope and a processor is also provided. The endoscope includes: (i) an axis having a distal end, the axis being adapted for insertion into a patient and positioned at a surgical site; (ii) an orientation sensor located near the distal end and configured to generate an orientation signal set based on the endoscope's position during use; and (iii) an imaging module positioned at the distal end and operable to acquire an image dataset of the surgical site, wherein the image dataset includes one or more two-dimensional (2D) images. The processor is communicatively coupled to the endoscope and configured to: (i) receive the image dataset and the orientation signal set from the endoscope; (ii) determine a perspective dataset based on the orientation signal set, wherein the perspective dataset indicates the position of the endoscope during the acquisition of each of the one or more 2D images; (iii) perform image depth analysis to determine a 3D feature set for each of the one or more 2D images, wherein the 3D feature set includes pixel depth; (iv) create a 3D image dataset based on the one or more 2D images and the 3D feature set; and (v) associate the perspective dataset with the 3D image dataset, wherein the image depth analysis includes techniques selected from the group consisting of: (i) wavefront sampling techniques performed on a single image of the one or more 2D images; and (ii) passive stereo vision techniques performed on a set of two images of the one or more 2D images, wherein each image in the set of two images is associated with the same perspective data.

[0021] In some embodiments, the imaging module includes: (i) a single lens; (ii) an aperture plate positioned between a first side of the single lens and the surgical site, the aperture plate including one or more apertures offset from the optical axis of the single lens; and (iii) an image pane positioned on a second side of the single lens to receive reflected light from the surgical site via the one or more apertures and the single lens, wherein the image pane is configured to generate the image dataset based on the reflected light.

[0022] In some embodiments, (i) the one or more apertures include at least two apertures positioned on the aperture plate and offset from the optical axis of the single lens, and (ii) the aperture plate has a fixed orientation and orientation relative to the lens.

[0023] In one embodiment, (i) the one or more apertures include a single aperture offset from the optical axis of the single lens and positioned on the aperture plate, and (ii) the aperture plate is operable to rotate relative to the lens about its circular axis during image acquisition.

[0024] In another embodiment, the processor is configured to, when performing the image depth analysis,: (i) identify two or more unfocused images of the surgical site within the image dataset; (ii) determine the spatial relationship between the two or more unfocused images of the surgical site; and (iii) determine the pixel depth in the image dataset based on the spatial relationship between the two or more unfocused images.

[0025] In some embodiments, the imaging module includes two or more cameras, and each of the two or more cameras: (i) is statically positioned relative to every other camera among the two or more cameras; and (ii) is oriented to have an optical axis parallel to every other camera among the two or more cameras.

[0026] In some embodiments, the processor is further configured, when performing the image depth analysis, to: (i) identify points in a first image of the image dataset, wherein the points include portions of the surgical site present in both the first image captured by a first camera of the two or more cameras and a second image captured by a second camera of the two or more cameras; (ii) identify the points in the second image; (iii) determine the displacement of the points from the first image to the second image; and (iv) determine the pixel depth of the points based on the displacement.

[0027] In one embodiment, the processor is further configured to, when identifying the point in the second image: (i) determine the epipolar lines of the first image and the second image based on the static position of the first camera relative to the second camera; and (ii) search for the point in the second image along the epipolar lines while excluding portions of the second image that do not fall on the epipolar lines.

[0028] In another embodiment, the processor is further configured to: (i) associate the 3D image dataset and the view dataset with the coordinate system of the image-guided surgery system; and (ii) display the 3D image dataset during the image-guided surgery navigation procedure based on the association with the coordinate system.

[0029] In one implementation, (i) the orientation sensor is configured to generate the orientation signal set based on the position and orientation of the endoscope during use, (ii) the perspective dataset indicates the position and orientation of the endoscope during the acquisition of the image dataset, and (iii) the processor is further configured to provide the 3D image dataset and the perspective dataset to the image-guided surgical navigation system.

[0030] In another embodiment, the processor is further configured to: (i) receive input from a user defining a viewpoint relative to the surgical site; (ii) determine a first portion of the 3D image dataset showing the surgical site from the viewpoint based on the viewpoint identified within the viewpoint dataset; and (iii) display the first portion of the 3D image dataset on a display.

[0031] In another embodiment, the processor is further configured to: (i) receive an indirect 3D scan of the surgical site and a scan view dataset associated with the indirect 3D scan; (ii) determine a second portion of the indirect 3D scan showing the surgical site from the scan view dataset based on identifying the view within the scan view dataset; and (iii) simultaneously display the first portion of the 3D image dataset and the second portion of the indirect 3D scan on the display.

[0032] In another embodiment, (i) the indirect 3D scan of the surgical site includes image data acquired before surgery, and (ii) the 3D image dataset includes image data acquired after surgery.

[0033] In one implementation, (i) the indirect 3D scan of the surgical site includes preoperative image data, (ii) the 3D image dataset includes preoperative image data, and (iii) the processor is further configured to: (A) receive scan adjustment input from a user; and (B) reconfigure the association between the indirect 3D scan of the surgical site and the scan view dataset based on the scan adjustment input.

[0034] According to an embodiment of the invention, a method for three-dimensional (3D) imaging is also provided, the method comprising deploying a distal endoscope to a surgical site of a patient, the distal endoscope comprising: (i) an imaging module operable to acquire image data of the surgical site, wherein the acquired image data includes one or more two-dimensional (2D) images; and (ii) an orientation sensor proximal to the distal endoscope and configured to generate an orientation signal based on the position of the endoscope. An image dataset from the imaging module and an orientation signal set from the orientation sensor are received. A viewing angle dataset is determined based on the orientation signal set, wherein the viewing angle dataset indicates the position of the endoscope during acquisition of each of the one or more 2D images. Image depth analysis is performed to determine a 3D feature set for each of the one or more 2D images, wherein the 3D feature set includes pixel depth. A 3D image dataset is created based on the one or more 2D images and the 3D feature set, and the viewpoint dataset is associated with the 3D image dataset, wherein the image depth analysis includes techniques selected from the group consisting of: (i) wavefront sampling techniques performed on a single image in the one or more 2D images; and (ii) passive stereo vision techniques performed on a set of two images in the one or more 2D images, wherein each image in the set of two images is associated with the same viewpoint data.

[0035] Another embodiment of the present invention provides an image-guided surgery (IGS) navigation system including a processor, a memory, and a display, the processor being configured to: (a) receive an image dataset generated by a tracking endoscope, the image dataset including one or more two-dimensional (2D) images; (b) receive a perspective dataset generated by the tracking endoscope, wherein the perspective dataset indicates the position of the tracking endoscope during acquisition of the image dataset; (c) perform image depth analysis to determine a 3D feature set of the image dataset, wherein the 3D feature set includes pixel depths in the one or more 2D images; (d) create a 3D image dataset based on the one or more 2D images and the 3D feature set; (e) associate the perspective dataset with the 3D image dataset; and (f) cause the display to display the 3D image dataset from the selected perspective based on the perspective dataset including the selected perspective.

[0036] The invention will be more fully understood from the following detailed description of embodiments thereof, taken in conjunction with the accompanying drawings, wherein: Attached Figure Description

[0037] Figure 1A and Figure 1B (Collectively referred to as FIG1) is a schematic illustration of a medical system configured to correct outdated computed tomography (CT) images during the performance of invasive medical procedures, according to an embodiment of the present invention;

[0038] Figure 2 A flowchart illustrating a method for correcting outdated CT images according to an embodiment of the present invention;

[0039] Figure 3 A schematic diagram illustrating image slices of a three-dimensional CT image according to an embodiment of the present invention;

[0040] Figure 4 A schematic detail diagram illustrating the distal end of a probe inserted into a sinus cavity according to an embodiment of the present invention;

[0041] Figure 5 To illustrate the embodiment of the invention before updating CT images Figure 3 A schematic diagram of the region of interest in a CT image slice;

[0042] Figure 6 To illustrate the following after updating CT images according to an embodiment of the present invention Figure 5 A schematic diagram of the region of interest in a CT image slice;

[0043] Figure 7A schematic diagram of an exemplary surgical navigation system for use on a patient seated in an exemplary medical protocol chair, according to another embodiment of the present invention, is shown;

[0044] Figure 8 The compatibility of embodiments according to the present invention is shown. Figure 7 A perspective view of an exemplary instrument used in conjunction with a surgical navigation system;

[0045] Figure 9A An embodiment of the invention is shown with a single imaging device. Figure 8 A perspective view of an exemplary distal end of the device;

[0046] Figure 9B An embodiment according to the present invention is shown. Figure 9A A front view of the distal end;

[0047] Figure 10A An embodiment according to the present invention is shown. Figure 9A A perspective view of the distal end, where the flushing deflector has been removed to show the flushing channel;

[0048] Figure 10B An embodiment according to the present invention is shown. Figure 9A A frontal view of the distal end, with the flushing deflector removed to show the flushing channel;

[0049] Figure 11 An embodiment of the invention is shown for use Figure 9A A schematic diagram of an exemplary static sampling aperture configuration for an imaging device;

[0050] Figure 12 It shows the use of Figure 9A A schematic diagram of an exemplary active sampling aperture configuration for an imaging device;

[0051] Figure 13 The embodiments of the present invention are shown to be executable to utilize Figure 9A An exemplary set of steps for an imaging device to generate a three-dimensional image model;

[0052] Figure 14A A set of imaging devices is shown. Figure 8 An exemplary alternative perspective view of the distal end of the device;

[0053] Figure 14B An embodiment according to the present invention is shown. Figure 14A A front view of the distal end;

[0054] Figure 15A An embodiment according to the present invention is shown. Figure 14AA set of exemplary overlapping field-of-view side diagram views of a set of imaging devices;

[0055] Figure 15B An embodiment according to the present invention is shown. Figure 14A A top view solution of a set of overlapping fields of view;

[0056] Figure 16 The embodiments of the present invention are shown to be executable to utilize Figure 14A A set of exemplary steps for a set of imaging devices to generate a three-dimensional image model;

[0057] Figure 17 An illustration is provided that shows the relationship between parallax and depth in stereoscopic imaging;

[0058] Figure 18 An illustration is shown that depicts the epipolar lines in stereoscopic imaging according to an embodiment of the present invention; and

[0059] Figure 19 It shows that it can be executed to utilize Figure 7 The flowchart shows a set of steps that generate and use a 3D image model in a surgical navigation system.

[0060] The accompanying drawings are not intended to be limiting in any way, and various embodiments of the invention can be conceived to be implemented in many other ways, including those not necessarily shown in the drawings. The drawings, which are incorporated in and constitute a part of this specification, illustrate several aspects of the invention and, together with the description, serve to explain the principles of the invention; however, it should be understood that the invention is not limited to the explicit arrangements shown. Detailed Implementation

[0061] Overview

[0062] According to one aspect of the invention, intraoperative medical imaging systems such as computed tomography (CT) imaging systems are modalities that are typically unavailable in clinical practice when performing otolaryngological (ENT) procedures. Therefore, ENT physicians performing procedures typically rely on previously acquired medical images, such as CT images. CT images can be used to generate real-time guidance (i.e., “maps”) that allows the physician performing the ENT medical procedure to navigate the medical probe within a cavity of interest in the patient’s head.

[0063] However, in some cases, such as during or after surgical ENT procedures such as scraping the obstructing cartilage or bone at the cavity wall, CT images no longer reflect the true anatomical structure of the cavity.

[0064] The embodiments of the present invention described below provide systems and methods for updating outdated medical images, such as outdated CT images of cavities of a patient's organs, which are also referred to herein as three-dimensional images.

[0065] In some implementations, during protocol updates, a camera located at the distal end of the medical probe and operating in the probe's coordinate system visualizes the location within cavities, including open spaces and organs. The processor uses the visualizations provided by the camera from different orientations and directions to estimate the location. The processor derives different orientations and directions based on signals received from orientation and direction (P&D) sensors mounted at the distal end of the medical probe.

[0066] Use multi-view Figure 3 In the angular partitioning model, the processor reconstructs the visual location by reconstructing the distance from the camera to the fail position. This type of multi-view... Figure 3 The triangular partitioning model can use, for example, a name such as "triangulateMultiview". A function that returns the position of a 3D world point, which corresponds to a point matched on multiple images taken using a camera calibrated with P&D.

[0067] To ensure that CT images reflect the latest (i.e., true) anatomical structure of the cavity, the processor registers the coordinate system associated with the CT imaging system with the coordinate system associated with the medical probe (e.g., with the sensor coordinate system). This registration enables the processor to identify one or more voxels in the 3D image at indicated locations, and when the identified voxels have density values ​​in the received 3D image that do not correspond to newly formed open spaces (e.g., formed by an ENT shaving procedure) (as visualized by the camera and as utilized by the processor using multiview), the processor can determine the voxels' location. Figure 3 When estimating the density values ​​of the identified voxels using the triangular partitioning model, the density values ​​are updated to correspond to the new open space.

[0068] Medical imaging systems typically use different visual effects to represent open spaces and body tissues. For example, a CT system may represent open spaces in black, hard body tissue in white, and soft body tissue in gray (various shades). Since a camera located at the distal edge of the medical probe can optically image only the wall tissue within the open space, a system implementing embodiments of the present invention can correct for any optical imaging locations of cavities not previously presented as open spaces in updated three-dimensional (3D) CT images. Furthermore, embodiments of the present invention can also present the entered location as an open space in updated 3D images if the camera has entered a location that was not previously an open space.

[0069] Other aspects of the disclosed invention provide imaging techniques, such as wavefront sampling and passive stereo vision, for generating 3D image data and 3D models from single-camera and multi-camera 2D imaging systems. Compared to 3D imaging systems with independent dynamic focusing cameras, such techniques are used to generate 3D functional imaging systems with small size and low complexity. Such imaging techniques (e.g., systems) are implemented at the distal end of surgical instruments with static or flexible axes while reducing size and power requirements, and can be used to generate 3D images of surgical sites within the human body. Such 3D imaging can be used to generate synthetic image sets available during IGS navigation to provide comparison of preoperative and postoperative image sets.

[0070] In some embodiments, a 3D endoscopic imaging system is provided, comprising: (a) an endoscope including: an orientation sensor; and an imaging module positioned at the distal end of the endoscope and operable to acquire an image dataset of a surgical site, wherein the image dataset includes one or more two-dimensional (2D) images; and (b) a processor communicatively coupled to the endoscope and configured to:

[0071] (i) Receive image datasets and orientation signal sets from the endoscope.

[0072] (ii) Determine the viewpoint dataset based on the orientation signal set, wherein the viewpoint dataset indicates the position of the endoscope during the acquisition of each of one or more 2D images.

[0073] (iii) Perform image depth analysis to determine a 3D feature set for each of one or more 2D images, wherein the 3D feature set includes pixel depth.

[0074] (iv) Create a 3D image dataset based on one or more 2D images and 3D feature sets, and

[0075] (v) Associate the viewpoint dataset with the 3D image dataset;

[0076] Image depth analysis includes techniques selected from a group of the following:

[0077] (i) Wavefront sampling techniques performed on a single image in one or more 2D images, and

[0078] (ii) A passive stereo vision technique performed on a set of two images in one or more 2D images, wherein each image in the set of two images is associated with the same viewpoint data.

[0079] Embodiments of the present invention disclose medical image (such as CT images) correction using visualization techniques assisted by P&D tracking of cavity location. Using these embodiments eliminates the need for repetitive imaging procedures such as CT imaging procedures, which typically require large and expensive operating facilities.

[0080] CT image correction using P&D tracking-assisted optical visualization

[0081] The following description of certain examples of the invention is not intended to limit the scope of the invention. Other examples, features, aspects, embodiments, and advantages of the invention will be apparent to those skilled in the art from the following description, which is shown by way of example, and a best mode is contemplated for carrying out the invention. It will be appreciated that the invention can have other different and obvious aspects, all of which are not departing from the invention. Therefore, the drawings and descriptions should be regarded as substantially illustrative and not restrictive.

[0082] It should be understood that the terms "proximal" and "distal" as used herein are relative to the clinician holding the handpiece assembly. Therefore, the end effector is distal relative to the closer handpiece assembly. It should also be understood that, for convenience and clarity, spatial terms such as "top" and "bottom" are used in relation to the clinician holding the handpiece assembly. However, surgical instruments are used in many orientations and locations, and these terms are not intended to be restrictive or absolute.

[0083] It should also be understood that any one or more of the teachings, expressions, types, examples, etc., described herein may be combined with any one or more of the other teachings, expressions, types, examples, etc., described herein. Therefore, the following teachings, expressions, types, examples, etc., should not be considered separate from each other. Various suitable ways in which the teachings herein can be combined will be apparent to those skilled in the art, referring to the teachings herein. Such modifications and variations are intended to be included within the scope of the claims.

[0084] Figure 1A and Figure 1B (Collectively referred to as FIG1) is a schematic illustration of a medical system 11 configured to correct outdated computed tomography (CT) images during invasive medical procedures according to an embodiment of the present invention. In the example shown in FIG1, the medical system 11 includes a medical imaging system comprising a mobile CT scanner 21, a console 34, and a medical probe 36. In the embodiments described herein, it is assumed that the medical probe 36 is used for ENT diagnostic or therapeutic procedures, such as a minimally invasive catheter-based sinus surgery performed on patient 26. Alternatively, with necessary modifications, the medical probe 36 may be used for other therapeutic and / or diagnostic purposes.

[0085] like Figure 1A As shown, prior to performing invasive medical procedures on patient 26, CT scanner 21 generates electrical signals including image data of the patient's internal cavities (e.g., nasal cavity or paranasal sinuses) and transmits the generated image data to console 34. Computed tomography scanner 21 generates image data in image coordinate system 28, including X-axis 31, Y-axis 33, and Z-axis 35.

[0086] like Figure 1B As shown, the medical probe 36 includes a handle 38 that can be grasped and manipulated by an operator 43 to insert the distal end 41 of the medical probe into the cavity of the patient 26, such as the nasal cavity or paranasal sinuses.

[0087] The distal end 41 includes a camera 45 that generates image data in response to a scene observed by the camera. The distal end 41 also includes a magnetic field sensor 60 that generates signals providing the orientation and orientation of the distal end, as described below. In the configuration shown in FIG1, the console 34 includes a processor 44 that converts the image data received from the camera 45 into an image 46 and presents the image on a display 48 as information about medical procedures.

[0088] Based on signals received from the medical probe 36 and other components of the medical system 20, the console 34 drives the display 48 to update the image 46, thereby presenting the current orientation of the distal end 41 within the patient's head and status information and instructions regarding the ongoing medical procedure. The processor 44 stores the data representing the image 46 in the memory 50. In some embodiments, the operator 40 may manipulate the image 46 using one or more input devices 52.

[0089] Processor 44 typically includes a general-purpose computer with suitable front-end and interface circuitry for receiving signals from medical probe 36 and controlling other components of console 34. Processor 44 can be software-programmed to perform the functions described herein. This software can be downloaded to console 34 electronically, for example, via a network, or it can be provided on a non-transitory tangible medium such as an optical storage medium, magnetic storage medium, or electronic storage medium. Alternatively, some or all of the functions of processor 44 can be performed by dedicated or programmable digital hardware components.

[0090] In the embodiments described herein, the medical system 11 uses magnetic orientation sensing to determine the orientation and direction coordinates of the distal end 41 of the medical probe 36 within the patient 26. To achieve magneto-based orientation and direction sensing, the console 34 includes drive circuitry 56 for driving a field generator 58 to generate a magnetic field within the probe organ of the patient 26. Typically, the field generator 58 includes coils positioned below the patient at a known orientation outside the patient 26. These coils generate a magnetic field within a predetermined working space containing cavities such as paranasal sinuses. A magnetic field sensor 60 (also referred to herein as an orientation and direction (P&D) sensor 60) within the distal end 41 of the medical probe 36 generates an electrical signal in response to the magnetic field from the coils, thereby enabling the processor 44 to determine the orientation and direction of the distal end 41 within the working volume.

[0091] Magnetic orientation tracking technology is described, for example, in U.S. Patents 5,391,199, 6,690,963, 5,443,489, 6,788,967, 5,558,091, 6,172,499, and 6,177,792, the disclosures of which are incorporated herein by reference. A signal generated by the magnetic field sensor 60 indicates the current position of the distal end 41 in a sensor coordinate system 62 defined by the orientation of the generator 58, and the system 62 is assumed to include an X-axis 64, a Y-axis 66, and a Z-axis 68. In the example shown in FIG1, the X-axis 64 approximately corresponds to the X-axis 31, the Y-axis 66 approximately corresponds to the Y-axis 33, and the Z-axis 68 approximately corresponds to the Z-axis 35. Such systems and techniques are combined with other aspects of the invention (in conjunction with...). Figure 7 The systems and technologies described herein are similar to or the same as those described above.

[0092] In the embodiment described herein, the medical system 11 uses a camera 45 to image a portion of the cavity. As described above, a sensor 60 is used to derive the orientation of the camera 45 as it images the cavity and the corresponding direction the camera is aiming. This is based on P&D tracking and utilizes multi-view imaging. Figure 3 In the triangular partitioning model, processor 44 is able to determine (i) the location of a portion of the wall of the cavity in which the distal end is located, and (ii) the location of the imaging position on the cavity wall relative to the camera (including camera 45 itself), all of which are located in the open space; and is able to update the CT image accordingly.

[0093] Figure 2 A flowchart illustrating a method for correcting outdated CT images is provided. Figure 3 , Figure 4 , Figure 5 and 6The diagram illustrates a method according to an embodiment of the present invention. In acquisition step 80, processor 44 acquires image data from CT scanner 21, stores the image data in memory 50, and generates a three-dimensional image based on the acquired image data. Step 80 is typically performed before performing an updated CT image procedure on patient 26.

[0094] Figure 3 This is a schematic illustration of an image slice 101 that can be acquired in step 80 using a three-dimensional computed tomography image. The processor 44 generates the image slice 101 in response to receiving image data from the CT scanner 21. In operation, the processor 44 typically incorporates the image slice 101 into the image 46.

[0095] exist Figure 3 In the example shown, image slice 101 comprises a two-dimensional “slice” of the head 103 of patient 26. As shown in Example 106, image slice 101 comprises voxels 105 of a three-dimensional image corresponding to a three-dimensional location in a computed tomographic image of the head 103. Processor 44 typically renders each given voxel 105 using different visual effects corresponding to the density detected at the three-dimensional location corresponding to the given voxel. Examples of density include, but are not limited to, open space, hard organ tissue, and soft organ tissue, and examples of visual effects include, but are not limited to, color, shading (e.g., different shades of gray), and patterns (e.g., gradients, pictures, textures, lines, dots, and boxes).

[0096] As shown in Figure 107, voxels 105 can be distinguished by appending letters to the identifier numbers, thus including voxels 105A-105C. Figure 3 In the example shown, voxel 105A indicates open space and is presented in black, voxel 105B indicates hard organ tissue (e.g., bone) and is presented in white, and voxel 105C indicates soft tissue (e.g., fat, muscle, cartilage, and brain tissue) and is presented in gray. While embodiments described herein depict an image slice 101 comprising three voxels 105 (i.e., voxels 105A-105C) with three different visual effects, presenting voxels with any number of visual effects representing any number of densities is considered to be within the spirit and scope of the invention.

[0097] Image slice 101 also includes region of interest 109. (See reference...) Figure 5 and Figure 6 As described herein, the region of interest 109 includes the cavity wall 47 location with outdated CT values ​​(i.e., density values), which the processor 44 can update using the embodiments described herein.

[0098] Returning to the flowchart, in order to initiate the update procedure, operator 43 manipulates handle 38 such that, in insertion step 82, the distal end 41 enters the cavity of the patient 26's organ. It should be understood that there is typically a delay between steps 80 and 82, which can be several days or even weeks.

[0099] Next, in registration step 84, processor 44 registers image coordinate system 28 with sensor coordinate system 62. Registration can be performed by any suitable method known in the art. During registration, each three-dimensional orientation indicated by the signal generated by magnetic field sensor 60 corresponds to one or more voxels in the CT image acquired in step 80.

[0100] In the visual signal receiving step 86, the processor 44 receives a visual signal from the camera 45 indicating the position of the cavity wall 47, and in the recognition step 88, the processor uses registration and multi-view... Figure 3 An angular partitioning model is used to identify one or more voxels in a CT image corresponding to an indicated location.

[0101] Figure 4 A schematic detail diagram of the distal end 41 inserted into the sinus cavity 111 is shown. As the distal end moves, a camera 45 in the distal end 41 acquires images of the cavity wall 47 from different perspectives. Based on the visual signals of the wall 47 acquired by the camera 45, and as described above, multi-view... Figure 3 In the angular partitioning model, processor 44 estimates the position 47A of the cavity wall. It should be understood that any position on the corresponding line of sight of camera 45 to position 47A, such as position 49, and the position of camera 45 itself, must be an open space within the cavity.

[0102] Returning to the flowchart, in the first comparison step 90, if any voxel among one or more identified voxels has an outdated CT density value—that is, a CT density value of tissue rather than a CT density value of open space (newly formed through the ENT procedure)—then in the correction step 92, the processor 44 corrects the outdated voxel value to represent the voxel as (newly formed) open space. For example, if a given identified voxel has an outdated CT density value corresponding to hard or soft tissue, the processor 44 may update the density value to correspond to open space. If, in step 90, none of the voxels among one or more identified voxels have an outdated CT value, the method proceeds to step 94.

[0103] In an embodiment of the invention, updating one or more updated voxels in an image slice updates a three-dimensional computed tomography image stored in memory 50.

[0104] After completing step 92, the control program proceeds to the second comparison step 94.

[0105] In the second comparison step 94, if operator 40 has completed the update procedure, i.e., the operator has sufficiently inspected cavity 47 or other cavities, the method ends. However, if operator 40 has not completed the update procedure, the method proceeds to step 86.

[0106] Figure 5 This is a schematic diagram illustrating the region of interest 118 of image slice 101 before updating, as in the computed tomography image acquired in step 80. Figure 5 In the example shown, the blocking bone 120 includes voxel 105B (i.e., dense bone) surrounding the internal region of voxel 105A (i.e., the hollow region of bone growth), and voxel 105C includes soft tissue.

[0107] Figure 6 To illustrate after updating the computed tomography images (i.e., after completing...) Figure 2 A schematic diagram of the region of interest 118 of image slice 101 after the steps of the flowchart. Figure 6 As shown, region of interest 118 includes updated region 150. Region 150 includes a region estimated by processor 44 based on visualization from camera 45 and using a multi-view triangle partitioning model, which is a newly formed open space. Region 150 is formed, for example, by an ENT shaving process, which removes... Figure 5 Some of the blocking bones in the 120 blocking bones.

[0108] Display 48 typically presents a subset of voxels 105 that processor 44 can update during update procedures using the embodiments described herein. As described above, image slice 101 includes a two-dimensional “slice” of a three-dimensional image generated by processor 44 in response to receiving image data from mobile computed tomography scanner 21. Since the operator moves the distal end 41 in three-dimensional space (i.e., three-dimensional coordinate systems 28 and 62), it should be understood that additional voxels 105 (i.e., not included in the two-dimensional image slice currently being presented on display 48) may exist, the corresponding density values ​​of which are updated by processor 44 in response to the position of cavity wall 47 detected by camera 45.

[0109] Exemplary Image-Guided Surgical Navigation System

[0110] According to another aspect of the invention, when performing medical procedures within the head (H) of a patient (P), it may be desirable to obtain information about the orientation of the instrument within the head (H) of the patient (P), especially when the instrument is in a position where it is difficult or impossible to obtain an endoscopic view of the working elements of the instrument within the head (H) of the patient (P).

[0111] Figure 7An exemplary IGS navigation system (10) according to an embodiment of the present invention is shown, capable of performing ENT procedures using image guidance. In addition to having, or replacing, the components and operability described herein, the IGS navigation system (10) may be constructed and operated according to at least some of the teachings of the following patents: U.S. Patent 7,720,521, entitled “Methods and Devices for Performing Procedures within the Ear, Nose, Throat and Paranasal Sinuses”, published May 18, 2010, the disclosure of which is incorporated herein by reference; and U.S. Patent Publication 2014 / 0364725, entitled “Systems and Methods for Performing Image Guided Procedures within the Ear, Nose, Throat and Paranasal Sinuses”, published December 11, 2014, the disclosure of which is incorporated herein by reference.

[0112] The IGS navigation system (10) of this example includes a field generator assembly (20) comprising a set of magnetic field generators (24) integrated into a horseshoe frame (22). The field generators (24) are operable to generate alternating magnetic fields of different frequencies around the head (H) of a patient (P). In this example, a navigation wire (40) is inserted into the head (H) of the patient (P). The navigation wire (40) may be a stand-alone device or may be positioned at other locations on an end effector or medical device, such as a surgical cutting instrument or dilator. In this example, the frame (22) is mounted to a seat (30) in which the patient (P) sits, such that the frame (22) is located near the head (H) of the patient (P). By way of example only, the seat (30) and / or field generator assembly (20) may be configured and operated in accordance with at least some of the teachings in U.S. Publication 2018 / 0310886 entitled “Apparatus to Secure Field Generating Device to Chair”, published on November 1, 2018, the disclosure of which is incorporated herein by reference.

[0113] The IGS navigation system (10) of this example also includes a processor (12) that controls the field generator (24) and other components of the IGS navigation system (10). For example, the processor (12) is operable to drive the field generator (24) to generate an alternating electromagnetic field; and to process signals from the navigation wire (40) to determine the position of sensors in the navigation wire (40) within the head (H) of the patient (P). The processor (12) includes processing units (e.g., a set of electronic circuits arranged to evaluate and execute software instructions using combinational logic circuits or other similar circuits) that communicate with one or more memories. The processor (12) of this example is mounted in a console (18) that includes operating controls (14) including a keyboard and / or pointing devices such as a mouse or trackball. During the execution of surgical procedures, the physician interacts with the processor (12) using the operating controls (14).

[0114] The navigation wire (40) includes a sensor (not shown) that positions itself in response to an alternating magnetic field generated by a field generator (24). A coupling unit (42) is attached to the proximal end of the navigation wire (40) and configured to provide communication of data and other signals between the console (18) and the navigation wire (40). The coupling unit (42) can provide wired or wireless communication of data and other signals.

[0115] In this example, the sensor for the navigation wire (40) includes at least one coil located at the distal end of the navigation wire (40). When such a coil is positioned within an alternating electromagnetic field generated by a field generator (24), the alternating magnetic field can generate a current in the coil, and this current can be transmitted along an electrical conduit in the navigation wire (40) and further transmitted to the processor (12) via a coupling unit (42). This phenomenon enables the IGS navigation system (10) to determine the position of the distal end of the navigation wire (40) or other medical devices (e.g., dilatation instruments, surgical cutting instruments, etc.) in three-dimensional space (i.e., within the head (H) of the patient (P)). To achieve this, the processor (12) executes an algorithm to calculate the position coordinates of the distal end of the navigation wire (40) based on the orientation correlation signal of the coil in the navigation wire (40). Although the orientation sensor is located in the wire (40) in this example, such orientation sensors can be integrated into a variety of other types of devices, including those described in more detail below.

[0116] The processor (12) uses software stored in its memory to calibrate and operate the IGS navigation system (10). Such operations include driving the field generator (24), processing data from the navigation guidewire (40), processing data from the operation controls (14), and driving the display screen (16). In some implementations, operations may also include monitoring and enforcing one or more safety features or functions of the IGS navigation system (10). The processor (12) is also operable to provide video in real time via the display screen (16), which shows the orientation of the distal end of the navigation guidewire (40) relative to a video camera image of the patient's head (H), a scanned image of the patient's head (H) (e.g., CT, MRI, or other X-ray or indirect imaging methods), and / or a computer-generated three-dimensional model of the anatomical structures within and near the patient's nasal cavity. The display screen (16) may display such images simultaneously and / or overlay them on each other during the surgical procedure. Such display images may also include a graphical representation of an instrument (such as a navigation wire (40)) inserted into the patient's head (H), allowing the operator to view a virtual drawing of the instrument in its actual position in real time. By way of example only, the display (16) may provide images based on at least some of the teachings in U.S. Publication 2016 / 0008083 entitled “Guidewire Navigation for Sinuplasty”, published on January 14, 2016, the disclosure of which is incorporated herein by reference. Endoscopic images may also be provided on the display (16) if the operator is also using an endoscope.

[0117] When the instrument is combined with the navigation guidewire (40), the images provided by the display screen (16) can help guide the operator to turn and further manipulate such instruments within the patient's head (H). It should also be understood that other components of the surgical instruments described below and other types of surgical instruments can be combined with sensors such as those of the navigation guidewire (40).

[0118] Endoscope with single-camera 3D imaging

[0119] Figure 8An exemplary endoscope (100) according to an embodiment of the present invention is shown, which can be used with a surgical navigation system or other surgical system such as an IGS navigation system (10). The endoscope (100) includes a body (102) that can be held during use. A set of controls (103) is positioned at a distal end of the body (102). An axis (104) extends distally from the body (102). The axis (104) includes a flexure portion (108) and a distal end (106). As will be described in more detail below, the distal end (106) may include one or more feature structures, such as a camera, a light source, irrigation and aspiration, and / or a channel through which flexible tools can be deployed. The endoscope (100) also includes an orientation sensor (110) located between the flexure portion (108) and the distal end (106). The orientation sensor (110) may include one or more coils configured to generate an orientation indication signal in response to an alternating electromagnetic field generated by the field generator 24, such that the navigation system (10) can track the three-dimensional orientation of the distal end (106) within the head (H) of the patient (P) in real time, similar to the orientation tracking of the navigation guidewire (40) described above.

[0120] The control (103) is operable to drive rotation of the shaft (104) about the longitudinal axis of the shaft (104). The control (103) is also operable to drive deflection of the distal end (106) away from the longitudinal axis of the shaft (104) at the flexure portion (108). By way of example only, the control circuit (103) may be configured and operated according to at least some of the teachings in U.S. Publication 2019 / 0015645 entitled “Adjustable Instrument for Dilation of Anatomical Passageway”, published January 17, 2019, the disclosure of which is incorporated herein by reference. By way of further example only, the endoscope (100) may also include one or more controls for activating characteristic structures of the endoscope (100); or controls for advancing, retracting, and rotating a flexible tool deployed via the endoscope (100) to the surgical site; or any other type of control that will be apparent to a person skilled in the art from the teachings herein.

[0121] Figure 9A and Figure 9BExemplary distal ends (300) of the distal ends (106) of an endoscope (100) according to embodiments of the present invention are shown. The distal end (300) may be statically positioned at the distal end of a shaft (104); or it may be positioned at the distal end of a flexible or deflectable shaft extending from or through a channel of the distal end (106). The distal end (300) includes an instrument channel (302) through which flexible instruments and tools can be deployed once the distal end (300) is positioned in the desired location. By way of example only, the channel (302) may be sized to accommodate dilation catheters, shaving instruments, ablation instruments, and / or various other types of instruments that will be apparent to those skilled in the art from the teachings herein. The distal end 300 also includes a wavefront imaging device (304) and a set of lamps (306) (e.g., LEDs, optical fibers, infrared light sources, etc.). The distal end (300) also includes a set of flush deflectors (308). Each flush deflector (308) is positioned proximally to a corresponding flush channel (e.g., Figure 10A and Figure 10B The two figures show the flushing channel (310) visible in the diagram, and each figure shows the distal end (300) of one or both flushing deflectors (308) removed. The flushing deflectors (308) are adapted to receive and direct water or other liquid from the flushing channel (310) to wash the wavefront imaging device (304), the lamp (306), or both.

[0122] The wavefront imaging device (304) may include a digital camera (e.g., one or more image sensors) or imaging device capable of acquiring image data and transmitting it to a processor (12); or another device capable of storing and processing image data to generate 3D image data for an IGS navigation system (10). While some conventional 3D image acquisition devices may require two or more offset imaging devices capable of actively and independently focusing on a target, the wavefront imaging device (304) of this example is capable of generating 3D image data from a single camera or image sensor using wavefront sampling techniques. To generate image data compatible with wavefront sampling techniques, the wavefront imaging device (304) includes one or more apertures positioned between the target and the camera lens; and is off-axis or off-axis of the lens. When imaging a target using the wavefront imaging device (304), a single image containing multiple out-of-focus depictions of the target is generated, which are offset from each other based on the orientation of the one or more apertures. The distance between each out-of-focus image can be used to calculate the depth or distance to the imaged target. Images acquired by the wavefront imaging device (304) can be provided as input to wavefront sampling algorithms (such as Frigerio's multi-image program) to generate 3D depth maps and 3D models of the imaging object.

[0123] Figure 11 An example of a wavefront imaging apparatus (304) implementing a static optical system (320) according to an embodiment of the present invention is shown. The static optical system (320) includes a lens (326) having a first static aperture (328a) and a second static aperture (328b) positioned between a lens (326) and a target (330). The apertures (328a, 328b) are diametrically opposed and offset from the optical axis of the lens (326). Two or more apertures can be used, with a larger number of apertures improving the accuracy of the resulting 3D image, but requiring additional time for target imaging and image data processing. A smaller number of apertures can shorten the time required for imaging and processing, but at the cost of a loss of accuracy in the resulting 3D image. The aperture size can be increased or decreased to control the depth of field of the acquired image. Decreasing the distance between the aperture and the lens (326) can also produce improved wavefront sampling results.

[0124] During imaging, the lens (326) receives a first out-of-focus image (332a) and a second out-of-focus image (332b) of the target (330) through the aperture (328a, 328b), and guides the first out-of-focus image (332a) and the second out-of-focus image (332b) onto the image plane (322). The image plane (322) may be provided by a CCD, CMOS or other image sensor. The image offset (336) between the resulting images (332a, 332b) can then be determined and used as part of a wavefront sampling technique to determine pixel depth and generate a 3D model of the target (330).

[0125] Figure 12Another example of a wavefront imaging device (304) implemented as an active optics system (340) according to an embodiment of the present invention is shown. The active optics system (340) includes a lens (326) having a single aperture (344a) positioned on a rotatable aperture plate (342) between the lens (326) and the target (330). The aperture plate (342) can be rotated electronically and automatically during imaging to move the single aperture (344a) from its indicated orientation to a second orientation (344b), a third orientation (344c), etc., throughout the rotation. In this example, the aperture plate (342) rotates about a rotation axis coaxial with the optical axis of the lens (326). As with the static optics system (320), the lens (326) receives an unfocused image of the target (330) during the rotation of the aperture plate (342). Therefore, the first unfocused image (346a), the second unfocused image (346b), the third unfocused image (346c), etc., are projected onto the image plane (322) at offset locations along the rotation path (348). This image data acquired by the image plane (322) can then be used with wavefront sampling techniques to determine pixel depth and generate a 3D model of the target (330).

[0126] Figure 13 A set of exemplary steps (400) according to an embodiment of the invention are shown, which can be performed to generate a three-dimensional image model using an imaging device such as imaging device 304. Image data can be generated as described above. Figure 11 and Figure 12 The data is acquired in the manner described in the context (box 402) and processed into 3D image data by applying a wavefront sampling algorithm (e.g., Frigerio's multi-image procedure or another similar algorithm). For example, such a process may include defining an anchor image to serve as the basis for determining the offset of other images (box 404). The anchor image may be an arbitrarily chosen image, i.e., the image most centrally located relative to two or more other images; or it may be chosen based on other criteria. Optical flow methods can then be used to calculate the pixel displacement between the anchor image and other images (box 406). A circle may be fitted based on points across the entire image (box 408), and a rotation diameter (or other offset value) may be determined (box 410). The rotation diameter can then be used to calculate the pixel depth of multiple pixels in the image (box 412). The pixel depth can then be used to construct a 3D model of the imaged object or target (box 414).

[0127] The distal end (300) and wavefront imaging device (304) are advantageous because they provide a solution for 3D imaging of the surgical site with relatively low complexity (e.g., compared to a 3D camera with multiple independent and actively oriented cameras); and they can be mounted on the end of the instrument, such as at the distal end (106), or on a flexible or deflectable axis. The advantage of reduced complexity is particularly realized when the wavefront imaging device (304) is implemented using a static optical system (320), which can be implemented with small physical size requirements and minimal support requirements, due to low power consumption and the absence of moving parts (e.g., such as an aperture plate (342) or independently oriented cameras).

[0128] As described above, the endoscope (100) of this example includes an orientation sensor (110) operable to provide a signal indicating the real-time orientation of the distal end (106) in three-dimensional space. When the imaging device (304) acquires images, image data from the imaging device (304) can be stored together with orientation data from the orientation sensor (110), allowing the IGS navigation system (10) (and / or some other computer system) to associate each acquired image with its corresponding location in three-dimensional space. This enables the IGS navigation system (10) to associate images acquired using the imaging device (304) with one or more preoperative images (e.g., CT or MRI scans, 3D mapping, etc.). This aspect of the invention, together with the dual-camera 3D imaging described below, can be used in conjunction with the image correction aspect of the invention described above.

[0129] Endoscope with dual-camera 3D imaging

[0130] Figure 14A and Figure 14BEach of the following illustrates another exemplary distal end (500) that can be used as a distal end (106) of an endoscope (100) according to other embodiments of the invention. The distal end (500) may be statically positioned at the distal end of a shaft (104); or it may be positioned at the distal end of a flexible or deflectable shaft that extends from or can be advanced and retracted through a channel of the distal end (106). The distal end (500) includes an instrument channel (502) having similar features and functions to the instrument channel (302); a lamp (506) having similar features and functions to the lamp (306); and a set of flushing channels (510) having similar features and functions to the flushing channel (310). The distal end (500) also includes a first imaging device (504a) and a second imaging device (504b), which may include a digital camera (e.g., one or more image sensors) or other digital imaging device, a coupling unit (42), or another device of an IGS navigation system (10) capable of acquiring image data at the distal end (500) and transmitting it via an axis or flexible axis to a processor (12). Reference Figure 14B As can be seen, the imaging devices (504a, 504b) are offset from each other along the first dimension (512) and are located in the same orientation along the second dimension (514).

[0131] Figure 15A and Figure 15B A schematic view of a distal end (500) for imaging a target using passive stereo vision, an embodiment of the invention, is shown. Passive stereo vision is a triangulation method for acquiring images from two or more different viewpoints using two or more imaging devices. Images generated using passive stereo vision can be converted into 3D image data algorithms, such as photogrammetric algorithms that determine pixel depth based on epipolar geometry. Figure 15A As can be seen, the first imaging device (504a) and the second imaging device (504b) are arranged on the distal end (500) so that they have parallel optical axes. When guided to the target (520), the first imaging device (504a) will acquire the first field of view (522) of the target (520), while the second imaging device (504b) will acquire the second field of view (526) of the target (520).

[0132] Because the two imaging devices (504a, 504b) have parallel optical axes (e.g., not independently focused or oriented to a convergence point), the images acquired by these devices may contain overlapping portions of the target (520). For example, Figure 15A and Figure 15BA first field of view (522), a second field of view (526), ​​and a shared field of view (524) are shown, in which the target (520) will be acquired by two imaging devices. Reference Figure 15B As can be seen, the target (520) is positioned essentially centered within the shared field of view (524). However, the target (520) is positioned to the right of the first field of view (522) and to the left of the second field of view (526). This parallax, or the distance by which the target (520) is offset or shifted from the first field of view (522) to the second field of view (526), ​​can be used to determine the distance between the imaging device and the target, and subsequently the depth of pixels or portions of the acquired image can be determined.

[0133] Examples of determining and using parallax to define depth for passive triangulation. Figure 16 An exemplary set of steps (600) for applying passive triangulation to image data according to an embodiment of the invention is shown. Image data of the target can be acquired (box 602) by two or more statically arranged cameras, such as a first imaging device (504a) and a second imaging device (504b), thereby generating two different images of the target from two different orientations. Points (e.g., one or more pixels or portions of the image) can be picked up within one image (box 604), and matching points (e.g., a portion of the target, such as target (520), shared between the two images) are searched in the second image. Figure 9B (As shown). Performing image analysis on points within the first image to find matching points throughout the second image can be both inefficient and inaccurate, resulting in high processing demands and an increased likelihood of false positive matches.

[0134] To improve the speed and accuracy of matching, the characteristics of the epipolar lines associated with two images can be determined (box 606). Epipolar geometry is a concept that can be applied to two images acquired by a camera with parallel optical axes to determine the parallax or distance between two matching pixels or points in the images without having to search the entire image for matching. Figure 17 and Figure 19 An illustration is shown according to an embodiment of the invention, illustrating a polar geometry used in stereoscopic imaging.

[0135] refer to Figure 17The illustration (700) shows the relationship between parallax (e.g., the distance between two matching points in an image) and depth (e.g., the distance from the camera to the target or a pixel or point within the target), with a first camera (704) and a second camera (706) arranged such that they have parallel optical axes (e.g., lines FB and GD) passing through their respective lenses, which are arranged on the same lens plane (703). Using this arrangement, target A (702) lies within the field of view of each camera. The distance between the optical axes of the cameras is BC+CD, and triangles ACB and BFE have similar characteristics but different dimensions. The triangular geometry is applied... Figure 17 The displacement between two matching points or pixels in an image can be expressed as the distance between the camera optical axes multiplied by the distance from the lens plane (703) to the image sensor divided by the depth or distance to the target (702), or displacement = (BD*BF) / AC. After determining the displacement or parallax, this formula can alternatively be used to determine or solve for the distance to the target (702) or AC.

[0136] To determine the displacement between points or pixels in an image without searching the entire image, one can apply... Figure 18 The principle of epipolar geometry is illustrated. Since the cameras (704, 706) have fixed orientations and parallel optical axes relative to each other, an epipolar line (712) extending across the first image (708) and the second image (710) can be determined, along which matching points or pixels can be found. Therefore, when matching points on the first image (708) with points on the second image (710), it is only necessary to search for portions of the second image (710) that fall on the epipolar line (712), rather than the entire second image (710). By performing matching in this way, the speed and accuracy of matching points across the entire image can be significantly improved.

[0137] Following the matching points (box 608), by applying epipolar geometry or using an alternative matching method, the displacement or parallax between the matching points can be determined (box 610), which can then be used to determine the depth or distance between the imaging device and the target (box 612). Once the depth can be determined (box 612), a 3D image dataset and a 3D model can be constructed (box 614).

[0138] As described above, the endoscope (100) of this example includes an orientation sensor (110) operable to provide a signal indicating the real-time orientation of the distal end (106) in three-dimensional space. When the imaging devices (504a, 504b) acquire images, image data from the imaging devices (504a, 504b) can be stored together with orientation data from the orientation sensor (110), allowing the IGS navigation system (10) (and / or some other computer system) to associate each acquired image with its corresponding location in three-dimensional space. This enables the IGS navigation system (10) to associate images acquired using the imaging devices (504a, 504b) with one or more preoperative images (e.g., CT or MRI scans, 3D mapping, etc.).

[0139] Methods for integrating 3D imaging and navigation

[0140] After generating the 3D image data as described above, such image data can be used to provide additional features during IGS navigation. For example, Figure 19 A flowchart illustrating a set of steps (800) that can be performed to integrate a 3D model or 3D image data with IGS navigation according to an embodiment of the invention is shown. After acquiring image data (box 802) using appropriate imaging modules (e.g., waveform imaging device (304) at the distal end (300), a group of cameras (504a, 504b) at the distal end (500)) and constructing a 3D model or other 3D image dataset (box 804), such data can be used in various ways. For example, a synthetic image set can be created by combining 3D image data with other images and data-associated image sets using location data such as orientation and orientation (box 806). This may include, for example, associating the constructed 3D model (box 804) with a set of CT images acquired preoperatively, a set of 2D images acquired during the procedure using an orientation-tracking 2D endoscope, or other similar information, so that matching perspectives of the surgical site can be displayed simultaneously from multiple image sets.

[0141] For example, a preoperative comparison (box 808) can be created, which compares a preoperatively acquired CT scan or other 3D model with a constructed 3D model (box 804). This can be used to assist in planning or preparing surgical procedures, or to verify the accuracy and configuration of one or more 3D models. For instance, comparing the constructed 3D model (box 804) with a set of CT images can help identify missing or incorrect image or depth data in one or both models, incorrectly correlated positional data in one or both models, or other errors that can be corrected by re-imaging to generate new 3D image data or by reconfiguring or recalibrating to correct positional data.

[0142] For example, a postoperative comparison (box 810) can be created, comparing postoperative CT scans, 3D models, or other image data with a constructed 3D model (box 804) to aid in assessing the success of the surgical procedure. The 3D model constructed before surgery (box 804) can be compared with the 3D model constructed after surgery (box 804); or with postoperative CT scans or 2D imaging of the surgical site. Such comparative data can help clinicians assess the success or completeness of the surgical procedure by physically comparing anatomical structures from different perspectives available in 3D imaging.

[0143] By way of further example only, 3D image data acquired or otherwise generated by the distal end (200) or distal end (300) may be used in accordance with the teachings of at least some of the following patents: U.S. Provisional Patent Application 62 / 782,608, filed December 20, 2018, entitled “3D Scanning of Nasal Tract with Deflectable Endoscope,” the disclosure of which is incorporated herein by reference; U.S. Patent 8,199,988, published June 12, 2012, entitled “Method and Apparatus for Combining 3D Dental Scans with Other 3D Data Sets,” the disclosure of which is incorporated herein by reference; and / or U.S. Patent 8,821,158, published September 2, 2014, entitled “Method and Apparatus for Matching Digital Three-Dimensional Dental Models with Digital Three-Dimensional Cranio-Facial CAT Scan Records,” the disclosure of which is incorporated herein by reference.

[0144] Exemplary combinations

[0145] The following examples illustrate various non-exhaustive ways in which the teachings herein can be combined or applied. It should be understood that the following examples are not intended to limit the scope of any claims that may be provided at any time in this patent application or a subsequent filing thereof. No disclaimer is intended. The following examples are provided merely for illustrative purposes. It is contemplated that the various teachings herein may be arranged and applied in a variety of other ways. It is also contemplated that some variations may omit certain features mentioned in the following examples. Therefore, none of the aspects or features mentioned below should be considered definitive unless otherwise expressly indicated, for example, by the inventor or a successor of the inventor of interest, at a later date. If any claim set forth in this patent application or a subsequent filing related to this patent application includes additional features beyond those mentioned below, such additional features should not be presumed to have been added for any reason related to patentability.

[0146] Example 1

[0147] A three-dimensional (3D) imaging system includes: an endoscope comprising: an axis having a distal end, the axis being adapted for insertion into a patient and positioned at a surgical site on the patient; an orientation sensor proximate to the distal end and configured to generate an orientation signal set based on the position of the endoscope during use; an imaging module positioned at the distal end and operable to acquire an image dataset of the surgical site, wherein the image dataset includes one or more two-dimensional (2D) images; and a processor communicatively coupled to the endoscope and configured to: receive the image dataset and the orientation signal set from the endoscope; determine a perspective dataset based on the orientation signal set, wherein the perspective dataset indicates the position of the endoscope during the acquisition of the image dataset; perform image depth analysis to determine a 3D feature set of the image dataset, wherein the 3D feature set includes pixel depths in the one or more 2D images; create a 3D image dataset based on the one or more 2D images and the 3D feature set; and associate the perspective dataset with the 3D image dataset.

[0148] Example 2

[0149] According to the 3D imaging system of Embodiment 1, the imaging module includes: a single lens; an aperture plate positioned between a first side of the single lens and the surgical site, the aperture plate including one or more apertures offset from the optical axis of the single lens; and an image pane positioned on a second side of the single lens to receive reflected light from the surgical site via the one or more apertures and the single lens, wherein the image pane is configured to generate the image dataset based on the reflected light.

[0150] Example 3

[0151] According to the 3D imaging system of Embodiment 2, wherein: the one or more apertures include at least two apertures positioned on the aperture plate and offset from the optical axis of the single lens, and the aperture plate has a fixed orientation and orientation relative to the lens.

[0152] Example 4

[0153] According to one or more of Embodiments 2 to 3, in a 3D imaging system, the one or more apertures include a single aperture offset from the optical axis of the single lens and positioned on the aperture plate, and the aperture plate is operable to rotate relative to the lens about its circular axis during image acquisition.

[0154] Example 5

[0155] According to one or more of the 3D imaging systems described in Embodiments 2 to 4, wherein the processor is configured to, when performing the image depth analysis,: identify two or more unfocused images of the surgical site within the image dataset; determine the spatial relationship between the two or more unfocused images of the surgical site; and determine the pixel depth in the image dataset based on the spatial relationship between the two or more unfocused images.

[0156] Example 6

[0157] According to any one or more of the embodiments 1 to 5, the 3D imaging system wherein the imaging module includes two or more cameras, and wherein each of the two or more cameras is statically positioned relative to every other camera among the two or more cameras and is oriented to have an optical axis parallel to every other camera among the two or more cameras.

[0158] Example 7

[0159] According to the 3D imaging system of Embodiment 6, wherein the processor is further configured to, when performing the image depth analysis: identify points in a first image of the image dataset, wherein the points include portions of the surgical site present in both the first image acquired by a first camera of the two or more cameras and a second image acquired by a second camera of the two or more cameras; identify the points in the second image; determine the displacement of the points from the first image to the second image; and determine the pixel depth of the points based on the displacement.

[0160] Example 8

[0161] According to the 3D imaging system of Embodiment 7, wherein the processor is further configured to, when identifying the point in the second image: determine the epipolar lines of the first image and the second image based on the static position of the first camera relative to the second camera; and search for the point in the second image along the epipolar lines while excluding portions of the second image that do not fall on the epipolar lines.

[0162] Example 9

[0163] According to any one or more of Embodiments 1 to 8, the 3D imaging system wherein the processor is further configured to: associate the 3D image dataset and the viewpoint dataset with the coordinate system of the image-guided surgery system, and display the 3D image dataset during the image-guided surgery navigation procedure based on the association with the coordinate system.

[0164] Example 10

[0165] According to one or more of Embodiments 1 to 9, the 3D imaging system wherein: the orientation sensor is configured to generate the orientation signal set based on the position and orientation of the endoscope during use, the viewpoint dataset indicates the position and orientation of the endoscope during the acquisition of the image dataset, and the processor is further configured to provide the 3D image dataset and the viewpoint dataset to an image-guided surgical navigation system.

[0166] Example 11

[0167] According to one or more of the 3D imaging systems described in Embodiments 1 to 10, wherein the processor is further configured to: receive input from a user defining a viewpoint relative to the surgical site; determine a first portion of the 3D image dataset showing the surgical site from the viewpoint based on the viewpoint identified within the viewpoint dataset; and display the first portion of the 3D image dataset on a display.

[0168] Example 12

[0169] According to the 3D imaging system of Embodiment 11, the processor is further configured to: receive an indirect 3D scan of the surgical site and a scan view dataset associated with the indirect 3D scan; determine a second portion of the indirect 3D scan showing the surgical site from the scan view dataset based on identifying the view within the scan view dataset; and simultaneously display the first portion of the 3D image dataset and the second portion of the indirect 3D scan on the display.

[0170] Example 13

[0171] According to the 3D imaging system of Embodiment 12, the indirect 3D scan of the surgical site includes image data acquired before surgery, and the 3D image dataset includes image data acquired after surgery.

[0172] Example 14

[0173] According to one or more of Embodiments 12 to 13, the 3D imaging system wherein: the indirect 3D scan of the surgical site includes preoperative image data, the 3D image dataset includes preoperative image data, and the processor is further configured to: receive scan adjustment input from a user; and reconfigure the association between the indirect 3D scan of the surgical site and the scan view dataset based on the scan adjustment input.

[0174] Example 15

[0175] A method for three-dimensional (3D) imaging includes: deploying a distal end of an endoscope to a surgical site of a patient, the distal end comprising: an imaging module operable to acquire image data of the surgical site, wherein the acquired image data includes one or more two-dimensional (2D) images; and an orientation sensor proximal to the distal end and configured to generate an orientation signal based on the position of the endoscope; receiving an image dataset from the imaging module and an orientation signal set from the orientation sensor; determining a perspective dataset based on the orientation signal set, wherein the perspective dataset indicates the position of the endoscope during the acquisition of the image dataset; performing image depth analysis to determine a 3D feature set of the image dataset, wherein the 3D feature set includes pixel depths in the one or more 2D images of the image dataset; creating a 3D image dataset based on the one or more 2D images of the image dataset and the 3D feature set; and associating the perspective dataset with the 3D image dataset.

[0176] Example 16

[0177] The method according to Embodiment 15 further includes: associating the 3D image dataset and the view dataset with the coordinate system of the image-guided surgery system; and displaying the 3D image dataset during the image-guided surgery navigation procedure based on the association with the coordinate system.

[0178] Example 17

[0179] The method according to any one or more of Embodiments 15 to 16 further includes: receiving input from a user defining a viewpoint relative to the surgical site; determining a first portion of the 3D image dataset showing the surgical site from the viewpoint based on the viewpoint identified within the viewpoint dataset; and displaying the first portion of the 3D image dataset on a display.

[0180] Example 18

[0181] The method according to embodiment 17 further includes: receiving an indirect 3D scan of the surgical site and a scan view dataset associated with the indirect 3D scan; determining a second portion of the indirect 3D scan showing the surgical site from the scan view dataset based on identifying the view within the scan view dataset; and simultaneously displaying the first portion of the 3D image dataset and the second portion of the indirect 3D scan on the display.

[0182] Example 19

[0183] The method according to embodiment 18 further includes: receiving scan adjustment input from a user; and reconfiguring the association between the indirect 3D scan of the surgical site and the scan view dataset based on the scan adjustment input; wherein: the indirect 3D scan of the surgical site includes image data acquired before surgery, and the 3D image dataset includes image data acquired before surgery.

[0184] Example 20

[0185] An image-guided surgery (IGS) navigation system includes a processor, a memory, and a display. The processor is configured to: receive an image dataset generated by a tracking endoscope, the image dataset comprising one or more two-dimensional (2D) images; receive a perspective dataset generated by the tracking endoscope, wherein the perspective dataset indicates the position of the tracking endoscope during acquisition of the image dataset; perform image depth analysis to determine a 3D feature set of the image dataset, wherein the 3D feature set includes pixel depths in the one or more 2D images; create a 3D image dataset based on the one or more 2D images and the 3D feature set; associate the perspective dataset with the 3D image dataset; and cause the display to display the 3D image dataset from the selected perspective based on the perspective dataset including the selected perspective.

[0186] Example 21

[0187] A system includes: a medical probe configured for insertion into a cavity of a patient's organ; an orientation and direction sensor located in the medical probe, the orientation and direction sensor operating in a sensor coordinate system; a camera located at a distal edge of the medical probe, the camera operating in the sensor coordinate system; and a processor configured to: receive a three-dimensional image of the cavity, including an open space and organ tissue, from an imaging system operating in an image coordinate system; receive from the medical probe a signal indicating the orientation and corresponding direction of the distal edge of the medical probe within the cavity; receive a corresponding visualized location within the cavity from the camera of the probe; register the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualized location; and update the density value of the identified voxel to correspond to the open space when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space.

[0188] Example 22

[0189] The system according to Embodiment 21, wherein the imaging system includes a computed tomography scanner.

[0190] Example 23

[0191] According to the system of embodiment 21, the orientation and direction sensor includes a magnetic field sensor.

[0192] Example 24

[0193] According to any one of embodiments 21 to 23, the processor is configured to form a correspondence between the density value and the visual effect, wherein a given visual effect corresponds to a given density value indicating the open space.

[0194] Example 25

[0195] According to the system of embodiment 24, the visual effect is selected from a group of colors, shadows and patterns.

[0196] Example 26

[0197] According to the system of embodiment 24, the processor is configured to use the visual effects to present the three-dimensional image on a display.

[0198] Example 27

[0199] According to the system of embodiment 26, wherein the given visual effect includes a first given visual effect, and wherein, before updating the density value, the processor is configured to render the three-dimensional image by using a second given visual effect different from the first given visual effect to render the one or more identified voxels.

[0200] Example 28

[0201] According to the system of embodiment 27, when updating the density value, the processor is configured to render the three-dimensional image by using the first given visual effect to render the one or more identified voxels.

[0202] Example 29

[0203] According to the system of embodiment 28, wherein the processor is configured to use multi-view Figure 3 The angular segmentation model extracts the distance from the location to the camera from the visual signal.

[0204] Example 30

[0205] A method includes: receiving a three-dimensional image of a cavity of a patient's organ, comprising open space and organ tissue, from an imaging system operating in an image coordinate system; receiving from a medical probe having orientation and orientation sensors and a camera, wherein the probe operates in a sensor coordinate system and is inserted into the cavity; a signal indicating the orientation and corresponding direction of a distal edge of the medical probe within the cavity; and a corresponding visualization location within the cavity; registering the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualization location; and updating the density value of the identified voxel to correspond to the open space when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space.

[0206] Example 31

[0207] A computer software product that operates in conjunction with a probe configured for insertion into a cavity of a patient's organ and includes orientation and direction sensors operating in a sensor coordinate system and a camera operating in the sensor coordinate system at the distal edge of the medical probe. The product includes a non-transitory computer-readable medium storing program instructions that, when read by a computer, cause the computer to: receive a three-dimensional image of the cavity, including an open space and organ tissue, from an imaging system operating in an image coordinate system; receive signals from the medical probe indicating the orientation and corresponding direction of the distal edge of the medical probe within the cavity; receive a corresponding visualized location of the cavity wall; register the image coordinate system with the sensor coordinate system to identify one or more voxels in the three-dimensional image at the visualized location; and update the density value of the identified voxel to correspond to the open space when the identified voxel has a density value in the received three-dimensional image that does not correspond to the open space.

[0208] Miscellaneous

[0209] While the embodiments described herein are processors that use a camera to correct outdated CT images using magnetic P&D sensing and optical visualization, the use of other types of orientation sensing and visualization to correct other types of medical images is considered to be within the spirit and scope of this invention. For example, ultrasound visualization can be used with an ultrasound transducer instead of a camera. Furthermore, while the embodiments described herein primarily relate to ENT procedures, the methods and systems described herein can also be used in other applications, such as other cavities of organs in the body.

[0210] It should be understood that any one or more of the teachings, expressions, embodiments, examples, etc., described herein may be combined with any one or more of the other teachings, expressions, embodiments, examples, etc., described herein. Therefore, the aforementioned teachings, expressions, embodiments, examples, etc., should not be considered in isolation from each other. Various suitable ways in which the teachings herein can be combined will be apparent to those skilled in the art upon reference to the teachings herein. Such modifications and variations are intended to be included within the scope of the claims.

[0211] It should be understood that any patent, patent publication, or other public material allegedly incorporated herein by reference, whether in whole or in part, is incorporated only to the extent that the incorporated material does not conflict with any existing definitions, statements, or other public material set forth in this disclosure. Therefore, and to the extent necessary, the disclosures expressly listed herein replace any conflicting material incorporated herein by reference. Any material, or part thereof, allegedly incorporated herein by reference that conflicts with any existing definitions, statements, or other public material set forth herein will be incorporated only to the extent that the incorporated material does not conflict with any existing public material.

[0212] The devices disclosed herein may be designed for single-use disposal or for multiple-use. In either or both cases, these devices may be repaired for reuse after at least one use. Repair may include any combination of the following steps: disassembling the device, then cleaning or replacing specific parts, and subsequently reassembling. Specifically, the device may be disassembled, and any number of specific components or parts may be replaced or removed selectively in any combination. While cleaning and / or replacing specific parts, the device may be reassembled in a repair facility for subsequent use, or reassembled by a surgical team prior to surgery. Those skilled in the art will appreciate that device repair can utilize a variety of techniques for disassembly, cleaning / replacement, and reassembly. The use of such techniques and the resulting repaired devices are within the scope of this application.

[0213] By way of example only, the types described herein can be processed prior to surgical procedures. First, new or used instruments can be obtained and cleaned as needed. The instruments can then be sterilized. In one sterilization technique, the instruments are placed in a closed, sealed container (such as a plastic or TYVEK bag). The container and instruments can then be placed in a radiation field that can penetrate the container, such as gamma radiation, X-rays, or high-energy electrons. The radiation kills bacteria on the instruments and in the container. The sterilized instruments can then be stored in a sterile container. The sealed container keeps the instruments sterile until it is opened in the surgical facility. Any other techniques known in the art can also be used to sterilize the devices, including but not limited to beta or gamma radiation, ethylene oxide, or vapor.

[0214] While various embodiments of the invention have been shown and described, further improvements to the methods and systems described herein can be achieved by suitable modifications made by those skilled in the art without departing from the scope of the invention. Several such possible modifications have been mentioned, and other modifications will be apparent to those skilled in the art. For example, the examples, types, geometries, materials, dimensions, ratios, steps, etc., discussed above are exemplary and not essential. Therefore, the scope of the invention should be considered in accordance with the following claims and should be understood as not being limited to the details of the structures and operations shown and described in the specification and drawings.

Claims

1. A three-dimensional (3D) imaging system, comprising: (a) an endoscope, the endoscope comprising: (i) a shaft having a distal tip, the shaft being adapted to be inserted into a patient and positioned at a surgical site in the patient, (ii) a position sensor proximate the distal tip and configured to generate a set of position signals based on the position of the endoscope during use, and (iii) an imaging module positioned at the distal tip and operable to acquire an image dataset of the surgical site, wherein the image dataset comprises one or more two-dimensional (2D) images; and (b) a processor communicatively coupled to the endoscope and configured to: (i) receiving the image data set and the position signal set from the endoscope, (ii) determining a view angle dataset based on the set of position signals, wherein the view angle dataset indicates a position of the endoscope during acquisition of each of the one or more 2D images, (iii) performing image depth analysis to determine a 3D feature set for each of the one or more 2D images, wherein the 3D feature set comprises pixel depth, (iv) creating a 3D image dataset based on the one or more 2D images and the 3D feature set, and (v) associating the view dataset with the 3D image dataset; Wherein the image depth analysis comprises a technique selected from the group consisting of: (i) a wavefront sampling technique performed on a single image of the one or more 2D images, and (ii) a passive stereoscopic technique performed on a set of two images of the one or more 2D images, wherein each image in the set of two images is associated with the same viewing angle data.

2. The 3D imaging system according to claim 1, wherein the imaging module comprises: (i) Single lens, (ii) an aperture plate positioned between a first side of the single lens and the surgical site, the aperture plate comprising one or more apertures offset from an optical axis of the single lens, and (iii) an image pane positioned on a second side of the single lens to receive reflected light from the surgical site via the one or more apertures and the single lens, wherein the image pane is configured to generate the image dataset based on the reflected light.

3. The 3D imaging system according to claim 2, wherein: (i) the one or more apertures include at least two apertures, the at least two apertures are positioned on the aperture plate and are offset from the optical axis of the single lens, and (ii) The aperture plate has a fixed position and orientation relative to the lens.

4. The 3D imaging system according to claim 2 or 3, wherein: (i) the one or more apertures include a single aperture positioned on the aperture plate offset from the optical axis of the single lens, and (ii) the aperture plate is operable to rotate relative to the lens about a circular axis of the aperture plate during image acquisition.

5. The 3D imaging system according to any one of claims 2 to 4, wherein the processor is configured to: (i) identifying two or more out-of-focus images of the surgical site within the image dataset, (ii) determining a spatial relationship between the two or more unfocused images of the surgical site, and (iii) determining the pixel depth in the image dataset based on the spatial relationship between the two or more unfocused images.

6. The 3D imaging system according to any one of claims 1 to 5, wherein the imaging module comprises two or more cameras, and wherein each of the two or more cameras: (i) being statically positioned relative to every other camera of the two or more cameras, (ii) is oriented to have a parallel optical axis with every other camera of the two or more cameras.

7. The 3D imaging system of claim 6, wherein the processor is further configured to, when performing the image depth analysis: (i) identifying a point in a first image in the image dataset, wherein the point comprises a portion of the surgical site that is present in both the first image acquired by a first camera of the two or more cameras and a second image acquired by a second camera of the two or more cameras, (ii) identifying said point in said second image, (iii) determining the displacement of the point from the first image to the second image, and (iv) determining the pixel depth of the point based on the displacement.

8. The 3D imaging system of claim 7, wherein the processor is further configured to, when identifying the point in the second image: (i) determining epipolar lines of the first image and the second image based on the static orientation of the first camera relative to the second camera, and (ii) searching for the point in the second image along the epipolar line while excluding portions of the second image that do not fall on the epipolar line.

9. The 3D imaging system according to any one of claims 1 to 8, wherein the processor is further configured to: (i) associating the 3D image dataset and the view dataset with a coordinate system of an image-guided surgery system, and (ii) displaying the 3D image dataset during an image-guided surgery navigation procedure based on the association with the coordinate system.

10. The 3D imaging system according to any one of claims 1 to 9, wherein: (i) the position sensor is configured to generate the set of position signals based on the position and orientation of the endoscope during use, (ii) the view angle dataset indicates the position and orientation of the endoscope during acquisition of the image dataset, and (iii) the processor is further configured to provide the 3D image dataset and the view angle dataset to an image-guided surgical navigation system.

11. The 3D imaging system according to any one of claims 1 to 10, wherein the processor is further configured to: (i) receiving input from a user defining a viewing angle relative to the surgical site, (ii) determining, based on identifying the view angle within the view angle dataset, a first portion of the 3D image dataset showing the surgical site from the view angle, and (iii) displaying the first portion of the 3D image dataset on a display.

12. The 3D imaging system of claim 11, wherein the processor is further configured to: (i) receiving an indirect 3D scan of the surgical site and a scan view dataset associated with the indirect 3D scan, (ii) determining a second portion of the indirect 3D scan showing the surgical site from the viewing angle based on identifying the viewing angle within the scan viewing angle dataset, and (iii) simultaneously displaying the first portion of the 3D image dataset and the second portion of the indirect 3D scan on the display.

13. The 3D imaging system of claim 12, wherein: (i) the indirect 3D scan of the surgical site includes image data acquired before surgery, and (ii) The 3D image dataset includes image data acquired after surgery.

14. The 3D imaging system according to claim 12 or 13, wherein: (i) the indirect 3D scan of the surgical site includes image data acquired before the surgery, (ii) the 3D image data set includes image data acquired before surgery, and (iii) the processor is further configured to: (A) receiving scan adjustment input from a user, and (B) reconfiguring the association between the indirect 3D scan of the surgical site and the scan view dataset based on the scan adjustment input.

15. An image guided surgery (IGS) navigation system comprising a processor, a memory and a display, wherein the processor is configured to: (a) receiving an image dataset generated by a tracked endoscope, the image dataset comprising one or more two-dimensional (2D) images; (b) receiving a view data set generated by the tracking endoscope, wherein the view data set indicates a position of the tracking endoscope during acquisition of the image data set; (c) performing image depth analysis to determine a 3D feature set for the image dataset, wherein the 3D feature set comprises pixel depths in the one or more 2D images; (d) creating a 3D image dataset based on the one or more 2D images and the 3D feature set; (e) associating the viewing angle dataset with the 3D image dataset; as well as (f) causing the display to display the 3D image dataset from the selected viewing angle based on the viewing angle dataset including the selected viewing angle.

Images