2d path probe visualization

Abstract

In one embodiment, a medical device includes a medical instrument configured to move within a passageway in a patient's body; a position tracking system for tracking coordinates of the medical instrument within the body; a display screen; a processor that registers the position tracking system and a 3D computed tomography (CT) image of at least a portion of the body within a common frame of reference, finds a 3D path of the passageway through which the medical instrument passes from a start point to an end point within the common frame of reference, calculates a direction from the medical instrument to the 3D path in response to the tracked coordinates, and renders and concurrently displays on the display screen a respective 2D CT slice based on the 3D CT image, including a respective 2D indication of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slice.

Description

TECHNICAL FIELD

[0001] The present application relates to medical systems, and in particular, but not exclusively, to path visualization. BACKGROUND

[0002] In image guided surgery (IGS), a practitioner uses an instrument that is tracked in real time within the body, so that the position and / or orientation of the instrument can be rendered on images of the patient's anatomy during the surgical procedure. In many IGS cases, the images of the patient are prepared in one modality (e.g., magnetic resonance imaging (MRI) or computed tomography (CT)), and the instrument tracking uses a different modality, such as electromagnetic tracking. For the tracking to be effective, the frames of reference of the two modalities must be registered to each other.

[0003] U.S. Patent Publication 2011 / 0236868 to Bronstein et al. describes a method of performing a computerized simulation of an image-guided procedure. The method can include receiving medical image data of a particular patient. A patient-specific digital image-based model of the patient's anatomy can be generated based on the medical image data. The computerized simulation of the image-guided procedure can be performed using the digital image-based model. The medical image data, the image-based model, and a simulated medical tool model can be displayed simultaneously.

[0004] U.S. Patent Publication 2017 / 0151027 to Walker et al. describes systems and methods for driving a flexible medical instrument to a target in an anatomical space with robotic assistance. The flexible instrument can have a tracking sensor embedded therein. An associated robotic control system can be provided that is configured to register the flexible instrument with an anatomical image using data from the tracking sensor and to identify one or more moves suitable for navigating the instrument toward an identified target. In some embodiments, the robotic control system drives or assists in driving the flexible instrument to the target.

[0005] U.S. Patent Publication 2016 / 0174874 to Averbuch et al. describes a registration method whereby an initial registration is made using a sensor-based method, and whereby, upon starting navigation of an endoscope, an image-based registration method is used to more accurately maintain registration between the endoscope position and previously acquired images. A six-degree-of-freedom position sensor is placed on the probe in order to reduce the number of previously acquired images that must be compared to real-time images obtained from the endoscope.

[0006] U.S. Patent Publication 2005 / 0228250 to Bitter et al. describes a user interface that includes an image region divided into multiple views for viewing corresponding 2-dimensional and 3-dimensional images of an anatomical region. A tool control panel can be opened and accessed simultaneously. A segmentation panel enables automatic segmentation of components of a displayed image within a user-specified intensity range or based on a predetermined intensity.

[0007] U.S. Patent Publication 2007 / 0276214 to Dachille et al. describes an imaging system for automatic segmentation and visualization of medical images and includes an image processing module that automatically processes image data using a set of instructions that identify a target object in the image data and process the image data according to a specified protocol, a rendering module that automatically generates one or more images of the target object based on one or more of the instructions, and a digital archive for storing the one or more generated images. The image data can be DICOM-formatted image data, where the imaging processing module extracts and processes metadata in a DICOM field of the image data to identify the target object. The image processing module directs a segmentation module to segment the target object using processing parameters specified by one or more of the instructions.

[0008] U.S. Patent 5,371,778 to Yanof et al. describes a CT scanner that noninvasively examines a volumetric region of a subject and generates volumetric image data indicative of the volumetric region. An object memory stores data values corresponding to each voxel of the volumetric region. An affine transform algorithm operates on a visible face of the volumetric region to convert the face from object space to a projection of the face onto an observation plane in image space. An operator console includes operator controls for selecting an angular orientation of the projection image of the volumetric region relative to the observation plane (i.e., the plane of the video display). A cursor positioning trackball enters i and j coordinate positions in image space that are converted to a cursor crosshair display on the projection image. A depth dimension k between the observation plane and the volumetric region in an observation direction normal to the observation plane is determined. The (i,j,k) image space position of the cursor is operated on by inverting the selected transform for identifying corresponding (x,y,z) cursor coordinates in object space. The cursor coordinates in object space are converted to corresponding addresses of the object memory for traversing transaxial, coronal, and sagittal planes through the volumetric region.

[0009] U.S. Patent 10,188,465 to Gliner et al. describes a method that includes receiving a computed tomography scan of at least a portion of a patient’s body and identifying scan voxels that correspond to an in-vivo region traversable by a probe inserted therein. The method also includes displaying the scan on a screen and marking a selected origin and destination for the probe thereon. A processor finds a path from the origin to the destination that consists of a contiguous set of the identified voxels. The processor also generates a representation of an outer surface of the body using the scan and displays the representation on the screen. The processor then renders the outer surface region surrounding the path locally transparent in the displayed representation so as to make visible on the screen internal body structures in the vicinity of the path.

[0010] U.S. Patent Publication 2018 / 0303550 to Altmann et al. describes a method for visualization that includes registering a position tracking system of at least a portion of a patient’s body and a three-dimensional (3D) computed tomography (CT) image in a common frame of reference. A position and orientation of at least one virtual camera are specified in the common frame of reference. Coordinates of a medical tool moving within a passageway in the body are tracked using the position tracking system. A virtual endoscope image of the passageway in the body based on the 3D CT image is rendered and displayed from the specified position and orientation, including an animated representation of the medical tool positioned in the virtual endoscope image according to the tracked coordinates. SUMMARY

[0011] According to embodiments of the present disclosure, there is provided a medical apparatus that includes a medical instrument configured to move within a passageway in a body of a patient; a position tracking system configured for tracking coordinates of the medical instrument within the body; a display screen; a processor configured to register the position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body in a common frame of reference, find a 3D path of the medical instrument through the passageway from a given origin to a given destination in the common frame of reference, compute a direction from the medical instrument to the 3D path in response to the tracked coordinates, and render and concurrently display on the display screen respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slices.

[0012] According to embodiments of the present disclosure, the respective 2D indications of the direction from the medical instrument to the 3D path include respective arrows.

[0013] Also in accordance with embodiments of the present disclosure, the processor is configured to render and concurrently display, on the display screen, respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slices, and a representation of the medical instrument in response to the tracking coordinates.

[0014] Additionally, in accordance with embodiments of the present disclosure, the processor is configured to calculate, in response to the tracking coordinates, a direction from the medical instrument to a nearest point of the 3D path.

[0015] Further, in accordance with embodiments of the present disclosure, the processor is configured to calculate, in response to the tracking coordinates, a 3D vector from the medical instrument to the 3D path.

[0016] Also in accordance with embodiments of the present disclosure, the processor is configured to render and concurrently display, on the display screen, respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the 3D vector from the medical instrument to the 3D path projected onto the respective 2D CT slices.

[0017] Also in accordance with embodiments of the present disclosure, the processor is configured to render and concurrently display, on the display screen, three respective two-dimensional (2D) CT slices based on the 3D CT image, including three respective 2D indications of the direction from the medical instrument to the 3D path projected onto the three respective 2D CT slices.

[0018] Additionally, in accordance with embodiments of the present disclosure, the three respective 2D slices include a coronal view, a sagittal view, and a transverse view, respectively.

[0019] Further, in accordance with embodiments of the present disclosure, the processor is configured to calculate, in response to the tracking coordinates, a distance from the medical instrument to the 3D path, and to render and concurrently display, on the display screen, respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the direction and the distance from the medical instrument to the 3D path projected onto the respective 2D CT slices.

[0020] Also in accordance with embodiments of the present disclosure, the processor is configured to render and concurrently display, on the display screen, respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slices, and a virtual endoscopic image of the passageway in the body based on the 3D CT image viewed from at least one respective position of at least one respective virtual camera, including an animated representation of the medical instrument positioned in the virtual endoscopic image in accordance with the tracking coordinates, and a 3D indication of the direction from the medical instrument to the 3D path.

[0021] Also in accordance with embodiments of the present disclosure, the position tracking system comprises an electromagnetic tracking system including one or more magnetic field generators positioned around the body part and a magnetic field sensor located at a distal end of the medical instrument.

[0022] In accordance with another embodiment of the present disclosure, a medical method is also provided that includes tracking coordinates of a medical instrument within a body of a patient using a position tracking system, the medical instrument being configured to move within a passageway in the body of the patient, registering the position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body in a common frame of reference, finding a 3D path of the medical instrument through the passageway from a given starting point to a given end point in the common frame of reference, computing a direction from the medical instrument to the 3D path in response to the tracked coordinates, and rendering and concurrently displaying respective two-dimensional (2D) CT slices on a display screen based on the 3D CT image, including a respective 2D indication of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slice.

[0023] Additionally, in accordance with embodiments of the present disclosure, the respective 2D indication of the direction from the medical instrument to the 3D path includes a respective arrow.

[0024] Further, in accordance with embodiments of the present disclosure, the rendering and concurrent displaying includes rendering and concurrently displaying respective 2D CT slices on the display screen based on the 3D CT image, including a respective 2D indication of the 3D vector from the medical instrument to the 3D path projected onto the respective 2D CT slice, and a representation of the medical instrument in response to the tracked coordinates.

[0025] Also in accordance with embodiments of the present disclosure, the computing includes computing a direction from the medical instrument to a nearest point of the 3D path in response to the tracked coordinates.

[0026] Also in accordance with embodiments of the present disclosure, the computing includes computing a 3D vector from the medical instrument to the 3D path in response to the tracked coordinates.

[0027] Additionally, in accordance with embodiments of the present disclosure, the rendering and concurrent displaying includes rendering and concurrently displaying respective 2D CT slices on the display screen based on the 3D CT image, including a respective 2D indication of the 3D vector from the medical instrument to the 3D path projected onto the respective 2D CT slice.

[0028] Further, in accordance with embodiments of the present disclosure, the rendering and concurrent displaying includes rendering and concurrently displaying three respective two-dimensional (2D) CT slices on the display screen based on the 3D CT image, including three respective 2D indications of the direction from the medical instrument to the 3D path projected on the three respective 2D CT slices.

[0029] Additionally, according to embodiments of the present disclosure, the three respective 2D slices include a coronal view, a sagittal view, and a transverse view, respectively.

[0030] Still according to embodiments of the present disclosure, the method further comprises calculating a distance from the medical instrument to the 3D path in response to the tracked coordinates, and wherein rendering and concurrently displaying comprises rendering and concurrently displaying, on the display screen, respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the direction and distance from the medical instrument to the 3D path projected onto the respective 2D CT slices.

[0031] Additionally, according to embodiments of the present disclosure, rendering and concurrently displaying comprises rendering and concurrently displaying, on the display screen, respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slices, and virtual endoscopic images of the passageway in the body based on the 3D CT image viewed from at least one respective position of at least one respective virtual camera, including an animated representation of the medical instrument positioned in the virtual endoscopic images according to the tracked coordinates, and a 3D indication of the direction from the medical instrument to the 3D path.

[0032] Still according to another embodiment of the present disclosure, there is also provided a software product comprising a non-transitory computer readable medium having stored therein program instructions that, when read by a central processing unit (CPU), cause the CPU to track coordinates of a medical instrument within a body of a patient using a position tracking system, the medical instrument being configured to move within a passageway in the body of the patient, register the position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body in a common frame of reference, find a 3D path of the medical instrument through the passageway from a given starting point to a given end point in the common frame of reference, calculate a direction from the medical instrument to the 3D path in response to the tracked coordinates, and render and concurrently display, on a display screen, respective two-dimensional (2D) CT slices based on the 3D CT image, including respective 2D indications of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slices. BRIEF DESCRIPTION OF DRAWINGS

[0033] The present application will be understood more fully from the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1 is a partial block diagram view of a medical system constructed and operative in accordance with an embodiment of the present application;

[0035] Figure 2 is a flowchart of steps in a method of three-dimensional path visualization for use in an apparatus according to Figure 1 embodiments of the present application.

[0036] Figure 3 A method for finding a path through a passageway for a device including using Figure 1 A flowchart of steps of a method for finding a path through a passageway for a device including using

[0037] Figures 4-9 A schematic diagram showing steps of a method of a flowchart of Figure 3 A flowchart of steps of a method for finding a path through a passageway for a device including using

[0038] Figure 10 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 1 A flowchart of steps of a method for finding a path through a passageway for a device including using

[0039] Figure 11 A schematic diagram showing steps of a method of a flowchart of Figure 12 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 10 A flowchart of steps of a method for finding a path through a passageway for a device including using

[0040] Figure 13 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 1 A flowchart of steps of a method for finding a path through a passageway for a device including using

[0041] Figure 14 A schematic diagram showing steps of a method of a flowchart of Figure 15 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 13 A flowchart of steps of a method for finding a path through a passageway for a device including using

[0042] Figure 16 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 1 A flowchart of steps of a method for finding a path through a passageway for a device including using

[0043] Figure 17 A schematic diagram showing steps of a method of a flowchart of Figure 18 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 16 A flowchart of steps of a method for finding a path through a passageway for a device including using A flowchart of steps of a method for finding a path through a passageway for a device including using

[0044] A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 19 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 1 A flowchart of steps of a method for finding a path through a passageway for a device including using A flowchart of steps of a method for finding a path through a passageway for a device including using

[0045] A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 20 A schematic diagram showing steps of a method of a flowchart of Figure 19 A flowchart of steps of a method for finding a path through a passageway for a device including using A flowchart of steps of a method for finding a path through a passageway for a device including using

[0046] A schematic virtual endoscopic image rendered and displayed by a device of Figures 21-23 A schematic virtual endoscopic image rendered and displayed by a device of Figure 1 A flowchart of steps of a method for finding a path through a passageway for a device including using A flowchart of steps of a method for finding a path through a passageway for a device including using

[0047] A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 24 A flowchart of steps of a method for finding a path through a passageway for a device including using Figure 1 A flowchart of steps of a method for finding a path through a passageway for a device including using A flowchart of steps of a method for finding a path through a passageway for a device including using

[0048] Figure 25 is a schematic diagram showing Figure 24 a combination of 2D and 3D path visualizations of steps of the method of the flowchart of DETAILED DESCRIPTION

[0049] SUMMARY

[0050] During a medical procedure in a nasal passageway, such as a sinus dilation procedure, it is not possible to directly observe conditions in the sinus without inserting an endoscope into the sinus. However, insertion of an endoscope is problematic because of the tight spaces involved and the additional cost of the endoscope. Furthermore, endoscopes for nasal passageways are typically rigid instruments that cannot be steered or provide a view back from the sinus cavity to the sinus opening.

[0051] Embodiments of the invention described herein solve this problem by generating virtual endoscope views of the procedure from virtual cameras, similar to what an actual endoscope positioned at a virtual camera location within the nasal passageway would see. The virtual endoscope views show the anatomy and a medical instrument moving through the anatomy. As the medical instrument moves along the passageway, the virtual cameras used to generate the virtual endoscope views are passed from one virtual camera to another in response to tracked coordinates of the medical instrument.

[0052] These virtual endoscope views can be used, for example, to visualize the position and orientation of a guidewire relative to the anatomy and other instruments such as a suction tool or a shaving tool (a debrider).

[0053] Moving from one virtual camera to another provides a more stable view of the anatomy than placing a virtual camera on the distal tip of the medical instrument, from which the virtual camera would always "move" as the distal tip moves, causing the video of the anatomy to jump around or be choppy, which is difficult to track.

[0054] Furthermore, although the embodiments disclosed below are specifically directed to visualization within a nasal passageway, the principles of the invention can be similarly applied within other spaces of the body, particularly within narrow passageways where actual optical endoscopes are not available or difficult to use.

[0055] Prior to the medical procedure, CT images of the patient's head including the sinuses are acquired, as well as a position tracking system, such as an electromagnetic tracking system, registered to the CT images. A position sensor is attached to the distal end of the guidewire or other instrument, and thus tracks the position and orientation of the distal end relative to the registered CT images as the distal end is inserted into the sinus. The CT images of the head are processed to generate and display images of a 3D volume of the nasal passageway.

[0056] Within this 3D volume, an operator of the imaging system, such as a surgeon performing a sinus dilation procedure, can select a start point and an end point of a 3D path along which to navigate a medical instrument. A suitable 3D path from the start point to the end point is computed, for example using a path finding algorithm and data from the CT images, indicating which voxels of the CT images comprise material suitable to be traversed, such as air or liquid.

[0057] The computed 3D path is automatically divided into segments, and turning points between segments are above a threshold turning value. Virtual cameras are positioned around these turning points. If there is no line of sight between virtual cameras positioned at turning points and / or the distance between turning points exceeds a given value, additional virtual cameras can be automatically positioned.

[0058] The orientation of the optical axis of each virtual camera is computed. The orientation can be computed using any suitable method. In some embodiments, the orientation can be computed based on the average direction of the vector from the virtual camera to the location of the next virtual camera along the path. In other embodiments, the orientation can be computed as a direction parallel to the path at the location of the respective virtual camera. The field of view of the virtual camera can be fixed, for example to 90 degrees or any suitable value, or set according to the outer limits of the relevant segment of the path served by the respective virtual camera, with additional tolerance to allow deviation from the path.

[0059] In some embodiments, the position of the virtual camera can be offset backwards in the direction opposite to the computed average direction. The virtual camera can be offset backwards by any suitable distance, for example until the camera is offset backwards into solid material, such as tissue or bone. Offsetting the virtual camera backwards can result in a better view of the medical instrument within the respective virtual endoscope image, particularly when the medical instrument is very close to the respective virtual camera, and can result in a better view of the surrounding anatomy.

[0060] As the medical tool moves through the passageway, a respective virtual camera is selected for rendering and displaying a respective virtual endoscope image according to a camera selection method. In some embodiments, the camera selection method comprises finding the camera closest to the tracked coordinates of the medical instrument, and then finding on which side of the bisector (plane) associated with the closest camera the tracked coordinates fall. If the tracked coordinates fall on the side of the bisector further downstream from the computed path (in the direction of travel of the medical instrument) from the closest camera, the closest camera is selected for rendering. If the tracked coordinates fall on the side of the bisector of the closest current virtual camera, the current virtual camera continues to provide its endoscope image. The bisector associated with the closest camera can be defined as a plane perpendicular to the computed path at the point of the closest camera. In other embodiments, the passageway can be divided into a plurality of regions based on the segments, with the virtual camera being selected according to the region in which the tracked coordinates are disposed.

[0061] The transition between two virtual cameras, and thus the transition between the associated virtual endoscope images, can be a smooth transition or an abrupt transition. In some embodiments, a smooth transition between two respective virtual cameras can be performed by finding the positions of additional virtual cameras along the path between the two virtual cameras, and then successively rendering respective transition virtual endoscope images viewed from the additional virtual camera positions.

[0062] While 3D images can be used to help a physician navigate a medical instrument along a computed path, some physicians feel uncomfortable with 3D images for this purpose.

[0063] Embodiments of the present invention address the above problems by rendering and displaying two-dimensional (2D) CT slices of a CT image, e.g., 2D coronal, sagittal, and transverse slices of a 3D CT image, on a display, with the direction of medical instrument movement displayed on each 2D image in order to return the medical instrument to the computed path. In some embodiments, a 3D image showing a representation of the medical tool, such as a virtual endoscope image, can be displayed together with the 2D CT slices.

[0064] In some embodiments, a processor computes, in response to tracked coordinates of a medical instrument, a direction, and optionally a distance and / or a 3D vector, from the medical instrument to a 3D path, e.g., to the nearest point on the 3D path. The processor renders and concurrently displays, on a display screen, respective 2D CT slices based on a 3D CT image, including respective 2D indications, e.g., arrows or other symbols, of the direction, and optionally the distance, and / or the 3D vector, from the medical instrument to the 3D path projected onto the respective 2D CT slices, and optionally a representation of the medical instrument in response to the tracked coordinates. In some embodiments, the respective 2D slices include coronal, sagittal, and transverse views, respectively.

[0065] System Description

[0066] Referring now to Figure 1 is a partially schematic, partially block diagram view of portions of a medical device 20 constructed and operative in accordance with an embodiment of the present invention. In the following description, it is assumed that a medical instrument 21 of the device 20 is used to perform a medical procedure on a patient 22. The medical instrument 21 is configured to move within a passageway in the body of the patient 22.

[0067] The medical device 20 includes a position tracking system 23 configured to track the coordinates of the medical instrument 21 within the body. In some embodiments, the position tracking system 23 includes an electromagnetic tracking system 25 that includes one or more magnetic field generators 26 positioned around a body part and one or more magnetic field sensors 32 located at the distal end of the medical instrument 21. In one embodiment, the magnetic field sensors 32 include single-axis coils and dual-axis coils that act as magnetic field sensors and are tracked by the electromagnetic tracking system 25 during the procedure. In order for the tracking to be effective, in the device 20, a CT (computed tomography) image of the patient 22 is registered with the frame of reference of the electromagnetic tracking system 25, see Figure 2 and Figure 3 While the CT image can generally include a magnetic resonance imaging (MRI) image or a fluoroscopy image, in the description herein, by way of example, the image is assumed to include a fluoroscopy CT image. In some embodiments, the position tracking system 23 can use any suitable tracking method, such as tracking the coordinates of the medical instrument 21 based on current or impedance distribution on surface electrodes, or based on ultrasound transducers.

[0068] Prior to and during the sinus procedure, a magnetic radiator assembly 24 included in the electromagnetic tracking system 25 is positioned under the patient's head. The magnetic radiator assembly 24 includes magnetic field generators 26 that are fixed in place and transmit an alternating magnetic field into the area 30 in which the patient's 22 head is located. The potential generated by the single-axis coil of the magnetic field sensor 32 in the area 30 in response to the magnetic field enables the position and orientation of the single-axis coil to be measured in the frame of reference of the magnetic tracking system. The position can be measured in three linear dimensions (3D), and the orientation can be measured for two axes that are orthogonal to the axis of symmetry of the single-axis coil. However, the orientation of the single-axis coil relative to its axis of symmetry cannot be determined from the potential generated by the coil.

[0069] The same is true for each of the two coils of the dual-axis coil of the magnetic field sensor 32. That is, for each coil, the position in 3D can be measured, as can the orientation relative to two axes that are orthogonal to the coil's axis of symmetry, but the orientation of the coil relative to its axis of symmetry cannot be determined.

[0070] By way of example, the radiators 26 of the assembly 24 are arranged in a generally horseshoe shape around the head of the patient 22. However, alternative configurations of the radiators of the assembly 24 will be apparent to those skilled in the art, and all such configurations are considered to be included within the scope of the present invention.

[0071] Prior to surgery, the registration of the frame of reference of the magnetic tracking system and the CT image can be performed by positioning the magnetic sensor at a known location of the image, such as the end of the patient's nose. However, as noted in Figure 2 In more detail, any other convenient system for frame of reference registration can be used.

[0072] The elements of the apparatus 20, including the emitter 26 and the magnetic field sensor 32, are under the general control of a system processor 40. The processor 40 can be mounted in a console 50 that includes operating controls 58, which typically include a keypad and / or pointing device such as a mouse or trackball. The console 50 is connected to the emitter and magnetic field sensor 32 via one or more cables 60 and / or wirelessly. The physician 54 uses the operating controls 58 to interact with the processor 40 while using the apparatus 20 to perform a medical procedure. In performing the procedure, the processor can present the results of the procedure on a display screen 56.

[0073] The processor 40 operates the apparatus 20 using software stored in a memory 42. The software can be downloaded to the processor 40 in electronic form, for example over a network, or alternatively or additionally, the software can be provided and / or stored on a non-transitory tangible medium, such as a magnetic, optical, or electronic memory.

[0074] Reference is now made to Figure 2 which is a flowchart of steps in a three-dimensional path visualization method used in the apparatus 20 in accordance with an embodiment of the present application. Figure 1 The flowchart 70.

[0075] The position tracking system 23( Figure 1 ) is configured to track (block 72) coordinates of a distal end of the medical instrument 21( Figure 1 ) within the body. The processor 40( ) is configured to register (block 74) the position tracking system 23 and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body within a common frame of reference. The registration can be performed by any suitable registration technique. For example, but not limited to, the registration methods described in U.S. Patent Publication 2017 / 0020411 or 2019 / 0046272. As described in the latter patent publication, for example, the processor 40 can analyze the CT image to identify respective locations of the patient's eyes in the image, thereby defining line segments connecting these respective locations. In addition, the processor 40 identifies a subset of voxels in the CT that cover a bone portion of the head along a second line segment parallel to the first line segment and a third line segment perpendicular to the first line segment. The physician 54 positions the probe in the vicinity of the bone portion and thus measures locations on the surface of the head that cover the bone portion. The processor 40 computes a correspondence between these measured locations and the subset of voxels in the CT image and thus registers the magnetic tracking system 25 with the CT image.

[0076] The processor 40 is configured to find (block 76) a 3D path of the medical instrument 21 through a passageway from a given starting point to a given end point. The steps of block 76 are described in more detail with reference to the path finding method of Figures 3-9 .

[0077] The processor 40 is configured to compute (block 78) segments of the computed 3D path. The processor 40 is configured to compute (block 80) respective different positions along the 3D path of the respective virtual camera in response to the computed segments. The steps of block 78 and block 80 are described in more detail with reference to the path finding method of Figures 10-12 .

[0078] The processor 40 is configured to select (block 82) the respective virtual camera for rendering the respective virtual endoscope image in response to the tracking coordinates of the medical instrument 21 and the respective position of the respective virtual camera within the common frame of reference. As the medical instrument 21 moves along the 3D path (which can be at a distance from either side of the path as the medical instrument 21 is not locked to the path), the virtual camera providing the virtual endoscope image is selected in accordance with the tracking coordinates of the medical instrument 21 and the control is passed from one virtual camera to another virtual camera continuously as the medical instrument 21 moves along the path. The steps of block 82 can be repeated intermittently, for example, each time new tracking coordinates are received, for example, in the range between 10 milliseconds to 100 milliseconds, such as 50 milliseconds. The steps of block 82 are described in more detail with reference to the path finding method of Figures 13-15 .

[0079] The processor 40 is configured to compute (block 84) a respective orientation of the respective virtual camera. The orientation of the camera is typically a 3D orientation and is defined with respect to a respective optical axis of the respective virtual camera. In other words, the orientation of the camera is a measure of the direction in which the camera faces the optical target. The position of the virtual camera can also be offset backwards as described in detail with reference to Figure 16 and Figure 18 . The orientation and / or the backwards movement can be computed as part of the process of selecting the camera, either before the camera is selected or when the medical instrument 21 leaves the field of view of the virtual camera currently providing the virtual endoscope image.

[0080] The processor 40 is configured to display (block 86) the respective virtual endoscope image on the display screen 56 (Fig. 1) in response to the selection of the respective virtual camera. The steps of block 86 are described in more detail with reference to the path finding method of Figure 1) the respective virtual endoscope images of the passageway in the body observed from the respective positions and orientations of the respective virtual cameras, including an animated representation of the medical instrument 21 positioned in the respective virtual endoscope images according to the tracking coordinates. In other words, based on the position and orientation of the medical instrument 21 and the pre-acquired CT data, the processor 40 at the step of block 86 renders and presents the images on the display screen 56 as they would be captured by the respective virtual cameras. The image rendered by any given selected virtual camera is a projection onto the virtual image plane of the camera of the portion of the 3D volume that would be visible from the camera position. The step of block 86 refers to Figures 21-23 are described in more detail.

[0081] Reference is now made to Figure 3 which is a flowchart 90 of the steps of a method of finding a path through a passageway using the apparatus 20. Figure 1 Figures 4-9 is a schematic diagram showing the steps of the method of the flowchart. The pre-planning components described with reference to the flowchart 90 are typically implemented prior to performing an invasive surgical procedure on the patient 22 Figure 3 and determining an optimal path for subsequent invasive medical instruments 21 Figure 1 ) in the procedure. It is assumed that the pre-planning is performed by the physician 54 Figure 1 . Figure 1

[0082] In an initial step (block 100 of the flowchart 90), a computed tomography (CT) X-ray scan of the sinuses of the patient 22 is performed and the data from the scan is acquired by the processor 40. As is known in the art, the scan includes two-dimensional X-ray “slices” of the patient 22 and the combination of the slices results in a three-dimensional voxel, each voxel having a Hounsfield unit, which is a measure of radiodensity determined by the CT scan.

[0083] In an image generation step (block 102), the physician 54 Figure 1 displays the results of the scan on the display screen 56 Figure 1 . As is known in the art, the results can be displayed as a series of two-dimensional (2D) slices, typically along planes parallel to the sagittal, coronal and / or transverse planes of the patient 22, although other planes are possible. The direction of the planes can be selected by the physician 54.

[0084] The displayed results are typically grayscale images and an example is provided in Figure 4 which is a slice parallel to the coronal plane of the patient 22. The grayscale values from black to white can be correlated to the Hounsfield units (HU) of the corresponding voxels, such that when applied to Figure 4 ​​When the image is displayed, air having HU = -1000 can be assigned black, and dense bone having HU = 3000 can be assigned white.

[0085] As is known in the art, the value of the Hounsfield unit for any other material or species (such as dense bone), other than the values for air and water, which are by definition -1000 and 0, respectively, depends inter alia on the spectrum of the irradiating X-rays used to produce the CT scans referred to herein. In turn, the spectrum of the X-rays depends on many factors, including the potential applied to the X-ray generator (in kilovolts (kV)), and the composition of the anode of the generator. For the sake of clarity, in the present disclosure, the Hounsfield unit values for specific materials or species are as given in Table I below.

[0086] Species / Materials Hounsfield Units Air -1000 Soft Tissue -300 to -100 Fat -50 Water 0 Blood +30 to +45 Dense Bone +3000

[0087] However, the numerical values of the HU for the specific species given in Table I, other than air and water, should be understood to be purely exemplary, and one of ordinary skill in the art will be able to modify these exemplary values depending on the species and X-ray machine used to generate the CT images referred to herein, without undue experimentation.

[0088] In general, the translation between HU values and gray scale values is encoded into the DICOM (Digital Imaging and Communications in Medicine) file that is the output of a CT scan from a given CT machine. For the sake of clarity, in the description below, the correlation of HU = -1000 with black, HU = 3000 with white, and intermediate HU values with corresponding intermediate gray scale levels is used, but it should be understood that this correlation is arbitrary. For example, the correlation can be "reversed", i.e., HU = -1000 can be assigned to white, HU = 3000 to black, and intermediate HU values to corresponding intermediate gray scale levels. Thus, one of ordinary skill in the art will be able to adjust the description herein to include other correlations between Hounsfield units and gray scale levels, and it is assumed that all such correlations are encompassed within the scope of the present invention.

[0089] In a marking step (block 104), the physician 54( Figure 1 ) marks the intended starting point, where he / she is going to insert the medical instrument 21( Figure 1 ) into the patient 22( Figure 1) and the end point is labeled where the distal end of the medical instrument 21 is expected to stop. Both points can be on the same 2D slice. Alternatively, each point can be on a different slice. Typically, but not necessarily, both points are in air, i.e., where HU = -1000, and the end point is typically, but not necessarily, at the junction of air and the liquid or tissue shown in the slice. An example of an end point not at such a junction is when the point can be in the middle of an air-filled lumen.

[0090] Figure 5 The start point 150 and the end point 152 are shown as labeled by the physician on the same 2D slice, and for clarity, it is assumed that these points are the points used in the remaining description of the flowchart, unless otherwise noted. Typically, the start and end points are displayed in a non-gray scale color, such as red.

[0091] In the permitted path definition step (block 106), the physician defines a range of Hounsfield units to be used by the path finding algorithm (mentioned below) as acceptable voxel values in finding a path from the start point 150 to the end point 152. The defined range typically includes HU equal to -1000, corresponding to air or void in the path; the defined range can also include HU greater than -1000, for example, the range can be defined as given by expression (1):

[0092] {HU | -1000 < HU < U} (1)

[0093] where U is a value chosen by the physician.

[0094] For example, U can be set to +45, so that the taken path can include water, fat, blood, soft tissue, as well as air or void. In some embodiments, the range can be set by the processor 40 ( Figure 1 ) without physician intervention.

[0095] It is not required that the defined range of values be a continuous range, and the range can be disjointed, including one or more sub-ranges. In some embodiments, the sub-ranges can be selected to include particular types of material. An example of a disjointed range is given by expression (2):

[0096] {HU | HU = -1000 or A < HU < B} (2)

[0097] where A, B are values chosen by the physician.

[0098] For example, A and B can be set to equal -300 and -100, respectively, so that the taken path can include air or void and soft tissue.

[0099] The method of selecting a HU range can include any suitable method, including but not limited to numerical and / or material names and / or grayscale scales. For example, in the case of selection by grayscale, the physician 54( Figure 1 ) can select one or more regions of the CT image, and the equivalent HU of the grayscale values of the selected regions are included in the acceptable HU range for the voxels whose path is to be determined by the pathfinding algorithm.

[0100] In the case of selection by name, the physician can be presented with a table of named species. The presented table is typically similar to Table I, but without the column providing values in Hounsfield units. The physician can select one or more named species from the table, in which case the equivalent HU of the selected named species are included in the acceptable HU range for the voxels whose path is to be determined by the pathfinding algorithm.

[0101] In the pathfinding step (block 108), the processor 40( Figure 1 ) implements a pathfinding algorithm to find one or more shortest paths for the medical instrument 21( Figure 1 ) to travel between the start point 150 and the end point 152. The algorithm assumes that traversable voxels in the path include any voxels having a HU within the HU range defined in the step of block 106, and voxels having HU values outside of this defined range act as barriers in any path found. While the pathfinding algorithm used can be any suitable algorithm capable of determining the shortest path within a three-dimensional maze, the inventors have found that a Flood Fill algorithm, Dijkstra's algorithm, or extensions such as the A* algorithm provide better results in terms of speed of computation and accuracy of determining the shortest path than other algorithms such as Floyd's algorithm or variants thereof.

[0102] In some embodiments, the pathfinding step includes consideration of the mechanical properties and dimensions of the medical instrument 21( Figure 1 ). For example, in the disclosed embodiments, the medical instrument 21 can be limited in the range of radii of curvature possible when it is bent. In determining possible paths for the medical instrument 21, the processor 40( Figure 1 ) ensures that no portion of the path defines a radius less than this range of radii.

[0103] In another disclosed embodiment, the processor 40( Figure 1 ) considers the mechanical properties and dimensions of the medical instrument 21( Figure 1The mechanical properties of the medical instrument 21 allow different parts of the medical instrument 21 to have different ranges of radii of curvature. For example, the end of the possible path can have a smaller radius of curvature than the possible radii of curvature of the proximal part of the medical instrument 21. However, the distal end of the medical instrument 21 can be more flexible than the proximal part and can have sufficient flexibility to accommodate a smaller radius of curvature so that the possible path is acceptable.

[0104] In considering the possible radii of curvature of the medical instrument 21 Figure 1 ) and the different radii of curvature of the possible paths, the processor 40 Figure 1 ) takes into account which parts of the path need to be traversed by different parts of the medical instrument 21 and the radii of curvature that are reachable by the medical instrument 21 as the distal end of the medical instrument 21 moves from the start point 150 to the end point 152.

[0105] In another disclosed embodiment, the processor 40 Figure 1 ) ensures that the path diameter D is always larger than the measured diameter d of the medical instrument 21. As is known in the art, the confirmation can be implemented at least in part, e.g., by the processor 40, using an erosion / dilation algorithm to find voxels within the range defined in the step of block 106.

[0106] In a superimposition step (block 110), the shortest path found in the step of block 108 is superimposed on the image displayed on the display screen 56. Figure 1 The shortest path 154 is shown as having been superimposed on the image of Figure 6 between the start point 150 and the end point 152. Typically, the path 154 is displayed in a non-gray scale color, which can or can not have the same color as the colors of the start and end points. In the event that more than one shortest path is found in the step of block 108, all such paths can be superimposed on the image, typically in different non-gray scale colors.

[0107] Typically, the found path traverses more than one 2D slice, in which case the superimposition can be implemented by incorporating the found path into all the 2D slices that are relevant, i.e., all the 2D slices that the path traverses. Alternatively or in addition, an at least partially transparent 3D image can be generated from the scanned 2D slices and the found path can be superimposed on the 3D image. The at least partially transparent 3D image can be formed on a representation of the outer surface of the patient 22, as described in more detail below.

[0108] Figure 5 is a representation of the outer surface 180 of the patient 22 according to an embodiment of the application. The processor 40 Figure 7) using the CT scan data acquired in the step of block 100 to generate a representation of the outer surface using the fact that air has a HU value of -1000, while skin has a HU value that is significantly different therefrom. For example, assume that the representation 180 is formed in a plane that is parallel to the coronal plane of the patient 22, i.e., the x-y plane of the frame of reference 184 defined by the patient 22, whose axes are also drawn in Figure 1 and Figure 7 .

[0109] Figure 8 The boundary plane 190 and the boundary region 192 according to an embodiment of the present application are schematically shown. From the perspective of the physician 54 ( Figure 8 ) the processor 40 ( Figure 1 ) optionally delineates the regions of the representation 180 to be rendered transparent, as well as those regions that are to be left "as is". To perform the delineation, the physician uses the boundary perimeter 194 of the region 192 to define the boundary plane 190 and the boundary region 192 in the boundary plane.

[0110] For the sake of clarity, the following description assumes that the boundary plane is parallel to the x-y plane of the frame of reference 184, as Figure 1 schematically shown, and which has the equation given below:

[0111] z = z bp (3)

[0112] As described below, the processor 40 uses the boundary plane and the boundary region 192 to determine which elements of the surface 180 are rendered partially transparent, and which elements are not so rendered.

[0113] The processor 40 determines elements of the surface 180 ( Figure 8 ) that have a value z > z bp , and that, when projected along the z-axis, lie within the boundary region 192. The processor 40 then renders the elements transparent, so that these elements are no longer visible in the surface 180. For example in Figure 7 the tip of the nose 196 of the patient 22 has a value z > z bp , so that the dashed line 198 near the tip of the patient's nose shows the portion of the outer surface 180 that is no longer visible when the image of the surface is presented on the display screen 56 ( Figure 8 ).

[0114] Because the elements described above are rendered transparent, the elements of the surface 180 that have a value z < z bpAnd elements of surface 180 that were within boundary region 192 when projected along the z-axis are now visible, and thus shown in the image. Prior to the partial transparency rendering, the "now visible" elements were not visible because they were occluded by surface elements. The now visible elements include elements of shortest path 154, such as Figure 1 as shown.

[0115] Figure 9 After the partial transparency rendering of the elements of the surface within boundary region 192 is shown schematically (in Figure 9 ), surface 180 as shown on display screen 56 (in Figure 8 ). For clarity, a dashed circle 194A corresponding to boundary perimeter 194 (in Figure 1 ) has been superimposed on the image, and frame of reference 184 is also plotted in the figure. Due to the transparent rendering of the elements within circle 194A, region 200 within the circle now shows internal structures (in Figure 8 ) of patient 22 derived from the CT tomographic data received in the step of block 100.

[0116] Figure 1 Shortest path 154 is also plotted. Due to the transparent rendering of the elements within circle 194A, a portion of the path is now visible in the image of surface 180, and has been plotted as a white solid line 202. Portions of the path that were not visible due to their occlusion by elements of surface 180 that have not yet been transparently rendered are indicated as a white dashed line 204.

[0117] It will be appreciated that in the case shown in Figure 9 and Figure 7 , the image shown on display screen 56 is a view of patient 22 along the z-axis of the x-y plane.

[0118] The above description provides one example of the application of partial transparency to view a shortest path derived from tomographic data, in this case, the partial transparency is formed with respect to a plane parallel to the coronal plane of patient 22. It will be appreciated that due to the three-dimensional nature of the tomographic data, the data can be manipulated so that embodiments of the present application can view shortest path 154 using partial transparency formed with respect to substantially any plane through patient 22, and can be defined in frame of reference 184.

[0119] In forming the partial transparency, the size and location of boundary plane 190 and boundary region 192 can be varied to enable physician 54 (in Figure 9 ) to also view shortest path 154 and internal structures in the vicinity of path 154.

[0120] Doctor 54 can change the orientation of boundary plane 190, for example, to enhance the visibility of a particular internal structure. Although boundary plane 190 is generally parallel to the plane of the image presented on display screen 56, this is not necessary, so if, for example, doctor 54 wants to see more detail of a particular structure, she / he can rotate boundary plane 190 so that it is no longer parallel to the image plane.

[0121] In some cases, the HU value / grayscale range selected in step 106 includes areas other than air, such as areas corresponding to soft tissue and / or mucus. The path 154 found in step 108 may include such areas, and in this case, for medical device 21 traversing path 154... Figure 1 These areas may need to be cleaned, for example, through debridement. In the optional warning step (box 112), the physician 54 ( Figure 1 ) Receive notification of the existence of a path 154 region that is not in the air, for example by highlighting the relevant portion of path 154 and / or by other visual or auditory cues.

[0122] Although the above description assumes that the CT scan is an X-ray scan, it should be understood that embodiments of the present invention include using MRI (magnetic resonance imaging) tomography images to find the shortest path.

[0123] Therefore, referring again to flowchart 90, in the case of MRI images, where Hounsfield values ​​may no longer be directly applicable, in step 106, physician 54 ( Figure 1 The grayscale value range (grayscale value of the MRI image) is defined, and the path-finding algorithm uses this range as the acceptable voxel values ​​when finding a path from the starting point 150 to the ending point 152. In the step of box 108, the path-finding algorithm assumes that traversable voxels in the path include any voxels having grayscale values ​​within the grayscale range defined in the step of box 106, and that voxels having grayscale values ​​outside this defined range act as barriers in any path found. Other modifications described above for adapting to the use of MRI images instead of X-ray CT images will be apparent to those skilled in the art, and all such modifications should be considered to be included within the scope of this invention.

[0124] See now Figure 1 . Figures 10-12 For including use Figure 10 The flowchart 300 shows the steps in the method by which device 20 calculates the segment of path 154 and calculates the position of virtual camera 320 along path 154. Figure 1 and Figure 11 To show Figure 12 The flowchart 300 is a schematic diagram of the steps of the method.

[0125] The processor 40( Figure 10 ) is configured to find (block 302) turning points 324 in the 3D path 154 that are above a threshold turn, and in response to the found turning points 324, to compute segments 322 of the 3D path 154 and respective different positions along the 3D path 154 of the respective virtual cameras 320, as Figure 1 shown.

[0126] The steps of block 302 are now described in sub-steps below.

[0127] The processor 40( Figure 11 ) is configured to compute (block 304) the segments 322 based on an n-dimensional multi-segment simplification. In some embodiments, the n-dimensional multi-segment simplification includes, by way of example, the Ramer-Douglas-Peucker algorithm, the Visvalingam-Whyatt algorithm, or the Reumann-Witkam. Any suitable algorithm that simplifies an n-dimensional multi-segment to a smaller-sized multi-segment can be used. The algorithm generally analyzes the path 154 to remove small turning points, while leaving larger turning points, such that the larger turning points 324 define the segments 322 between the turning points 324. The threshold turn value corresponding to turning points removed from the path 154 can be set by configuring parameters of the algorithm used. For example, the input parameter of the Ramer-Douglas-Peucker algorithm can be set to about 0.08.

[0128] In other embodiments, the processor 40 is configured to use any suitable algorithm to compute the segments 322 such that turning points below the threshold turn value are removed.

[0129] The processor 40( Figure 1 ) is configured to position (block 306) the virtual cameras 320 at or around the turning points 324 between the segments 322, and at the start point 150 of the path 154 and optionally at the end point 152. Figure 1 The path 154 in the passageway 328 is shown to have been simplified with one turning point 324 and two segments 322. Three virtual cameras 320 have been placed at the start point 150, the turning point 324, and the end point 152 on the path 154, respectively. Small turning points (indicated using dashed-elliptical shapes 326) on the path 154 have been removed by the n-dimensional multi-segment simplification to leave the turning points 324.

[0130] Processor 40 is configured to inspect (box 308) the line of sight between two adjacent virtual cameras 320 and, in response to an obstruction of the line of sight, position one or more virtual cameras 320 between the two adjacent virtual cameras 320. The line of sight can be inspected by examining voxels of a 3DCT image to determine whether material obstructs the line of sight between adjacent virtual cameras 320. The type of material considered to obstruct or not obstruct the line of sight can be determined by calculation as referenced. Figure 11 The steps of block 106 are the same as those described in path 154. In some embodiments, different criteria may be used. For example, the physician 54 may define the material obstructing the line of sight as bone and hard tissue, thereby defining air, liquid, and soft tissue as materials that do not obstruct the line of sight. In some cases, the physician 54 may define the material obstructing the line of sight as bone, hard tissue, and soft tissue. Alternatively, the physician 54 may set a material that does not obstruct the line of sight, such as air or liquid, instead of specifying a material that obstructs the line of sight. In some embodiments, the processor 40 is configured to check that the direct line of sight between two adjacent virtual cameras 320 is not obstructed. In other embodiments, the processor 40 may be configured to check that the direct line of sight between two adjacent virtual cameras 320 is not obstructed, including checking that a given tolerance around the line of sight between the two virtual cameras 320 is not obstructed. The given tolerance around the line of sight can have any suitable value. For example, Figure 3 It is shown that the line of sight between virtual camera 320-2 and virtual camera 320-3 along section 322-2 is blocked by a portion of organization 330. Figure 11 It shows that another virtual camera 320-4 has been added between virtual cameras 320-2 and 320-3. Figure 12 It is also shown that although the direct line of sight between virtual cameras 320-1 and 320-2 is not obstructed, the line of sight extended by the given tolerance is obstructed by a portion of the organization 332 when the given tolerance around the line of sight is taken into account. Figure 11 This shows that another virtual camera 320-5 has been added between virtual cameras 320-1 and 320-2. Once the additional virtual camera 320 has been added, the processor 40 can check the line of sight (or extended line of sight) between adjacent virtual cameras 320 based on the initial virtual camera 320 plus the additional virtual camera 320.

[0131] The processor 40 is optionally configured to locate (box 310) one or more additional virtual cameras 320 located in one or more segments 322 in response to the distance between existing virtual cameras 320 exceeding a limit. Figure 12It is shown that virtual camera 320-6 and virtual camera 320-7 have been added to section 322-1 in response to the distance between existing virtual cameras 320 exceeding a limit. The limit can be any suitable limit, such as but not limited to in the range of 1 mm to 20 mm, such as 4 mm. The additional cameras 320 are typically evenly spaced between existing virtual cameras 320.

[0132] Reference is now made to Figure 12 . Figures 13-15 A flowchart 340 of steps in a method of selecting cameras for a device 20 comprising use Figure 13 of a camera. Figure 1 and Figure 14 A schematic diagram of steps of the method of flowchart 340 is shown. Figure 15 Processor 40

[0133] is configured to calculate (block 342) for respective ones of virtual cameras 320 respective bisectors 350 Figure 13 . In some embodiments, processor 40 is configured to calculate respective bisectors 350 as respective planes perpendicular to 3D path 154 at respective locations of respective virtual cameras 320 on 3D path 154. Figure 1 Bisectors 350 for each virtual camera 320 are shown. Bisectors 350 can be calculated at any time after path 154 has been calculated, until when medical instrument 21 is in close proximity to respective virtual camera 320, as described by the steps of block 344 below. Figure 14

[0134] Processor 40 is configured to find (block 344) the virtual camera 320 closest to the distal end of medical instrument 21 in response to the tracked coordinates of medical instrument 21 and the known locations of virtual cameras 320 (e.g., virtual camera 320-7) Figure 14 ​). Processor 40 is configured to find (block 346) which side of the bisector line 350 of the nearest virtual camera 320 (e.g., virtual camera 320-7) the tracking coordinates fall on. Processor 40 is configured to select (block 348) one of the virtual cameras 320 to use for rendering the virtual endoscope image based on which side of the bisector line 350 of the nearest virtual camera 320 (e.g., virtual camera 320-7) the tracking coordinates of the medical instrument 21 fall on. If the tracking coordinates fall on the side of the bisector line 350 closer to the current virtual camera 320 (e.g., virtual camera 320-6), then the current virtual camera (e.g., virtual camera 320-6) is still used. If the tracking coordinates fall on the side of the bisector line 350 further from the current virtual camera 320 (e.g., virtual camera 320-6), then the next virtual camera 320 along the path (i.e., virtual camera 320 (e.g., virtual camera 320-7)) is selected as the new virtual camera 320. The steps of blocks 344-346 are repeated intermittently.

[0135] The steps of blocks 344-348 are illustrated in Figure 1 . Figure 15 Two virtual cameras 320-6 and 320-7 are illustrated. Virtual camera 320-6 is the current virtual camera used to render and display the virtual endoscope image of the passageway 328. Virtual camera 320-7 is further downstream of the path 154 relative to the direction of travel of the medical instrument 21 than virtual camera 320-6. In other words, virtual camera 320-7 is closer to the end point 152 than virtual camera 320-6. Figure 15 Figure 11 Various possible example positions 352 of the distal end of the medical instrument 21 are illustrated. All of the positions 352 are closer to virtual camera 320-7 than virtual camera 320-6. Thus, for all of the positions 352 as illustrated in Figure 15 , it will be found that virtual camera 320-7 is the closest virtual camera 320 in the steps of block 344. Once the closest virtual camera 320-7 is found, the position of the distal end of the medical instrument 21 relative to the bisector line 350 of the nearest virtual camera 320-7 is checked in the steps of block 346. In the example of Figure 15 , position 352-6 is on the side of the bisector line 350 closest to virtual camera 320-6 (the current virtual camera) (indicated with arrow 354), and position 352-7 is on the other side of the bisector line 350 further from the current virtual camera 320-6 (indicated with arrow 356). In Figure 15 ​In the example of FIG. 3, according to the steps of block 348, if the tracking coordinates of the distal end of medical instrument 21 are in any of positions 352-6 or similar positions, then the current virtual camera 320-6 will remain as the selected virtual camera. If the tracking coordinates of the distal end of medical instrument 21 are in any of positions 352-7 or similar positions, then virtual camera 320-7 is selected as the new virtual camera.

[0136] Thus, processor 40 is configured to select a respective virtual camera 320 for rendering a respective virtual endoscope image in response to the tracking coordinates (positions 352) of medical instrument 21 ( Figure 15 ) falling on which side of a respective bisector 350 of the respective virtual camera 320 closest to the tracking coordinates.

[0137] In other embodiments, the passageway can be divided into multiple regions based on segments 322, with virtual cameras 320 being selected according to the region in which the tracking coordinates are disposed.

[0138] Referring now to Figure 1 , which is a flowchart 360 of steps in a method of computing an orientation and offsetting a virtual camera 320 position using a device 20. Figure 16 Referring now to Figure 1 and Figure 17 , which are schematic diagrams showing steps in the method of flowchart 360 of Figure 18 .

[0139] The orientation of the optical axis of each virtual camera 320 is computed. The orientation can be computed at any time after the position of virtual camera 320 has been computed. In some embodiments, the orientation of virtual camera 320 can be computed at the time each virtual camera 320 is selected for use as medical instrument 21 ( Figure 16 ) moving along path 154. One method of computing the orientation of a virtual camera 320 is now described as follows.

[0140] Processor 40 ( Figure 1 ) is configured to select (block 362) a position 370 on path 154 from one virtual camera 320-2 to another virtual camera 320-4, as shown in Figure 1 Position 370 can be selected to include or not include the position of virtual camera 320-4. Position 370 can be selected by dividing segment 322 between virtual camera 320-2 and virtual camera 320-4 into subsegments. Alternatively, position 370 can be selected by measuring a given distance along path 154 from virtual camera 320-2 to each position 370. In some embodiments, position 370 can be selected using points that define path 154 when path 154 is generated. Processor 40 ( Figure 17) configured to define a vector 372 from the virtual camera 320-2 to the location 370, and to compute (block 364) an average direction 374 of the vectors 372 as shown. Thus, the processor 40 is configured to compute an average direction of vectors along the 3D path 154 from the location of the virtual camera 320-2 to different points (e.g., the location 370). The processor 40 Figure 1 ) is configured to compute (block 366) an orientation of the virtual camera 320-2 in response to the computed average direction 374. In other words, the orientation of the optical axis of the virtual camera 320 is computed to be the average direction 374. In other embodiments, the orientation can be computed to be a direction parallel to the path at the respective virtual camera 320-2 location. Figure 17

[0141] In some embodiments, the respective location of the respective virtual camera 320 (e.g., the virtual camera 320-2) can be offset backwards, for example, in the opposite direction 376 to the respective average direction 374 of the respective virtual camera 320, as shown. Offsetting the virtual camera 320 back can result in a better view of the medical instrument 21 within the respective virtual endoscope image, particularly when the medical instrument 21 is in close proximity to the respective virtual camera 320, and can result in a better view of the surrounding anatomy. Thus, the processor 40 Figure 1 ) is configured to offset (block 368) the location of the virtual camera 320-2 in the direction 376 opposite the computed average direction 374 to a new location 380, as shown. The degree of offset can be fixed, for example, a predetermined number of millimeters, for example, in a range between 0.2 mm and 2 mm, for example, 1.3 mm. Alternatively, the degree of offset can be limited by the surrounding anatomy 378, such that the camera 320-2 is offset as far back as possible, as long as it is not pushed into bone or tissue, which is defined by the physician 54. The type of material considered by the “surrounding anatomy” can be the same or different from the material criteria for defining a path blocked by the medical instrument 21 as described with reference to the steps of block 106 of FIG. 1. Figure 18 Figure 1 Figure 18 The field of view of the respective virtual camera 320 can be set to any suitable respective value. The field of view of each virtual camera 320 can be fixed, for example, in a range of values between 25 degrees - 170 degrees, for example, 90 degrees. The field of view of any one virtual camera 320 can be set according to the external limits of the segment 322 of the path 254 that the virtual camera 320 covers (e.g., from

[0142] The field of view of the respective virtual camera 320 can be set to any suitable respective value. The field of view of each virtual camera 320 can be fixed, for example, in a range of values between 25 degrees - 170 degrees, for example, 90 degrees. The field of view of any one virtual camera 320 can be set according to the external limits of the segment 322 of the path 254 that the virtual camera 320 covers (e.g., from Figure 3 ​​​the additional tolerance of the virtual camera 320 (such as a given angular tolerance, e.g., add X degrees to the outer limit, where X can be any suitable value, e.g., in the range of 5 degrees and 90 degrees). In some embodiments, by analyzing the surrounding anatomy 378 around the segment 322, the field of view can be set to cover all anatomy in the segment 322 until the next virtual camera. The material types considered for the "surrounding anatomy" can be the same as described with reference to Figure 17 The material criteria used to define the path blocked by the medical instrument 21 can be the same or different than described with reference to the step of block 106.

[0143] Reference is now made to Figure 3 which is a flowchart 390 of steps in a rendering transition method in the system 20 including the Figure 19 Reference is now made to Figure 1 which is a schematic diagram illustrating the steps of the method of the flowchart 390 of Figure 20

[0144] The transition between two virtual cameras 320 and thus the transition between the associated virtual endoscopic images can be a smooth transition or a sharp transition. In some embodiments, a smooth transition between two respective virtual cameras can be performed by performing the following steps. The transition is described between virtual camera 320-2 and virtual camera 320-4 by way of example.

[0145] The processor 40 Figure 19 is configured to find (block 392) the positions of additional virtual cameras 396 to be added between the current virtual camera 320-2 and the next virtual camera 320-4, as shown in Figure 1 The positions of the additional virtual cameras 396 can be found in a similar manner as the selection of the position 370 described with reference to Figure 20 and Figure 16 Any suitable number of additional virtual cameras 396 can be selected. A larger number of additional virtual cameras 396 will generally result in a smoother transition.

[0146] The processor 40 Figure 17 is configured to display (block 394) a respective transition virtual endoscopic image of the body passageway 328 observed from the respective position of the respective additional virtual camera 396 disposed between the two adjacent virtual cameras 320-2, 320-4 based on continuous rendering, Figure 1 on the display screen 56 Figure 20 ​) on the display screen 56 (block 394) between the two respective virtual endoscope images from the two respective adjacent virtual cameras 320-2, 320-4. Each of the transition virtual endoscope images can be displayed on the display screen 56 for any suitable duration, e.g., a duration in the range of 20 milliseconds to 40 milliseconds.

[0147] Referring now to Figure 1 , they are the illustrative virtual endoscope images 398 rendered and displayed by the device 20. As previously mentioned with reference to Figures 21-23 , the processor 40 ( Figure 1 ) is configured to render and display the respective virtual endoscope images 398 on the display screen 56 (block 394) of the passageway 328 in the body viewed from the respective positions and orientations of the respective virtual cameras 320 ( Figure 2 ) according to the animated representation 400 of the medical instrument 21 ( Figure 1 ) positioned in the respective virtual endoscope images 398 based on the 3D CT images. Figure 1 Figure 11

[0148] Figure 1 The virtual endoscope image 398-1 is shown viewed from the position of one virtual camera 320. The representation 400 of the medical instrument 21 is shown in Figures 21-22 in the passageway 328 along the path 154 at one position, and in Figure 21 in the passageway 328 along the path 154 at one more forward position. The representation of the path 154 still being traversed is shown using an arrow, which disappears as the medical instrument 21 moves along the path 154. Any suitable line or symbol can be used to represent the path 154. Figure 22 The virtual endoscope image 398-2 is shown viewed from the position of another virtual camera 320.

[0149] The virtual endoscope images 398 can be rendered using volume visualization techniques that generate the virtual endoscope images 398 in a 3D viewing volume (e.g., a cone projected outward from the position of the relevant virtual camera 320) from tissue image data (e.g., HU of voxels based on a 3D CT scan). The images 398 can be rendered based on known colors of the tissue. Certain materials such as liquids or even soft tissue can be selected to be transparent, while all other higher density materials can be rendered according to their natural colors. Alternatively, even liquids and soft tissue with higher density materials can be rendered according to their natural colors of the respective materials. In some embodiments, the physician 54 ( Figure 23 ​​The rendering parameters of the virtual endoscope image 398 discussed above can be set. In some embodiments, the surrounding anatomy 378 can be rendered, while other anatomy can be ignored. The material types considered for the "surrounding anatomy" can be the same or different from those described with respect to the step of block 106 as described in detail above with reference to Figure 1

[0150] The image 398 above is presented for illustrative purposes only, and other kinds of images can be rendered and displayed as well in accordance with the principles of the application.

[0151] Reference is now made to Figure 3 and Figure 24 . Figure 25 is a flowchart 500 of steps in a two-dimensional path visualization method for use in the apparatus 20 of Figure 24 in accordance with embodiments of the application. Figure 1 is a schematic diagram of combined 2D and 3D path visualization 512 illustrating the steps of the method of the flowchart 500 of Figure 25 .

[0152] The position tracking system 23 Figure 24 is configured to track (block 502) coordinates of the medical instrument 21 Figure 1 within the body. The processor 40 Figure 1 registers (block 504) the position tracking system 23 and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body in a common frame of reference. The steps of block 504 are substantially the same as the steps of block 74 described in detail above with reference to Figure 1 .

[0153] The processor 40 Figure 2 is configured to find (block 506) a 3D path 154 Figure 25 of the medical instrument 21 Figure 1 through a passageway 328 Figure 9 from a given starting point 150 Figure 1 to a given ending point 152 Figure 9 in the common frame of reference. The steps of block 506 are substantially the same as the steps of block 76 described in detail above with reference to Figure 25 .

[0154] The processor 40 Figure 2 is configured to compute (block 508) a 2D path 154 Figure 1 ​the direction and optional distance and / or 3D vector from the distal end of the medical instrument 21 to the 3D path 154. In some embodiments, the processor 40 is configured to calculate the direction and optional distance and / or 3D vector from the distal end of the medical instrument 21 to the nearest point of the 3D path 154 in response to the tracked coordinates of the medical instrument 21.

[0155] The processor 40( Figure 1 ) is configured to render and concurrently display on the display screen 56( Figure 1 ) the respective two-dimensional (2D) CT slices 514, including respective 2D indications 516 of the direction, and optional distance, and / or 3D vector from the distal end of the medical instrument 21 to the 3D path 154 projected onto the respective 2D CT slices 514, and optionally a representation 518 of the medical instrument 21 in response to the tracked coordinates of the medical instrument 21, based on the 3D CT images.

[0156] The 2D CT slices 514 are generated from the 3D CT images in accordance with the tracked coordinates of the distal end of the medical instrument 21 such that the 2D CT slices 514 are planes intersecting in the 3D CT images at the tracked coordinates. In some embodiments, the processor 40 is configured to render and concurrently display on the display screen 56( Figure 1 ) three respective two-dimensional (2D) CT slices 514, including respective 2D indications 516 of the direction, and optional distance, and / or 3D vector from the distal end of the medical instrument 21 to the 3D path 154 (of the nearest point) projected onto the three respective 2D CT slices 514, and optionally a representation 518 of the medical instrument 21 in response to the tracked coordinates of the medical instrument 21. In some embodiments, the three respective 2D slices include a coronal view slice 514-1, a sagittal view slice 514-2, and a transverse view slice 514-3, respectively.

[0157] The 2D indications 516 can include any suitable symbol, such as an arrow or a pointer. The 2D indications 516 can indicate only the direction to the path 154, and thus all 2D indications 516 can be the same length regardless of the distance to the path 154. In some embodiments, the 2D indications 516 can indicate the direction and distance (or 3D vector) to the path 154, and the 2D indications 516 can be sized (length and / or width) in accordance with the distance to the path 154. The 2D indications 516 represent the direction and optional distance or vector of the path projected onto the respective 2D CT slices 514. For example, based on the 3D path 154 being a straight line, the 2D indications 516 can be straight arrows or pointers. In some embodiments, the 2D indications 516 can be curved arrows or pointers, such as when the 3D path 154 is a curved line. Figure 1The 3D vectors are projected onto the x-y plane to generate 2D indications 516 for coronal view slices 514-1, onto the y-z plane to generate 2D indications 516 for sagittal view slices 514-2, and onto the x-z plane to generate 2D indications 516 for transverse view slices 514-3, shown in reference frame 184.

[0158] The representation 518 of the medical instrument 21 Figure 8 ) can include any suitable symbol, e.g., a cross or a circle.

[0159] In some embodiments, the processor 40 Figure 1 ) is configured to render and concurrently display on the display screen: (a) the respective 2D CT slices 514, including the respective 2D indications 516 of the direction from the medical instrument 21 to the 3D path 154 projected onto the respective 2D CT slices 514; and a 3D CT image-based virtual endoscopic image of the passageway 328 in the body viewed from at least one respective position of at least one respective virtual camera 520, including an animated representation 522 of the medical instrument 21 positioned in the virtual endoscopic image 520 according to the tracking coordinates, and a 3D indication 524 of the direction from the medical instrument 21 to the 3D path 154. The virtual endoscopic image 520 can be rendered and displayed according to the methods described above with reference to Figure 1 Figures 2-23 .

[0160] The above images 514, 520 are presented for illustrative purposes only, and other kinds of images can be rendered and displayed according to the principles of the present application as well.

[0161] As used herein, the term "about" or "approximately" with respect to any numerical or range of values indicates suitable dimensional tolerances that allow the components or elements to function for their intended purpose as described herein. More specifically, "about" or "approximately" can refer to a range of values ±20% of the recited value, e.g., "about 90%" can refer to a range of values from 72% to 108%.

[0162] For clarity, various features of the present application described in the context of separate embodiments can also be provided in combination in a single embodiment. Conversely, various features of the present application described in the context of a single embodiment can also be provided separately or in any appropriate

[0163] The above embodiments are cited by way of example, and the present application is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present application includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons of skill in the art upon reading the foregoing description and which are not disclosed in the prior art.

Claims

1. A medical device comprising: a medical instrument configured to move within a passageway in a body of a patient; a position tracking system configured to track coordinates of the medical instrument within the body; a display screen; and a processor configured to: register the position tracking system and a 3D CT image of at least a portion of the body within a common frame of reference; find, within the common frame of reference, a 3D path of the medical instrument through the passageway from a given starting point to a given ending point; compute, responsive to the tracked coordinates, a direction from the medical instrument to the 3D path; and render and concurrently display, on the display screen: based on respective 2D CT slices of the 3D CT image, a respective 2D indication of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slice; and based on the 3D CT image, a virtual endoscopic image of the passageway in the body viewed from at least one respective position of at least one respective virtual camera, including an animated representation of the medical instrument positioned in the virtual endoscopic image according to the tracked coordinates, and a 3D indication of the direction from the medical instrument to the 3D path. The respective 2D indication of the direction from the medical instrument to the 3D path comprises a respective arrow.

2. The apparatus of claim 1, wherein, The processor is configured to render and concurrently display, on the display screen, based on the respective 2D CT slices of the 3D CT image, a respective 2D indication of the direction from the medical instrument to the 3D path projected onto the respective 2D CT slice, and a representation of the medical instrument responsive to the tracked coordinates.

3. The apparatus of claim 1, wherein, The processor is configured to compute, responsive to the tracked coordinates, a direction from the medical instrument to a nearest point of the 3D path.

4. The apparatus of claim 1, wherein, The processor is configured to compute, responsive to the tracked coordinates, a 3D vector from the medical instrument to the 3D path.

5. The apparatus of claim 1, wherein, The processor is configured to render and concurrently display, on the display screen, based on the respective 2D CT slices of the 3D CT image, the respective 2D indication of the 3D vector from the medical instrument to the 3D path projected onto the respective 2D CT slice.

6. The apparatus of claim 5, wherein, The processor is configured to render and concurrently display, on the display screen, based on three respective 2D CT slices of the 3D CT image, three respective 2D indications of the direction from the medical instrument to the 3D path projected onto the three respective 2D CT slices.

7. The apparatus of claim 1, wherein, The three respective 2D CT slices respectively comprise a coronal view, a sagittal view, and a transverse view.

8. The apparatus of claim 7, wherein, The processor is configured to:

9. The apparatus of claim 1, wherein, compute, responsive to the tracked coordinates, a distance from the medical instrument to the 3D path; and render and concurrently display, on the display screen, based on the respective 2D CT slices of the 3D CT image, a respective 2D indication of the distance from the medical instrument to the 3D path projected onto the respective 2D CT slice. render and concurrently display on a display screen the respective 2D CT slices, including respective 2D indications of the direction and the distance from the medical instrument projected onto the respective 2D CT slices to the 3D path, based on the 3D CT images.

10. The apparatus of claim 1, wherein, The position tracking system comprises an electromagnetic tracking system comprising one or more magnetic field generators positioned around a portion of the body and a magnetic field sensor located at a distal end of the medical instrument.

11. A software product comprising a non-transitory computer readable medium having stored therein program instructions which, when read by a CPU, cause the CPU to: track coordinates of a medical instrument within a body of a patient using a position tracking system, the medical instrument being configured to move within a passageway in the body of the patient; register the position tracking system and a 3D CT image of at least a portion of the body within a common frame of reference; find a 3D path of the medical instrument through the passageway from a given starting point to a given end point within the common frame of reference; compute a direction from the medical instrument to the 3D path in response to the tracked coordinates; and render and concurrently display on a display screen: respective 2D CT slices based on the 3D CT images, including respective 2D indications of the direction from the medical instrument projected onto the respective 2D CT slices to the 3D path; and virtual endoscopic images of the passageway in the body based on the 3D CT images from at least one respective position of at least one respective virtual camera, including an animated representation of the medical instrument positioned in the virtual endoscopic images in accordance with the tracked coordinates, and a 3D indication of the direction from the medical instrument to the 3D path.

Images