3D path detection visualization

Abstract

In one embodiment, an apparatus includes a medical device, a position tracking system that tracks coordinates of the device in a pathway in a body, and a processor for aligning the system with a 3DCT image of at least a portion of the body, finding a path of the device through the pathway, calculating segments of the path, calculating corresponding positions along the path of corresponding virtual cameras in response to the calculated segments, selecting the corresponding virtual camera for rendering a corresponding virtual endoscopic image in response to the tracking coordinates, calculating a corresponding orientation of the corresponding virtual camera, and rendering and displaying a virtual endoscopic image based on the 3DCT image of the pathway in the body observed from the corresponding position and orientation of the corresponding virtual camera, including an animated representation of the device positioned in the corresponding virtual endoscopic image according to the tracking coordinates.

Description

Technical Field

[0001] The present invention relates to medical systems and, in particular, but not exclusively, to pathway visualization. Background Art

[0002] In image-guided surgery (IGS), practitioners use instruments that are tracked in real time within the body, allowing the instrument's position and / or orientation to be visualized on images of the patient's anatomy during the surgical procedure. In many IGS scenarios, patient images are prepared using one modality, such as magnetic resonance imaging (MRI) or computed tomography (CT), and instrument tracking is performed using a different modality, such as electromagnetic tracking. For tracking to be effective, the reference frames of the two modalities must be registered to each other.

[0003] US Patent Publication 2011 / 0236868 to Bronstein et al. describes a method for performing a computerized simulation of an image-guided procedure. The method may include receiving medical image data for a specific patient. A patient-specific digital image-based model of the specific patient's anatomy may be generated based on the medical image data. The digital image-based model may be used to perform a computerized simulation of the image-guided procedure. The medical image data, the image-based model, and the simulated medical tool model may be displayed simultaneously.

[0004] US Patent Publication 2017 / 0151027 to Walker et al. describes a system and method for robotically assisting in driving a flexible medical instrument to a target in an anatomical space. The flexible instrument may have a tracking sensor embedded therein. An associated robotic control system may be provided that is configured to register the flexible instrument with an anatomical image using data from the tracking sensor and identify one or more movements suitable for navigating the instrument toward the identified target. In some embodiments, the robotic control system drives or assists in driving the flexible instrument to the target.

[0005] U.S. Patent Publication 2016 / 0174874 to Averbuch et al. describes a registration method whereby a sensor-based method is used for initial registration, and whereby upon beginning to navigate the endoscope, an image-based registration method is used to more accurately maintain registration between the endoscope position and previously acquired images. A six-degree-of-freedom position sensor is placed on the probe in order to reduce the number of previously acquired images that must be compared with the real-time image obtained from the endoscope.

[0006] US Patent Publication 2005 / 0228250 to Bitter et al. describes a user interface including an image region divided into a plurality of views for viewing corresponding 2D and 3D images of an anatomical region. A tool control panel can be opened and accessed simultaneously. The segmentation panel enables automatic segmentation of components of the displayed image within a user-specified intensity range or based on a predetermined intensity.

[0007] U.S. Patent Publication No. 2007 / 0276214 to Dachille et al. describes an imaging system for automatic segmentation and visualization of medical images, comprising an image processing module for automatically processing image data using a set of instructions for identifying a target object in the image data and processing the image data according to a specified protocol, a rendering module for automatically generating one or more images of the target object based on one or more instructions in the instructions, and a digital archive for storing the one or more generated images. The image data may be DICOM formatted image data, wherein the imaging processing module extracts and processes metadata in the DICOM domain of the image data to identify the target object. The image processing module directs the segmentation module to segment the target object using processing parameters specified by the one or more instructions in the instructions.

[0008] U.S. Patent No. 5,371,778 to Yanof et al. describes a CT scanner that non-invasively examines a volumetric region of a subject and generates volumetric image data indicative of the volumetric region. An object memory stores data values ​​corresponding to each voxel of the volumetric region. An affine transformation algorithm operates on a visible face of the volumetric region to transform the face from object space into a projection of the face on an observation plane in image space. An operator console includes operator controls for selecting the angular orientation of a projected image of the volumetric region relative to the observation plane (i.e., the plane of a video display). A cursor positioning trackball inputs i and j coordinate positions in image space, which are converted to cursor crosshairs displayed on the projected image. A depth dimension k is determined between the observation plane and the volumetric region in a viewing direction perpendicular to the observation plane. The (i, j, k) image space position of the cursor is manipulated by inverting the selected transformation to identify corresponding (x, y, z) cursor coordinates in object space. The cursor coordinates in object space are converted to corresponding addresses in the object memory for transverse, coronal, and sagittal planes through the volumetric region.

[0009] U.S. Patent 10,188,465 to Gliner et al. describes a method comprising receiving a computed tomography scan of at least a portion of a patient's body and identifying voxels in the scan corresponding to regions of the body that are traversable by a probe inserted therein. The method further comprises displaying the scan on a screen and marking thereon selected start and end points for the probe. A processor finds a path from the start point to the end point, the path consisting of a continuous set of the identified voxels. The processor also generates a representation of the outer surface of the body using the scan and displays the representation on the screen. The processor then renders an outer surface region surrounding the path partially transparent in the displayed representation so that internal structures of the body near the path are visible on the screen.

[0010] US Patent Publication 2018 / 0303550 by Altmann et al. describes a method for visualization that includes registering a position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of a patient's body within a common reference frame. The position and orientation of at least one virtual camera are specified within the common reference frame. The position tracking system is used to track the coordinates of a medical tool moving within a pathway in the body. A virtual endoscopic image based on the 3D CT image of the pathway in the body is rendered and displayed from the specified position and orientation, including an animated representation of the medical tool positioned in the virtual endoscopic image according to the tracked coordinates. Summary of the Invention

[0011] According to one embodiment of the present disclosure, a medical device is also provided, which includes a medical device configured to move within a pathway within a patient's body, a position tracking system configured to track the coordinates of the medical device within the body, a display screen, and a processor configured to align the position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body within a common reference frame, find a 3D path of the medical device through the pathway from a given starting point to a given end point, calculate segments of the 3D path, calculate corresponding different positions along the 3D path of a corresponding virtual camera in response to the calculated segments, select a corresponding virtual camera for rendering a corresponding virtual endoscopic image in response to the tracking coordinates of the medical device and the corresponding position of the corresponding virtual camera within the common reference frame, calculate a corresponding orientation of the corresponding virtual camera, and render and display on the display screen a virtual endoscopic image based on the 3D CT image of the pathway in the body observed from the corresponding position and orientation of the corresponding virtual camera, including an animated representation of the medical device positioned in the corresponding virtual endoscopic image according to the tracking coordinates.

[0012] Also according to an embodiment of the present disclosure, the processor is configured to find turning points in the 3D path that turn above a threshold, and calculate segments of the 3D path and corresponding different positions along the 3D path of the corresponding virtual camera in response to the found turning points.

[0013] Also in accordance with an embodiment of the present disclosure, the processor is configured to position at least one of the virtual cameras in the middle of one of the segments in response to a distance between adjacent ones of the virtual cameras exceeding a limit.

[0014] Additionally, according to an embodiment of the present disclosure, the processor is configured to check a line of sight between two adjacent ones of the virtual cameras, and position at least one of the virtual cameras between the two adjacent ones of the virtual cameras in response to the line of sight being blocked.

[0015] Furthermore, according to an embodiment of the present disclosure, the processor is configured to calculate the segments based on n-dimensional multi-line segment simplification.

[0016] Also in accordance with an embodiment of the present disclosure, the n-dimensional multi-line segment simplification includes a Ramer–Douglas–Peucker algorithm.

[0017] Also in accordance with an embodiment of the present disclosure, the processor is configured to calculate a corresponding bisector of a corresponding virtual camera of the virtual cameras, and select a corresponding virtual camera for rendering a corresponding virtual endoscopic image in response to which side the tracking coordinates of the medical device fall relative to the corresponding bisector of the corresponding virtual camera of the virtual cameras closest to the tracking coordinates.

[0018] In addition, according to an embodiment of the present disclosure, the processor is configured to calculate the respective bisectors as respective planes perpendicular to the 3D path at respective ones of the positions of the virtual cameras on the 3D path.

[0019] In addition, according to an embodiment of the present disclosure, the processor is configured to calculate an average direction of vectors from corresponding positions in the positions of corresponding virtual cameras in the virtual cameras to different points along the 3D path, and calculate corresponding orientations in the orientations of corresponding virtual cameras in the virtual cameras in response to the calculated average direction.

[0020] Also in accordance with an embodiment of the present disclosure, the processor is configured to shift corresponding ones of the positions of corresponding ones of the virtual cameras in a direction opposite to the calculated average direction.

[0021] Also in accordance with an embodiment of the present disclosure, the processor is configured to render and display on a display screen a transition between two corresponding virtual endoscopic images of the virtual endoscopic images of two corresponding adjacent virtual cameras among the virtual cameras based on continuously rendering corresponding transitional virtual endoscopic images of the intra-body passage observed from corresponding positions of a corresponding additional virtual camera disposed between the two corresponding adjacent virtual cameras among the virtual cameras.

[0022] Additionally, according to an embodiment of the present disclosure, a position tracking system includes an electromagnetic tracking system comprising one or more magnetic field generators positioned about a body part and a magnetic field sensor at a distal end of the medical device.

[0023] According to another embodiment of the present disclosure, a medical method is also provided, the method including tracking coordinates of a medical device within a patient's body using a position tracking system, the medical device being configured to move within a passage in the patient's body, aligning the position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body in a common reference frame, finding a 3D path of the medical device through the passage from a given starting point to a given end point, calculating segments of the 3D path, calculating corresponding different positions along the 3D path of a corresponding virtual camera in response to the calculated segments, selecting a corresponding virtual camera for rendering a corresponding virtual endoscopic image in response to the tracked coordinates of the medical device and the corresponding position of the corresponding virtual camera in the common reference frame, calculating a corresponding orientation of the corresponding virtual camera, and rendering and displaying on a display screen a corresponding virtual endoscopic image of the passage in the body based on the 3D CT image as viewed from the corresponding position and orientation of the corresponding virtual camera, including an animated representation of the medical device positioned in the corresponding virtual endoscopic image according to the tracking coordinates.

[0024] Furthermore, according to an embodiment of the present disclosure, the method includes finding a turning point in the 3D path that turns above a threshold, and wherein segments of the 3D path and corresponding different positions along the 3D path of the corresponding virtual camera are calculated in response to the found turning points.

[0025] Also in accordance with an embodiment of the present disclosure, the method includes positioning at least one of the virtual cameras in the middle of one of the segments in response to a distance between adjacent ones of the virtual cameras exceeding a limit.

[0026] Also in accordance with an embodiment of the present disclosure, the method includes checking a line of sight between two adjacent ones of the virtual cameras, and positioning at least one of the virtual cameras between the two adjacent ones of the virtual cameras in response to the line of sight being blocked.

[0027] Additionally, according to an embodiment of the present disclosure, the method includes calculating the segments based on n-dimensional multi-line segment simplification.

[0028] Furthermore, according to an embodiment of the present disclosure, n-dimensional multi-line segment simplification includes a Ramer–Douglas–Peucker algorithm.

[0029] Also according to an embodiment of the present disclosure, the method includes calculating a corresponding bisector of corresponding ones of the virtual cameras, and wherein the selecting includes selecting the corresponding virtual camera for rendering the corresponding virtual endoscopic image in response to which side the tracking coordinates of the medical device fall relative to the corresponding one of the bisectors of the corresponding one of the virtual cameras that is closest to the tracking coordinates.

[0030] Also according to an embodiment of the present disclosure, calculating the respective bisectors includes calculating the respective bisectors as respective planes perpendicular to the 3D path at respective ones of the positions of the virtual cameras on the 3D path.

[0031] In addition, according to an embodiment of the present disclosure, the method includes calculating an average direction of vectors from corresponding positions in the positions of corresponding virtual cameras in the virtual cameras to different points along the 3D path, and wherein calculating the corresponding orientations includes calculating corresponding orientations in the orientations of corresponding virtual cameras in the virtual cameras in response to the calculated average direction.

[0032] Furthermore, according to an embodiment of the present disclosure, the method includes offsetting corresponding ones of the positions of corresponding ones of the virtual cameras in a direction opposite to the calculated average direction.

[0033] Also according to an embodiment of the present disclosure, the method includes rendering and displaying on a display screen a transition between two corresponding virtual endoscopic images of the virtual endoscopic images of two corresponding adjacent virtual cameras among the virtual cameras based on continuously rendering corresponding transitional virtual endoscopic images of the intra-body passage observed from corresponding positions of a corresponding additional virtual camera disposed between the two corresponding adjacent virtual cameras among the virtual cameras.

[0034] Still another embodiment of the present disclosure provides a software product comprising a non-transitory computer-readable medium storing program instructions that, when read by a central processing unit (CPU), cause the CPU to track coordinates of a medical device within a patient's body using a position tracking system, the medical device being configured to move within a pathway in the patient's body, register the position tracking system with a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body within a common reference frame, find a 3D coordinate of the medical device through the pathway from a given starting point to a given end point, and path, calculating segments of the 3D path, calculating respective different positions along the 3D path of the respective virtual cameras in response to the calculated segments, selecting the respective virtual cameras for rendering the respective virtual endoscopic images in response to the tracked coordinates of the medical instrument and the respective positions of the respective virtual cameras within a common reference frame, calculating respective orientations of the respective virtual cameras, and rendering and displaying on a display screen the respective virtual endoscopic images based on the 3D CT images of the pathway in the body as viewed from the respective positions and orientations of the respective virtual cameras, including an animated representation of the medical instrument positioned in the respective virtual endoscopic images according to the tracked coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0036] Figure 1 is a partially schematic, partially block diagram illustration of a medical system constructed and operative in accordance with an embodiment of the present invention;

[0037] Figure 2 is according to an embodiment of the present invention included in Figure 1 A flow chart of the steps in the method of three-dimensional path visualization used in an apparatus;

[0038] Figure 3 To include the use Figure 1 A flow chart of the steps of a method for a device to find a path through a pathway.

[0039] Figure 4-Figure 9 To show Figure 3 A flowchart is a schematic diagram of the steps of the method;

[0040] Figure 10 To include the use Figure 1 A flowchart of the steps in a method for a device to calculate a path segment and calculate a position of a virtual camera along the path;

[0041] Figure 11 and Figure 12 It shows Figure 10 A flowchart is a schematic diagram of the steps of the method;

[0042] Figure 13To include the use Figure 1 A flowchart of the steps in a method of selecting a camera for a device;

[0043] Figure 14 and Figure 15 It shows Figure 13 A flowchart is a schematic diagram of the steps of the method;

[0044] Figure 16 To include the use Figure 1 A flowchart of steps in a method for computing an orientation and offsetting a virtual camera position for a device;

[0045] Figure 17 and Figure 18 It shows Figure 16 A flowchart is a schematic diagram of the steps of the method;

[0046] Figure 19 To include the use Figure 1 A flowchart of the steps in a method of device rendering transition;

[0047] Figure 20 To show Figure 19 a flowchart of the steps of the method; and

[0048] Figure 21-23 is Figure 1 Schematic virtual endoscopic image rendered and displayed by the device. DETAILED DESCRIPTION

[0049] Overview

[0050] During medical procedures within the nasal passages, such as sinus dilation surgery, it is impossible to directly visualize the sinuses without inserting an endoscope into the sinuses. However, insertion of an endoscope is problematic due to the tight spaces involved and the additional cost of the endoscope. Furthermore, endoscopes used in the nasal passages are typically rigid instruments that cannot be turned or provide a view from the sinus cavity back to the sinus opening.

[0051] The embodiments of the present invention described herein address this problem by generating a virtual endoscopic view of the procedure from a virtual camera, similar to the view that an actual endoscope positioned at the virtual camera's location within the nasal passage would see. The virtual endoscopic view displays the anatomical structure and the medical instrument moving through it. As the medical instrument moves along the pathway, the virtual cameras used to generate the virtual endoscopic view are transferred from one virtual camera to another in response to the tracking coordinates of the medical instrument.

[0052] These virtual endoscopic views can be used, for example, to visualize the position and orientation of the guidewire relative to the anatomy and other instruments such as suction tools or shaving tools (debriders).

[0053] Moving from one virtual camera to another provides a more stable view of the anatomy than placing the virtual camera on the distal tip of the medical instrument (whereby the virtual camera is always "moving"), because the movement of the distal tip causes the video of the anatomy to jump or stutter, which is difficult to track.

[0054] Furthermore, although the embodiments disclosed below relate specifically to visualization within the nasal passages, the principles of the invention may be similarly applied within other spaces of the body, particularly within narrow passages where actual optical endoscopes are unavailable or difficult to use.

[0055] Prior to a medical procedure, a CT image of the patient's head, including the sinuses, is acquired, along with a position tracking system, such as an electromagnetic tracking system, that is registered with the CT image. A position sensor is attached to the distal end of a guidewire or other instrument, and thus, as the distal end is inserted into the sinuses, the position and orientation of the distal end relative to the registered CT image is tracked. The CT image of the head is processed to generate and display a 3D volumetric image of the nasal passages.

[0056] Within this 3D volume, an operator of the imaging system (such as a surgeon performing a sinus dilation procedure) can select a start point and an end point for a 3D path along which to navigate a medical instrument. The appropriate 3D path from the start point to the end point is calculated, for example, using a path-finding algorithm and data from the CT image indicating which voxels of the CT image contain suitable materials to be traversed, such as air or liquid.

[0057] The calculated 3D path is automatically divided into segments, and the turning points between the segments are above a threshold turning value. Virtual cameras are positioned around these turning points. If there is no line of sight between the virtual cameras positioned at the turning points and / or the distance between the turning points exceeds a given value, additional virtual cameras can be automatically positioned.

[0058] The orientation of the optical axis of each virtual camera is calculated. Any suitable method can be used to calculate the orientation. In some embodiments, the orientation can be calculated based on the average direction of the vector from the virtual camera to the position of the next virtual camera along the path. In other embodiments, the orientation can be calculated as a direction parallel to the path at the position of the corresponding virtual camera. The field of view of the virtual camera can be fixed, for example, fixed to 90 degrees or any suitable value, or set according to the outer limits of the relevant section of the path served by the corresponding virtual camera, with additional tolerance to allow deviation from the path.

[0059] In some embodiments, the position of the virtual camera can be offset backward in a direction opposite to the calculated average direction. The virtual camera can be offset backward by any suitable distance, for example, until the camera is offset backward against solid material, such as tissue or bone. Offsetting the virtual camera backward can result in a better view of the medical device within the corresponding virtual endoscopic image, particularly when the medical device is very close to the corresponding virtual camera, and can result in a better view of surrounding anatomical structures.

[0060] As the medical tool moves through the pathway, a corresponding virtual camera is selected according to a camera selection method for rendering and displaying a corresponding virtual endoscopic image. In some embodiments, the camera selection method includes finding the camera closest to the tracking coordinates of the medical instrument, and then finding which side of the bisector (plane) associated with the nearest camera the tracking coordinates fall on. If the tracking coordinates fall on the side of the bisector that is further downstream from the calculated path (in the direction of travel of the medical instrument) of the nearest camera, the nearest camera is selected for rendering. If the tracking coordinates fall on the side of the bisector of the nearest current virtual camera, the current virtual camera continues to provide its endoscopic image. The bisector associated with the nearest camera can be defined as a plane perpendicular to the calculated path at the point of the nearest camera. In other embodiments, the pathway can be divided into multiple regions based on segments, where the virtual camera is selected based on the region in which the tracking coordinates are set.

[0061] The transition between the two virtual cameras, and therefore the transition between the associated virtual endoscopic images, can be a smooth transition or a sharp transition. In some embodiments, a smooth transition between the two corresponding virtual cameras can be performed by finding the position of an additional virtual camera on the path between the two virtual cameras, and then continuously rendering the corresponding transition virtual endoscopic images viewed from the additional virtual camera position.

[0062] System Description

[0063] Now see Figure 1 , which is a partially schematic, partially block diagram illustration of a medical device 20 constructed and operative in accordance with an embodiment of the present invention. In the following description, it is assumed that a medical device 21 of the device 20 is used to perform a medical procedure on a patient 22. The medical device 21 is configured to move within a passageway in the body of the patient 22.

[0064] The medical device 20 includes a position tracking system 23 that is configured to track the coordinates of the medical device 21 within the body. In some embodiments, the position tracking system 23 includes an electromagnetic tracking system 25 that includes one or more magnetic field generators 26 positioned around the body part and one or more magnetic field sensors 32 at the distal end of the medical device 21. In one embodiment, the magnetic field sensors 32 include a single-axis coil and a dual-axis coil that act as magnetic field sensors and are tracked by the electromagnetic tracking system 25 during the procedure. In order for the tracking to be effective, in the device 20, a CT (computed tomography) image of the patient 22 is registered with the reference frame of the electromagnetic tracking system 25, as described in more detail. Figure 2 and Figure 3 Although CT images may generally include magnetic resonance imaging (MRI) images or fluoroscopic images, in the description herein, the images are assumed to include fluoroscopic CT images by way of example. In some embodiments, position tracking system 23 may track the coordinates of medical device 21 using any suitable tracking method, such as based on current or impedance distribution on body surface electrodes, or based on an ultrasonic transducer.

[0065] Before and during sinus surgery, a magnetic radiator assembly 24, included in an electromagnetic tracking system 25, is positioned beneath the patient's head. Magnetic radiator assembly 24 includes a magnetic field generator 26 that is fixed in position and transmits an alternating magnetic field into a region 30 where the patient's 22 head is located. The potential generated by a single-axis coil of a magnetic field sensor 32 in region 30 in response to the magnetic field enables the position and orientation of the single-axis coil to be measured in the reference frame of the magnetic tracking system. Position can be measured in three linear dimensions (3D), and orientation can be measured with respect to two axes orthogonal to the axis of symmetry of the single-axis coil. However, the orientation of the single-axis coil relative to its axis of symmetry cannot be determined from the potential generated by the coil.

[0066] The same is true for each of the two coils of the dual-axis coil of magnetic field sensor 32. That is, for each coil, the position in 3D can be measured, and the orientation relative to two axes orthogonal to the coil's axis of symmetry can also be measured, but the orientation of the coil relative to its axis of symmetry cannot be determined.

[0067] By way of example, radiators 26 of assembly 24 are arranged in a generally horseshoe shape about the head of patient 22. However, alternative configurations of the radiators of assembly 24 will be apparent to those skilled in the art, and all such configurations are considered to be within the scope of the present invention.

[0068] Prior to surgery, registration of the magnetic tracking system's reference frame with the CT image can be performed by positioning a magnetic sensor at a known location in the image, such as the end of the patient's nose. Figure 2 As described in more detail, any other convenient system for reference frame registration may be used.

[0069] The components of device 20, including radiator 26 and magnetic field sensor 32, are under the overall control of a system processor 40. Processor 40 may be mounted in a console 50 that includes operating controls 58, which typically include a keypad and / or a pointing device, such as a mouse or trackball. Console 50 is connected to radiator and magnetic field sensor 32 via one or more cables 60 and / or wirelessly. A physician 54 uses operating controls 58 to interact with processor 40 while performing a medical procedure using device 20. As the procedure is performed, the processor may present the results of the procedure on display screen 56.

[0070] Processor 40 uses software stored in memory 42 to operate device 20. The software may be downloaded to processor 40 in electronic form, for example, over a network, or alternatively or additionally, the software may be provided and / or stored on non-transitory tangible media such as magnetic, optical or electronic memory.

[0071] Now see Figure 2 , which is according to an embodiment of the present invention included in Figure 1 Flowchart 70 of steps in a three-dimensional path visualization method used in apparatus 20 .

[0072] Position tracking system 23( Figure 1 ) is configured to track (block 72) the medical device 21 ( Figure 1 )'s distal end in the body. Processor 40 ( Figure 1 ) aligns the position tracking system 23 and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body in a common reference frame (box 74). The alignment can be performed by any suitable alignment technique. For example, but not limited to, the alignment method described in U.S. Patent Publication 2017 / 0020411 or 2019 / 0046272. As described in the latter patent publication, for example, the processor 40 can analyze the CT image to identify the corresponding positions of the patient's eyes in the image, thereby defining a line segment connecting these corresponding positions. In addition, the processor 40 identifies a subset of voxels in the CT that covers the bone portion of the head along a second line segment parallel to the first line segment and a third line segment perpendicular to the first line segment. The physician 54 positions the probe near the bone portion and thereby measures the position on the surface of the head covering the bone portion. The processor 40 calculates the correspondence between these measured positions in the CT image and the voxel subset, and thereby aligns the magnetic tracking system 25 with the CT image.

[0073] The processor 40 is configured to find (block 76) a 3D path of the medical device 21 through the passage from a given starting point to a given end point. Figure 3-Figure 9 The path finding method is described in more detail.

[0074] The processor 40 is configured to calculate (block 78) segments of the calculated 3D path. The processor 40 is configured to calculate (block 80) corresponding different positions along the 3D path of the corresponding virtual camera in response to the calculated segments. The steps of blocks 78 and 80 refer to Figure 10-12 Described in more detail.

[0075] The processor 40 is configured to select (block 82) a corresponding virtual camera for rendering a corresponding virtual endoscopic image in response to the tracking coordinates of the medical device 21 and the corresponding positions of the corresponding virtual cameras within a common reference frame. As the medical device 21 moves along the 3D path (which may be a distance to either side of the path because the medical device 21 is not locked to the path), a virtual camera that provides a virtual endoscopic image is selected based on the tracking coordinates of the medical device 21, and control is continuously passed from one virtual camera to another as the medical device 21 moves along the path. The steps of block 82 may be repeated intermittently, for example, each time new tracking coordinates are received, for example, in a range of between 10 milliseconds and 100 milliseconds, such as 50 milliseconds. The steps of block 82 refer to Figure 13-15 Described in more detail.

[0076] The processor 40 is configured to calculate (block 84) the respective orientations of the respective virtual cameras. The orientations of the cameras are typically 3D orientations and are defined relative to the respective optical axes of the respective virtual cameras. In other words, the orientation of the camera is a measure of the direction in which the camera faces the optical target. The position of the virtual camera may also be as described in Figure 16 and Figure 18 The orientation and / or backward movement may be calculated as part of the process of selecting a camera, before selecting a camera, or when the medical instrument 21 leaves the field of view of the virtual camera currently providing the virtual endoscopic image.

[0077] The processor 40 is configured to display the Figure 1) renders and displays (block 86) corresponding virtual endoscopic images based on the 3D CT images of the pathway in the body as viewed from the corresponding position and orientation of the corresponding virtual camera, including an animated representation of the medical instrument 21 positioned in the corresponding virtual endoscopic image according to the tracking coordinates. In other words, based on the position and orientation of the medical instrument 21 and the previously acquired CT data, the processor 40 renders the corresponding images at the step of block 86 as they are captured by the corresponding virtual camera and presents the images on the display screen 56. The image rendered by any given selected virtual camera is a projection of a portion of the 3D volume that would be visible from the camera position onto the camera's virtual image plane. The step of block 86 refers to Figure 21-23 Described in more detail.

[0078] Now see Figure 3 , which includes the use of Figure 1 Flowchart 90 of the steps of a method of device 20 finding a path through a passage. Figure 4-Figure 9 To show Figure 3 90 is a schematic diagram of the steps of the method. Referring to the flowchart 90, the pre-planning components are typically described in the patient 22 ( Figure 1 ) is performed before performing an invasive surgical procedure and determines the subsequent invasive medical device 21 ( Figure 1 ) is the optimal path. Assume that the pre-planning is performed by the physician 54 ( Figure 1 ).

[0079] In an initial step (box 100 of flowchart 90), a computed tomography (CT) X-ray scan of the sinuses of patient 22 is performed, and data from the scan is acquired by processor 40. As is known in the art, the scan comprises two-dimensional X-ray "slices" of patient 22, and the combination of the slices produces three-dimensional voxels, each having a Hounsfield unit, which is a measure of radiation intensity determined by the CT scan.

[0080] In the image generation step (box 102), the physician 54 ( Figure 1 ) displays the scan results on the display screen 56 ( Figure 1 As is known in the art, the results can be displayed as a series of two-dimensional (2D) slices, typically along planes parallel to the sagittal, coronal, and / or transverse planes of the patient 22, but other planes are also possible. The orientation of the planes can be selected by the physician 54.

[0081] The displayed result is usually a grayscale image and is Figure 4 An example is provided in , which is a slice parallel to the coronal plane of patient 22. The grayscale values ​​from black to white can be related to the Hounsfield units (HU) of the corresponding voxels, such that when applied to Figure 4When viewing an image of a human eye, air with HU=-1000 may be assigned black, and dense bone with HU=3000 may be assigned white.

[0082] As is known in the art, the value of Hounsfield units for any substance or species (such as compact bone), other than the values ​​for air and water (which are defined as -1000 and 0, respectively), depends, among other things, on the spectrum of the irradiating X-rays used to generate the CT scans referred to herein. The spectrum of the X-rays, in turn, depends on many factors, including the potential applied to the X-ray generator (in kilovolts (kV)) and the composition of the generator's anode. For clarity, in this disclosure, Hounsfield unit values ​​for specific substances or species are given in Table 1 below.

[0083] Species / Substance Hounsfield Unit Air -1000 soft tissue -300 to -100 Fat -50 water 0 blood +30 to +45 compact bone +3000

[0084] However, the values ​​of HU for specific species (other than air and water) given in Table 1 should be understood to be purely exemplary, and a person skilled in the art will be able to modify these exemplary values ​​depending on the species and X-ray machine used to generate the CT images mentioned herein without undue experimentation.

[0085] Typically, the translation between HU values ​​and grayscale values ​​is encoded in a DICOM (Digital Imaging and Communications in Medicine) file, which is the output of a CT scan from a given CT machine. For clarity, in the following description, a correlation of HU = -1000 with black, HU = 3000 with white, and intermediate HU values ​​with corresponding intermediate grayscale levels is used, but it should be understood that such correlations are merely arbitrary. For example, the correlations could be "reversed," i.e., HU = -1000 could be assigned to white, HU = 3000 to black, and intermediate HU values ​​to corresponding intermediate grayscale levels. Thus, one of ordinary skill in the art will be able to adapt the description herein to include other associations between Hounsfield units and grayscale levels, and all such associations are assumed to be included within the scope of the present invention.

[0086] In the marking step (block 104), the physician 54 ( Figure 1 ) marks the intended starting point where he / she places the medical device 21 ( Figure 1 )Insert patient 22( Figure 1) and marks the intended endpoint where the distal end of the medical device 21 will rest. Both points can be on the same 2D slice. Alternatively, each point can be on a different slice. Typically, but not necessarily, both points are in air, i.e., where HU = -1000, and the endpoint is typically, but not necessarily, at the junction of air and the liquid or tissue shown in the slice. An example of an endpoint not being at such a junction is when the point may be in the middle of an air-filled cavity.

[0087] Figure 5 The start point 150 and end point 152 are shown as marked by the physician on the same 2D slice, and for clarity, unless otherwise stated, it is assumed that these points are the points used in the remaining description of the flowchart. Typically, the start point and end point are shown in a non-grayscale color, such as red.

[0088] In the permissible path definition step (box 106), the physician defines a range of Hounsfield units to be used by the path finding algorithm (mentioned below) as acceptable voxel values ​​when finding a path from the start point 150 to the end point 152. The defined range typically includes HU equal to -1000, corresponding to air or voids in the path; the defined range may also include HU greater than -1000. For example, the range may be defined as given by expression (1):

[0089] {HU|-1000≤HU≤U} (1)

[0090] where U is a value chosen by the physician.

[0091] For example, U may be set to +45 so that the path taken may include water, fat, blood, soft tissue, and air or voids. In some embodiments, the range may be determined by the processor 40 ( Figure 1 ) settings without physician intervention.

[0092] The defined range of values ​​need not be a continuous range, and the range may be disjoint, comprising one or more sub-ranges. In some embodiments, the sub-ranges may be selected to include specific types of materials. An example of a disjoint range is given by expression (2):

[0093] {HU|HU=-1000 or A≤HU≤B} (2)

[0094] Where A and B are the values ​​chosen by the physician.

[0095] For example, A and B may be set equal to -300 and -100, respectively, so that the path taken may include air or voids and soft tissue.

[0096] The method of selecting the HU range may include any suitable method, including but not limited to numbers and / or material names and / or grayscale. For example, in the case of selecting by grayscale, the physician 54 ( Figure 1 ) One or more regions of the CT image can be selected, and the equivalent HU of the grayscale value of the selected region is contained in an acceptable HU range for determining the voxels of the path by the path finding algorithm.

[0097] In the case of selection by name, a table of named species can be presented to the physician. The presented table is generally similar to Table 1, but without the column providing the value in Hounsfield units. The physician can select one or more named species from the table, in which case the equivalent HU of the selected named species is within the acceptable HU range for the voxel to be pathed by the pathfinding algorithm.

[0098] In the path finding step (block 108), the processor 40 ( Figure 1 ) implements a path finding algorithm to find the medical device 21 between the starting point 150 and the end point 152 ( Figure 1 ) to follow. The algorithm assumes that traversable voxels in a path include any voxel with a HU within the HU range defined in the step of block 106, and that voxels with HU values ​​outside of this defined range act as barriers in any path found. While the path-finding algorithm used may be any suitable algorithm capable of determining the shortest path within a three-dimensional maze, the inventors have found that a Flood Fill algorithm, Dijkstra's algorithm, or an extension of the A* algorithm, for example, provides better results in terms of computational speed and accuracy in determining the shortest path than other algorithms (e.g., Floyd's algorithm or variants thereof).

[0099] In some embodiments, the path finding step includes taking into account the mechanical properties and dimensions of the medical device 21 ( Figure 1 For example, in the disclosed embodiment, the medical device 21 may be constrained to a range of possible curvature radii when it is bent. When determining the possible paths that the medical device 21 may follow, the processor 40 ( Figure 1 ) ensures that no part of the path is constrained to a radius smaller than this radius.

[0100] In another disclosed embodiment, the processor 40 ( Figure 1 ) Consider medical devices 21( Figure 1) that allow different portions of the medical device 21 to have different ranges of curvature radii. For example, the end of a possible path may have a smaller radius of curvature than the possible radius of curvature of the proximal portion of the medical device 21. However, the distal end of the medical device 21 may be more flexible than the proximal portion and may be flexible enough to accommodate the smaller radius of curvature so that the possible path is acceptable.

[0101] When considering medical devices 21( Figure 1 ) and different curvature radii of possible paths, the processor 40 ( Figure 1 ) takes into account which parts of the path need to be traversed by different parts of the medical device 21 as the distal end of the medical device 21 moves from the starting point 150 to the end point 152, and the curvature radius that the medical device 21 can achieve.

[0102] In another disclosed embodiment, the processor 40 ( Figure 1 ) ensures that the path diameter D is always greater than the measured diameter d of the medical device 21. As is known in the art, the validation may be performed at least in part, for example, by the processor 40 using an erosion / dilation algorithm to find voxels within the range defined in the step of block 106.

[0103] In the superimposition step (box 110 ), the shortest path found in the box 108 step is superimposed on the image displayed on the display screen 56 . Figure 6 It is shown that the starting point 150 and the end point 152 have been superimposed on Figure 5 The shortest path 154 is displayed on the image. Typically, the path 154 is displayed in a non-grayscale color, which may or may not be the same color as the starting and ending points. In the event that more than one shortest path is found at block 108, all such paths may be superimposed on the image, typically in different non-grayscale colors.

[0104] Typically, the found path traverses more than one 2D slice, in which case superposition can be performed by combining the found path into all relevant 2D slices, i.e., all 2D slices traversed by the path. Alternatively or in addition, an at least partially transparent 3D image can be generated from the scanned 2D slices, and the found path can be superimposed on the 3D image. The at least partially transparent 3D image can be formed on a representation of the outer surface of the patient 22, as described in more detail below.

[0105] Figure 7 is a representation of the outer surface 180 of the patient 22 according to an embodiment of the present invention. The processor 40 ( Figure 1) uses the CT scan data acquired in the step of box 100 to generate a representation of the outer surface by using the fact that the HU value of air is -1000, while the skin has a significantly different HU value. For example, assume that the representation 180 is formed in a plane parallel to the coronal plane of the patient 22, that is, parallel to the xy plane of the reference frame 184 defined by the patient 22, whose axis is also Figure 7 and Figure 8 Draw in.

[0106] Figure 8 Schematically illustrates a boundary plane 190 and a boundary region 192 according to an embodiment of the present invention. Figure 1 ) in the direction of the processor 40 ( Figure 1 ) Optionally delineate areas of the representation 180 that are to be rendered transparent, and those areas that are to be left “as is.” To perform the delineation, the physician defines a boundary plane 190 and a boundary region 192 in a boundary plane using a boundary perimeter 194 of the region 192 .

[0107] For clarity, the following description assumes that the boundary plane is parallel to the xy plane of the reference frame 184, as shown in FIG. Figure 8 It is shown schematically and has the equation given below:

[0108] z=z bp (3)

[0109] As described below, processor 40 uses bounding planes and bounding regions 192 to determine which elements of surface 180 are to be rendered as partially transparent, and which elements are not to be so rendered.

[0110] Processor 40 determines the elements of surface 180 ( Figure 7 ), which has z>z bp and are located within the boundary region 192 when projected along the z-axis. The processor 40 then renders the elements as transparent, thus making them no longer visible in the surface 180. For example, Figure 8 , the nose tip 196 of patient 22 has a value z>z bp , thus dashed line 198 near the tip of the patient's nose illustrates the portion of outer surface 180 that is no longer visible when an image of the surface is presented on display screen 56 ( Figure 1 ).

[0111] Because the elements described above are rendered transparent, with a value of z <z bpAnd the elements of surface 180 that are within boundary region 192 are now visible when projected along the z-axis and are therefore displayed in the image. Before the local transparent rendering, the "now visible" elements were not visible because they were obscured by the surface elements. The now visible elements include the elements of shortest path 154, such as Figure 9 shown.

[0112] Figure 9 Schematically shows after local transparent rendering of elements of the surface within the boundary area 192 ( Figure 8 ), on the display screen 56( Figure 1 ) is shown on the surface 180. For clarity, the corresponding boundary perimeter 194 ( Figure 8 ) has been superimposed on the image, and the reference frame 184 is also drawn in the figure. Due to the transparent rendering of the elements within the circle 194A, the area 200 within the circle now shows the internal structure of the patient 22 derived from the CT tomography data received in the step of box 100 ( Figure 1 ).

[0113] Figure 9 Also drawn is shortest path 154. Due to the transparent rendering of the elements within circle 194A, a portion of the path is now visible in the image of surface 180 and has been drawn as a solid white line 202. The portion of the path that is not visible because it is obscured by elements of surface 180 that have not been rendered transparently is represented by a dashed white line 204.

[0114] It should be understood that Figure 7 and Figure 9 In the illustrated case, the image shown on display screen 56 is a view of patient 22 viewed along the z-axis of the xy plane.

[0115] The above description provides one example of applying local transparency to view a shortest path derived from tomographic data, in this case, the local transparency is formed relative to a plane parallel to the coronal plane of patient 22. It should be understood that due to the three-dimensional nature of the tomographic data, the data can be manipulated such that embodiments of the present invention can use local transparency formed relative to substantially any plane through patient 22 to view the shortest path 154, and can be defined in reference frame 184.

[0116] When forming the local transparency, the size and position of the boundary plane 190 and the boundary area 192 can be changed to make the doctor 54 ( Figure 1 ) can also view the shortest path 154 and the internal structure near the path 154.

[0117] Physician 54 can change the orientation of bounding plane 190, for example, to enhance visibility of specific internal structures. While bounding plane 190 is typically parallel to the plane of the image presented on display screen 56, this is not required, so if, for example, physician 54 wants to see more detail of a specific structure, she / he can rotate bounding plane 190 so that it is no longer parallel to the image plane.

[0118] In some cases, the HU value / grayscale range selected in the step of block 106 includes regions other than air, such as regions corresponding to soft tissue and / or mucus. The path 154 found in the step of block 108 may include such regions, and in such cases, for the medical device 21 ( Figure 1 ), it may be necessary to clear these areas, such as by debridement. In the optional warning step (box 112), the doctor 54 ( Figure 1 ) is notified of the presence of areas of path 154 that are not in air, for example by highlighting the relevant portion of path 154 and / or by other visual or auditory cues.

[0119] While the above description assumes a CT scan as an X-ray scan, it will be appreciated that embodiments of the present invention include finding the shortest path using MRI (magnetic resonance imaging) tomographic images.

[0120] Thus, referring back to flowchart 90, in the case of MRI images, where Hounsfield values ​​may no longer be directly applicable, in step 106, the physician 54 ( Figure 1 ) defines a range of grayscale values ​​(of the MRI image) that the path-finding algorithm uses as acceptable voxel values ​​when finding a path from the start point 150 to the end point 152. In the step of block 108, the path-finding algorithm assumes that traversable voxels in the path include any voxels with a grayscale within the grayscale range defined in the step of block 106, and that voxels with grayscale values ​​outside of this defined range act as barriers in any path found. Other variations of the above description to accommodate the use of MRI images rather than X-ray CT images will be apparent to those of ordinary skill in the art, and all such variations should be considered to be within the scope of the present invention.

[0121] Now see Figure 10-12 . Figure 10 To include the use Figure 1 Flowchart 300 of steps in a method for device 20 to calculate segments of path 154 and calculate positions of virtual camera 320 along path 154. Figure 11 and Figure 12 To show Figure 10 Flowchart 300 is a schematic diagram of the steps of the method.

[0122] Processor 40( Figure 1 ) is configured to find (block 302) a turning point 324 in the 3D path 154 that is above a threshold turn, and in response to the found turning point 324, calculate a segment 322 of the 3D path 154 and a corresponding different position along the 3D path 154 of the corresponding virtual camera 320, as Figure 11 shown.

[0123] The sub-steps of the step of block 302 are now described below.

[0124] Processor 40( Figure 1 ) is configured to calculate (block 304) segments 322 based on n-dimensional multiline segment simplification. In some embodiments, n-dimensional multiline segment simplification includes, by way of example, a Ramer–Douglas–Peucker algorithm, a Visvalingam–Whyatt algorithm, or a Reumann-Witkam algorithm. Any suitable algorithm that simplifies n-dimensional multiline segments into polylines of smaller size may be used. The algorithm generally analyzes the path 154 removing small turning points while leaving larger turning points such that the larger turning points 324 define segments 322 between the turning points 324. The threshold turning value corresponding to the turning points removed from the path 154 may be set by configuring the parameters of the algorithm used. For example, the input parameters of the Ramer–Douglas–Peucker algorithm may be set to approximately 0.08.

[0125] In other embodiments, processor 40 is configured to calculate segment 322 using any suitable algorithm such that turning points below a threshold turning value are removed.

[0126] Processor 40( Figure 1 ) is configured to position (box 306) the virtual camera 320 at or around the turning point 324 between the segments 322, and at the starting point 150 and optionally at the end point 152 of the path 154. Figure 11 Path 154 in passage 328 is shown simplified with one turning point 324 and two segments 322. Three virtual cameras 320 have been placed on path 154 at the start point 150, turning point 324, and end point 152, respectively. The small turning point on path 154 (indicated by dashed oval 326) is removed by n-dimensional multi-segment simplification, leaving turning point 324.

[0127] The processor 40 is configured to check (block 308) the line of sight between two adjacent virtual cameras 320 and position one or more virtual cameras 320 between the two adjacent virtual cameras 320 in response to the line of sight being blocked. The line of sight may be checked by examining voxels of the 3DCT image to determine whether there is material that blocks the line of sight between the adjacent virtual cameras 320. The type of material that is considered to block or not block the line of sight may be compared to the type of material that is considered to block or not block the line of sight when calculating the reference image. Figure 3 The steps of box 106 are the same as those of path 154 described above. In some embodiments, different criteria may be used. For example, the doctor 54 may set the material that blocks the line of sight to bone and hard tissue, thereby limiting air, liquid, and soft tissue as materials that do not block the line of sight. In some cases, the doctor 54 may set the material that blocks the line of sight to bone, hard tissue, and soft tissue. Alternatively, instead of specifying a material that blocks the line of sight, the doctor 54 may set a material that does not block the line of sight, such as air or liquid. In some embodiments, the processor 40 is configured to check that the direct line of sight between two adjacent virtual cameras 320 is not blocked. In other embodiments, the processor 40 may be configured to check that the direct line of sight between two adjacent virtual cameras 320 is not blocked, including checking that a given tolerance around the line of sight between the two virtual cameras 320 is not blocked. The given tolerance around the line of sight can have any suitable value. For example, Figure 11 The line of sight between virtual camera 320 - 2 and virtual camera 320 - 3 along segment 322 - 2 is shown to be blocked by a portion of tissue 330 . Figure 12 It is shown that another virtual camera 320 - 4 has been added between the virtual camera 320 - 2 and the virtual camera 320 - 3 . Figure 11 It is also shown that although the direct line of sight between virtual camera 320 - 1 and virtual camera 320 - 2 is not blocked, when a given tolerance around the line of sight is considered, the line of sight extended by the given tolerance is blocked by a portion of tissue 332 . Figure 12 Another virtual camera 320-5 is shown to have been added between virtual camera 320-1 and virtual camera 320-2. Once the additional virtual camera 320 has been added, processor 40 may check the line of sight (or extended line of sight) between adjacent virtual cameras 320 based on the initial virtual camera 320 plus the additional virtual camera 320.

[0128] Processor 40 is optionally configured to position (block 310 ) one or more additional virtual cameras 320 intermediate one or more sections 322 in response to a distance between existing virtual cameras 320 exceeding a limit. Figure 123. Virtual cameras 320-6 and 320-7 are shown as being added to segment 322-1 in response to the distance between existing virtual cameras 320 exceeding a limit. The limit may be any suitable limit, such as, but not limited to, within the range of 1 mm to 20 mm, such as 4 mm. The additional cameras 320 are typically evenly spaced between the existing virtual cameras 320.

[0129] Now refer to Figure 13-15 . Figure 13 To include the use Figure 1 Flowchart 340 of the steps in a method of selecting a camera for device 20. Figure 14 and Figure 15 To show that Figure 13 Flow chart 340 is a schematic diagram of the steps of the method.

[0130] Processor 40( Figure 1 ) is configured to calculate (block 342) a corresponding bisector 350 ( Figure 14 In some embodiments, processor 40 is configured to compute respective bisectors 350 as respective planes perpendicular to 3D path 154 at respective locations of respective virtual cameras 320 on 3D path 154 . Figure 14 A bisector 350 is shown for each virtual camera 320. The bisector 350 may be calculated at any time after the path 154 has been calculated, until when the medical device 21 is in close proximity to the corresponding virtual camera 320, as described below at block 344.

[0131] The processor 40 is configured to find (block 344) the virtual camera 320 (e.g., virtual camera 320-7) closest to the distal end of the medical device 21 in response to the tracked coordinates of the medical device 21 and the known positions of the virtual cameras 320 ( Figure 1). The processor 40 is configured to find (block 346) which side of the bisector 350 of the nearest virtual camera 320 (e.g., virtual camera 320-7) the tracking coordinates fall on. The processor 40 is configured to select (block 348) one of the virtual cameras 320 for rendering the virtual endoscopic image based on which side of the bisector 350 of the nearest virtual camera 320 (e.g., virtual camera 320-7) the tracking coordinates of the medical device 21 fall on. If the tracking coordinates fall on the side of the bisector 350 that is closer to the current virtual camera 320 (e.g., virtual camera 320-6), the current virtual camera (e.g., virtual camera 320-6) is still used. If the tracking coordinates fall on the side of the bisector 350 that is farther away from the current virtual camera 320 (e.g., virtual camera 320-6), the next virtual camera 320 along the path (i.e., virtual camera 320 (e.g., virtual camera 320-7)) is selected as the new virtual camera 320. The steps of blocks 344-346 are repeated intermittently.

[0132] The steps in boxes 344-348 are Figure 15 Shown in. Figure 15 Two virtual cameras 320-6 and 320-7 are shown. Virtual camera 320-6 is the current virtual camera used to render and display the virtual endoscopic image of pathway 328. Relative to the direction of travel of medical device 21, virtual camera 320-7 is further downstream in path 154 than virtual camera 320-6. In other words, virtual camera 320-7 is closer to endpoint 152 ( Figure 11 ). Figure 15 Various possible example positions 352 of the distal end of the medical device 21 are shown. All positions 352 are closer to the virtual camera 320-7 than to the virtual camera 320-6. Figure 15 Of all the positions 352 shown, virtual camera 320-7 will be found to be the closest virtual camera 320 in the step of box 344. Once the closest virtual camera 320-7 is found, the position of the distal end of the medical instrument 21 relative to the bisector 350 of the closest virtual camera 320-7 is checked in the step of box 346. Figure 15 In the example of , position 352-6 is located on one side of the bisector 350 closest to virtual camera 320-6 (the current virtual camera) (indicated by arrow 354), and position 352-7 is located on the other side of the bisector 350 farther from the current virtual camera 320-6 (indicated by arrow 356). Figure 15In the example shown in FIG. 3 , if the tracking coordinates of the distal end of the medical device 21 are at any of the positions 352-6 or similar positions, the current virtual camera 320-6 will remain as the selected virtual camera according to the steps of block 348. If the tracking coordinates of the distal end of the medical device 21 are at any of the positions 352-7 or similar positions, the virtual camera 320-7 will be selected as the new virtual camera.

[0133] Therefore, the processor 40 is configured to respond to the medical device 21 ( Figure 1 ) falls on which side of the corresponding bisector 350 of the corresponding virtual camera 320 that is closest to the tracking coordinates, to select the corresponding virtual camera 320 for rendering the corresponding virtual endoscopic image.

[0134] In other embodiments, the passage may be divided into a plurality of regions based on the segments 322, wherein the virtual camera 320 is selected according to the region in which the tracking coordinates are set.

[0135] Now see Figure 16 , which includes the use of Figure 1 Flowchart 360 of steps in a method of computing the orientation and offset of a virtual camera 320 position on a device 20. See also Figure 17 and Figure 18 , they are shown Figure 16 Flow chart 360 is a schematic diagram of the steps of the method.

[0136] The orientation of the optical axis of each virtual camera 320 is calculated. The orientation can be calculated at any time after the position of the virtual camera 320 has been calculated. In some embodiments, the orientation of the virtual camera 320 can be calculated after each virtual camera 320 is selected to be used as a medical device 21 (moved along the path 154) Figure 1 ) is calculated. A method for calculating the orientation of a virtual camera 320 is now described below.

[0137] Processor 40( Figure 1 ) is configured to select (block 362) a position 370 on the path 154 from one virtual camera 320-2 to another virtual camera 320-4, such as Figure 17 As shown. Positions 370 may be selected to include or exclude the position of virtual camera 320-4. Positions 370 may be selected by dividing segment 322 between virtual camera 320-2 and virtual camera 320-4 into subsegments. Alternatively, positions 370 may be selected by measuring a given distance from virtual camera 320-2 to each position 370 along path 154. In some embodiments, when path 154 is generated, positions 370 may be selected using points defining path 154. Processor 40( Figure 1) is configured to define a vector 372 from the virtual camera 320-2 to the position 370, and calculates (block 364) as Figure 17 The average direction 374 of the vector 372 is shown. Thus, the processor 40 is configured to calculate the average direction of the vector from the position of the virtual camera 320-2 to different points (e.g., the position 370) along the 3D path 154. The processor 40 ( Figure 1 ) is configured to calculate (block 366) an orientation of the virtual camera 320-2 in response to the calculated average direction 374. In other words, the orientation of the optical axis of the virtual camera 320 is calculated as the average direction 374. In other embodiments, the orientation may be calculated as a direction parallel to the path at the location of the corresponding virtual camera 320-2.

[0138] In some embodiments, the respective positions of the respective virtual cameras 320 (e.g., virtual camera 320-2) may be offset, for example, in opposite directions 376 backwards to the respective average directions 374 of the respective virtual cameras 320, as shown in FIG. Figure 18 As shown. Offsetting the virtual camera 320 back can produce a better view of the medical device 21 within the corresponding virtual endoscopic image, especially when the medical device 21 is very close to the corresponding virtual camera 320, and can produce a better view of the surrounding anatomical structures. Therefore, the processor 40 ( Figure 1 ) is configured to shift (block 368) the position of the virtual camera 320-2 to a new position 380 in a direction 376 opposite to the calculated average direction 374, as Figure 18 As shown. The degree of offset can be fixed, for example, a predetermined number of millimeters, such as within a range between 0.2 mm and 2 mm, such as 1.3 mm. Alternatively, the degree of offset can be limited by the surrounding anatomy 378 so that the camera 320-2 is offset as much as possible as long as it is not pushed into bone or tissue, as defined by the physician 54. The type of material considered by "surrounding anatomy" can be as described in detail in the accompanying drawings. Figure 3 The material standards used to define the path of the obstructing medical device 21 may be the same or different as described in the step of box 106.

[0139] The field of view of the respective virtual cameras 320 can be set to any suitable respective value. The field of view of each virtual camera 320 can be fixed, for example, within a range of values ​​between 25 degrees and 170 degrees, such as 90 degrees. The field of view of any one virtual camera 320 can be set according to the outer limits of the segment 322 of the path 254 that the virtual camera 320 covers (e.g., from Figure 17The vector 372 deviates from or by finding the outer limits of the segment 322, the additional tolerance of the virtual camera 320 (such as a given angular tolerance, for example, adding X degrees to the outer limits, where X can be any suitable value, for example, in the range of 5 degrees to 90 degrees). In some embodiments, by analyzing the surrounding anatomical structures 378 around the segment 322, the field of view can be set to cover all anatomical structures in the segment 322 until the next virtual camera. The material type considered for the "surrounding anatomical structures" can be the same as that of FIG. Figure 3 The material standards used to define the path of the obstructing medical device 21 may be the same or different as described in the step of box 106.

[0140] Now see Figure 19 , which includes Figure 1 Flowchart 390 of the steps in the rendering transition method in the system 20. Figure 20 , which shows Figure 19 Flow chart 390 is a schematic diagram of the steps of the method.

[0141] The transition between the two virtual cameras 320, and therefore the transition between the associated virtual endoscopic images, can be a smooth transition or a sharp transition. In some embodiments, a smooth transition between two corresponding virtual cameras can be performed by performing the following steps. By way of example, the transition between virtual camera 320-2 and virtual camera 320-4 is described.

[0142] Processor 40( Figure 1 ) is configured to find (block 392) the position of an additional virtual camera 396 to be added between the current virtual camera 320-2 and the next virtual camera 320-4, such as Figure 20 As shown. Figure 16 and Figure 17 The positions of the additional virtual cameras 396 are found in a similar manner to the selection of the positions 370 described. Any suitable number of additional virtual cameras 396 may be selected. A larger number of additional virtual cameras 396 will generally produce a smoother transition.

[0143] Processor 40( Figure 1 ) is configured to continuously render corresponding transitional virtual endoscopic images of the intracorporeal passage 328 observed from corresponding positions of corresponding additional virtual cameras 396 disposed between two adjacent virtual cameras 320-2, 320-4 ( Figure 20 ), on the display screen 56( Figure 1) and displays (block 394) a transition between two respective virtual endoscopic images of two respective adjacent virtual cameras 320-2, 320-4. Each of the transitional virtual endoscopic images may be displayed on the display screen 56 for any suitable duration, for example, a duration in the range of 20 milliseconds to 40 milliseconds.

[0144] Now see Figure 21-23 , which are composed of Figure 1 Schematic virtual endoscopic image 398 rendered and displayed by the device 20. As previously seen Figure 2 As mentioned, the processor 40 ( Figure 1 ) is configured to display the display 56 ( Figure 1 ) is rendered and displayed on the corresponding virtual camera 320 ( Figure 11 ) of the passage 328 in the body observed based on the 3D CT image, including the medical device 21 ( Figure 1 ) animation shows 400

[0145] Figure 21-22 A virtual endoscopic image 398-1 is shown as viewed from the position of a virtual camera 320. A representation 400 of the medical instrument 21 is shown in FIG. Figure 21 , shown at a location along path 154 in passage 328, and at Figure 22 3, it is shown at a more advanced position along the path 154 in the passage 328. The path 154 remains traversed using an arrow, which disappears as the medical device 21 moves along the path 154. Any suitable line or symbol may be used to represent the path 154. Figure 23 A virtual endoscopic image 398 - 2 viewed from the position of another virtual camera 320 is shown.

[0146] The virtual endoscopic image 398 can be rendered using a volume visualization technique that generates a virtual endoscopic image 398 in a 3D view volume (e.g., a cone projected outward from the position of the associated virtual camera 320) from tissue image data (e.g., based on the HU of voxels from a 3D CT scan). The image 398 can be rendered based on the known color of the tissue. Certain materials such as liquids or even soft tissues can be selected to be transparent, while all other denser materials can be rendered according to the natural color of the denser material. Alternatively, even liquids and soft tissues with denser materials can be rendered according to the natural color of the corresponding material. In some embodiments, the physician 54 ( Figure 1) can set the rendering parameters of the virtual endoscopic image 398 discussed above. In some embodiments, the surrounding anatomical structure 378 can be rendered, while other anatomical structures can be ignored. The material type considered for "surrounding anatomical structure" can be the same as that of FIG. Figure 3 The steps of block 106 are described above, and the material criteria used to define the path 154 cannot pass through are the same or different.

[0147] The above image 398 is presented for illustration purposes only, and other kinds of images may likewise be rendered and displayed in accordance with the principles of the present invention.

[0148] As used herein, the term "about" or "approximately" for any numerical value or range indicates a suitable dimensional tolerance that allows the component or collection of elements to achieve its intended purpose as described herein. More specifically, "about" or "approximately" can refer to a range of ±20% of the value of the recited value, for example, "about 90%" can refer to a range of values ​​from 72% to 108%.

[0149] For clarity, various features of the invention described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for simplicity, various features of the invention are described in the context of a single embodiment and may also be provided separately or in any suitable subcombination.

[0150] The above embodiments are cited by way of example, and the present invention is not limited by what has been specifically shown and described hereinabove. On the contrary, the scope of the present invention includes combinations and subcombinations of the various features described above, as well as variations and modifications thereof, which will occur to those skilled in the art upon reading the above description and which are not disclosed in the prior art.

Claims

1. A medical device comprising: a medical device configured to move within a passageway in a patient's body; a position tracking system configured to track coordinates of the medical device within the body; Display screen; as well as a processor configured to: registering the position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body within a common reference frame; finding a 3D path of the medical device through the passage from a given starting point to a given end point; calculating segments of the 3D path; calculating respective different positions along the 3D path of respective virtual cameras in response to the calculated segments; selecting the respective virtual camera for rendering a respective virtual endoscopic image in response to the tracked coordinates of the medical instrument and the respective position of the respective virtual camera within the common reference frame; calculating corresponding orientations of the corresponding virtual cameras; as well as Rendering and displaying on the display screen the corresponding virtual endoscopic image based on the 3D CT image of the passage in the body observed from the corresponding position and orientation of the corresponding virtual camera, including an animated representation of the medical instrument positioned in the corresponding virtual endoscopic image according to the tracking coordinates.

2. The device according to claim 1, wherein The processor is configured to: find a turning point in the 3D path that turns above a threshold; and calculate a segment of the 3D path and the corresponding different positions along the 3D path of the corresponding virtual camera in response to the found turning point.

3. The device according to claim 2, wherein The processor is configured to position at least one of the virtual cameras in the middle of one of the segments in response to a distance between adjacent ones of the virtual cameras exceeding a limit.

4. The device according to claim 2, wherein The processor is configured to: check a line of sight between two adjacent ones of the virtual cameras; and position at least one of the virtual cameras between the two adjacent ones of the virtual cameras in response to the line of sight being blocked.

5. The apparatus according to claim 2, wherein The processor is configured to calculate the segments based on n-dimensional multi-line segment simplification.

6. The device according to claim 5, wherein The n-dimensional multi-line segment simplification includes the Ramer–Douglas–Peucker algorithm.

7. The apparatus according to claim 1, wherein The processor is configured to: calculating corresponding bisectors of corresponding virtual cameras of the virtual cameras; and In response to which side the tracking coordinates of the medical instrument fall relative to a respective one of the bisectors of the respective one of the virtual cameras closest to the tracking coordinates, the respective virtual camera is selected for rendering a respective virtual endoscopic image.

8. The apparatus according to claim 7, wherein The processor is configured to calculate the respective bisectors as respective planes perpendicular to the 3D path at respective ones of the positions of respective ones of the virtual cameras on the 3D path.

9. The apparatus according to claim 1, wherein The processor is configured to: calculating an average direction of vectors from respective ones of the positions of respective ones of the virtual cameras to different points along the 3D path; and Respective ones of the orientations of respective ones of the virtual cameras are calculated in response to the calculated average direction.

10. The apparatus according to claim 9, wherein The processor is configured to shift respective ones of the positions of respective ones of the virtual cameras in a direction opposite to the calculated average direction.

11. The apparatus according to claim 1, wherein The processor is configured to render and display on the display screen a transition between two corresponding virtual endoscopic images of the two corresponding adjacent virtual cameras among the virtual cameras based on continuously rendering corresponding transitional virtual endoscopic images of the passage in the body observed from corresponding positions of a corresponding additional virtual camera arranged between the two corresponding adjacent virtual cameras among the virtual cameras.

12. The apparatus according to claim 1, wherein The position tracking system includes an electromagnetic tracking system comprising one or more magnetic field generators positioned about the portion of the body and a magnetic field sensor at a distal end of the medical device.

13. A software product comprising a non-transitory computer-readable medium storing program instructions, which, when read by a central processing unit (CPU), cause the CPU to: tracking coordinates of a medical device within a body of a patient using a position tracking system, the medical device being configured to move within a passageway in the body of the patient; registering the position tracking system and a three-dimensional (3D) computed tomography (CT) image of at least a portion of the body within a common reference frame; finding a 3D path of the medical device through the passage from a given starting point to a given end point; calculating segments of the 3D path; calculating respective different positions along the 3D path of respective virtual cameras in response to the calculated segments; selecting the respective virtual camera for rendering a respective virtual endoscopic image in response to the tracked coordinates of the medical instrument and the respective position of the respective virtual camera within the common reference frame; calculating corresponding orientations of the corresponding virtual cameras; as well as Rendering and displaying on a display screen the corresponding virtual endoscopic image based on the 3D CT image of the passage in the body as viewed from the corresponding position and orientation of the corresponding virtual camera, including an animated representation of the medical instrument positioned in the corresponding virtual endoscopic image according to the tracking coordinates.

Images