System and method for detecting and correcting patient and / or imaging system movement for target coverage

Abstract

Systems and methods for visualizing navigation of a medical device relative to a target using a live fluoroscopic video and updating a location of the target overlaid on the live fluoroscopic video. These systems and methods involve updating a 2D position of a target coverage based on patient movement sensed by a patient reference sensor. In various aspects, a medical device tip may be automatically detected, thus eliminating the need for a frame of reference. A setup procedure is automatically performed in response to detecting movement of the fluoroscopic imaging system. The pose estimation may be optimized by using an aperiodic grid. The systems and methods may incorporate a digital zoom that may translate a fluoroscopic image to focus on the target. The systems and methods may involve tracking which portion of the target is iteratively covered by a previous procedure.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to and the benefits of U.S. Application No. 63 / 416,334, filed on October 14, 2022, and U.S. Application No. 63 / 429,147, filed on December 1, 2022. Technical Field

[0003] The present disclosure relates to the field of visualizing the navigation of a medical device, such as a biopsy tool or an ablation tool, relative to a target, confirming the relative position, and monitoring and compensating for movement of the patient and / or live imaging. Background Art

[0004] There are several commonly used medical methods (such as endoscopy or minimally invasive surgery) for treating various diseases affecting organs, including the liver, brain, heart, lungs, gallbladder, kidneys and bones. Typically, clinicians use one or more imaging methods such as magnetic resonance imaging (MRI), ultrasound imaging, computed tomography (CT) or fluoroscopy to identify the area of ​​interest in the patient's body and the final target for biopsy or treatment and navigate to the area of ​​interest and the final target. In some operations, preoperative scanning can be used for target identification and intraoperative guidance. However, real-time imaging may be required to obtain a more accurate and current image of the target area. In addition, real-time image data showing the current position of the medical device relative to the target and its surroundings may be required to navigate the medical device to the target in a safe and accurate manner (e.g., without causing damage to other organs or tissues).

[0005] For example, endoscopic approaches have proven useful in navigating to areas of interest within a patient's body, and particularly to areas within the body's luminal network, such as the lungs. To enable endoscopic approaches, and more particularly bronchoscopic approaches in the lungs, endobronchial navigation systems have been developed that use previously acquired MRI data or CT image data to generate a three-dimensional (3D) rendering, model, or volume of a particular body part, such as the lungs.

[0006] The resulting volume generated from the MRI scan or CT scan is then used to create a navigation plan to facilitate advancement of a navigation catheter (or other suitable medical device) through the bronchoscope and the branches of the patient's bronchus to the area of ​​interest. A positioning or tracking system, such as an electromagnetic (EM) tracking system, may be used in conjunction with, for example, the CT data to facilitate guiding the navigation catheter through the branches of the bronchus to the area of ​​interest. In some cases, the navigation catheter may be positioned adjacent to the area of ​​interest of the branching lumen network or within one of the airways within the area of ​​interest to provide access for one or more medical devices.

[0007] However, a 3D volume of a patient's lungs generated from a previously acquired scan (such as a CT scan) may not provide an adequate basis for accurately guiding a medical device or instrument to a target during a navigation procedure. In some cases, inaccuracies are caused by deformation of the patient's lungs during surgery relative to the lungs when the previously acquired CT data was acquired. This deformation (CT to body divergence) can be caused by many different factors including, for example, changes in the body when transitioning between sedated and non-sedated states, changes in the patient's posture with the bronchoscope, pushing of the bronchoscope on tissue, different lung volumes (e.g., CT scans were acquired during inspiration, while navigation was performed during exhalation), different bed locations, different days, etc.

[0008] Therefore, another imaging modality may be employed to help visualize the medical device and target in real time and enhance in vivo navigation procedures. Furthermore, in order to accurately and safely navigate a medical device to a remote target, such as for biopsy or treatment, both the medical device and the target should be visible in the guidance system. Summary of the invention

[0009] The technology disclosed herein generally relates to systems and methods for: visualizing navigation of a medical device relative to a target using live fluoroscopic video, repeating a setup procedure in response to detecting movement of a fluoroscopic imaging system, and updating the position of a target overlaid on the live fluoroscopic video in response to detecting movement of a patient.

[0010] In one aspect, the present disclosure provides a method. The method includes: executing a setup procedure, the setup procedure including determining a first position of a tip of a catheter in a reference frame of a live fluoroscopic video; and determining a position of a fluoroscopic imaging system based on the first position of the tip of the catheter in the reference frame. The method also includes receiving a live fluoroscopic video from the fluoroscopic imaging system, and displaying the live fluoroscopic video. The method also includes projecting a three-dimensional (3D) coordinate from an electromagnetic (EM) sensor disposed at the tip of the catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, which generates the projected 3D coordinates, thereby overlaying a target on the live fluoroscopic video. The method also includes determining that the tip of the catheter is not at or near the projected 3D coordinates at a second position in the live fluoroscopic video, and in response to determining that the tip of the catheter is not at or near the projected 3D coordinates at the second position in the live fluoroscopic video, determining that the fluoroscopic imaging system has moved and repeating the setup procedure. The method also includes receiving the position of the patient from one or more second EM sensors disposed on the patient, and determining that the patient has moved based on the position of the patient. The method also includes, in response to determining that the patient has moved, determining an updated position of the target, and overlaying the target on the live fluoroscopic video at the updated position.

[0011] Specific implementations of the method may include one or more of the following features. The method may include determining that the catheter is not aligned with the target, and in response to determining that the catheter is not aligned with the target, updating the appearance of the target overlaying the live fluoroscopic video. Updating the appearance of the target may include changing the color of the target, highlighting the target, or applying a line pattern to the target. The method may include translating the view of the live fluoroscopic video so that the target is centered in the view of the live fluoroscopic video. Determining that the patient has moved may include determining a reference position of one or more second EM sensors, and determining that a distance between the position of the patient and the reference position is greater than a threshold distance.

[0012] The method may include determining possible translations of the fluoroscopic imaging system based on a periodic grid appearing in the live fluoroscopic video; selecting from the possible translations a translation whose projected 3D coordinates most closely match a second position of the tip of the catheter in the live fluoroscopic video, thereby producing a selected translation; and determining a position of the fluoroscopic imaging system based on a first position of the tip of the catheter in the reference frame and the selected translation. The method may include tracking a portion of the target that has been biopsied and displaying the portion of the target that has been biopsied. The fluoroscopic imaging system may be a 3D fluoroscope.

[0013] In another aspect, the present disclosure provides a system for guiding navigation of a biopsy tool within a patient. The system includes an electromagnetic (EM) field generator that generates an electromagnetic field, a first EM sensor disposed at the tip of a catheter, one or more second EM sensors disposed on the patient, and a display. The system also includes a processor and a memory having instructions stored thereon, wherein when the processor executes the instructions, the processor executes a setup procedure, the setup procedure including: determining a first position of the tip of the catheter in a reference frame of a live fluoroscopic video; and determining a position of a fluoroscopic imaging system based on the first position of the tip of the catheter in the reference frame.

[0014] When the processor executes the instructions, the processor also receives live fluoroscopic video from the fluoroscopic imaging system, displays the live fluoroscopic video, projects three-dimensional (3D) coordinates from an electromagnetic (EM) sensor disposed at the tip of the catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system to generate projected 3D coordinates, overlays a target on the live fluoroscopic video, determines that the tip of the catheter is not at or near the projected 3D coordinates at a second position in the live fluoroscopic video, and in response to determining that the tip of the catheter is not at or near the projected 3D coordinates at the second position in the live fluoroscopic video, determines that the fluoroscopic imaging system has moved and repeats the setup procedure.

[0015] When the processor executes the instructions, the processor also receives the position of the patient from one or more second EM sensors disposed on the patient, determines that the patient has moved based on the position of the patient, determines an updated position of the target in response to determining that the fluoroscopic imaging system has moved or that the patient has moved, and overlays the target on the live fluoroscopic video at the updated position.

[0016] Specific implementations of the system may include one or more of the following features. When the processor executes the instructions, the processor may calculate a patient coordinate reference system based on the EM field sensed by the EM sensor. When the processor executes the instructions, the processor may determine that the catheter is not aligned with the target, and in response to determining that the catheter is not aligned with the target, update the appearance of the target overlaying the live fluoroscopic video. Updating the appearance of the target overlaying the live fluoroscopic video may include changing the color of the target, highlighting the target, or applying a line pattern to the target.

[0017] In various aspects, when the processor executes the instructions, the processor may translate the view of the live fluoroscopic video so that the target is in the center of the view of the live fluoroscopic video. When the processor executes the instructions, the processor determines possible translations of the fluoroscopic imaging system based on a periodic grid appearing in the live fluoroscopic video; selects from the possible translations a translation whose projected 3D coordinates most closely match a second position of the tip of the catheter in the live fluoroscopic video, thereby generating a selected translation; and determines a position of the fluoroscopic imaging system based on the first position of the tip of the catheter in the reference system and the selected translation. When the processor executes the instructions, the processor may track a portion of the target that has been biopsied and display the portion of the target that has been biopsied. The fluoroscopic imaging system may be a 3D fluoroscope.

[0018] In another aspect, the present disclosure provides a system. The system includes an electromagnetic (EM) field generator that generates an electromagnetic field, one or more EM sensors disposed on a patient's body, and a display. The system also includes a processor coupled to the display and a memory coupled to the processor and having instructions stored thereon, which, when executed by the processor, causes the processor to display a screen on the display, the screen including a three-dimensional (3D) view of a 3D model of a target observed from the perspective of the tip of a medical device, and a live two-dimensional (2D) fluoroscopic view showing the medical device in the screen. The instructions, when executed by the processor, may also cause the processor to overlay a target marker corresponding to the 3D model of the target on the live 2D fluoroscopic view, determine that the patient's body has moved based on one or more signals received from the one or more EM sensors, and update the 2D position of the target marker overlaying the live 2D fluoroscopic view in response to determining that the patient's body has moved.

[0019] Specific implementations of the system may include one or more of the following features. Determining that the patient's body has moved may include determining that an EM sensor of one or more EM sensors disposed on the patient has moved greater than a threshold amount. Determining that the patient's body has moved may include determining that a combination of two or more EM sensors of the EM sensors disposed on the patient has moved greater than a threshold amount in a particular direction.

[0020] In another aspect, the present disclosure provides a method. The method includes: detecting a non-periodic grid of markers in at least one fluoroscopic image captured by a fluoroscopic imaging system; generating a non-periodic grid of detected markers; and determining a position of the fluoroscopic imaging system based on the non-periodic grid of detected markers. The method also includes displaying a live fluoroscopic video from the fluoroscopic imaging system; projecting a three-dimensional (3D) coordinate from an electromagnetic (EM) sensor disposed at the tip of a catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, thereby generating projected 3D coordinates; and overlaying a target on the live fluoroscopic video. The method also includes determining that the tip of the catheter is not at or near the projected 3D coordinates at a second position in the live fluoroscopic video, and in response to determining that the tip of the catheter is not at or near the projected 3D coordinates at the second position in the live fluoroscopic video, determining that the fluoroscopic imaging system has moved. The method also includes, in response to determining that the fluoroscopic imaging system has moved, repeating the detection of the non-periodic grid of markers and determining the position of the fluoroscopic imaging system.

[0021] The details of one or more aspects of the present disclosure are set forth in the following drawings and description. Other features, objectives, and advantages of the techniques described in the present disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various aspects of the present disclosure are described below with reference to the accompanying drawings, in which:

[0023] Figure 1 is a schematic diagram of a system for navigating to a target via a luminal network according to the present disclosure;

[0024] Figure 2 is a screenshot of an example of a user interface for confirming local registration according to the present disclosure;

[0025] Figure 3 is a screenshot of an example of a navigation user interface with a pop-up message asking whether to continue with a target overlay feature according to the present disclosure;

[0026] Figure 4 is a screenshot of an example of a navigation user interface without a target overlay feature according to the present disclosure;

[0027] Figure 5 is a screen shot of an example of a navigation user interface with a target overlay feature according to the present disclosure;

[0028] Figure 6 is a screenshot of an example of a navigation user interface showing a screen that appears when a "Target Overlay" tab is selected in accordance with the present disclosure;

[0029] Figure 7 is a flow chart of an example of a method for visualizing a medical procedure on live fluoroscopic video according to the present disclosure;

[0030] Figure 8 is a screenshot of an exemplary navigation user interface showing a target marker overlaid on a real-time two-dimensional (2D) fluoroscopic view in accordance with the present disclosure;

[0031] Fig. 9 is a flow chart illustrating another method for updating the position of a target overlaid with live fluoroscopic video according to the present disclosure;

[0032] Fig.10 It shows that it can be Fig. 9 Schematic diagram of a non-periodic grid of radiopaque markers used in the method;

[0033] Fig.11 is a flow chart illustrating a method of tracking and displaying a biopsied portion of a target;

[0034] Fig.12 is a flow chart of an example of a method of visualizing navigation of a medical device relative to a target; and

[0035] Fig.13 is a schematic diagram of a system for navigating to a target and displaying a user interface according to the present disclosure. DETAILED DESCRIPTION

[0036] For example, clinicians can use fluoroscopic imaging systems to visualize the navigation of medical devices and confirm the placement of medical devices after the medical devices are navigated to the desired location. However, although fluoroscopic images show high-density objects such as metal tools, bones, and large soft tissue objects such as the heart, fluoroscopic images may not clearly show small soft tissue objects of interest such as lesions. In addition, fluoroscopic images are two-dimensional projections. Therefore, X-ray volume reconstruction is required to achieve the identification of soft tissue objects and the navigation of medical devices to these objects.

[0037] There are several solutions to provide 3D or volume reconstruction. One solution is a CT machine, which combines multiple X-ray projections from known, calibrated X-ray source positions into a volume through an algorithm in which soft tissue is more visible. For example, a CT machine can be used with iterative scanning during a procedure to provide guidance through the body until one or more tools reach the target. This is a cumbersome procedure because it requires several complete CT scans, a dedicated CT room, and blind navigation between scans. In addition, due to the high level of ionizing radiation, each scan requires staff to leave the room and expose the patient to the radiation. Another solution is a cone beam CT machine. However, cone beam CT machines are expensive and, like CT machines, can only provide blind navigation between scans, require multiple iterative navigations, and require staff to leave the room. In some exemplary aspects, the systems and methods of the present disclosure combine the benefits of a CT machine and a fluoroscopic imaging system to help clinicians navigate medical devices to targets including small soft tissue objects.

[0038] In an electromagnetic navigation procedure, planning, registration, and navigation are performed to ensure that a medical device (e.g., a biopsy tool) follows a planned path to a target (e.g., a lesion) so that a biopsy or treatment of the target can be completed. After the navigation phase, a fluoroscopic image can be captured and used in a local registration process to reduce CT-to-body divergence. After the local registration process, the locatable guide can be removed from the extended working channel, and a medical device (e.g., a biopsy tool) can be introduced into the extended working channel and navigated to the target to perform a biopsy or treatment of the target (e.g., a lesion).

[0039] In the process of navigating a medical device to a target, a clinician may use a live 2D fluoroscopic view to visualize the position of the medical device relative to the target. Although the medical device may be visible in the live fluoroscopic view, some targets (e.g., lesions) may not be visible in the live fluoroscopic view. And electromagnetic navigation cannot be used because some medical devices may not include sensors. In addition, the user interface for advancing or navigating the medical device toward the target does not provide sufficient information about the medical device relative to the target, including when the medical device is close to the target.

[0040] The present disclosure features a user interface that overlays a 2D target marker or representation corresponding to a three-dimensional model of a target identified in a CT scan onto a live 2D fluoroscopic view, enabling a clinician to visualize the location of a medical device tip relative to the target. Since the live fluoroscopic view with the target overlay is a two-dimensional view and does not necessarily show whether the medical device is above or below the target, the same user interface also includes a three-dimensional medical device tip view of the 3D model of the target, enabling the clinician to confirm that the medical device is not above or below the target.

[0041] The user interface also provides a graphical indication of whether the medical device is aligned with the target in three dimensions. For example, when the medical device is aligned with the target in three dimensions, the user interface shows the target coverage in a first color (e.g., green). On the other hand, when the medical device is not aligned with the target in three dimensions, the user interface shows the target coverage in a second color (e.g., orange or red) different from the first color.

[0042] According to aspects of the present disclosure, visualization of in vivo navigation of a medical device (eg, a biopsy tool) toward a target (eg, a lesion) may be part of a larger workflow of a navigation system, such as an electromagnetic navigation system. Figure 1 is a perspective view of an exemplary system for facilitating navigation of a medical device (e.g., a biopsy tool) to a soft tissue target via an airway of a lung. The system 100 may be further configured to construct fluoroscopically based three-dimensional volume data of a target region from a 2D fluoroscopic image. The system 100 may be further configured to facilitate access of the medical device to the target region and to determine the position of the medical device relative to the target by using electromagnetic navigation bronchoscopy (ENB).

[0043] One aspect of the system 100 is a software component for reviewing computed tomography (CT) image data that has been acquired separately from the system 100. Review of the CT image data allows a user to identify one or more targets, plan a path to the identified targets (planning phase), navigate the catheter 102 to the targets using a user interface (navigation phase), and confirm placement of the sensor 104 relative to the targets. One such EMN system is the ELECTROMAGNETIC NAVIGATION BRONCHOSCOPY currently sold by Medtronic PLC. ® The target may be tissue of interest identified by examining CT image data during the planning phase. After navigation, a medical device (such as a biopsy tool or other tool) may be inserted into the catheter 102 to obtain a tissue sample from tissue located at or near the target.

[0044] like Figure 1As shown, catheter 102 is part of a catheter guide assembly 106. In practice, catheter 102 is inserted into a bronchoscope 108 to access the luminal network of a patient P. Specifically, catheter 102 of catheter guide assembly 106 can be inserted into a working channel of bronchoscope 108 for navigation through the luminal network of the patient. A locatable guide 110 including a sensor 104 is inserted into catheter 102 and locked in place so that sensor 104 extends a desired distance beyond the distal tip of catheter 102. The position and orientation of sensor 104 relative to a reference coordinate system within an electromagnetic field, and thus the distal portion of catheter 102, can be derived. Catheter guide assembly 106 is currently marketed by Medtronic under the trade name SUPERDIMENSION ® Surgical Suite or EDGE TM Surgical kits are marketed and sold and are believed to be usable with the present disclosure.

[0045] The system 100 generally includes an operating table 112 configured to support a patient P; a bronchoscope 108 configured to be inserted into the airway of the patient P through the mouth of the patient P; a monitoring device 114 coupled to the bronchoscope 108 (e.g., a video display for displaying video images received from a video imaging system of the bronchoscope 108); a positioning or tracking system 116, a patient reference sensor 118, and a transmitter pad 120, the transmitter pad including a plurality of incorporated markers; and a computer system 122 including software and / or hardware for facilitating identification of a target, path planning to a target, navigation of a medical device to a target, and / or confirmation and / or determination of placement of the catheter 102 or a suitable device therethrough relative to the target. The computer system 122 may be similar to Fig.13 The workstation 1501 may be configured to execute operations including Figure 7 , Fig. 9 , Fig.11 and Fig.12 The method of the present disclosure including the method of .

[0046] Also included in this particular aspect of the system 100 is a fluoroscopic imaging system 124 capable of acquiring fluoroscopic or X-ray images or videos of the patient P. The images, image sequences, or videos captured by the fluoroscopic imaging system 124 may be stored within the fluoroscopic imaging system 124, or transmitted to the computer system 122 for storage, processing, and display. Additionally, the fluoroscopic imaging system 124 may be moved relative to the patient P so that images may be acquired from different angles or perspectives relative to the patient P to create a sequence of fluoroscopic images (such as a fluoroscopic video). The pose of the fluoroscopic imaging system 124 relative to the patient P and at the time of capturing the images may be estimated via markers incorporated with the transmitter pad 120. The markers are positioned below the patient P, between the patient P and the operating table 112, and between the patient P and the radiation source or sensing unit of the fluoroscopic imaging system 124. The markers incorporated with the transmitter pad 120 may be two separate elements that may be coupled in a fixed manner or alternatively may be manufactured as a single unit. The fluoroscopic imaging system 124 may include a single imaging system or more than one imaging system.

[0047] The computer system 122 can be any suitable computing device including a processor and a storage medium, wherein the processor is capable of executing instructions stored on the storage medium. The computer system 122 may also include a database configured to store patient data, a CT data set including a CT image, a fluoroscopic data set including a fluoroscopic image and a video, a fluoroscopic 3D reconstruction, a navigation plan, and any other such data. Although not explicitly shown, the computer system 122 may include an input, or may be otherwise configured to receive a CT data set, a fluoroscopic image or video, and other data described herein. Additionally, the computer system 122 includes a display configured to display a graphical user interface. The computer system 122 may be connected to one or more networks, through which one or more databases may be accessed.

[0048] Regarding the planning phase, the computer system 122 uses the previously acquired CT image data to generate and view a three-dimensional model or rendering of the airway of the patient P, so that the identification of the target on the three-dimensional model can be performed (automatically, semi-automatically or manually), and the path through the airway of the patient P to the tissue located at and around the target is determined. More specifically, the CT images acquired from the previous CT scan are processed and assembled into a three-dimensional CT volume, and then the three-dimensional CT volume is used to generate a three-dimensional model of the airway of the patient P. The three-dimensional model can be displayed on a display associated with the computer system 122, or in any other suitable manner. Using the computer system 122, various views of the three-dimensional model or the enhanced two-dimensional image generated by the three-dimensional model are presented. The enhanced two-dimensional images can have certain three-dimensional capabilities because they are generated from three-dimensional data. The three-dimensional model can be manipulated to facilitate the identification of the target on the three-dimensional model or the two-dimensional image, and the selection of a suitable path through the airway of the patient P to enter the tissue located at the target can be performed. Once selected, the path plan, the three-dimensional model, and the images derived therefrom can be saved and exported to the navigation system for use during one or more navigation phases. One such planning software is ILLUMISITE, currently sold by Medtronic. ® Planning kit.

[0049] With respect to the navigation phase, registration of the images and the navigation path are performed using a six-degree-of-freedom electromagnetic positioning or tracking system 116 or other suitable system for determining position, but other configurations are also contemplated. The tracking system 116 includes a tracking module, a patient reference sensor 118, and a transmitter pad 120 (including markers). The tracking system 116 is configured for use with the locatable guide 110 and, in particular, the sensor 104. As described above, the locatable guide 110 and the sensor 104 are configured for insertion through the catheter 102 into the airway of the patient P (with or without the bronchoscope 108), and can be selectively locked relative to each other via a locking mechanism.

[0050] The transmitter pad 120 is positioned below the patient P. The transmitter pad 120 generates an electromagnetic field around at least a portion of the patient P, within which the position of the patient reference sensor 118 and the sensor 104 can be determined using the tracking system 116. A second electromagnetic sensor 126 can also be incorporated into the end of the catheter 102. The second electromagnetic sensor 126 can be a five-degree-of-freedom sensor or a six-degree-of-freedom sensor. One or more of the patient reference sensors 118 are attached to the chest of the patient "P". The six-degree-of-freedom coordinates of the reference sensor 118 are sent to a computer system 122 (which includes appropriate software) where they are used to calculate a patient reference coordinate system. Registration is typically performed to coordinate the position of the three-dimensional model and two-dimensional image of the planning phase with the airway of the patient "P" observed through the bronchoscope 108, and to allow accurate knowledge of the position of the sensor 104 when the navigation phase is performed, even in portions of the airway that the bronchoscope 108 cannot reach.

[0051] Registration of the position of the patient P on the transmitter pad 120 can be performed by moving the sensor 104 through the airway of the patient P. More specifically, data relating to the position of the sensor 104 as the locatable guide 110 moves through the airway is recorded using the transmitter pad 120, the patient reference sensor 118, and the tracking system 116. The shape resulting from the position data is compared with the internal geometry of the passage of the three-dimensional model generated in the planning phase, and based on the comparison, a positional correlation between the shape and the three-dimensional model is determined, for example, using software on the computer system 122. Among other things, the software identifies non-tissue spaces (e.g., cavities filled with air) in the three-dimensional model. The software aligns or registers an image representing the position of the sensor 104 with the three-dimensional model and / or a two-dimensional image generated by the three-dimensional model, based on the recorded position data and the assumption that the locatable guide 110 is still positioned in the non-tissue space in the airway of the patient P. Alternatively, manual registration techniques may be employed by navigating the bronchoscope 108 with the sensor 104 to a pre-specified location in the lungs of the patient P and manually associating the images from the bronchoscope with the model data of the three-dimensional model.

[0052] Although described herein with respect to an EMN system using an EM sensor, the present disclosure is not limited thereto and may be used in conjunction with a flexible sensor, an ultrasound sensor, or in the absence of a sensor. Additionally, the methods described herein may be used in conjunction with a robotic system to enable a robotic actuator to drive a catheter 102 or a bronchoscope 108 close to a target.

[0053] After registering the patient P to the image data and the path plan, a user interface is displayed in the navigation software, which sets the path that the clinician should follow to reach the target. As depicted in the user interface, once the catheter 102 has been successfully navigated to the vicinity of the target, the locatable guide 110 can be unlocked and removed from the catheter 102, thereby leaving the catheter 102 in place as a guide channel for guiding medical devices (including but not limited to optical systems, ultrasound probes, marker placement tools, biopsy tools, ablation tools (i.e., microwave ablation devices), laser probes, cryogenic probes, sensor probes, and aspiration needles) to the target. The medical device can then be inserted through the catheter 102 and navigated to the target or a specific area adjacent to the target.

[0054] Prior to inserting the medical device through the catheter 102, a local registration process may be performed for each target to reduce CT to body divergence. During the capture phase of the local registration process, a sequence of fluoroscopic images may be captured and acquired via the fluoroscopic imaging system 124, optionally by the user and according to directions displayed via the computer system 122. A fluoroscopic 3D reconstruction may then be generated via the computer system 122. The generation of the fluoroscopic 3D reconstruction is based on the sequence of fluoroscopic images and the projection of the structure of the markers incorporated with the transmitter pad 120 onto the sequence of images. One or more slices of the 3D reconstruction may then be generated based on the preoperative CT scan and via the computer system 122. The 3D reconstruction and one or more slices of the fluoroscopic 3D reconstruction may then be displayed to the user on a display, optionally simultaneously, via the computer system 122. The slices of the 3D reconstruction may be presented on a user interface in a scrollable format, wherein the user is able to scroll through the slices continuously.

[0055] During the marking phase of the local registration process, the clinician may be guided to identify and mark the target while using the 3D reconstructed slices as a reference. The user may also be guided to identify and mark the navigation catheter tip in the sequence of fluoroscopic 2D images. The offset between the target location and the navigation catheter tip may then be determined or calculated via the computer system 122. The offset may then be used to correct the navigation catheter on the display (e.g., in the image that can be selected by selecting Figure 4 The 3D model of the target area may be used to determine the position and / or orientation of the target in the peripheral navigation screen (shown as a “Peripheral Navigation” tab 401 for viewing), and / or correct the registration between the 3D model and the tracking system 116 in the target area, and / or generate a local registration between the 3D model and the fluoroscopic 3D reconstruction in the target area.

[0056] During the confirmation phase of the local registration process, the fluoroscopic 3D reconstruction is displayed in the confirmation screen 202, such as Figure 2The confirmation screen 202 includes a slider 208 that can be selected and moved by the user to review a fluoroscopic 3D reconstructed video loop showing the marked target and navigation catheter tip from different viewing angles. After confirming that the markers are present on the target and navigation catheter tip throughout the video, the clinician can select an "Accept" button 210, at which point the local registration process ends and the position of the navigation catheter is updated. The clinician can then use, for example, Figure 4 The navigation view in the peripheral navigation screen is shown to fine-tune the alignment of the navigation catheter with the target before starting an endoscopic procedure.

[0057] After the local registration process, the clinician or robot can insert the medical device into the catheter 102 and advance the medical device toward the target. While advancing the medical device toward the target, the clinician can view a user interface screen that includes:

[0058] (a) A 3D medical device tip view of a 3D model of the target based on a preoperative CT scan, and

[0059] (b) A live 2D fluoroscopic view with a target marker corresponding to a 3D model of the target overlaid thereon. This user interface screen not only allows the clinician to view the medical device in real time, but also allows the clinician to view whether the medical device is aligned with the target. The user interface screen may also provide a graphical indication of whether the medical device is aligned with the target in three dimensions. For example, when the medical device is aligned with the target in three dimensions, the user interface shows the target overlay in a first color (e.g., green). On the other hand, when the medical device is not aligned with the target in three dimensions, the user interface shows the target overlay in a second color (e.g., orange or red) different from the first color.

[0060] Figure 2 2 is a screen shot of a confirmation screen 202 of an exemplary local registration user interface that appears during the confirmation phase of the local registration process. The confirmation screen 202 displays a navigation catheter tip marker 204 and a target marker 206 that were previously marked by the clinician during the marking phase of the local registration process. After the clinician selects the “Accept” button, Figure 3 The navigation user interface of the present invention is displayed with a pop-up message 302. The pop-up message 302 may include buttons 304, 306 that enable the clinician to select whether to use the target coverage feature to guide the navigation of the medical device (e.g., biopsy tool) to the target. Specifically, the clinician can select button 304 to continue with the target coverage feature, or the clinician can select button 306 to continue without using the target coverage feature.

[0061] When the clinician selects button 306, the Figure 4400 (which was previously displayed before the local registration process was performed), thereby showing the adjustments (if any) to the position and / or orientation of the navigation catheter tip 405 as a result of the position registration process. The peripheral navigation screen 401 includes a local CT view 402, a 3D navigation catheter tip view 404, a 3D map view 406, and a bronchoscope view 408. The peripheral navigation screen 401 also includes a local registration user control 403 that enables the user to apply local registration and / or restart local registration. The user interface 400 also includes a "central navigation" tab 411 and a "target alignment" tab 412, which can be selected separately to perform central navigation or target alignment, respectively.

[0062] When the clinician selects the local registration user control 403, the user interface 400 displays the peripheral navigation screen and Figure 5 Target Coverage tab 502 is shown. When target coverage tab 502 is selected, as shown in FIG. Figure 6 Shown is a display target overlay screen including a live 2D fluoroscopic view or video 602 of the catheter 102 in the body of the patient "P".

[0063] Figure 7 7 is a flow chart of an example of a method for visualizing a medical procedure on a live fluoroscopic video (e.g., in a live fluoroscopic view). At block 702, local registration is performed. A fluoroscopic sweep for local registration may be performed within about 30 to 60 degrees to generate a 3D volume reconstruction. In various aspects, a fluoroscopic sweep for local registration may be performed within a range of about 200 degrees.

[0064] At box 704, a setup procedure is performed. The setup procedure may include adjusting the C-arm fluoroscopy so that the C-arm fluoroscopy is properly aligned with the patient's body, and marking and confirming the markers of the locatable guide, such as the tip of the locatable guide, in the reference frame of the live fluoroscopic view. In other aspects, a 3D fluoroscopy or a cone beam computed tomography (CBCT) imaging system may be used instead of the C-arm fluoroscopy. The reference frame may be captured when the fluoroscopic imaging system 124 (e.g., the fluoroscopy) is in the AP position and at an angle to the AP position without hindering the practicality of the method of the present disclosure. The reference frame may include projections of structures of markers or fiducials (e.g., radiopaque markers or marker beads) for determining position information (e.g., the position of the fluoroscopic imaging system). The fiducial structure may be a non-periodic fiducial structure that provides the ability to identify a precise position when evaluating a small area of ​​the fiducial structure captured in the "field of view" by the fluoroscopy 124. Fig.10An example of a reference system 1000 including a projection of a non-periodic structure of radiopaque markers 1002 is shown, which provides more accurate position information than a periodic grid of radiopaque markers.

[0065] In the case where the grid is periodic, a setup procedure may be required to improve the C-arm source pose estimate from a single fluoroscopic image and solve for the C-arm translation relative to the antenna. The setup procedure may include marking or determining a first position of the tip of the catheter in a reference frame of the live fluoroscopic video; and determining the position of the fluoroscopic imaging system based on the first position of the tip of the catheter in the reference frame. The tip of the catheter may be marked manually, or the tip of the catheter may be automatically determined using a segmentation algorithm or any other suitable algorithm or image recognition process for detecting the tip of the catheter. If the tip of the catheter is automatically detected, a reference frame may not be required, and the setup procedure may occur automatically whenever movement of the fluoroscopic imaging system is detected. In the case where the grid is non-periodic, the C-arm source pose may be estimated without a setup procedure, for example, without requiring a clinician or user to mark the tip of the catheter in the reference frame of the live fluoroscopic video. At box 706, the live fluoroscopic video is displayed in a user interface as shown herein. At box 708, the position of the target in the live fluoroscopic video is determined.

[0066] After determining the location of the target in the live fluoroscopic video at box 708, a marker is displayed overlaid on the live fluoroscopic video at the determined location of the target at box 710. The method 700 may include panning the view of the live fluoroscopic video so that the target is in the center of the view of the live fluoroscopic video. Alternatively, the view may be automatically panned according to any other suitable automatic panning protocol that depends on the determined location of the target and / or medical tool (e.g., catheter) in the fluoroscopic video. For example, the panning protocol may specify that the target is in a preset quadrant of the view of the live fluoroscopic video. At box 712, the computer system 122 determines whether there is a change in the alignment of the catheter 102 (e.g., the extended working channel) with the target. If there is a change in the alignment of the catheter 102 with the target, the appearance of the marker is updated at box 714. The update of the appearance of the marker may include changing the color of the marker. In the case where the marker is circular or elliptical, the update of the appearance of the marker may include changing the dashed line type of the line forming the circle or ellipse.

[0067] exist Figure 8, the tip 604 of the catheter 102 is shown aligned with the target marker 606 displayed in the live 2D fluoroscopic image 602. In addition, the medical device tip view 608 depicts a view as if the camera were located at the tip 604 of the catheter 102. The medical device tip view 608 presents a three-dimensional (3D) representation of the target 610. If the tip 604 of the catheter 102 is nearly aligned with the target, the target marker 606 may be displayed in a first color (e.g., green) and overlaid on the live 2D fluoroscopic image 602. On the other hand, if the tip 604 of the catheter 102 is not aligned with the target (e.g., as shown in FIG. 5 ), the target marker 606 may be displayed in a first color (e.g., green) and overlaid on the live 2D fluoroscopic image 602. Figure 4 4, wherein the spherical target is set to the right of the center of the 3D navigation catheter tip view 404), the target marker 606 can be displayed in a different color (e.g., orange or red). Similarly, in the medical device tip view 608 where the tip 604 of the catheter 102 is not aligned with the target, the 3D representation of the target 610 will appear offset in the image and only a portion or all of it is visible in the view. In addition, the color can also be changed according to the severity of the misalignment of the tip 604 of the catheter 102 with the target as described above.

[0068] The medical device tip view 608 may also include a text box 611 that displays text indicating the distance between the tip 604 of the catheter 102 and the center of the target. In various aspects, the computer system 122 may calculate the distance by aligning a 3D model of the luminal network (which may be based on a CT scan and include the target) with the live fluoroscopic view or finding a correspondence between the 3D model of the luminal network and the live fluoroscopic view and measuring the distance between the tip 604 and the center of the 3D model of the target using, for example, image processing. In finding a correspondence between the 3D model and the live fluoroscopic view, a fluoroscopic 3D reconstruction generated and marked during the local registration process may be used. In some aspects, the distance is measured from the center or outer edge of the target. The target overlay screen also includes a target overlay toggle button 614 that, when selected, displays a target overlay view such as Figure 8 The target marker 606 is toggled between being shown and not being shown.

[0069] exist Figure 8, the tip 604 of the catheter 102 is shown aligned with the target marker 606 displayed in the live 2D fluoroscopic image 602. In addition, the medical device tip view 608 depicts a view as if the camera were located at the tip 604 of the catheter 102. The medical device tip view 608 presents a three-dimensional representation of the target 610. If the tip 604 of the catheter 102 is nearly aligned with the target, the target marker 606 may be displayed in a first color (e.g., green) and overlaid on the live 2D fluoroscopic image 602. On the other hand, if the tip 604 of the catheter 102 is not aligned with the target (e.g., as shown in FIG. 5 ), the target marker 606 may be displayed in a first color (e.g., green) and overlaid on the live 2D fluoroscopic image 602. Figure 4 4, wherein the spherical target is set to the right of the center of the 3D navigation catheter tip view 404), the target marker 606 can be displayed in a different color (e.g., orange or red). Similarly, in the medical device tip view 608 where the tip 604 of the catheter 102 is not aligned with the target, the 3D representation of the target 610 will appear offset in the image and only a portion or all of it is visible in the view. In addition, the color can also be changed according to the severity of the misalignment of the tip 604 of the catheter 102 with the target as described above.

[0070] The medical device tip view 608 may also include a text box 611 that displays text indicating the distance between the tip 604 of the catheter 102 and the center of the target. In various aspects, the computer system 122 may calculate the distance by aligning a 3D model of the luminal network (which may be based on a CT scan and include the target) with the live fluoroscopic view or finding a correspondence between the 3D model of the luminal network and the live fluoroscopic view and measuring the distance between the tip 604 and the center of the 3D model of the target using, for example, image processing. In finding a correspondence between the 3D model and the live fluoroscopic view, a fluoroscopic 3D reconstruction generated and marked during the local registration process may be used. In some aspects, the distance is measured from the center or outer edge of the target. The target overlay screen also includes a target overlay toggle button 614 that, when selected, displays a target overlay view such as Figure 8 The target marker 606 is toggled between being shown and not being shown.

[0071] Go to again Figure 7 If there is no change in the alignment of the catheter 102 with the target, the computer system 122 determines whether movement of the patient's body is detected at block 716. The movement of the patient's body may be detected by one or more patient reference sensors (e.g., Figure 1If movement of the patient's body is detected at block 716, method 700 returns to block 708 to determine a new position of the target in the live fluoroscopic video. The new position of the target in the live fluoroscopic video may be determined based on position information provided by one or more patient reference sensors 118.

[0072] If no movement of the patient's body is detected at block 716, the method 700 determines at block 718 whether there is movement of the C-arm fluoroscopy or a change in the optical magnification of the camera or image intensifier of the C-arm fluoroscopy. Movement of the C-arm fluoroscopy may include translation and / or rotation of the C-arm fluoroscopy. A periodic grid of radiopaque markers 612 or a non-periodic grid of radiopaque markers adjacent to the transmitter pad 120 may be used (e.g., Fig.10 The method 700 may be used to determine the C-arm translation and / or rotation based on the movement of the positionable guiding antenna (the non-periodic grid shown in FIG. 7 ). If there is movement of the C-arm fluoroscopy scope or a change in the optical magnification of the camera or image intensifier of the C-arm fluoroscopy scope at block 718, the method 700 returns to block 704 to re-execute the setup procedure or re-estimate the position of the fluoroscopic imaging system based on the non-periodic grid. Otherwise, the method 700 returns to block 710 to display a marker overlaid on the live fluoroscopic video at the location of the target determined at block 708.

[0073] In some aspects, computer system 122 may monitor the movement of both the fluoroscopic imaging system and the patient in parallel, and may update the position of the target overlaid on the live fluoroscopic video based on the patient's movement.

[0074] In some aspects, the computer system 122 can detect movement of the fluoroscopic imaging system without using a reference frame. Fig. 9 A method 900 is shown for determining whether a fluoroscopic imaging system and / or a patient's body has moved, in which case the position of a target overlaid on a live fluoroscopic video is updated based on the movement of the patient's body. In some aspects, the movement of the fluoroscopic imaging system is determined without using a reference frame.

[0075] In one aspect, method 900 includes continuously projecting the 3D coordinates of the catheter tip EM sensor onto the live fluoroscopic video with the position of the fluoroscopic imaging system solved from the setup phase. Thus, at box 902, which may be performed in the setup phase, the pose of the fluoroscopic imaging system is estimated based on data from a first electromagnetic (EM) sensor disposed at the tip of the catheter and the 3D coordinates of the tip of the catheter (e.g., a locatable guide, EWC, or biopsy tool) are determined. At box 903, the live fluoroscopic video is displayed, and a target or a representation of the target is overlaid on the live fluoroscopic video. At box 904, the 3D coordinates of the catheter tip are projected onto the live fluoroscopic video based on the pose estimate. At box 906, the position of the tip of the catheter is detected in the live fluoroscopic video. In one example, to detect the position of the catheter tip, image analysis may be employed to identify pixels in the live 2D fluoroscopic image 602 having a Hounsfield unit value greater than a threshold, above which the Hounsfield unit value corresponds to a radiopaque catheter. The last connected pixel of the pixels making up the catheter 102 may be identified as the tip 604 of the catheter 102. Other processes may also be used to detect the tip 604 of the catheter 102 in the live fluoroscopic video without departing from the scope of the present disclosure. For example, an automatic computer vision algorithm or a user interface that enables a user to manually mark the catheter tip may be used to detect whether the EM sensor (catheter tip) is visible in the fluoroscopic video.

[0076] At box 908, method 900 determines whether the fluoroscopic imaging system 124 has moved based on whether the projected 3D coordinates match or closely match the position of the catheter tip detected in the live fluoroscopic video. If method 900 determines at box 908 that the fluoroscopic scope has moved, the setup procedure including box 902 is repeated. The new position of the fluoroscopic imaging system can be determined based at least in part on determining the translation. Due to the periodic grid markers, there are a discrete number of possible translations that may have occurred. The computer system 122 can evaluate the possible translations and select from the possible translations the translation in which the projection of the EM sensor on the catheter tip matches the catheter tip detection in the live fluoroscopic video. On the other hand, if method 900 determines at box 908 that the fluoroscopic scope has not moved, method 900 proceeds to box 912.

[0077] At block 912, the computer system 122 receives the position of the patient's body from one or more second EM sensors disposed on the patient's body. At block 916, the method 900 determines whether the patient's body moves based on the position of the patient's body. If the method 900 determines at block 916 that the patient's body moves based on the position of the patient's body, then at block 918, an updated position of a target to overlay the live fluoroscopic video is determined based on the position of the patient's body, at block 920, the target is overlaid on the live fluoroscopic video at the updated position of the target, and the method 900 returns to block 904. If the method 900 determines that the patient's body moves, then the method 900 returns to block 904.

[0078] Fig.11 A method 1100 for tracking a biopsy procedure is depicted. At box 1102, a target is divided into segments. The target may be divided into segments by determining the shape and volume of the target from a preoperative image (e.g., a computed tomography (CT) image), and the target may be divided into segments based on the shape and volume of the target. The method 1100 may include displaying the target and enabling a clinician to mark a segment to be biopsied in the volume. The method 1100 then determines at box 1104 whether the segment has been biopsied, automatically (e.g., by an image processing algorithm) or manually (e.g., by a user marking a biopsy segment via a user interface), and records at box 1106 that the segment has been biopsied. At box 1108, the method 1100 determines whether all segments have been biopsied. If all segments have been biopsied, the method 1100 returns to box 1104 to determine whether another target segment has been biopsied. Otherwise, the method 1100 displays a message on the user interface at block 1110 that all segments have been biopsied.

[0079] Fig.12 is a flowchart of an example of a method 1200 that can be implemented Figure 7 The computer system 122 may determine whether the medical device tip is aligned with the target by aligning a 3D model of the cavity network (which may be based on a CT scan and includes the target) with a live fluoroscopic view or finding a correspondence between the 3D model of the cavity network and the live fluoroscopic view; and determining whether the medical device tip is aligned with the 3D model of the target based on the determined alignment or correspondence, and applying, for example, image processing, to determine whether the medical device tip is aligned with the target. In the process of aligning the 3D model with the live fluoroscopic view or finding a correspondence between the 3D model and the live fluoroscopic view, the fluoroscopic 3D reconstruction generated and marked in the local registration process may be used.

[0080] When the computer system 122 determines that the medical device tip is aligned with the target, the computer system 122 sets the target marker color to green at box 1406; otherwise, the computer system 122 sets the target marker color to orange at box 1408. At box 1210, the computer system 122 displays a live 2D fluoroscopic view in a target overlay screen, which shows at least the medical device. At box 1212, the computer system 122 displays a target marker set with a color overlaid on the live 2D fluoroscopic view. At box 1214, the computer system 122 displays a 3D virtual target from the perspective of the medical device tip in the same target overlay screen, the 3D virtual target corresponding to the target marker. Boxes 1204 to 1212 can be repeated until the medical device tip is placed in the center of the target, or until the biopsy or other treatment is completed. This final navigation allows the user to use fluoroscopic navigation technology to obtain a live image with the target marked on the live image, which enables the user to see the degree to which the medical device tip is aligned with the target to ensure that the medical device tip reaches the target to obtain a sample of the target or perform treatment on the target.

[0081] In another aspect of the present disclosure, in a local registration (e.g., Figure 7 After the fluoroscopic 3D reconstruction is performed (block 702 of FIG. 1 ), the computer system 122 may perform an image analysis of the fluoroscopic 3D reconstruction to determine the placement angle of the fluoroscopic imaging system 124 to best engage the target overlay tab. In this aspect of the present disclosure, after identifying the tip of the catheter 102 in both slices of the fluoroscopic 3D reconstruction, and identifying the target in the fluoroscopic 3D reconstruction and determining the relative positions of the tip of the catheter 102 and the target in the fluoroscopic 3D reconstruction, the computer system 122 performs an image analysis of the 3D reconstruction to determine a slice of the 3D reconstruction in which the catheter and the target are visible. This may be a slice in which both the target and the catheter 102 are most visible or most visible above a certain minimum threshold. It will be understood by those skilled in the art that there will be slices in which one or the other (or both) of the catheter or the target is not visible, and that the computer system 122 may ignore those images when performing this analysis.

[0082] After analyzing the remaining slices of the fluoroscopic 3D reconstruction, one of these slices is identified as most clearly depicting both the catheter 102 and the target. Once the slice of the 3D reconstruction is determined, the position of the fluoroscopic imaging system 124 (e.g., angle with respect to the patient P or the operating table 112) can be determined at which a corresponding 2D fluoroscopic image (such as live 2D fluoroscopic image 602) can be captured. Prior to engaging the "target overlay" tab 502, this position of the fluoroscopic imaging system 124 can be presented to the clinician on the user interface so that the clinician can manually move the fluoroscopic imaging system 124 to the position. Alternatively, the fluoroscopic imaging system 124 can receive an indication of the position determined by the computer system 122 and automatically drive the fluoroscopic imaging system 124 to the position so that when the "target overlay" tab 502 is selected, the live 2D fluoroscopic image 602 is acquired at this predetermined optimal position to view the catheter 102 and the target.

[0083] Another aspect of the present disclosure is to enable the use of a zoom feature that may be built into the fluoroscopic imaging system 124. Figure 6 , the live 2D fluoroscopic image 602 includes a plurality of radiopaque markers 612. These radiopaque markers 612 may be placed on or embedded in the transmitter pad 120. The distance between the radiopaque markers is fixed and known to the computer system 122. Since the distance between the radiopaque markers 612 is known, if the distance between any of the markers exceeds the known distance, the computer system 122 may determine that the zoom feature of the fluoroscopic imaging system 124 is engaged. The exact amount of zoom that has been engaged may be determined by comparing the spacing of the radiopaque markers 612 in the live 2D fluoroscopic image 602 with the known spacing of the radiopaque markers 612 in the transmitter pad 120. Once the amount of zoom is determined, the computer system 122 may calculate the offset of the relative position of the tip 604 of the catheter 102 and the target so that the target marker 606 may be accurately displayed in the live 2D fluoroscopic image 602 regardless of the zoom change from when the local registration process was performed.

[0084] Reference now Fig.13 , which is a schematic diagram of a system 1300 configured to communicate with a Fig.12The system 1300 may include a workstation 1301 and optionally a fluoroscopic imaging system or fluoroscope 1315. In some aspects, the workstation 1301 may be coupled directly or indirectly to the fluoroscope 1315, for example, via wireless communication. The workstation 1301 may include a memory 1302, a processor 1304, a display 1306, and an input device 1310. The processor 1304 may include one or more hardware processors. The workstation 1301 may optionally include an output module 1312 and a network interface 1008. The memory 1302 may store applications 1318 and image data 1314. The applications 1318 may include executable by the processor 1304 for performing operations including Figure 7 , Fig. 9 , Fig.11 and Fig.12 The method of the present disclosure is an instruction of the method.

[0085] The application 1318 may also include a user interface 1316. The image data 1314 may include a CT scan, a generated fluoroscopic 3D reconstruction of the target area, and / or any other fluoroscopic image data of the 3D reconstruction and / or one or more generated slices. The processor 1304 may be coupled to the memory 1302, the display 1306, the input device 1310, the output module 1312, the network interface 1308, and the fluoroscope 1315. The workstation 1301 may be a fixed computer system such as a personal computer, or a portable computer system such as a tablet computer. The workstation 1301 may be embedded in multiple computer systems.

[0086] The memory 1302 may include any non-transitory computer-readable storage medium for storing data and / or software, including instructions, which are executable by the processor 1304 and which control the operation of the workstation 1301 and in some aspects may also control the operation of the fluoroscopy scope 1315. According to the present disclosure, the fluoroscopy scope 1315 can be used to capture a sequence of fluoroscopic images based on which a fluoroscopic 3D reconstruction is generated and to capture live 2D fluoroscopic views. In one aspect, the memory 1302 may include one or more storage devices, such as solid-state storage devices, for example flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, the memory 1302 may include one or more mass storage devices connected to the processor 1304 via a mass storage controller (not shown) and a communication bus (not shown).

[0087] Although the description of computer-readable media contained herein refers to solid-state storage devices, it should be understood by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor 1304. That is, computer-readable storage media can include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable storage media can include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-ray or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store the desired information and can be accessed by the workstation 1301.

[0088] Thus, when the application 1318 is executed by the processor 1304, the display 1306 may be caused to present a user interface 1316. The user interface 1316 may be configured to present to the user a single screen that includes a three-dimensional (3D) view of the 3D model of the target from the perspective of the tip of the medical device, a live two-dimensional (2D) fluoroscopic view showing the medical device, and an overlay corresponding to the 3D model of the target, such as, for example, Figure 8 Target marker on live 2D fluoroscopic view shown. User interface 1316 may also be configured to display the target marker in different colors depending on whether the medical device tip is aligned with the target in three dimensions.

[0089] The network interface 1308 may be configured to connect to a network, such as a local area network (LAN) consisting of a wired network and / or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet. The network interface 1308 may be used to connect between the workstation 1301 and the fluoroscopic scope 1315. The network interface 1308 may also be used to receive image data 1314. The input device 1310 may be any device through which a user can interact with the workstation 1301, such as, for example, a mouse, a keyboard, a foot pedal, a touch screen, and / or a voice interface. The output module 1312 may include any connection port or bus, such as, for example, a parallel port, a serial port, a universal serial bus (USB), or any other similar connection port known to those skilled in the art. Based on the above and with reference to the various drawings, it will be understood by those skilled in the art that certain modifications may also be made to the present disclosure without departing from the scope of the present disclosure.

[0090] Although detailed aspects are disclosed herein, the disclosed aspects are merely examples of the disclosure that may be embodied in various forms and in various aspects. For example, aspects of an electromagnetic navigation system in combination with a target coverage system and method are disclosed herein; however, the target coverage system and method may also be applied to other navigation or tracking systems or methods known to those skilled in the art. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the disclosure in almost any suitable specific structure.

[0091] Although various aspects of the disclosure have been shown in the accompanying drawings, it is not intended to limit the disclosure to these aspects, as the disclosure is intended to be as broad as the art allows and the specification should be read in the same manner. Therefore, the above description should not be interpreted as limiting, but merely as an illustration of various aspects. Those skilled in the art will be able to envision other modifications within the scope and spirit of the claims appended hereto.

[0092] Various aspects of the present disclosure may also be described with reference to the following numbered paragraphs:

[0093] 1. A method, comprising:

[0094] Execute a setup procedure, the setup procedure comprising:

[0095] determining a first position of the tip of the catheter in a reference frame of the live fluoroscopic video; and

[0096] determining a position of a fluoroscopic imaging system based on the first position of the tip of the catheter in the reference system;

[0097] receiving live fluoroscopic video from a fluoroscopic imaging system;

[0098] displaying the live fluoroscopic video;

[0099] projecting three-dimensional (3D) coordinates from an electromagnetic (EM) sensor disposed at the tip of the catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, thereby generating projected 3D coordinates;

[0100] overlaying a target on the live fluoroscopic video;

[0101] determining that a second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates;

[0102] in response to determining that the second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates, determining that the fluoroscopic imaging system has moved and repeating the setup procedure;

[0103] receiving a position of the patient from one or more second EM sensors disposed on the patient;

[0104] determining, based on the position of the patient, that the patient has moved;

[0105] In response to determining that the patient has moved, determining an updated position of the target; and

[0106] The target is overlaid on the live fluoroscopic video at the updated position.

[0107] 2. The method according to paragraph 1, further comprising:

[0108] determining that the catheter is not aligned with the target; and

[0109] In response to determining that the catheter is not aligned with the target, an appearance of the target overlaid on the live fluoroscopic video is updated.

[0110] 3. The method of paragraph 2, wherein updating the appearance of the target comprises: changing the color of the target, highlighting the target, or applying a line pattern to the target.

[0111] 4. The method of paragraph 1 further comprising: panning the view of the live fluoroscopic video so that the target is in the center of the view of the live fluoroscopic video.

[0112] 5. The method of paragraph 1, wherein determining that the patient has moved comprises:

[0113] determining a reference position of the one or more second EM sensors; and

[0114] It is determined that a distance between the patient's position and the reference position is greater than a threshold distance.

[0115] 6. The method according to paragraph 2, further comprising:

[0116] determining a possible translation of the fluoroscopic imaging system based on a periodic grid appearing in the live fluoroscopic video;

[0117] selecting from the possible translations a translation whose projected 3D coordinates most closely match the second position of the tip of the catheter in the live fluoroscopic video, thereby producing a selected translation; and

[0118] The position of the fluoroscopic imaging system is determined based on the first position of the tip of the catheter in the reference system and the selected translation.

[0119] 7. The method according to paragraph 1, further comprising:

[0120] tracking the portion of the target that has been biopsied; and

[0121] The portion of the target that has been biopsied is displayed.

[0122] 8. The method of paragraph 1, wherein the fluoroscopic imaging system is a 3D fluoroscopy scope.

[0123] 9. A system for guiding navigation of a biopsy tool within a patient, the system comprising:

[0124] an electromagnetic (EM) field generator configured to generate an electromagnetic field;

[0125] a first EM sensor disposed at a tip of the catheter;

[0126] one or more second EM sensors disposed on the patient;

[0127] monitor;

[0128] Processor; and

[0129] A memory having instructions stored thereon, wherein when the instructions are executed by the processor, the processor:

[0130] Execute a setup procedure, the setup procedure comprising:

[0131] determining a first position of the tip of the catheter in a reference frame of the live fluoroscopic video; and

[0132] determining a position of a fluoroscopic imaging system based on the first position of the tip of the catheter in the reference system;

[0133] receiving live fluoroscopic video from a fluoroscopic imaging system;

[0134] displaying the live fluoroscopic video;

[0135] projecting three-dimensional (3D) coordinates from an electromagnetic (EM) sensor disposed at the tip of the catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, thereby generating projected 3D coordinates;

[0136] overlaying a target on the live fluoroscopic video;

[0137] determining that a second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates;

[0138] in response to determining that the second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates, determining that the fluoroscopic imaging system has moved and repeating the setup procedure;

[0139] receiving a position of the patient from the one or more second EM sensors disposed on the patient;

[0140] determining, based on the position of the patient, that the patient has moved;

[0141] In response to determining that the fluoroscopic imaging system has moved or the patient has moved, determining an updated position of the target; and

[0142] The target is overlaid on the live fluoroscopic video at the updated position.

[0143] 10. The system of paragraph 9, wherein, when the instructions are executed by the processor, the processor further calculates a patient coordinate reference system based on the EM field sensed by the EM sensor.

[0144] 11. The system of paragraph 9, wherein when the instructions are executed by the processor, the processor further:

[0145] determining that the catheter is not aligned with the target; and

[0146] In response to determining that the catheter is not aligned with the target, an appearance of the target overlaid on the live fluoroscopic video is updated.

[0147] 12. The system of paragraph 11, wherein updating the appearance of the target overlaying the live fluoroscopic video comprises: changing a color of the target, highlighting the target, or applying a line pattern to the target.

[0148] 13. The system of paragraph 9, wherein, when the instructions are executed by the processor, the processor further translates the view of the live fluoroscopic video so that the target is centered in the view of the live fluoroscopic video.

[0149] 14. The system of paragraph 9, wherein when the instructions are executed by the processor, the processor further:

[0150] determining a possible translation of the fluoroscopic imaging system based on a periodic grid appearing in the live fluoroscopic video;

[0151] selecting from the possible translations a translation whose projected 3D coordinates most closely match the second position of the tip of the catheter in the live fluoroscopic video, thereby producing a selected translation; and

[0152] The position of the fluoroscopic imaging system is determined based on the first position of the tip of the catheter in the reference system and the selected translation.

[0153] 15. The system of paragraph 9, wherein when the instructions are executed by the processor, the processor further:

[0154] tracking the portion of the target that has been biopsied; and

[0155] The portion of the target that has been biopsied is displayed.

[0156] 16. The system of paragraph 9, wherein the fluoroscopic imaging system is a 3D fluoroscopy scope.

[0157] 17. A system, comprising:

[0158] an electromagnetic (EM) field generator configured to generate an electromagnetic field;

[0159] one or more EM sensors disposed on a body of a patient;

[0160] monitor;

[0161] a processor coupled to the display; and

[0162] a memory coupled to the processor and having instructions stored on the memory that, when executed by the processor, cause the processor to:

[0163] displaying on the display a screen including a three-dimensional (3D) view of a 3D model of the target from the perspective of the tip of the medical device;

[0164] displaying in the screen a live two-dimensional (2D) fluoroscopic view showing the medical device;

[0165] overlaying a target marker corresponding to the 3D model of the target on the live 2D fluoroscopic view;

[0166] determining, based on one or more signals received from the one or more EM sensors, that the body of the patient has moved; and

[0167] In response to determining that the body of the patient has moved, a 2D position of the target marker overlaying the live 2D fluoroscopic view is updated.

[0168] 18. The system of paragraph 17, wherein determining that the body of the patient has moved comprises determining that an EM sensor of the one or more EM sensors disposed on the patient has moved greater than a threshold amount.

[0169] 19. The system of paragraph 17, wherein determining that the body of the patient has moved comprises determining that a combination of two or more of the EM sensors disposed on the patient has moved in a particular direction by more than a threshold amount.

[0170] 20. A method comprising:

[0171] detecting an aperiodic grid of markers in at least one fluoroscopic image captured by a fluoroscopic imaging system, thereby producing an aperiodic grid of detected markers;

[0172] determining a position of the fluoroscopic imaging system based on the detected aperiodic grid of markers;

[0173] displaying live fluoroscopic video from the fluoroscopic imaging system;

[0174] projecting three-dimensional (3D) coordinates from an electromagnetic (EM) sensor disposed at the tip of a catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, thereby generating projected 3D coordinates;

[0175] overlaying a target on the live fluoroscopic video;

[0176] determining that a second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates;

[0177] In response to determining that the second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates, determining that the fluoroscopic imaging system has moved; and

[0178] In response to determining that the fluoroscopic imaging system has moved, detecting the non-periodic grid of markers is repeated and the position of the fluoroscopic imaging system is determined.

Claims

1. A method, comprising: Execute a setup procedure, the setup procedure comprising: determining a first position of the tip of the catheter in a reference frame of the live fluoroscopic video; and determining a position of a fluoroscopic imaging system based on the first position of the tip of the catheter in the reference system; receiving live fluoroscopic video from a fluoroscopic imaging system; displaying the live fluoroscopic video; projecting three-dimensional (3D) coordinates from an electromagnetic (EM) sensor disposed at the tip of the catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, thereby generating projected 3D coordinates; overlaying a target on the live fluoroscopic video; determining that a second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates; in response to determining that the second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates, determining that the fluoroscopic imaging system has moved and repeating the setup procedure; receiving a position of the patient from one or more second EM sensors disposed on the patient; determining, based on the position of the patient, that the patient has moved; In response to determining that the patient has moved, determining an updated position of the target; and The target is overlaid on the live fluoroscopic video at the updated position.

2. The method according to claim 1, further comprising: determining that the catheter is not aligned with the target; as well as In response to determining that the catheter is not aligned with the target, an appearance of the target overlaid on the live fluoroscopic video is updated.

3. The method of claim 2, wherein updating the appearance of the target comprises: Change the color of the target, highlight the target, or apply a line pattern to the target.

4. The method according to claim 1, further comprising: The view of the live fluoroscopy video is translated so that the target is in the center of the view of the live fluoroscopy video.

5. The method of claim 1 , wherein determining that the patient has moved comprises: determining a reference position of the one or more second EM sensors; as well as It is determined that a distance between the position of the patient and the reference position is greater than a threshold distance.

6. The method according to claim 2, further comprising: determining a possible translation of the fluoroscopic imaging system based on a periodic grid appearing in the live fluoroscopic video; selecting from the possible translations a translation whose projected 3D coordinates most closely match the second position of the tip of the catheter in the live fluoroscopic video, thereby producing a selected translation; as well as The position of the fluoroscopic imaging system is determined based on the first position of the tip of the catheter in the reference system and the selected translation.

7. The method according to claim 1, further comprising: tracking the portion of the target that has been biopsied; as well as The portion of the target that has been biopsied is displayed.

8. The method of claim 1, wherein the fluoroscopic imaging system is a 3D fluoroscopy scope.

9. A system for guiding navigation of a biopsy tool within a patient, the system comprising: an electromagnetic (EM) field generator configured to generate an electromagnetic field; a first EM sensor disposed at a tip of the catheter; one or more second EM sensors disposed on the patient; monitor; processor; and A memory having instructions stored thereon, wherein when the instructions are executed by the processor, the processor: Execute a setup procedure, the setup procedure comprising: determining a first position of the tip of the catheter in a reference frame of the live fluoroscopic video; and determining a position of a fluoroscopic imaging system based on the first position of the tip of the catheter in the reference system; receiving live fluoroscopic video from a fluoroscopic imaging system; displaying the live fluoroscopic video; projecting three-dimensional (3D) coordinates from an electromagnetic (EM) sensor disposed at the tip of the catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, thereby generating projected 3D coordinates; overlaying a target on the live fluoroscopic video; determining that a second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates; in response to determining that the second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates, determining that the fluoroscopic imaging system has moved and repeating the setup procedure; receiving a position of the patient from the one or more second EM sensors disposed on the patient; determining, based on the position of the patient, that the patient has moved; In response to determining that the fluoroscopic imaging system has moved or the patient has moved, determining an updated position of the target; and The target is overlaid on the live fluoroscopic video at the updated position.

10. The system according to claim 9, wherein: When the instructions are executed by the processor, the processor further calculates a patient coordinate reference system based on the EM field sensed by the EM sensor.

11. The system according to claim 9, wherein: When the instructions are executed by the processor, the processor further: determining that the catheter is not aligned with the target; and In response to determining that the catheter is not aligned with the target, an appearance of the target overlaid on the live fluoroscopic video is updated.

12. The system of claim 11, wherein updating the appearance of the target overlaying the live fluoroscopic video comprises: Change the color of the target, highlight the target, or apply a line pattern to the target.

13. The system according to claim 9, wherein: When the instructions are executed by the processor, the processor also translates the view of the live fluoroscopy video so that the target is in the center of the view of the live fluoroscopy video.

14. The system according to claim 9, wherein: When the instructions are executed by the processor, the processor further: determining a possible translation of the fluoroscopic imaging system based on a periodic grid appearing in the live fluoroscopic video; selecting from the possible translations a translation whose projected 3D coordinates most closely match the second position of the tip of the catheter in the live fluoroscopic video, thereby producing a selected translation; as well as The position of the fluoroscopic imaging system is determined based on the first position of the tip of the catheter in the reference system and the selected translation.

15. The system according to claim 9, wherein: When the instructions are executed by the processor, the processor further: tracking the portion of the target that has been biopsied; and The portion of the target that has been biopsied is displayed.

16. The system of claim 9, wherein the fluoroscopic imaging system is a 3D fluoroscopy scope.

17. A system, comprising: an electromagnetic (EM) field generator configured to generate an electromagnetic field; one or more EM sensors disposed on a body of a patient; monitor; a processor coupled to the display; and a memory coupled to the processor and having instructions stored on the memory that, when executed by the processor, cause the processor to: displaying on the display a screen including a three-dimensional (3D) view of a 3D model of the target from the perspective of the tip of the medical device; displaying in the screen a live two-dimensional (2D) fluoroscopic view showing the medical device; overlaying a target marker corresponding to the 3D model of the target on the live 2D fluoroscopic view; determining, based on one or more signals received from the one or more EM sensors, that the body of the patient has moved; as well as In response to determining that the body of the patient has moved, a 2D position of the target marker overlaying the live 2D fluoroscopic view is updated.

18. The system of claim 17, wherein determining that the body of the patient has moved comprises: It is determined that an EM sensor of the one or more EM sensors disposed on the patient has moved greater than a threshold amount.

19. The system of claim 17, wherein determining that the body of the patient has moved comprises: It is determined that a combination of two or more of the EM sensors disposed on the patient have moved in a particular direction by more than a threshold amount.

20. A method comprising: detecting an aperiodic grid of markers in at least one fluoroscopic image captured by a fluoroscopic imaging system, thereby producing an aperiodic grid of detected markers; determining a position of the fluoroscopic imaging system based on the detected aperiodic grid of markers; displaying live fluoroscopic video from the fluoroscopic imaging system; projecting three-dimensional (3D) coordinates from an electromagnetic (EM) sensor disposed at the tip of a catheter onto the live fluoroscopic video based on the position of the fluoroscopic imaging system, thereby generating projected 3D coordinates; overlaying a target on the live fluoroscopic video; determining that a second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates; responsive to determining that the second position of the tip of the catheter in the live fluoroscopic video is not at or near the projected 3D coordinates, determining that the fluoroscopic imaging system has moved; as well as In response to determining that the fluoroscopic imaging system has moved, detecting the non-periodic grid of markers is repeated and the position of the fluoroscopic imaging system is determined.

Images