System and method for confirming position or orientation of medical device relative to target

Abstract

Systems and methods for visually verifying whether a medical device is located inside a target or directed toward the target using intraoperative imaging when the medical device is located at or near the target. These systems and methods involve performing a fluoroscopic scan on a patient having a medical device placed thereon; reconstructing a volume based on fluoroscopic scanning; displaying an initial slice of the volume from which the user starts a search; and receiving information identifying a tip and a target of the medical device in the volume as the user scrolls the slice of the volume. Scrolling the volume allows the user to determine a relationship between the medical device and the target. Alternatively, feedback is provided to a user by enhancing the marking of the tip of the medical device and the target and / or notifying the user of whether the tip of the medical device is inside the target or directed to the target.

Description

Technical Field

[0001] The present technology generally relates to systems and methods for navigation and position or orientation confirmation of surgical procedures. More specifically, the present disclosure relates to systems and methods for confirming the position or orientation of a medical device relative to a target using two-dimensional intraoperative X-ray images captured using a standard X-ray imaging device and a three-dimensional volume constructed from the two-dimensional intraoperative X-ray images. Background Art

[0002] There are a variety of common methods for treating various diseases that affect organs including the liver, brain, heart, lungs, and kidneys. Typically, clinicians use one or more imaging modalities, such as magnetic resonance imaging, ultrasound imaging, computed tomography (CT), and other imaging modalities, to identify areas of interest in a patient's body and ultimately identify targets for treatment.

[0003] Endoscopic methods have proven useful for navigating to areas of interest within a patient's body, and in particular, areas within the body's cavitary network, such as the lungs. To enable endoscopic methods, and more particularly bronchoscopic methods, within the lungs, endobronchial navigation systems have been developed that use previously acquired MRI data or CT image data to generate a three-dimensional rendering or volume of a particular body part, such as the lungs. Specifically, a three-dimensional rendering or volume rendering of a particular body part of a patient is generated using previously acquired images acquired from an MRI scan or CT scan of the patient.

[0004] The resulting volume generated from the MRI scan or CT scan is then used to create a navigation plan to facilitate advancing a navigation catheter (or other suitable device) through the bronchoscope and the patient's bronchial branches to the area of ​​interest. Electromagnetic tracking can be used in conjunction with the CT data to facilitate guiding the navigation catheter through the bronchial branches to the area of ​​interest. In some cases, the navigation catheter can be positioned within an airway of the branching cavity network, adjacent to the area of ​​interest, or within the area of ​​interest to provide access for one or more medical devices.

[0005] Therefore, in order to generate a navigation plan, or even to generate a three-dimensional rendering or volume rendering of a patient's anatomical structure (such as the lungs), the clinician is required to utilize an MRI system or a CT system to acquire the necessary image data for constructing the three-dimensional volume. MRI systems or CT-based imaging systems are extremely expensive and, in many cases, are not available at the same location where the navigation plan is generated or the navigated procedure is performed.

[0006] During navigation surgery, a fluoroscopic imaging device is usually located in the operating room. Clinicians can use a standard fluoroscopic imaging device to visualize and confirm the placement of the tool after the tool is navigated to the desired position. However, although standard fluoroscopic images show high-density objects (such as metal tools and bones) and large soft tissue objects (such as the heart), it is difficult for fluoroscopic images to distinguish small soft tissue objects of interest (such as lesions). Further, fluoroscopic images are only two-dimensional projections. In order to be able to see small soft tissue objects in three-dimensional space, X-ray volume reconstruction is required. There are multiple solutions for providing soft tissue three-dimensional volume reconstruction, such as CT and cone beam CT widely used in the medical community. These machines algorithmically combine multiple X-ray projections from known calibrated X-ray source positions into a three-dimensional volume visible to soft tissue.

[0007] In order to navigate a tool to a distant soft tissue target for biopsy or treatment, both the tool and the target should be visible in some kind of 3D guidance system. Most of these systems use some kind of X-ray device to see through the body. For example, during surgery, a CT machine can be used with iterative scans to provide guidance through the body until the tool reaches the target. This is a cumbersome procedure because it requires multiple complete CT scans, a dedicated CT room, and blind navigation between scans. In addition, each scan requires the staff to leave the room. Another option is a cone beam CT machine, which is available in some operating rooms and is somewhat easier to operate, but is expensive and, like CT, only provides blind navigation between scans, requires multiple iterations for navigation, and requires the staff to leave the room. Summary of the invention

[0008] The technology of the present disclosure generally relates to systems and methods for using intraoperative imaging to confirm whether a medical device is located within and / or aligned with a target when the medical device is located at or near the target.

[0009] In one aspect, the present disclosure provides a system comprising a computer system having a processor and a display that displays a graphical user interface. The computer system also has a computer-readable storage medium having instructions stored thereon, which, when executed by the processor, cause the processor to receive a sequence of intraoperative X-ray images. Each intraoperative X-ray image includes at least a portion of a medical device and a target. These instructions, when executed by the processor, also cause the processor to receive a marking of the tip of the medical device in at least two images in the sequence of intraoperative X-ray images; and to construct a three-dimensional (3D) reconstruction based on the sequence of intraoperative X-ray images and the marking of the tip of the medical device, the 3D reconstruction including the medical device and the target. These instructions, when executed by the processor, also cause the processor to display a slice through the tip of the medical device of the 3D reconstruction, determine the position of the target in the 3D reconstruction; and present feedback about the position of the tip of the medical device relative to the target.

[0010] Embodiments of the system may also include one or more of the following features. These instructions, when executed by the processor, may cause the processor to enhance a marking of the target or a marking of the medical device to show that the medical device is located inside or outside the target. These instructions, when executed by the processor, may cause the processor to present a message indicating that the medical device is located inside or outside the target. These instructions, when executed by the processor, may cause the processor to perform the following operations: determine that the tip of the medical device is outside the target; in response to determining that the tip of the medical device is outside the target, determine whether the tip of the medical device is aligned with the target; and present a message indicating whether the tip of the medical device is aligned with the target. These instructions, when executed by the processor, may cause the processor to perform operations: determine that the tip of the medical device is outside the target; in response to determining that the tip of the medical device is outside the target, determine the distance between the tip of the medical device and the target or the position of the tip of the medical device relative to the target; and present a message indicating the distance between the tip of the medical device and the target or the position of the tip of the medical device relative to the target.

[0011] In various aspects, the instructions, when executed by a processor, may cause the processor to perform the following operations: receive a preoperative computed tomography (CT) image of a target; construct a 3D model of the target based on the preoperative CT image; and overlay the 3D model of the target on the 3D reconstruction. The instructions, when executed by the processor, may cause the processor to perform the following operations: register the preoperative CT image with the 3D reconstruction; determine a position of the target in the 3D reconstruction based on the registration, thereby obtaining a determined target position; and overlay the 3D model of the target on the 3D reconstruction based on the determined target position.

[0012] In various aspects, the instructions, when executed by a processor, may cause the processor to perform the following operations: construct a 3D model of a medical device based on a sequence of intraoperative X-ray images; and superimpose the 3D model of the medical device on a 3D reconstruction. The instructions, when executed by the processor, may cause the processor to perform the following operations: receive a preoperative computed tomography (CT) image of a target; segment the target from the preoperative CT image, thereby obtaining a segmented target; and superimpose the segmented target on the 3D reconstruction. The instructions, when executed by the processor, may cause the processor to perform the following operations: receive a preoperative computed tomography (CT) image of a lung; receive at least one marker of a target on the preoperative CT image; and superimpose at least one marker of the target on the 3D reconstruction. The at least one marker of the target may represent a size or shape of the target.

[0013] In various aspects, the sequence of intraoperative X-ray images may be fluoroscopic images or cone beam CT images. The medical device may not include a position sensor. The medical device may be a biopsy tool.

[0014] In another aspect, the present disclosure provides a method. The method includes: receiving a sequence of intraoperative X-ray images; receiving a marking of a tip of a medical device in at least two images in the sequence of intraoperative X-ray images; and generating a three-dimensional (3D) reconstruction based on the sequence of intraoperative X-ray images and the marking of the tip of the medical device. The 3D reconstruction includes at least a portion of the medical device and a target. The method also includes: displaying a slice of the 3D reconstruction through the tip of the medical device; determining a position of the target in the 3D reconstruction; and presenting feedback about the position of the tip of the medical device relative to the target.

[0015] Implementations of the method may also include one or more of the following features. The method may include enhancing a marking of the target or a marking of the medical device to show that the medical device is located inside or outside the target. The method may include presenting a message indicating that the medical device is located inside or outside the target.

[0016] The method may include: determining that the tip of the medical device is outside the target; in response to determining that the tip of the medical device is outside the target, determining that the tip of the medical device is aligned with the target; in response to determining that the tip of the medical device is aligned with the target, determining the distance between the tip of the medical device and the target; and presenting a message indicating that the tip of the medical device is aligned with the target and the distance between the tip of the medical device and the target.

[0017] The method may include: determining that the tip of the medical device is outside the target; in response to determining that the tip of the medical device is outside the target, determining that the tip of the medical device is not aligned with the target; in response to determining that the tip of the medical device is not aligned with the target, determining the position of the tip of the medical device relative to the target; and presenting a message indicating that the tip of the medical device is not aligned with the target and the position of the tip of the medical device relative to the target.

[0018] In yet another aspect, the present disclosure provides another system comprising a computer system having a processor and a display for displaying a graphical user interface. The computer system also has a computer-readable storage medium having instructions stored thereon, which, when executed by the processor, causes the processor to receive a sequence of intraoperative X-ray images from an X-ray imaging device. Each intraoperative X-ray image includes at least a portion of a medical device and a target. These instructions, when executed by the processor, also cause the processor to estimate the posture of the X-ray imaging device based on the sequence of intraoperative X-ray images, thereby obtaining an estimated posture; and generate a three-dimensional (3D) volume based on the sequence of intraoperative X-ray images and the estimated posture. The 3D volume includes at least a portion of a medical device and a target. These instructions, when executed by the processor, also cause the processor to display a slice of the 3D volume, from which a user searches for the tip and target of the medical device in the slice of the 3D volume; receive position information of a scroll control object; and display other slices of the 3D volume corresponding to the position information.

[0019] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the technology described in the disclosure will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various aspects and embodiments of the present disclosure are described below with reference to the accompanying drawings, in which:

[0021] Figure 1 is a perspective view of an illustrative example of an electromagnetic navigation (EMN) system incorporating a fluoroscopic imaging device according to the present disclosure;

[0022] Figure 2A is a diagram illustrating a user interface showing a slice of a fluoroscopic 3D reconstruction centered on a target;

[0023] Figure 2B is a diagram illustrating a user interface showing a slice of a fluoroscopic 3D reconstruction centered on the tip of a medical device;

[0024] Figure 3 and Figure 4is a flow chart illustrating a method for visualizing a relationship between a tip of a medical device and a target;

[0025] Figure 5 is a diagram illustrating a user interface for marking a tip of a medical device in a fluoroscopic image of a portion of a lung;

[0026] Figure 6 is a diagram showing a user interface for displaying a reconstructed volume and marking objects;

[0027] Figure 7 is a diagram illustrating an example user interface for displaying enhancements to a marking of a tip of a medical device;

[0028] Figure 8 is a flow chart illustrating another method for visualizing the relationship between the tip of a medical device and a target;

[0029] Fig. 9 is demonstrated for use according to Figure 8 A method of providing a user interface for providing textual feedback to a user while displaying a reconstructed volume;

[0030] FIG. 10A to FIG. 10H are diagrams illustrating other examples of user interfaces implementing aspects of the methods described herein; and

[0031] Fig.11 is a diagram illustrating a system configured for use with the methods of the present disclosure. DETAILED DESCRIPTION

[0032] When navigating a medical device to a target, a fluoroscopic three-dimensional reconstruction or volume can be generated from a two-dimensional fluoroscopic image using limited angle tomosynthesis and displayed to help the clinician align the medical device with the target or confirm that the tip of the medical device is within the target. However, there is significant scattering in the anterior-posterior (AP) direction. Scattering makes it challenging to visually determine whether the tip of a medical device (e.g., a biopsy tool) is inside, above, or below a target (e.g., a lesion). This challenge does not exist when using a CT-like volume. In addition, during a biopsy, if localized bleeding and atelectasis occur, 3D visualization of the target may be significantly deteriorated.

[0033] Figure 2A and Figure 2B The challenge of understanding whether a medical device is above or below a target by scrolling through slices of a fluoroscopic 3D reconstruction is demonstrated. Figure 2A A slice of a fluoroscopic 3D reconstruction centered on the target is shown, and Figure 2B A slice of a fluoroscopic 3D reconstruction centered on the tip of the medical device is shown. Figure 2B As shown, the target is still visible on the slice centered at the tool tip. However, based on the planned target radius, the tool is slightly below the target.

[0034] The present disclosure relates to a system and method that supports a clinician in deciding whether to perform a medical procedure (e.g., a biopsy) at the correct location by providing the clinician with visual confirmation that the end portion of a medical device (e.g., a biopsy tool) is located within or directed toward a target. Confirmation can be provided via intraoperative imaging when the end portion of the medical device is located near the target.

[0035] The system and method of the present disclosure is performed after navigating the medical device to the vicinity of the target, and uses a 3D reconstruction generated based on intraoperative fluoroscopic images using limited angle tomosynthesis. The system and method of the present disclosure also eliminates the reliance on a positionable guide (LG) so that the medical device (e.g., a biopsy tool) can be placed in a position for performing a medical procedure (e.g., a biopsy).

[0036] These systems and methods include: performing a scan with a fluoroscope to obtain a sequence of fluoroscopic images; reconstructing a volume based on the sequence of fluoroscopic images; displaying an initial slice from which a clinician begins a search; and identifying a target and a tip of a medical device in a reconstructed volume by enabling a clinician to scroll through slices of the reconstructed volume. In some aspects, the relationship between the medical device and the target can be displayed so that the clinician can understand the relationship between the medical device and the target, for example, that the tip of the medical device is located within or aligned with the target.

[0037] Alternatively, in other aspects, additional visualization features are provided as feedback to the clinician. For example, the marking of the target and / or the tip of the medical device is enhanced and / or a message or notification is provided to the clinician indicating whether the tip of the medical device is located inside or outside the target. In various aspects, the systems and methods of the present disclosure estimate the orientation of the tip of the medical device, predict where the tip of the medical device will travel if the clinician advances the medical device along a trajectory that is aligned with the current position of the tip of the medical device and with the orientation of the tip of the medical device, and then displays the trajectory, which can also help the clinician understand whether the end portion of the medical device is aligned with the target.

[0038] Figure 1An electromagnetic navigation (EMN) system 100 is depicted that is configured to examine CT image data to identify one or more targets, plan a path to the identified targets (planning phase), navigate an extended working channel (EWC) 12 of a catheter assembly to the targets via a user interface (navigation phase), and confirm placement of the EWC 12 and a medical device (e.g., a biopsy tool) relative to the targets. One such EMN system is the ELECTROMAGNETIC NAVIGATION SYSTEM currently sold by Medtronic. The target may be tissue of interest identified during the planning phase by examining the CT image data. After navigation, a medical device, such as a biopsy tool or other tool, may be inserted into the EWC 12 to obtain a tissue sample from tissue located at or near the target.

[0039] like Figure 1 As shown, the EWC 12 is part of a catheter guide assembly 40. In practice, the EWC 12 is inserted into a bronchoscope 30 for accessing the luminal network of a patient P. Specifically, the EWC 12 of the catheter guide assembly 40 can be inserted into the working channel of the bronchoscope 30 to navigate through the luminal network of the patient. A locatable guide device (LG) 32 including a sensor 44 is inserted into the EWC 12 and locked in place so that the sensor 44 extends a desired distance beyond the distal tip of the EWC 12. The position and orientation of the sensor 44 relative to a reference coordinate system and thus the position and orientation of the distal portion of the EWC 12 within an electromagnetic field can be derived. The catheter guide assembly 40 is currently marketed by Medtronic under the trade name superDimension TM Surgical Suite or EDGE TM The surgical kit is marketed and sold and is believed to be usable with the present disclosure. For a more detailed description of the catheter guide assembly 40, reference is made to commonly owned U.S. Patent Publication No. 2014 / 0046315, U.S. Patent No. 7,233,820, and U.S. Patent No. 9,044,254 filed on March 15, 2013 by Ladtkow et al., the entire contents of each of which are hereby incorporated by reference herein.

[0040] The EMN system 100 generally includes: an operating table 20 configured to support a patient P; a bronchoscope 30 configured to be inserted into the airway of patient P through the mouth of patient P; a monitoring device 120 coupled to the bronchoscope 30 (e.g., a video display for displaying video images received from a video imaging system of the bronchoscope 30); a tracking system 50 including a tracking module 52, a plurality of reference sensors 54, and a transmitter pad 56; and a workstation or computer system 125 including software and / or hardware for facilitating identification of a target, path planning to a target, navigation of a medical device to a target, confirmation of placement of the EWC 12, and confirmation of placement of a medical device extending through and out of the EWC 12 relative to a target.

[0041] This particular aspect of the system 100 also includes a fluoroscopic imaging device 110 capable of acquiring a fluoroscopic image or an x-ray image or video of the patient P. The images, image sequences, or videos captured by the fluoroscopic imaging device 110 can be stored in the fluoroscopic imaging device 110 or transmitted to the computer system 125 for storage, processing, and display. In addition, the fluoroscopic imaging device 110 can be moved relative to the patient P so that images can be acquired from different angles or perspectives relative to the patient P to create a fluoroscopic video. In one aspect, the fluoroscopic imaging device 110 includes an angle measurement device 111, which is configured to measure the angle of the fluoroscopic imaging device 110 relative to the patient P. The angle measurement device 111 can be an accelerometer. The fluoroscopic imaging device 110 can include a single imaging device or more than one imaging device. In the case where the fluoroscopic imaging device 110 includes multiple imaging devices, each imaging device can be a different type of imaging device or the same type of imaging device. Further details about the fluoroscopic imaging device 110 are described in U.S. Patent No. 8,565,858, which is incorporated herein by reference in its entirety.

[0042] The computer system 125 can be any suitable computer system including a processor and a storage medium, wherein the processor is capable of executing instructions stored on the storage medium. The computer system 125 can further include a database configured to store patient data, a CT data set including a CT image, a fluoroscopic data set including a fluoroscopic image and a video, a navigation plan, and any other such data. Although not explicitly shown, the computer system 125 can include an input terminal, or can be otherwise configured to receive the CT data set, the fluoroscopic image or video, and other data described herein. In addition, the computer system 125 includes a display configured to display a graphical user interface. The computer system 125 can be connected to one or more networks, through which one or more databases can be accessed.

[0043] Regarding the planning phase, the computer system 125 utilizes previously acquired CT image data to generate and view a three-dimensional model of the airway of the patient P, enabling the identification (automatic, semi-automatic or manual) of targets on the three-dimensional model, and allowing the determination of a path through the airway of the patient P to reach tissue located at and around the target. More specifically, the CT images acquired from the previous CT scan are processed and assembled into a three-dimensional CT volume, which is then used to generate a three-dimensional model of the airway of the patient P. The three-dimensional model can be displayed on a display associated with the computer system 125, or in any other suitable manner. Using the computer system 125, various views of the three-dimensional model or enhanced two-dimensional images generated by the three-dimensional model are presented. The enhanced two-dimensional images can have some three-dimensional capabilities because they are generated from three-dimensional data. The three-dimensional model can be manipulated to facilitate the identification of targets on the three-dimensional model or two-dimensional image, and a suitable path can be selected to approach the tissue located at the target through the airway of the patient P. Once selected, the path plan, the three-dimensional model, and the images derived therefrom can be saved and exported to the navigation system for use during (multiple) navigation phases. One such planning software is superDimension, currently sold by Medtronic. TM Planning kit.

[0044] With respect to the navigation phase, a six-degree-of-freedom electromagnetic tracking system 50 (e.g., similar to those disclosed in U.S. Patent Nos. 8,467,589, 6,188,355, and published PCT Application Nos. WO 00 / 10456 and WO 01 / 67035 (the entire contents of each of which are incorporated herein by reference)) or other suitable positioning measurement system is used to perform registration of the image and navigation path, although other configurations are also contemplated. The tracking system 50 includes a tracking module 52, a plurality of reference sensors 54, and a transmitter pad 56. The tracking system 50 is configured for use with the locatable guide 32 (particularly the sensor 44). As described above, the locatable guide 32 and the sensor 44 are configured for insertion into the airway of the patient P through the EWC 12 (with or without the bronchoscope 30), and can be selectively locked relative to each other via a locking mechanism.

[0045] The transmitter pad 56 is positioned below the patient P. The transmitter pad 56 generates an electromagnetic field around at least a portion of the patient P within which the location of the reference sensor 54 and the sensor 44 can be determined using the tracking module 52. One or more of the reference sensors 54 are attached to the chest of the patient P. The six degree of freedom coordinates of the reference sensor 54 are sent to the computer system 125 (which includes appropriate software) where they are used to calculate the patient reference coordinate system. Registration, as described in detail below, is typically performed to coordinate the location of the three-dimensional model and two-dimensional image from the planning phase with the airway of the patient P as viewed through the bronchoscope 30, and to allow the navigation phase to proceed with precise knowledge of the location of the sensor 44 (even in portions of the airway that the bronchoscope 30 cannot reach). Further details of this registration technique and its implementation in cavity navigation can be found in U.S. Patent Application Publication No. 2011 / 0085720, the entire contents of which are incorporated herein by reference, although other suitable techniques are also contemplated.

[0046] Registration of the position of the patient P on the transmitter pad 56 is performed by moving the LG 32 through the airway of the patient P. More specifically, as the locatable guide 32 moves through the airway, data related to the position of the sensor 44 is recorded using the transmitter pad 56, the reference sensor 54, and the tracking module 52. The shape generated from the position data is compared with the internal geometry of the passage of the three-dimensional model generated in the planning stage, and the positional correlation between the shape and the three-dimensional model is determined based on the comparison, for example, using software on the computer system 125. In addition, the software can identify non-tissue space (e.g., air-filled cavities) in the three-dimensional model. The software aligns or registers the image representing the position of the sensor 44 with the three-dimensional model and the two-dimensional image generated from the three-dimensional model, which images are based on the recorded position data and the assumption that the locatable guide 32 remains located in the non-tissue space in the airway of the patient P. Alternatively, a manual registration technique can be utilized by navigating the bronchoscope 30 with the sensor 44 to a pre-specified position in the lungs of the patient P and manually associating the image from the bronchoscope with the model data of the three-dimensional model.

[0047] After registering the patient P to the image data and the path planning, a user interface is displayed in the navigation software that shows the path that the clinician needs to follow to reach the vicinity of the target using the tip of the EWC 12. One such navigation software is superDimension currently sold by Medtronic TM Navigation system.

[0048] Once the EWC 12 has successfully navigated proximate to a target depicted on the user interface, the locatable guide device 32 can be unlocked from the EWC 12 and removed, thereby leaving the EWC 12 in place as a guide channel for guiding medical devices to the target, including but not limited to optical systems, ultrasound probes, marker placement tools, biopsy tools, ablation tools (i.e., microwave ablation devices), laser probes, cryogenic probes, sensor probes, and aspiration needles.

[0049] Thus, navigating the medical device to the target can be divided into two stages. In the first stage, the EWC 12 is navigated close to the target. In the second stage, the medical device is fed through the EWC 12 until the distal portion of the medical device extends out of the EWC 12. The medical device is then aligned with and guided to the target using a fluoroscopic 3D reconstruction generated from a sequence of fluoroscopic images acquired while the medical device is aligned with and guided to the target.

[0050] Specifically, in a first stage, a tool (e.g., LG 32 inside the EWC 12) is navigated to a desired location in the patient P (e.g., near a target), and the tool is re-navigated using local registration corrections. This may include following path planning and using the above-described EMN system, bronchoscopic imaging, and / or fluoroscopic imaging using a fluoroscopic imaging device 110. Fluoroscopic imaging may include performing a first fluoroscopic scan of the EWC 12. When the fluoroscopic imaging device 110 rotates around the patient P, the first fluoroscopic scan acquires a sequence of first two-dimensional (2D) fluoroscopic images at different angles. In other aspects, intraoperative fluoroscopic imaging may be replaced by another suitable mode of intraoperative X-ray imaging, such as intraoperative cone beam computed tomography (CBCT), which may also be referred to as C-arm CT, cone beam volume CT, flat panel CT, or digital volume tomography (DVT). CBCT imaging involves X-ray computed tomography, in which the X-rays are divergent, forming a cone. In the case of CBCT, first 2D CBCT images are acquired while a CBCT imaging device (not shown) rotates around the patient P. Each first fluoroscopic image acquired by the fluoroscopic imaging device 110 may show radiopaque markers from a pattern of radiopaque markers on a transmitter pad 56 disposed under the patient P, for example.

[0051] After receiving the first fluoroscopic images, the pose of the fluoroscopic imaging device 110 (e.g., a C-arm fluoroscope) is estimated for each first fluoroscopic image. The pose estimation may include generating a probability map. The probability map indicates the probability that each pixel of each first fluoroscopic image belongs to a projection of a radiopaque marker of the transmitter pad 56, which may include multiple radiopaque markers. The radiopaque marker may be in the form of a two-dimensional (2D) structure of markers. The 2D structure of markers may include multiple spherical markers arranged in a two-dimensional grid pattern.

[0052] The probability map can be generated, for example, by feeding the image to a simple label detector (such as a Harris corner detector), which outputs a new image of smoothed density corresponding to the probability that each pixel belongs to a label. The probability map includes pixels or densities corresponding to the labels. In some aspects, the probability map can be scaled down (i.e., reduced in size) to simplify calculations.

[0053] Different candidates may be generated for projections of the marked structure on the image. Different candidates may be generated by virtually positioning the imaging device within a range of different possible poses. The possible poses of the fluoroscopic imaging device include the 3D position and orientation of the fluoroscopic imaging device. In some aspects, such a range may be limited based on the geometry and / or degrees of freedom of the imaging device. For each possible pose, a virtual projection of at least a portion of the marker is generated, as if the fluoroscopic imaging device actually captured an image of the marked structure when positioned in that pose.

[0054] Based on the image probability map, the candidate with the highest probability of the projection of the marked structure on the image is identified. Each candidate (i.e., the virtual projection of the marked structure) can be superimposed or associated with the probability map. Then, a probability score can be determined or a probability score can be associated with each marker projection of the candidate. In some aspects, the probability score can be positive or negative, that is, if the virtual marker projection falls into a pixel with a lower probability, a cost may be incurred. Then, the probability scores of all marker projections of the candidate can be summed, and a total probability score can be determined for each candidate. For example, if the marked structure is a two-dimensional grid, the projection will have a grid form. Each point of the projected grid will be located on at least one pixel of the probability map. If the point of the 2D grid candidate is located on the pixel with the highest density, that is, if its point is located on the projection of the center of the marker on the image, the probability score of the grid candidate is the highest. The candidate with the highest probability score can be determined as the candidate with the highest probability of the projection of the marked structure on the image. Then, the posture of the imaging device when capturing the image can be estimated based on the virtual posture of the imaging device used to generate the identified candidate.

[0055] The above-described posture estimation process is one possible posture estimation process; however, those skilled in the art will recognize that other methods and processes of posture estimation may be employed without departing from the scope of the present disclosure. As described above, the posture estimation process is performed on each image in the first fluoroscopic scan. The result of the processing is a determination of the posture of the fluoroscopic imaging device for each image acquired.

[0056] The following Figure 3 to Figure 1 0 shows an example of various aspects of the workflow in the second phase, using components of system 100 (including fluoroscopic imaging device 110 and computer system 125) to generate and display a fluoroscopic three-dimensional reconstruction showing a medical device and a target, and provide feedback to a clinician about the position and orientation of the tip of the medical device relative to the target. The feedback can include text or visual indicators of whether the tip of the medical device is aligned with the target and / or within the target. In various aspects, the position and orientation of the tip of the medical device relative to the target is determined without using an EM sensor disposed on or in the medical device.

[0057] In the second stage of navigation, the medical device is passed through the EWC 12 until the medical device extends out of the EWC 12. The medical device can be a biopsy tool or a therapeutic tool (e.g., a microwave ablation catheter). The second stage of navigation also includes confirming that the medical device is aligned with and / or within the target without updating the registration or performing electromagnetic (EM) corrections. The second stage of navigation also may not include adjusting or otherwise configuring the settings of the fluoroscope, such as centering the fluoroscope on the catheter or setting the fluoroscopic image orientation, because these functions are performed during the first stage of fluoroscopic navigation.

[0058] Figure 3The present invention is a flowchart of a method for verifying or confirming the position of a medical device (e.g., a biopsy tool or a treatment tool) relative to a target (e.g., a lesion) in a second stage of fluoroscopic navigation or other suitable intraoperative X-ray navigation (e.g., CBCT navigation) by providing visual feedback to a user via a display using a fluoroscopic image or other suitable intraoperative X-ray image (e.g., CBCT image) from a second scan of at least a portion of the medical device. At box 302, a sequence of second fluoroscopic images or other suitable second intraoperative X-ray images (e.g., second CBCT images) is obtained from a second scan of at least a portion of the medical device by a fluoroscopic imaging device or other suitable intraoperative X-ray imaging device (e.g., CBCT imaging device). At box 304, a pose of the fluoroscopic imaging device or other suitable intraoperative X-ray image (e.g., CBCT imaging device) is estimated based on the sequence of second fluoroscopic images or other suitable second intraoperative X-ray images. At box 306, a 3D reconstruction is generated based on the sequence of second fluoroscopic images or other suitable second intraoperative X-ray images and the estimated pose.

[0059] At box 307, a 3D reconstruction and a scroll control object for scrolling a slice of the 3D reconstruction are displayed, for example, via a user interface. At box 308, a slice of the 3D reconstruction is displayed, from which the user starts searching for the tip and target of the medical device in the 3D reconstruction slice. At box 310, position information of the scroll control object moved by the user during the search is received, and at box 312, other slices of the 3D reconstruction corresponding to different positions of the scroll control object are displayed (for example, the clinician uses a mouse or other input device to move the scroll control object to different positions). Then, optionally at box 314, a mark of the tip and target of the medical device in the slice of the 3D reconstruction is received. Alternatively, the user can scroll the 3D reconstruction to understand and / or analyze the relationship between the medical device and the target. If desired, method 300 may include displaying a button or similar control object that, when selected by the user, enables the user to mark the tip and target of the medical device to help the user analyze the relationship between the medical device and the target.

[0060] Alternatively, feedback may be provided to the user. Figure 4is a flow chart illustrating a method for providing feedback to a user regarding the position of the tip of a medical device relative to a target. At box 402, a sequence of second fluoroscopic images or other suitable second intraoperative X-ray images (e.g., CBCT images) is received from a second scan of at least a portion of a medical device via a C-arm fluoroscope or other suitable intraoperative imaging device (e.g., a CBCT imaging device). At box 404, a marking of the tip of a medical device (e.g., a biopsy tool) in at least two second fluoroscopic images in the sequence of second fluoroscopic images is received. In the case of CBCT imaging, marking of the tip of the medical device may alternatively be performed in the reconstructed volume. The marking may be received from a user interface, prompting the user to place the marking on the two second fluoroscopic images displayed in the user interface. For example, as Figure 5 As shown in FIG. 1 , a user interface 501 is displayed on a display of a computer system 125 wherein two second fluoroscopic images 510a, 510b acquired during a second fluoroscopic scan are presented and the clinician is prompted to mark the locations of the tips of tools 512a, 512b that have exited the EWC 511a, 511b in each of the fluoroscopic images 510a, 510b.

[0061] The user interface 501 includes a message prompting the user to use the trackball module to mark the tip of the tool 512a, 512b in each of the second fluoroscopic images 510a, 510b. Figure 5 In an example of a user interface of , markers 522a, 522b are placed on the tips of tools 512a, 512b. User interface 501 also includes a message asking whether the tip of the tool is visible in the second fluoroscopic images 510a, 510b. User interface 501 also includes a hyperlink text "Replace Image", which the clinician can select to replace one or both of the second fluoroscopic images 510a, 510b if the tip of tools 512a, 512b is not visible or difficult for the clinician to see. Alternatively, method 400 may include segmenting the tip of the medical device in the two second fluoroscopic images, and determining the position of the segmented tip of the medical device in the two second fluoroscopic images based on the segmentation of the tip of the medical device. Segmentation of the tip of the medical device may include segmenting the tip of the medical device using a neural network (e.g., a convolutional neural network).

[0062] Although it can be based on Figure 3 In the 3D reconstruction of the method 300 or in accordance with Figure 4 The method of performing marking of the tip of the medical device in a 2D image, but Figure 3 and Figure 4The marking of the tip of the medical device in the methods 300 and 400 may be performed in a 3D reconstruction and / or in one or more 2D images. Figure 3 The method may additionally or alternatively include marking or receiving a mark in one or more of the 2D images. Also, in another example, Figure 4 The method may additionally or alternatively include marking or receiving a marking of the tip of the medical device in the 3D reconstruction. Additionally or alternatively, the coordinates of the tip of the medical device may be automatically estimated using, for example, a suitable image recognition process (such as a segmentation algorithm).

[0063] At box 406, a second 3D reconstruction of the second fluoroscopic image is generated based on the markings of the tip of the medical device. At box 408, a slice through the tip of the medical device of the second 3D reconstruction is displayed. At box 410, the location of the target in the second 3D reconstruction is determined. This can be done by using Figure 6 The user interface 601 shown in FIG. 4 is used to determine the position of the target in the second 3D reconstruction.

[0064] A user interface may be displayed on a display of computer system 125, wherein the clinician is presented with the second 3D reconstruction and is asked to identify the location of the target in the second fluoroscopic 3D reconstruction. Figure 6 As shown in FIG. 1 , user interface 601 is displayed on a display of computer system 125, wherein a second fluoroscopic 3D reconstruction 602 is presented, and instructions 604 require the clinician to identify a target by scrolling through slices of the second 3D reconstruction and mark the location of the target in the second fluoroscopic 3D reconstruction 602. The clinician can then use input device 1110 ( Fig.11 ) (e.g., a trackball or trackpad) to place a marker 623 at a location on the target. Alternatively, the clinician can use input device 1110 to place ellipse 613 on the target to indicate the approximate edge of the target. The clinician can use input device 1110 to change the size and / or shape of ellipse 613 so that the clinician can ensure that ellipse 613 is very close to the edge of the target.

[0065] Additionally or alternatively, the location of the object may be obtained by registering the first 3D reconstruction with the second 3D reconstruction using a suitable method, such as a mutual information method.Since the user marked the object in the first reconstruction, the system knows the location of the object in the second reconstruction.

[0066] The user interface 601 also includes a scroll button 607 that the clinician can select and move left or right to switch to a different slice of the second 3D reconstruction. The clinician can move the scroll button 607 to search for a slice of the second 3D reconstruction that provides the clinician with the best view of the catheter 611, tool 612, and target so that the clinician can accurately place the marker 623.

[0067] Additionally or alternatively, the computer system 125 may execute an application that automatically segments the object to determine the location of the object. The segmentation may be performed using a suitable method such as a convolutional neural network (CNN).

[0068] The user interface 601 also includes a back button 606 that the clinician can select to return to a previous screen of the user interface 601. For example, the clinician can select the "back" button 606, which returns the user interface 601 to the "capture" screen, which the clinician can use to recapture the second fluoroscopic image if the clinician finds that the existing second fluoroscopic image is of poor quality. The user interface 601 also includes an "accept" button 608 that the clinician can select to confirm the placement of the markers 621 to 623 on the second fluoroscopic 3D reconstruction 602.

[0069] After the clinician selects the accept button 608, at block 412, feedback is presented on the display regarding the position of the tip of the medical device relative to the target. The feedback may include enhancing the markings of the target or the markings of the medical device to show that the medical device is inside or outside the target. For example, Figure 7 , a crosshair symbol 704 indicating the center of the target and an ellipse (e.g., circle 706) indicating the maximum size of the target can be superimposed on the 3D reconstruction 701. Additionally or alternatively, a trajectory 708 of the tip 702 of the medical device can be superimposed on the 3D reconstruction 701. The trajectory 708 can be determined based on the orientation of the medical device.

[0070] The orientation of the medical device can be determined according to a suitable method known to those skilled in the art. For example, a user can draw a line starting from the tip 702 of the medical device and back along the medical device. The user's line is then used to calculate the 3D orientation of the medical device. Alternatively, after determining the position of the tip 702 of the medical device, the orientation of the medical device can be estimated in multiple fluoroscopic 2D images. For example, the orientation of the medical device can be estimated in multiple fluoroscopic 2D images based on a gradient analysis near the tip 702 of the medical device. The 2D orientation of the medical device can then be combined into the 3D orientation of the tip 702 of the medical device. On the one hand, determining the position of the tip 702 of the medical device may include estimating the position of the tip 702. Feedback may include presenting a message indicating that the medical device is located inside or outside the target.

[0071] Figure 8 8 is a flow chart illustrating an example of a method 800 for providing visual feedback to a user via a display. At box 802, the method 800 determines that the tip of a medical device is located outside of a target. The method 800 may determine that the tip of the medical device is located outside of a target based on preoperative target information that is registered to a fluoroscopic 3D reconstruction that includes the tip of the medical device. The preoperative target information may be obtained from a planning phase during which the center coordinates of the target are collected manually or automatically. The target information from the planning phase may be a simple ellipsoid that encapsulates the target in a preoperative CT image and is marked by a user. Alternatively, the method 800 may include segmenting the target in the preoperative CT image to obtain a more accurate 3D representation of the target.

[0072] At block 804, method 800 determines whether the tip of the medical device is aligned with the target. If the tip of the medical device is aligned with the target, then at block 806, a distance between the tip of the medical device and the target is determined. Then, at block 808, a message indicating the alignment of the tip of the medical device with the target and the distance between the tip of the medical device and the target is presented on a display. For example, Fig. 9 , a message box 902 is displayed superimposed on the displayed 3D reconstruction 701, including the text "Tool aligned with target. Tip is 22 mm in front of the center of target". If the tip of the medical device is not aligned with the target, the position of the tip of the medical device relative to the target is determined at box 810. Then, at box 812, a message indicating that the tip of the medical device is not aligned with the target and the position of the tip of the medical device relative to the target is presented on the display.

[0073] In various aspects, method 800 may include: receiving a preoperative computed tomography (CT) image of a target; constructing a 3D model of the target based on the preoperative CT image; and superimposing the 3D model of the target on the 3D reconstruction. The method may include: registering the preoperative CT image with the 3D reconstruction; determining a position of the target in the 3D reconstruction based on the registration to obtain a determined target position; and superimposing the 3D model of the target on the 3D reconstruction based on the determined target position.

[0074] Method 800 may include: constructing a 3D model of a medical device based on a sequence of fluoroscopic images; and superimposing the 3D model of the medical device on the 3D reconstruction. Method 800 may include: receiving a preoperative computed tomography (CT) image of a target; segmenting the target from the preoperative CT image to obtain a segmented target; and superimposing the segmented target on the 3D reconstruction. Method 800 may include: receiving a preoperative computed tomography (CT) image of a lung; receiving at least one marker of the target on the preoperative CT image; and superimposing the at least one marker of the target on the 3D reconstruction. The at least one marker of the target may include a shape of the target or may represent a size of the target. In some aspects, the medical device does not include a position sensor.

[0075] FIG. 10A to FIG. 10H are diagrams illustrating other examples of user interfaces that implement aspects of the methodologies described herein. FIG. 10A to FIG. 10C A user interface for setting up the fluoroscope and capturing fluoroscopic images is shown. Fig. 10D and Fig. 10E Shows something like Figure 5 and Figure 6 Other examples of user interfaces for applying an ellipse 613 or other suitable marking on a target in a slice of the second fluoroscopic 3D reconstruction 602, and for applying markings 522a, 522b to the tips of tools 512a, 512b protruding from the EWC 511a, 511b in both fluoroscopic images 510a, 510b. Fig.10F An example of a user interface is shown that allows the clinician to scroll through slices of a 3D reconstruction to visually identify the tool tip and target. Figure 10G and Fig. 10H Another example of a user interface for enhancing 3D reconstruction with representations of targets (e.g., a target sphere in planning), tools, and tool trajectories is shown. Figure 10G and Fig. 10H As shown, the user interface may include a user control for rotating the augmented 3D reconstruction to visualize the tool and the tool trajectory relative to the target.

[0076] Reference now Fig.11, which is a diagram of a system 1100 configured for use with the methods of the present disclosure. The system 1100 may include a workstation 1101 that may be optionally connected to a fluoroscopic imaging device 110 ( Figure 1 ). In some aspects, the workstation 1101 can be coupled directly or indirectly to the fluoroscope 1115, for example, via wireless communication. The workstation 1101 can include a memory 1102, a processor 1104, a display 1106, and an input device 1110. The processor 1104 can include one or more hardware processors. The workstation 1101 can optionally include an output module 1112 and a network interface 1108. The memory 1102 can store application programs 1118 and image data 1114. The application programs 1118 can include programs executable by the processor 1104 for performing operations including Figure 3 , Figure 4 and Figure 8 Instructions for methods of the present disclosure.

[0077] The application 1118 may further include a user interface 1116. The image data 1114 may include a CT scan, a sequence of fluoroscopic images, a fluoroscopic 3D reconstruction, and / or any other imaging information. The processor 1104 may be coupled to the memory 1102, the display 1106, the input device 1110, the output module 1112, the network interface 1108, and the fluoroscope 1115. The workstation 1101 may be a fixed computer system such as a personal computer or a portable computer system such as a tablet computer. The workstation 1101 may be embedded in multiple computer devices.

[0078] The memory 1102 may include any non-transitory computer-readable storage medium for storing data and / or software including instructions that are executable by the processor 1104 and control the operation of the workstation 1101, and in some aspects, may also control the operation of the fluoroscope 1115. The fluoroscopic imaging device 110 may be used to capture a sequence of fluoroscopic images (generating a fluoroscopic 3D reconstruction based on the sequence of fluoroscopic images), and to capture real-time 2D fluoroscopic views according to the present disclosure. In one aspect, the memory 1102 may include one or more storage devices, such as solid-state storage devices, for example, flash memory chips. Alternatively, or in addition to one or more solid-state storage devices, the memory 1102 may include one or more mass storage devices connected to the processor 1104 via a mass storage controller (not shown) and a communication bus (not shown).

[0079] Although the description of computer-readable media contained herein refers to solid-state storage devices, it should be understood by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor 1104. That is, computer-readable storage media can include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable storage media can include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-ray or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the workstation 1101.

[0080] The application 1118, when executed by the processor 1104, can cause the display 1106 to present a user interface 1116. The user interface 1116 can be configured to present a single screen to the user that includes a three-dimensional (3D) rendering of a tool, lesion, and / or catheter of the present disclosure. The user interface 1116 can be further configured to display the lesion in different colors depending on whether the tool tip is aligned with the lesion in three dimensions.

[0081] The network interface 1108 can be configured to connect to a network, such as a local area network (LAN), a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet, which can be composed of a wired network and / or a wireless network. The network interface 1108 can be used to connect between the workstation 1101 and the fluoroscope 1115. The network interface 1108 can also be used to receive image data 1114. The input device 1110 can be any device that a user can use to interact with the workstation 1101, such as a mouse, a keyboard, a foot pedal, a touch screen, and / or a voice interface. The output module 1112 can include any connection port or bus, such as a parallel port, a serial port, a universal serial bus (USB), or any other similar connection port known to those skilled in the art.

[0082] In light of the above and with reference to the various figures in the accompanying drawings, those skilled in the art will appreciate that certain modifications may also be made to the present disclosure without departing from the scope of the present disclosure. For example, although the systems and methods are described as being usable with an EMN system for navigation through a cavity network (such as the lungs), the systems and methods described herein may be used with systems utilizing other navigation and treatment devices (such as percutaneous devices). Additionally, although the above systems and methods are described as being used in a patient's cavity network, it should be understood that the above systems and methods may be used in other target areas (such as the liver). Further, the above systems and methods may also be used for transthoracic needle aspiration biopsy.

[0083] Detailed aspects of the disclosure are disclosed herein. However, the disclosed aspects are merely examples of the disclosure, which may be embodied in various forms and aspects. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather should be construed merely as a basis for the claims and as a representative basis for teaching those skilled in the art to employ the disclosure in various ways in almost any appropriately detailed structure.

[0084] As can be appreciated, medical devices that can be positioned through one or more branching cavity networks of a patient to treat tissue, such as biopsy tools or energy devices (such as microwave ablation catheters), can prove useful in the field of surgery, and the present disclosure relates to systems and methods that can be used with such instruments, tools and devices. The cavity network can be accessed percutaneously or through natural cavities using navigation techniques. Additionally, navigation through the cavity network can be accomplished using image guidance. These image guidance systems can be separate or integrated with the biopsy tool or energy device or a separate entry tool, and can include MRI, CT, fluoroscopy, ultrasound, electrical impedance tomography, optical systems and / or device tracking systems. Methods for locating entry tools include EM, IR, echolocation, optics, etc. The tracking system can be integrated into an imaging device, wherein tracking is performed in a virtual space or fused with preoperative images or live images.

[0085] In some cases, the treatment target can be accessed directly from the lumen, such as the lining of the bronchus for treatment of COPD, asthma, lung cancer, etc. In other cases, the biopsy tool, energy device and / or additional access tool may need to penetrate the lumen and extend into other tissues to reach the target, such as for treatment of intraparenchymal disease. Final positioning and confirmation of energy device or tool placement can be performed by imaging and / or navigation guidance using standard fluoroscopic imaging devices in combination with the above-described methods and systems.

[0086] It should be understood that the various aspects disclosed herein may be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, according to examples, certain actions or events of any process or method described herein may be performed in a different order, may be added, merged, or omitted entirely (e.g., all described actions or events may not be necessary to implement these techniques). In addition, although for clarity, certain aspects of the present disclosure are described as being performed by a single module or unit, it should be understood that the technology of the present disclosure may be performed by a combination of units or modules associated with, for example, a medical device.

[0087] In one or more examples, the described techniques can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may include a non-transitory computer-readable medium, which corresponds to a tangible medium such as a data storage medium (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer).

[0088] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the above structures or any other physical structures suitable for implementing the described techniques. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0089] Although several embodiments of the disclosure are shown in the accompanying drawings, it is not intended to limit the disclosure thereto, as it is intended that the scope of the disclosure be as wide as the art will allow, and this specification should be read in the same manner. Therefore, the above description should not be construed as limiting, but rather as merely exemplary of specific embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the appended claims.

Claims

1. A system comprising: A computer system comprising a processor and a display configured to display a graphical user interface, and a computer-readable storage medium having stored thereon instructions, the instructions, when executed by the processor, causing the processor to perform the following operations: receiving a sequence of intraoperative X-ray images, each intraoperative X-ray image including at least a portion of a medical device and a target; receiving a marking of the tip of the medical device in at least two images in the sequence of intraoperative X-ray images; constructing a three-dimensional (3D) reconstruction based on the sequence of intraoperative X-ray images and the marking of the tip of the medical device, the 3D reconstruction including the medical device and the target; displaying a slice of the 3D reconstruction through the tip of the medical device; determining a position of the object in the 3D reconstruction; and Feedback is presented regarding the position of the tip of the medical device relative to the target.

2. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to enhance the marking of the target or the marking of the medical device to show that the medical device is located inside or outside the target.

3. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to present a message indicating that the medical device is located inside or outside the target.

4. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to perform the following operations: determining that the tip of the medical device is located outside of the target; In response to determining that the tip of the medical device is outside of the target, determining whether the tip of the medical device is aligned with the target; and A message is presented indicating whether the tip of the medical device is aligned with the target.

5. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to perform the following operations: determining that the tip of the medical device is located outside of the target; In response to determining that the tip of the medical device is located outside of the target, determining a distance between the tip of the medical device and the target or a position of the tip of the medical device relative to the target; and A message is presented indicating the distance of the tip of the medical device from the target or the position of the tip of the medical device relative to the target.

6. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to perform the following operations: receiving a preoperative computed tomography (CT) image of the target; constructing a 3D model of the target based on the preoperative CT image; and A 3D model of the object is superimposed on the 3D reconstruction.

7. The system of claim 6, wherein: The instructions, when executed by the processor, further cause the processor to perform the following operations: registering the preoperative CT image with the 3D reconstruction; Determining a position of the target in the 3D reconstruction based on the registration, thereby obtaining a determined target position; and Based on the determined position of the object, a 3D model of the object is superimposed on the 3D reconstruction.

8. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to perform the following operations: constructing a 3D model of the medical device based on the sequence of intraoperative X-ray images; and A 3D model of the medical device is superimposed on the 3D reconstruction.

9. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to perform the following operations: receiving a preoperative computed tomography (CT) image of the target; Segmenting the target from the preoperative CT image to obtain a segmented target; and The segmented object is superimposed on the 3D reconstruction.

10. The system of claim 1, wherein: The instructions, when executed by the processor, further cause the processor to perform the following operations: Receive preoperative computed tomography (CT) images of the lungs; receiving at least one marking of the target on the preoperative CT image; and At least one marker of the object is superimposed on the 3D reconstruction.

11. The system of claim 10, wherein: At least one marking of the object indicates a size or a shape of the object.

12. The system of claim 1, wherein: The sequence of intraoperative X-ray images is a fluoroscopic image or a cone beam CT image.

13. The system of claim 1, wherein: The medical device does not include a position sensor.

14. The system of claim 1, wherein: The medical device is a biopsy tool.

15. A method comprising: receiving a sequence of intraoperative X-ray images; receiving a marking of a tip of a medical device in at least two images in the sequence of intraoperative x-ray images; generating a three-dimensional (3D) reconstruction based on the sequence of intraoperative X-ray images and the marking of the tip of the medical device, the 3D reconstruction including at least a portion of the medical device and a target; displaying a slice of the 3D reconstruction through the tip of the medical device; determining a position of the target in the 3D reconstruction; as well as Feedback is presented regarding the position of the tip of the medical device relative to the target. 16 . The method of claim 15 , further comprising enhancing a marking of the target or a marking of the medical device to show that the medical device is located inside or outside the target.

17. The method of claim 15, further comprising presenting a message indicating that the medical device is located inside or outside the target.

18. The method of claim 15, further comprising: determining that the tip of the medical device is located outside of the target; In response to determining that the tip of the medical device is located outside of the target, determining that the tip of the medical device is aligned with the target; In response to determining that the tip of the medical device is aligned with the target, determining a distance between the tip of the medical device and the target; as well as A message is presented indicating that the tip of the medical device is aligned with the target and the distance between the tip of the medical device and the target.

19. The method of claim 15, further comprising: determining that the tip of the medical device is located outside of the target; In response to determining that the tip of the medical device is outside of the target, determining that the tip of the medical device is not aligned with the target; In response to determining that the tip of the medical device is not aligned with the target, determining a position of the tip of the medical device relative to the target; as well as A message is presented indicating that the tip of the medical device is not aligned with the target and the position of the tip of the medical device relative to the target.

20. A system comprising: A computer system comprising a processor and a display configured to display a graphical user interface, and a computer-readable storage medium having stored thereon instructions, the instructions, when executed by the processor, causing the processor to perform the following operations: receiving a sequence of intraoperative X-ray images from an X-ray imaging device, each intraoperative X-ray image including at least a portion of the medical device and a target; estimating a posture of the X-ray imaging device based on the sequence of intraoperative X-ray images, thereby obtaining an estimated posture; generating a three-dimensional (3D) volume based on the sequence of intraoperative X-ray images and the estimated pose, the 3D volume including at least a portion of the medical device and the target; displaying a slice of the 3D volume from which a user searches for the tip of the medical device and the target in the slice of the 3D volume; Receive the position information of the scroll control object; and Other slices of the 3D volume corresponding to the position information are displayed.

Images