Computed tomography fusion systems and methods

Abstract

Devices, systems and methods for generating a guidance map to aid in the in situ navigation of invasive medical devices incorporating positioning beacons are described. An example method for determining a representation of a vasculature structure of a patient includes determining a location of at least one beacon, which is removably affixed to a medical device, in the vasculature structure of the patient. The method further includes generating a 2D projection of the vasculature structure based on the location, a previously obtained representation of the vasculature structure, and an output image from a live X-ray device, and determining, using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output image from of the live X-ray device. The 2D projection and the motion or position of the vasculature structure are combined to generate the representation of the vasculature structure.

Description

International Application Attorney Docket No.: 123178.8008.WO00 COMPUTED TOMOGRAPHY FUSION SYSTEMS AND METHODS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 634,747 filed on April 16, 2024. The entire contents of the before-mentioned patent application are incorporated by reference as part of the disclosure of this application. TECHNICAL FIELD

[0002] This document generally relates to magnetic positioning, and more particularly, to enabling computed tomography fusion using magnetometers and beacons. BACKGROUND

[0003] A computed tomography scan (CT scan) is a medical imaging technique used to obtain detailed internal images of the body. CT scanners use a rotating X-ray tube and a row of detectors placed in a gantry to measure X-ray attenuations by different tissues inside the body. The multiple X-ray measurements taken from different angles are then processed on a computer using tomographic reconstruction algorithms to produce tomographic (cross-sectional) images of a body. CT scan can be used in patients with metallic implants or pacemakers. SUMMARY

[0004] Devices, systems, and methods for generating a guidance map to aid in the in situ navigation of invasive medical devices incorporating positioning beacons are described. The described embodiments advantageously allow, among other features and benefits, the physician and medical team to navigate with reduced reliance on imaging with ionizing radiation and injection of contrast agents, and to provide accurate position and orientation information to co- render previously obtained X-ray data with live X-ray images for real-time guidance and visualization.

[0005] In an example aspect, a system for determining a representation of a vasculature structure of a patient includes at least one beacon, an array of magnetic sensors, external to the patient, configured to sense the at least one beacon integrated into an invasive medical device, and at least one processor. In this system, the at least one processor is configured to determine a location of the at least one beacon in the vasculature structure, generate a two-dimensional (2D)International Application Attorney Docket No.: 123178.8008.WO00 projection of the vasculature structure based on the location, a previously obtained representation of the vasculature structure, and an output image from a live X-ray device, determine, using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output image from of the live X-ray device, generate, based on combining the 2D projection and the motion or position of the vasculature structure, the representation of the vasculature structure, and track, based on the representation of the vasculature structure, a location of the invasive medical device in the vasculature structure.

[0006] In another aspect, a method for determining a representation of a vasculature structure of a patient includes determining a location of at least one beacon, which is removably affixed to a medical device, in the vasculature structure of the patient. The method further includes generating a two-dimensional (2D) projection of the vasculature structure based on the location, a previously obtained representation of the vasculature structure, and an output image from a live X-ray device, and determining, using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output image from of the live X-ray device. The method then combines the 2D projection and the motion or position of the vasculature structure to generate the representation of the vasculature structure, and tracks, based on the representation, a location of the medical device in the vasculature structure.

[0007] In yet another example aspect, the above-described method may be implemented by an apparatus or device that includes a processor and / or memory.

[0008] In yet another example aspect, this method may be embodied in the form of processor-executable instructions and stored on a computer-readable program medium.

[0009] The subject matter described in this patent document can be implemented in specific ways that provide one or more of the following features. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 illustrates an example of an arterial vasculature, imaged under X-ray, using a contrast agent.

[0011] FIG. 2 illustrates an example embodiment of a system for incorporating a previously generated representation of the regions of interest of the cardiac vasculature and overlaying it atop a live X-ray image.

[0012] FIG. 3 is a flow diagram describing an example method for determining the relativeInternational Application Attorney Docket No.: 123178.8008.WO00 location and orientation of a previously generated representation of the regions of interest of the cardiac vasculature relative to a second two-dimensional image.

[0013] FIG. 4 illustrates an example of the placement of beacons on an invasive medical device within a patient’s heart.

[0014] FIG. 5A illustrates an example embodiment of a beacon, which includes a permanent magnet, integrated into an invasive medical device.

[0015] FIG. 5B illustrates an example embodiment of a beacon, which includes an electromagnet, integrated into an invasive medical device.

[0016] FIG. 5C illustrates an example embodiment of a beacon, which includes both an electromagnet and a permanent magnet, integrated into an invasive medical device.

[0017] FIG. 6 illustrates an example embodiment of an array of sensors capable of sensing beacons incorporated into an invasive medical device.

[0018] FIG. 7 is a flow diagram for an example method for magnet localization using a single magnet.

[0019] FIG. 8 is a flow diagram for an example refinement process used in magnet localization.

[0020] FIGS. 9A–9C illustrate examples of X-axis, Y-axis, and Z-axis motion, respectively, for a beacon.

[0021] FIGS. 10A and 10B illustrate examples of azimuthal angle motion and elevation angle motion, respectively, for a beacon.

[0022] FIG. 11 is a flow diagram for an example static registration method.

[0023] FIG. 12 illustrates an example of generating a mask of heart vasculature.

[0024] FIG. 13 illustrates an example of Z-axis motion due to breathing.

[0025] FIG. 14 illustrates an example of heart rotational motion direction due to beating.

[0026] FIG. 15 illustrates an example of heart rotational angle due to beating.

[0027] FIG. 16 is a flow diagram for an example method to determine the heart rotational direction and angle parameters.

[0028] FIG. 17 is a flow diagram for an example method to determine the deformation parameters for dynamic registration.

[0029] FIG. 18 illustrates an example of projective view rendering.

[0030] FIG. 19A illustrates an example of a three-dimensional representation of a patientInternational Application Attorney Docket No.: 123178.8008.WO00 heart.

[0031] FIG. 19B illustrates an example of a two-dimensional X-ray view of a patient heart.

[0032] FIG. 19C illustrates an example of the correspondence between key points in a two- dimensional rendering of a three-dimensional representation and the live X-ray image.

[0033] FIG. 20A illustrates an example of dynamic motion and deformation correction of a three-dimensional representation to match the two-dimensional X-ray view of a patient heart.

[0034] FIG. 20B illustrates an example of registration and overlay of a corrected three- dimensional representation over top of a two-dimensional X-ray view of a patient heart .

[0035] FIG. 20C illustrates an example of the correspondence between key points in a two- dimensional rendering of a deformed three-dimensional representation and the live X-ray image.

[0036] FIG. 21 illustrates a flowchart of an example method for determining a representation of a vasculature structure of a patient.

[0037] FIG. 22 is a block diagram illustrating an example system configured to implement embodiments of the disclosed technology. DETAILED DESCRIPTION

[0038] During invasive diagnostic and therapeutic procedures, it is often necessary or desirable to determine the location of the invasive medical device. For example, percutaneous cardiac interventions require a catheter to be navigated through the patient’s vasculature to the coronary artery where treatment is to occur. Traditionally, this is accomplished using contrast enhanced fluoroscopy, a time series of X-ray bursts whose attenuation is enhanced by injecting a material into the patient. However, there are significant limitations and undesired properties of fluoroscopy and contrast agents.

[0039] Section headings are used in the present document to improve readability of the description and do not in any way limit the discussion or the embodiments (and / or implementations) to the respective sections only.

[0040] 1 Introduction

[0041] Cardiovascular disease is the world’s largest disease burden. A major cause is atherosclerosis, a chronic inflammation of the arteries which causes them to harden and accumulate cholesterol plaques on the artery walls, constricting or blocking blood flow. When arteries in the heart become blocked, patients may undergo percutaneous coronary interventionsInternational Application Attorney Docket No.: 123178.8008.WO00 (PCIs) to clear blockages or the buildup of plaque on the arterial walls.

[0042] PCI belongs to a class of diagnostic and treatment procedures where a hollow catheter tube is inserted into the patient’s arteries, typically through the arm or thigh, and is guided by a physician through the vascular network into the heart. During PCI, the objective is to clear the blockage of arteries or insert a stent to widen the arterial cross-section, aiding in blood flow. Diagnostic procedures such as intravenous ultrasound (IVUS) may be performed in a similar fashion, where an ultrasonic probe is inserted on a catheter instead, to image the interior of arteries and assess the degree of blockage, often in preparation for PCI or other coronary intervention.

[0043] Angiographic diagnosis often precedes a PCI or IVUS procedure. This is an X-ray based imaging procedure used to survey the coronary vasculature. It often uses fluoroscopy, a medical imaging method which uses a continuous X-ray or X-ray bursts to capture a stream of real time images of a patient’s internal anatomical structures, giving the physician a video-like sequence of images, capturing the motion of the heart and the catheter. Similar to PCI, a catheter is guided to the coronary arteries, where it is used to inject a contrast agent into the arterial network. As it is often difficult to distinguish the blood vessels from the surrounding heart tissue, these contrast materials increase the attenuation of the vasculature they are injected into as seen in FIG. 1. These contrast agents are typically liquids containing high atomic number materials (e.g., iodine with atomic number 53 and gadolinium with atomic number 64).

[0044] Contrast agents are also often used in PCI to aid the physician in navigating the complex vasculature network of the heart, enabling them to deliver the catheter to the target artery with higher precision. However, care must be taken as using large amounts of contrast agent during a PCI procedure can lead to serious complications such as contrast-induced acute kidney injury (CI-AKI). Therefore, the physician is constantly balancing the desire to use more contrast to aid in navigation, with that to use less contrast to avoid patient injury and complications.

[0045] Another consideration is that X-rays are a form of ionizing radiation. The harmful effects of repeated exposure to and high doses of ionizing radiation are well known. Therefore, reducing the amount of exposure is highly beneficial for both the patient and health practitioners. During PCI, this means that the physician should use the minimum X-ray intensity to achieve acceptable contrast and limit the number of X-ray frames taken. This provides a second tradeoffInternational Application Attorney Docket No.: 123178.8008.WO00 for the physician between ease of navigation and reduced radiation dosage for both the patient and medical team.

[0046] The desire to use less contrast and less X-ray exposure provides a strong motivation to find technologies which both aid in physician navigation through the vasculature network while also minimizing the amount of contrast and radiation exposure necessary to carry out PCIs.

[0047] Another challenge associated with PCI is that motion from the beating of the heart and respiration results in motion of the catheter in the field of view of the fluoroscope. This further complicates the task of tracking the location of the catheter within the vasculature and navigating it to the treatment site.

[0048] To aid navigation and reduce the required contrast and X-ray dosages, roadmapping systems (that aid in navigation), visualize previously obtained X-ray images of the vasculature over live fluoroscope data, correcting for motion through image analysis and analysis of other signals such as electrocardiogram time series data. They can also aid in navigation by capturing maps of the local vasculature in situ, rather than during a separate procedure.

[0049] Some existing systems generate roadmaps by obtaining an initial contrast scan at a given fluoroscope angle, masking it, and then overlaying this on the live X-ray images. They maintain the overlay by tracking the curvature of the catheter and using it to register against the vasculature network. The disadvantage of this approach is that a two-dimensional X-ray image will integrate attenuation over all materials along the line of sight between the source and detector, resulting in additional blood vessels being visible which may not be near the treatment site but are located “behind” the vessel of interest, relative to the source. The reliance on image analysis to determine the curvature of the catheter and infer alignment can also result in ambiguities and artifacts that could be avoided with a direct measurement, such as jumping registration, missing vasculature elements and false tracking of implanted devices.

[0050] Other existing systems overlay a 3D computed tomography (CT) pre-scan atop live fluoroscope images, applying translation, rotation, and scaling to overlay the two sets of X-ray images by means of image analysis. While a generating a projection from a three-dimensional CT image can help deal with some of the ambiguities associated with the two-dimensional imaging case, the system still infers alignment based on an image-to-image overlay metric. Since no direct measurement is used, this is still prone to error.

[0051] Embodiments of the disclosed technology, among other features and benefits,International Application Attorney Docket No.: 123178.8008.WO00 overcome at least the following limitations of current systems:

[0052] – Reliance on indirect measurement of patient movement. Other technologies register pre-scan roadmaps to live X-ray images by inference using only information available from the live images themselves. This is indirect measurement which relies on information inferred from the X-ray images themselves rather than an external information source.

[0053] – Lack of three-dimensional data. Systems that rely solely on X-ray imaging to provide guidance use an inherently two-dimensional process. The X-rays travel through the patient, and contrast is generated by the integrated attenuation along the line of sight between the X-ray source and detector. Purely X-ray based systems can only overcome this by taking images at multiple viewing angles, such as in computed tomography imaging. This results in a loss of dimensionality. Direct measurement using additional sensors in situ can provide full three- dimensional information that could not otherwise be obtained without additional X-ray perspectives.

[0054] – Slow update rates. Since fluoroscopic systems typically obtain images on a 7-15 Hz update rate, the motion bandwidth that can be analyzed is limited to < 7.5 Hz. However, the bandwidth of cardiac motion can extend up to 50 Hz, which means these dynamics cannot be measured using fluoroscopy alone. While the roadmaps themselves only need to be refreshed at the update rate of the fluoroscope, the higher frequency motion dynamics are important for tracking and compensating for heart motion, which is a key purpose of these systems.

[0055] – Lower radiation dosage. Since the capture of fluoroscope images exposes both the patient and medical practitioners to harmful ionizing radiation, there is always a motivation to find technologies which can reduce total fluoroscope time during any procedure. While other roadmap technologies are employed with this same motivation, the method by which they register and update the roadmap positioning relies on fluoroscope imaging to perform these tasks. The disclosed technology introduces direct measurements to the system by means of additional sensors which do not require any X-ray imaging to obtain. This can reduce the total fluoroscope time needed as some system aspects can be updated without even obtaining an X-ray image.

[0056] – Lower contrast usage. The use of contrast agents in catheter guidance may inadvertently lead to excessive usage, which can result in renal injury. Thus, technologies which can provide guidance while requiring less contrast are also highly desirable. The use of directInternational Application Attorney Docket No.: 123178.8008.WO00 measurement provides a contrast-free information channel. This differs from X-ray based guidance systems which require contrast agents to image the vasculature against the surrounding tissue which has a similar material attenuation factor.

[0057] According to some embodiments of the disclosed technology, a cardiac pre-scan such as a CT scan, which is typically performed before the procedure for diagnostic purposes, is used to generate a roadmap which is then aligned and overlaid on a live X-ray image during the procedure. Overlay registration and updates are performed using positioning beacons which are inserted into the patient, incorporating an invasive medical device. In some embodiments, this is a magnetic or electrical source which is sensed and positioned using a plurality of sensors placed external to the patient. In some embodiments, live X-ray images are generated by a fluoroscope. An example embodiment of a system which implements this is shown in FIG. 2. As shown therein, an invasive device with an incorporated beacon is inserted into a patient and monitored by a sensor array and a live X-ray device. A processing unit determines the appropriate overlay of a pre-scan and renders a visualization of the pre-scan combined with the live X-ray image.

[0058] In some embodiments, the system will analyze the pre-scan data, beacon positions over a given period, and live X-ray images. It will then perform an initial registration where the pre-scan is scaled, translated, and rotated to align the pre-scan to the live image, and store these transformations in an internal model. This model will be updated to maintain alignment for future live X-ray frames. Then, a two-dimensional projection is rendered for visualization to aid in guiding the physician and medical team. Beacon data will continue to be captured and processed, which will be used to update the internal model. When additional live X-ray frames are captured, the internal model with updates will be rendered as a 2D overlay and visualized along with the live X-ray frames. An example flow diagram implanting such a process is shown in FIG. 3.

[0059] 2 Roadmap generation

[0060] In some embodiments, the roadmap is generated from a previous CT scan. In CT scanning, a 3D image is generated by imaging the patient in a tube with multiple X-ray images at different angles, usually by rotating the source in a corkscrew. During the procedure, the vasculature is imaged using a contrast agent, similar to a traditional 2D angiogram. However, the data generated by CT scanning is far richer than a set of 2D images, giving true 3D information. Based on the contrast, voxels containing vasculature can be masked and separated from theInternational Application Attorney Docket No.: 123178.8008.WO00 surrounding heart tissue.

[0061] Masked data is stored as a 3D representation. In some examples, the data is stored as a point cloud, with the location of each voxel stored as a set of x, y and z coordinates. In other examples, the data is mapped to a set of cubic B-splines and stored as a set of control point coordinates with associated spline parameters.

[0062] In other embodiments, a 3D roadmap is generated from a series of 2D X-ray images with the vasculature imaged under contrast. Images are taken at various angles about the heart and stored to generate the reconstruction. This is analogous to the method used by a dedicated CT scanner to create its 3D representation. 3D reconstruction can be performed algorithmically using one of several well-established methods such as filtered back projection or iterative reconstruction techniques. The three-dimensional data can then be masked similarly to the CT scan case.

[0063] In yet other embodiments, 2D roadmaps are extracted from live fluoroscopes during the procedure. For example, the fluoroscope can be triggered mechanically or electronically to save an image some specified delay after contrast is injected. This image is then analyzed, masked to extract the contrast containing pixels and then used as the overlay reference for data fusion. Such an image’s utility will be restricted to positions where the imaging source of the fluoroscope is in a similar position to where the image was taken. However, it is still possible to manipulate this image to form an effective overlay for surgical guidance.

[0064] For embodiments, such as the previously discussed 2D roadmaps, where the scan is obtained with the beacon tracking system in place and catheters placed within the heart volume, it is possible to simultaneously record the position of the beacons during roadmap acquisition. The absolute positions of the beacons during the scan can be associated with the beacons observed position in the roadmap image for later registration and data fusion.

[0065] 3 Positioning beacons

[0066] Embodiments of the disclosed technology include one or more position references, referred to as “beacons,” which are used to track the position and movement of points of interest, particularly within the body of the patient. Embodiments of these beacons include, but are not limited to, magnetic sources tracked with a magnetic positioning system (e.g., as described in Section 3), electric sources tracked with an impedance measurement, or physical fiducial markers which are tracked with photons. An example of a fiducial marker is a gold spheroidInternational Application Attorney Docket No.: 123178.8008.WO00 embedded inside the patient near the heart and tracked using the X-ray imaging.

[0067] Examples of magnetic source embodiments include electromagnets, permanent bar magnets, or both. These beacons may be placed on invasive medical devices which may carry other treatment devices and payloads, or on separate catheters which are placed within the heart vasculature. FIG. 4 shows an example of such an arrangement, wherein beacons are placed on an invasive medical device, e.g., a catheter or guidewire.

[0068] In some embodiments, as shown in FIG. 5A, the beacon is a permanent magnet, e.g., rare earth magnets and magnetized sections of catheter wire. In other embodiments, as shown in FIG. 5B, the beacon can be an electromagnet. In embodiments containing electromagnets, power is delivered from outside the body, along wires embedded in the catheter structure to the coil. These electromagnets emit a constant magnetic field or a time-varying, alternating current (AC) field, e.g., a sinusoid oscillating a single frequency, spread spectrum signals such as Gold codes or Zadoff-Chu sequences, and the like.

[0069] In yet other embodiments, both permanent magnets and electromagnets can be incorporated into the invasive medical device. For example, two magnets arranged along and perpendicular to the catheter axis, referred to as the “cross axis” and “long axis” respectively, can be used. In some examples, a permanent magnet with an axis pointing along the long axis with an AC magnet along the cross axis are used. In other examples, the AC magnet points along the long axis with the DC magnet along the cross axis. An example of the latter configuration is illustrated in FIG. 5C.

[0070] Embodiments using magnetic sources are localized as described in later sections, providing location and orientation information for the beacons as well as information regarding the motion of these locations within the heart volume and the heart itself. This enables tracking of the heart’s location and deformation as it beats, as well as during breathing.

[0071] 4 Magnetic positioning

[0072] Embodiments of the disclosed technology can be configured to track magnetic beacons within the body. These are located by means of a plurality of sensors placed external to the patient, as illustrated in FIG. 6.

[0073] In some embodiments, the beacon’s magnetic field approximates that of a magnetic dipole source, with the field obeying the equation:International Application Attorney Docket No.: 123178.8008.WO00 μ3^^̂^^^⃗ ⋅ ^^̂ − ^^^⃗^^^^⃗ = ^4π ^^ .

[0074] Since the magnetic the above equation is solved for theposition of the beacon ^⃗ and its.the vector ^⃗ is stored as a set ofthree coordinates x, y, and z. In other examples, the unit vector ^^ is stored as the azimuthal andelevation angles θ and ϕ respectively.

[0075] FIG. 7 is a flow diagram of an example method for localization of a magnetic beacon. As illustrated therein, the localization procedure begins with initializing the sensor array state, and loading the magnetic model constraints and initial parameter estimates. In an example, the data collection rate is every millisecond. In some embodiments, each datum that is collected may be processed. In other embodiments, the collected data may be downsampled (or upsampled) prior to processing. In the case of embodiments with time-varying magnetic signals, processing may include a demodulation step to obtain DC magnetic field levels.

[0076] As shown in FIG. 7, if the most recently computed convergence metric is less than a predetermined threshold, then the refinement procedure (further detailed in FIG. 8) is performed. If not, then the following series of operations (which constitute the geometric feature extraction process) are performed prior to performing the refinement procedure:

[0077] (a) an initial pole axis search is performed with a constrained orientation range;

[0078] (b) the residual error minimization along the target axis is performed.

[0079] The refinement procedure is followed by estimating the model parameters (e.g., x, y, z, θ, and ϕ) and the convergence metric. The estimated parameters are used to update the predicted sensor measurements in the feedback path illustrated in FIG. 7.

[0080] In some embodiments, the operations in the flow diagrams illustrated in FIG. 7 and FIG. 8 include: 1) Magnetic field measurements are taken at each sensor within the sensor array. 2) Earth and environmental magnetic fields are removed from each set of sensor measurements. In an example, for improved accuracy, each magnetometer can be individually calibrated with 6- or 10-parameter calibration for hard and soft iron impairments. 3) An approximate initial solution, based on the measurement of geometric parameters of the system, is generated.International Application Attorney Docket No.: 123178.8008.WO00 a) In some embodiments, specific features corresponding to system geometry are found within the sensor array data, leading to initial position and orientation estimates. This approach applies to systems with one or more magnetic beacons. b) In some embodiments, a best fit for the one or more magnetic beacons is determined from a look-up table (LUT). c) In some embodiments, (a) the geometric features and (b) the position and orientation estimates determined using the table lookup method are blended. 4) A parameterized model representing estimated magnetic beacon position parameters is generated. In some embodiments, e.g., FIG. 7 shows a model with 5 parameters, representing x, y, and z position of a single cylindrical dipole magnet plus azimuth and elevation. In the case of a magnetic dipole, the rotation about the magnet axis does not change the resulting magnetic field, so this parameter does not need to be modeled. By the convention chosen, for zero theta (yaw rotation about z-axis), the magnetic poles are aligned in the sensor reference frame x-direction, with x position of the positive magnetic pole being less than the x position of the negative magnetic pole. 5) Candidate model parameters can be used to predict the corresponding locations of each magnetic pole and the resulting magnetic field measurements at each sensor in the receive sensor array. 6) A convergence metric, representing the difference between the set of field measurements at each element of the sensor array and the predicted field measurement for each element of the sensor array, is calculated. The convergence metric is a function of the set of differences between the measured values and the predicted values. a) In some embodiments, the convergence metric is a nonlinear function. b) In some embodiments, the initial location estimates of position and orientation are a blend of the geometric feature analysis and the table lookup method, with the relative weights of each method determined by the convergence metric. 7) After initial position and orientation estimates have been made, precise estimates are made using a refinement stage (e.g., illustrated in FIG. 8) which successively updates the best candidate model parameters using the set of partial derivatives of the convergence metric relative to changes in each parameter, such that the system attempts to drive the convergence metric toward zero.International Application Attorney Docket No.: 123178.8008.WO00 a) In some embodiments, the absolute magnetic field strengths of the magnetic beacon or magnets are used in the convergence metric. b) In some embodiments, the measured and predicted magnetic fields are normalized relative to each other such that the total energy in the set of measured field value matches the total energy in the set of predicted field values. 8) In the case where the amount of motion from one snapshot in time to the next is small, the initial geometric estimator can be skipped, and the system can proceed directly to the refinement stage. This is analogous to the continuous tracking mode following initial acquisition for a GPS navigation system. a) In some embodiments, the choice of whether to perform full position acquisition or continue with tracking mode is made by calculating the convergence metric between the last estimated location and the new set of measurements. The tracking mode is used if this metric is below a threshold.

[0081] FIGS. 9A-9C and 10A-10B show an example of the five beacon parameters. In FIGS. 9A-9C, a times-series plot of the x, y, and z positions (in cm), respectively, of three magnetic beacons incorporated into an invasive medical device placed within a porcine heart are shown. The beating of the heart and breathing can be seen. FIG. 10A shows a time-series of the azimuthal angle of the beacon orientation, and FIG. 10B shows a time-series of the elevation angle (in degrees) of the beacon orientation.

[0082] 5 Data augmentation

[0083] While tracking and compensation are primarily driven by the tracking of beacons, additional sensors can be used to augment these operations. For example, motion from respiration or the heart beating can be simultaneously tracked by separate sensors and this data can be fused with the magnetic positioning data for more accurate inference. Additionally, or alternatively, information associated with the movement of the fluoroscopy table, X-ray arm, or other equipment can also be tracked.

[0084] In some embodiments, heartbeat data can be generated from electrical sensing through an electrocardiogram (ECG) system. In other embodiments, and based on the heart operating by circulating a current through the tissue, it is possible to instead track the heart beats magnetically with a sensor that is sufficiently sensitive to detect its signals, which are typically on the order of 10s or 100s of pT. The heart motion may also be inferred from the patient’s pulseInternational Application Attorney Docket No.: 123178.8008.WO00 using devices readily available in a medical setting such as an optical pulse monitor or a cuff monitor placed on the patient’s limb.

[0085] In some embodiments, breathing can be tracked by using a band sensor wrapped around the patient’s chest that is sensitive to expansion and contraction. In other embodiments, breathing can be tracked using an inertial measurement unit (IMU), which includes one or more of an accelerometer, a gyroscope, or a magnetometer. In yet other embodiments, breathing motion may be tracked optically using a camera system or a laser range-finding device, which tracks the movement of the patient’s chest.

[0086] In some embodiments, the movement and positions of the surgical equipment can be tracked using signals directly obtained from the equipment. For example, encoder positions of the C-arm or the timing of the X-ray pulses generated by the fluoroscope may be transmitted to the system as analog or digital signals.

[0087] In some embodiments, surgical equipment is tracked indirectly using auxiliary sensors and signals. For example, the C-arm position can be tracked using devices including, but not limited to, IMUs, additional magnetic sensors and beacons, or optical camera systems.

[0088] 6 Static registration

[0089] In some embodiments, the system first registers the translation, orientation, and scaling parameters to overlay the roadmap onto the live X-ray view. Dynamic adjustments to compensate for the motion of the beacons due to processes such as breathing and the beating of the heart are described in Section 5.

[0090] In some embodiments, the registration is defined by the 6-parameter model including translations x, y, and z, rotations in azimuth and elevation, and a scaling parameter. FIG. 11 shows a flow diagram of an example embodiment of a static registration method to align a three- dimensional model to a view using a two-dimensional X-ray image taken with the vasculature under contrast. As shown therein, FIG. 11 includes the following operations:

[0091] 1. Load the initial parameters for registration including the initial learning rate, a representation of the roadmap model in three-dimensions, including only the relevant vasculature, the system feature positions, including the C-arm position and table position in the world frame, the beacon positions in the world frame, the beacon trajectory history, system constraints for possible view positions and registration parameter bounds, and / or the initial guesses for the translation, rotation, and scaling parameters.International Application Attorney Docket No.: 123178.8008.WO00

[0092] 2. Load or capture a representation of the vasculature under contrast from the live X-ray source and create a mask of the vasculature from the image. In some examples, the representation may be a three-dimensional voxel grid or mesh. In other examples, the representation may be parametrically defined.

[0093] 3. Iteratively perform the following operations:

[0094] 3a. Generate a projective view of the roadmap in the live X-ray image pixel dimensions, given the current perspective parameters and form a mask, as illustrated in the examples shown in FIG. 12.

[0095] 3b. Compute the likelihood of the current projective view perspective configuration. In this example implementation the likelihood is computed using the set of viewparameters ^^^^, the pixels of the model view perspective determined by the view perspective^^^^, ^; ^^^^^the live image pixels ^!^^, ^^, and the beacon trajectory positions ^ "^^#⃗ $%& where thelikelihood has the form'= ()^!^^, ^^*^^^^, ^; ^^^^^+ ⋅ ()^"^^#⃗ $%&*^^^^+

[0096] with respect to each view parameter. If all gradients are below a given threshold and the learning rate is also below its specified threshold, continue to the next step of the algorithm. Otherwise, continue this iteration.

[0097] 3d. If the gradients are all below their given threshold, reduce the learning rate and begin a new iteration. Otherwise continue this iteration.

[0098] 3e. Update each of the view parameters by the gradient multiplied by the learning rate.

[0099] 4. Collect the final view parameters from the iterative method.

[0100] 5. Generate and return the static overlay image and configuration parameters.

[0101] In some embodiments, the time history of the beacon positions is used to form an estimate of the trajectory of the beacons and of the vasculature structure for registration purposes. In the inertial frame of the heart, where motion from breathing, heart beating, and other sources of motion are removed, the motion of the beacons describe its path through the vasculature. The likelihood that the trajectory matches the structure of a region of the vasculature measured in the pre-scan can be computed and the maximum likelihood used to determine which branch the beacon is currently in. In this manner, such embodiments use this information toInternational Application Attorney Docket No.: 123178.8008.WO00 determine the registration parameters in addition to, or in lieu of, imaging the vasculature under contrast.

[0102] 7 Dynamic motion compensation

[0103] In some embodiments, the initial registration is continually modified to maintain alignment between the model and live X-ray images which are obtained later in the procedure. This is to compensate for the fact that the heart is continually moving due to respiration and its own beating. This causes the correct overlay alignment to change over time and must be compensated for dynamically.

[0104] Respiration causes a mostly translational motion of the heart with a small rotational component. This is understandable as when the lungs inflate and deflate, the ribcage expands and contracts, causing the heart to move relative to the world frame.

[0105] On the other hand, the motion of the heart is more complex. In some embodiments, the heart is modeled to rotate about a single axis along with perturbative motion about the moment of rotation. However, some systems can consider a deformational component which corresponds to warpage of the heart’s surface relative to its rest state as it pumps. This deformation occurs due to the chambers of the heart expanding and contracting during the phases of a heartbeat. The rotational and translational components can be viewed as a motion of the entire heart while the deformational component is a local effect occurring for specific areas of the surface.

[0106] In some embodiments, to perform dynamic compensation, input data (that includes, but is not limited to, the current and historical beacon locations) is used to generate a motion of the model and continuously update that model to best align its projection to the current live X- ray image. These updates may include scaling, translations, and rotations to compensate for both respiration and breathing, as well as deformational updates to compensate for heart surface changes during different phases of the beat cycle.

[0107] In some embodiments, the scaling, translational and rotational components are treated separately from the deformation component. For the scaling, translational and rotational transformations, the entire roadmap is transformed as a whole with its internal structure preserved. In the case of the deformational transformation, local sections of the roadmap are treated separately to compensate for local deformations in the surface of the heart.International Application Attorney Docket No.: 123178.8008.WO00

[0108] 7.1 Rotational and translational motion

[0109] In some embodiments, the heart is modeled as a system with a coordinate system whose origin is at the centroid of the beacon locations and basis vectors defined by the axis along which the bulk system rotates, and the two vectors forming plane perpendicular to this axis. The main axis can be found, for example, by principal component analysis (PCA). The beacon positions may be expressed in this coordinate system.

[0110] In some embodiments, the motion due to breathing is modeled as a translation of the centroid and a rotation of the basis vector set. An example of translational motion due to breathing can be seen in FIG. 13. As the lungs expand and contract, the heart moves up and down but is also rotated slightly, as the pressure from the lungs causes the heart to tilt relative to the world frame. As normal human respiratory rates are roughly 12 to 20 breaths per minute, or 0.2-0.33 Hz, this motion is in the sub 1 Hz bandwidth.

[0111] In some embodiments, the bulk motion of the heart while beating is modeled as a rotation of a central vector about a single axis along with additional perturbative adjustments. A diagram depicting the direction of motion is shown in FIG. 14, where the direction of the arrow indicates the plane in which rotation occurs. FIG. 15 shows a time-series of the actual measured rotation angle (in degrees) of this motion, measured in a porcine heart with a magnetic beacon.

[0112] In some embodiments, the motion model for the bulk heart motion can be determined using the flow diagram shown in FIG. 16, which considers an example system with three beacons. As shown therein, the operations include:

[0113] a. Loading the reference parameters.

[0114] b. Retrieving the beacon positions for a predetermined time interval, e.g., 30 sec.

[0115] c. For each time interval, determining the leading basis vector describing the main heart motion direction, e.g., using singular value decomposition (SVD). Using the x, y and z co- ordinates, in the world frame, for each beacon "^, form the matrix "-- ".- "^-"-1 ".1 "^1^= ,"-. ".. "^. / = 0"-2".2"^24"-^ ".^ "^^ "-3 ".3 "^3

[0116] In the first matrix representation, the row and column indices are used. In the second, the indices are replaced with the explicit beacon index and vector component identifiers. With this matrix, determine the decompositionInternational Application Attorney Docket No.: 123178.8008.WO00 ^5 = 6Σ85

[0117] Herein, the main direction of motion is given by the first column vector of 8, which is designated ^-⃗ $%&.

[0118] To remove the ambiguity in the direction of the vectors ^^⃗, determining its signrelative to at the reference time interval ^^⃗$%^&, e.g.,:;<=^^^⃗$%& ∙ ^^⃗$%^&^

[0119] Then, multiplying the vectors ^^⃗$%& by their respective sign values and the matrix 8′ using the new ^^⃗$%&, including the sign changes.

[0120] e. Changing the beacon matrix to the new basis with ^@ = 8@^. In this basis, onecan form a vector describing the heart center position A^⃗ . This is formed from the components1J A^ = hCVE- ℎC = I ^-^

[0121] Herein, this beacon vectors projected ontothe primary direction of motion to form the heart center of motion A^⃗ .

[0122] f. Estimating the rotation of the centered vectors in time, thereby finding the following normal vectors: =^⃗ ^$%& = ^ H^⃗ $t − 1& × ^ H^⃗ $t&

[0123] The normalized vectors nP$%& lie on the unit ball with coordinates ^^, ^, Q^ but onlyhave two degrees of freedom, and they are mapped to a two dimensional subspace using a stereographic projection: ^^^R, S^ = T− ,−

[0124] g. Estimating the|R| and |S| using a kernel density estimator which will, due to normalization, map to the domain $0,1& for both coordinates. Themaximum corresponds to the axis of rotation ^R^W1, S^W1^ which is then mapped to threedimensions using an inverse stereographic transform 2R 2S −1 + R. + S.[

[0125] These

[0126] h. Projecting the centered vectors into the subspace perpendicular to =\]^_International Application Attorney Docket No.: 123178.8008.WO00 A^⃗ ]^_ = `^^#⃗ $%& − ^`^^#⃗ $%& ⋅ =\]^_^=\]^_

[0127] i. Determining^^^^$^^^^&⃗θ$%& = < TH$0^^&⃗ × ^ H^^^$^^%^^&⃗ U sinc- × H % e^^^^^^^ d fi

[0128] The even if the rotationoccurs with a negative

[0129] j. Identifying residual motion not due to rotation. Form a rotation matrix forrotation about =\]^_ by angle θ$%&, jk\^θ$%&^, determine the residual motion^^^^l⃗ $%& = A^⃗ $%& − jk\^m$%&^A^⃗ $0&

[0130] k. Returning the motion parameters, specifically the heart vector A^⃗ and therotation normal =\]^_ and ^^^^l⃗ $%&.

[0131] In some embodiments, the rotation angle is associated with a phase of the heartbeat and regressed with a function containing a periodic component.

[0132] In some embodiments, ECG data is used to determine the R-R peak interval, which is used in estimating the phase of the heartbeat for regression.

[0133] In some embodiments, the motion due to breathing is regressed with a function containing a periodic component.

[0134] In some embodiments, an initial model is used to predict future motion patterns using a predictive method that is updated as data is processed by the algorithm. In some examples, the predictive model is implemented as a Kalman filter. The positions, orientations, velocities, andaccelerations of the beacons are used as input to update the direction of the vector A^⃗ and therotation normal =\]^_. In other embodiments, the motion parameters are estimated over some time interval using a regression model and the regressed function is used as the predictor.

[0135] In the described embodiments, the motion generated by these transformations is applied to the roadmap, thereby translating and rotating its basis frame to match that measured for the current fluoroscope image. The current perspective of the X-ray source is then used to generate a 2D projection of the roadmap view which is overlaid on the live X-ray images.International Application Attorney Docket No.: 123178.8008.WO00

[0136] 7.2 Deformation correction

[0137] Embodiments of the disclosed technology may also correct for the local deformation of the heart’s surface during the heartbeat cycle. The heart’s surface is composed of muscle fibers that stretch and contract as the heart pumps, acting as an elastically deformable surface.

[0138] In some embodiments, the surface can be sampled as a mesh of points. A subset of these are chosen to be control points which are adjusted in position to deform the surface to conform the shape to match that of the heart during a specific phase of the heartbeat. The remaining points are then adjusted such that the energy due to elastic deformation is minimized.

[0139] The set of control points 8n = ^^^^ are moved, deforming the surface such that themesh points are transformed by a matrix op such that ^@p = op^p. Under this transformation thedeformation energy can be expressed as q= I I r so .^p ^^^ − op^^s.

[0140] Here, r^pis a between mesh points ^ and ^ .^ p

[0141] Since the beacons provide accurate measurements of the location of the heart surface for determining the deformed mesh, the control points in close proximity to beacon locations can be deformed such that the beacon position matches the local position in the mesh model. The remaining control points can be adjusted to minimize the energy.

[0142] An example implementation is shown in FIG. 17, which includes the following operations:

[0143] 1. Configure an initial learning rate.

[0144] 2. Load a heart surface model along with a set of control points on the surface.

[0145] 3. For each beacon, find all control points which are closer than a maximum distance from the beacon. Move these control points until the beacon sits on the heart surface.

[0146] 4. Iteratively perform the following procedure:

[0147] 4a. Compute the energy of the configuration according to Hooke’s law.

[0148] 4b. Compute the gradient of the energy with respect to the coordinates of each control point.

[0149] 4c. Calculate the magnitude of the gradient with respect to each coordinate. If all gradients are below a given threshold and the learning rate is also below itsInternational Application Attorney Docket No.: 123178.8008.WO00 specified threshold, continue to the next step of the algorithm. Otherwise, continue this iteration.

[0150] 4d. If the gradients are all below their given threshold, reduce the learning rate and begin a new iteration. Otherwise continue this iteration.

[0151] 4e. Update each of the coordinate positions by the gradient multiplied by the learning rate.

[0152] 5. Fix the positions of the control points. Interpolate the remaining points of the roadmap from the control point positions.

[0153] 8 Data fusion and visualization

[0154] In some embodiments, a three-dimensional roadmap model is overlaid on a two- dimensional X-ray image. In these cases, a two-dimensional view of the model must be generated corresponding to the view that would be seen from the perspective of the X-ray source. This corresponds to a projective geometry problem, an example of which is shown in FIG. 18, where a three-dimension object X is projected on to a plane to create a two-dimensional view X’. This view can be generated using the registered positioning relative to the X-ray source described in Section 7, or using projective geometry techniques.

[0155] In some embodiments, the projective view is rendered on a display device. Some example renderings include an overlay atop the live X-ray images, or a rendering overlaid with the real time beacon positions overlaid atop the projective view, and the like.

[0156] FIGS. 19A-19C show an example of rendering processes. A three-dimensional (3D) representation of the patient’s heart is illustrated in FIG. 19A. The representation is dynamically corrected to match the current view of the heart, including its motion and deformation due to beating and respiration, as illustrated in FIG. 19B. In some embodiments, matching is characterized by the distance between key points in the representation and image spaces and the matching metric is the Euclidean distance between the measured and rendered key points, e.g., t]ku]t^k .Is^ v − ^^v^ ^^ s

[0157] An examplein the undeformed reference rendering and the measured image is illustrated in FIG. 19C. The registration and deformation process produces a rendering that more closely matches the observed live X-ray image.

[0158] FIGS. 20A-20C show an example of a deformed three-dimensional representation. A three-dimensional representation of a patient’s deformed heart is shown in FIG. 20A. TheInternational Application Attorney Docket No.: 123178.8008.WO00 deformed representation must then be rendered as an overlay, and combined with the live X-ray view, as shown in FIG. 20B. In embodiments where the rendering metric includes the distances between key points, a lower metric is measured after deformation. This is illustrated in FIG. 20C where the positions of the key points match the live X-ray more closely than the undeformed model shown in FIG. 19C.

[0159] 9 Examples and implementations of the disclosed technology

[0160] FIG. 21 shows a flowchart of an example method for determining a representation of a vasculature structure of a patient. The method 2100 includes determining (2110) a location of at least one beacon, which is removably affixed to a medical device, in the vasculature structure, and generating (2120) a two-dimensional (2D) projection of the vasculature structure based on the location, a previously obtained representation of the vascular structure, and an output of a live X-ray device. The method further includes determining (2130), using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output of the live X-ray device, and generating (2140), based on combining the 2D projection and the motion or position of the vasculature structure, the representation of the vasculature structure. The method then includes tracking (2150), based on the representation of the vasculature structure, a location of the medical device in the vasculature structure.

[0161] The described features can be implemented to further provide one or more of the following technical solutions:

[0162] S1. A method for determining a representation of a vasculature structure of a patient, comprising: determining a location of at least one beacon in the vasculature structure, wherein the at least one beacon is removably affixed to a medical device; generating a two-dimensional (2D) projection of the vasculature structure based on the location, a previously obtained representation of the vasculature structure, and an output image from a live X-ray device; determining, using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output image from of the live X-ray device; generating, based on combining the 2D projection and the motion or position of the vasculature structure, the representation of the vasculature structure; and tracking, based on the representation of the vasculature structure, a location of the medical device in the vasculature structure.

[0163] S2. The method of solution S1, comprising: displaying, based on the representation, a visualization of the vasculature structure that excludes a relative motion of the patient.International Application Attorney Docket No.: 123178.8008.WO00

[0164] S3. The method of solution S2, wherein the relative motion of the patient comprises a heart beating, breathing, or a movement of the patient.

[0165] S4. The method of any of solutions S1 to S3, wherein the previously obtained representation of the vasculature structure comprises a deformation, and wherein the method comprises: estimating, based on the motion information, the deformation; and updating, prior to generating the representation, the 2D projection to incorporate an estimate of the deformation.

[0166] S5. The method of any of solutions S1 to S4, comprising: constraining, based on image information from the output image or structural information from the previously obtained representation, the representation.

[0167] S6. The method of solution S5, wherein the image information comprises a pixel intensity of the output image or an observed motion of the at least one beacon in a plurality of output images from the live X-ray device.

[0168] S7. The method of solution S5, wherein the structural information comprises at least one of: surface contours of the vasculature structure; a branching or a physical layout of the vasculature structure within a larger anatomical structure; or a deformation dynamic based on a material property or a physical orientation property.

[0169] S8. The method of any of solutions S1 to S7, comprising: tracking a time history of the location and an orientation of the at least one beacon.

[0170] S9. The method of any of solutions S1 to S8, wherein the vasculature structure comprises a heart of the patient, and wherein the method comprises: tracking, based on the representation of the vasculature structure, a motion of the heart using an adaptive tracking loop.

[0171] S10. A system for determining a representation of a vasculature structure of a patient, comprising: at least one beacon; an array of magnetic sensors, external to the patient, configured to sense the at least one beacon integrated into an invasive medical device; and at least one processor configured to: determine a location of the at least one beacon in the vasculature structure, generate a two-dimensional (2D) projection of the vasculature structure based on the location, a previously obtained representation of the vasculature structure, and an output image from a live X-ray device, determine, using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output image from of the live X-ray device, generate, based on combining the 2D projection and the motion or position of the vasculature structure, the representation of the vasculature structure, and track, based on theInternational Application Attorney Docket No.: 123178.8008.WO00 representation of the vasculature structure, a location of the invasive medical device in the vasculature structure.

[0172] S11. The system of solution S10, wherein the at least one beacon comprises a permanent magnet or an electromagnet.

[0173] S12. The system of solution S11, wherein an electromagnetic field associated with the at least one beacon comprises a constant electromagnetic field.

[0174] S13. The system of solution S11, wherein an electromagnetic field associated with the at least one beacon comprises a time-varying electromagnetic field.

[0175] S14. The system of solution S13, comprising: an arbitrary waveform generator configured to output a static or adjustable single-frequency signal that generates the time-varying electromagnetic field.

[0176] S15. The system of solution S13, comprising: an arbitrary waveform generator configured to output a multi-frequency spread-spectrum signal that generates the time-varying electromagnetic field.

[0177] In some embodiments, e.g., S14 and S15, the arbitrary waveform generator (e.g., the AWG70000B Series Arbitrary Waveform Generator from Tektronix) is part of an integrated system that includes the at least one beacon, the array of magnetic sensors, and the at least one processor.

[0178] S16. The system of any of solutions S10 to S15, wherein the at least one beacon comprises a permanent magnet with a first magnetic axis and an electromagnet with a second magnetic axis that is perpendicular to the first magnetic axis.

[0179] S17. The system of solution S16, wherein the first magnetic axis is parallel to an axis of the invasive medical device.

[0180] S18. The system of solution S16, wherein the second magnetic axis is parallel to an axis of the invasive medical device.

[0181] S19. The system of any of solutions S10 to S18, wherein the at least one beacon comprises an electric source or a physical object detectable by photons.

[0182] S20. The system of any of solutions S10 to S19, wherein the invasive medical device comprises an intravascular ultrasound catheter, a plasma ablation catheter, an obstruction removing catheter, or a percutaneous treatment catheter.

[0183] S21. The system of any of solutions S10 to S20, wherein tracking the location of theInternational Application Attorney Docket No.: 123178.8008.WO00 invasive medical device is further based on at least one signal associated with a biological process of the patient.

[0184] S22. The system of solution S21, wherein the biological process comprises a heart beat of the patient, and wherein the at least one signal comprises an electrical signal of an electrocardiogram (ECG), an output signal from magnetically sensing a heart’s current, or a pulse signal generated by an optical sensor or a pressure sensor.

[0185] S23. The system of solution S22, wherein an interval of the heart beat is monitored using the electrical signal of the ECG or a magnetocardiogram.

[0186] S24. The system of solution S21, wherein the biological process comprises breathing by the patient, and wherein the at least one signal comprises: an electrical signal indicative of a motion of a chest of the patient, a measurement signal corresponding to an acceleration, an orientation, or a magnetic field strength, or an optical signal.

[0187] S25. The system of solution S24, comprising: an inertial measurement unit (IMU) configured to generate the measurement signal; or an optical means configured to detect the optical signal, wherein the IMU comprises at least one of a magnetometer, an accelerometer, or a gyroscope, and wherein the breathing is monitored using a band sensitive to mechanical strain from the breathing, the IMU, or the optical means.

[0188] In some embodiments, e.g., S25, the inertial measurement unit (IMU, or IMMU when the magnetometer is incorporated) and / or the optical means are part of an integrated system that includes the at least one beacon, the array of magnetic sensors, and the at least one processor.

[0189] S26. The system of solution S25, wherein the optical means comprises a camera or a laser range-finding device.

[0190] S27. The system of any of solutions S10 to S26, wherein the previously obtained representation of the vasculature structure comprises at least one of a three-dimensional (3D) computed tomography (CT) scan, a collection of 2D X-ray images captured from multiple viewing angles, a single 2D X-ray image captured from a viewing angle of the live X-ray device, or another representation obtained from an imaging modality that excludes X-rays.

[0191] S28. The system of any of solutions S10 to S27, wherein a source or a sensor associated with the live X-ray device is monitored using: an analog signal or a digital signal generated by the source or the sensor, or an external monitoring system that reports on observed motion of the at least one beacon or at least one of the array of magnetic sensors.International Application Attorney Docket No.: 123178.8008.WO00

[0192] S29. A system comprising one or more processors that are configured to implement the method recited in one or more of solutions S1 to S9.

[0193] S30. A method for determining a representation of a vasculature structure of a patient that uses the system recited in one or more of solutions S10 to S28.

[0194] S31. A system for incorporating a previously generated representation of regions of interest of a cardiac vasculature and combining it with a live X-ray image such that features of both images are aligned for use in guiding an invasive medical device using information from positioning beacons placed on a plurality of invasive devices within the body.

[0195] S32. A method for determining a relative location and orientation of a previously generated representation of regions of interest of a cardiac vasculature relative to a two- dimensional image, wherein the determining is based on a location and orientation of positioning beacons incorporated into one or more invasive medical devices, and feature information from within the previously generated representation and the two-dimensional image.

[0196] S33. A method of visually representing a relative location and orientation of a previously generated representation of regions of interest of a cardiac vasculature relative to a two-dimensional image.

[0197] S34. A method for determining a relative location and orientation of a previously generated representation of regions of interest of a cardiac vasculature relative to a two- dimensional image, comprising: making a first determination of a relative location and orientation of a previously generated representation of the regions of interest of the cardiac vasculature relative to a two-dimensional image; making a second determination that the relative location and orientation has changed; and updating, based on the second determination, the relative location and orientation of the regions of interest of the cardiac vasculature.

[0198] S35. A method or system for generating a guidance map to aid in in situ navigation of invasive medical devices incorporating positioning beacons as described in this patent document.

[0199] FIG. 22 shows an example of a hardware platform 2200 that can be used to implement some of the techniques described in the present document. For example, the hardware platform 2200 can implement method 2100, or implement the various modules and algorithms described herein. The hardware platform 2200 includes a processor 2202 that can execute code to implement a method. The hardware platform 2200 includes a memory 2204 that is used to store processor-executable code and / or store data. The hardware platform 2200 further includesInternational Application Attorney Docket No.: 123178.8008.WO00 magnets 2206 and magnetometers 2208, which can communicate with the processor 2202 using leads or a wireless protocol. The processor 2202 is configured to implement localization, fusion, and registration algorithms. In some embodiments, the memory 2204 comprises multiple memories, some of which are exclusively used by the processor 2202 when implementing the fusion, localization, or registration algorithms.

[0200] Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine- readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing unit” or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0201] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by aInternational Application Attorney Docket No.: 123178.8008.WO00 communication network.

[0202] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and devices can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0203] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, these are optional. Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0204] While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0205] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirableInternational Application Attorney Docket No.: 123178.8008.WO00 results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0206] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

International Application Attorney Docket No.: 123178.8008.WO00 WHAT IS CLAIMED IS:

1. A method for determining a representation of a vasculature structure of a patient, comprising: determining a location of at least one beacon in the vasculature structure, wherein the at least one beacon is removably affixed to a medical device; generating a two-dimensional (2D) projection of the vasculature structure based on the location, a previously obtained representation of the vasculature structure, and an output image from a live X-ray device; determining, using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output image from of the live X-ray device; generating, based on combining the 2D projection and the motion or position of the vasculature structure, the representation of the vasculature structure; and tracking, based on the representation of the vasculature structure, a location of the medical device in the vasculature structure.

2. The method of claim 1, comprising: displaying, based on the representation, a visualization of the vasculature structure that excludes a relative motion of the patient.

3. The method of claim 2, wherein the relative motion of the patient comprises a heart beating, breathing, or a movement of the patient.

4. The method of any of claims 1 to 3, wherein the previously obtained representation of the vasculature structure comprises a deformation, and wherein the method comprises: estimating, based on the motion information, the deformation; and updating, prior to generating the representation, the 2D projection to incorporate an estimate of the deformation.

5. The method of any of claims 1 to 4, comprising: constraining, based on image information from the output image or structural information from the previously obtained representation, the representation.International Application Attorney Docket No.: 123178.8008.WO00 6. The method of claim 5, wherein the image information comprises a pixel intensity of the output image or an observed motion of the at least one beacon in a plurality of output images from the live X-ray device.

7. The method of claim 5, wherein the structural information comprises at least one of: surface contours of the vasculature structure; a branching or a physical layout of the vasculature structure within a larger anatomical structure; or a deformation dynamic based on a material property or a physical orientation property.

8. The method of any of claims 1 to 7, comprising: tracking a time history of the location and an orientation of the at least one beacon.

9. The method of any of claims 1 to 8, wherein the vasculature structure comprises a heart of the patient, and wherein the method comprises: tracking, based on the representation of the vasculature structure, a motion of the heart using an adaptive tracking loop.

10. A system for determining a representation of a vasculature structure of a patient, comprising: at least one beacon; an array of magnetic sensors, external to the patient, configured to sense the at least one beacon integrated into an invasive medical device; and at least one processor configured to: determine a location of the at least one beacon in the vasculature structure, generate a two-dimensional (2D) projection of the vasculature structure based on the location, a previously obtained representation of the vasculature structure, and an output image from a live X-ray device, determine, using motion information from the at least one beacon, a motion or position of the vasculature structure relative to the output image from of the live X-ray device, generate, based on combining the 2D projection and the motion or position of the vasculature structure, the representation of the vasculature structure, andInternational Application Attorney Docket No.: 123178.8008.WO00 track, based on the representation of the vasculature structure, a location of the invasive medical device in the vasculature structure.

11. The system of claim 10, wherein the at least one beacon comprises a permanent magnet or an electromagnet.

12. The system of claim 11, wherein an electromagnetic field associated with the at least one beacon comprises a constant electromagnetic field.

13. The system of claim 11, wherein an electromagnetic field associated with the at least one beacon comprises a time-varying electromagnetic field.

14. The system of claim 13, comprising: an arbitrary waveform generator configured to output a static or adjustable single- frequency signal that generates the time-varying electromagnetic field.

15. The system of claim 13, comprising: an arbitrary waveform generator configured to output a multi-frequency spread-spectrum signal that generates the time-varying electromagnetic field.

16. The system of any of claims 10 to 15, wherein the at least one beacon comprises a permanent magnet with a first magnetic axis and an electromagnet with a second magnetic axis that is perpendicular to the first magnetic axis.

17. The system of claim 16, wherein the first magnetic axis is parallel to an axis of the invasive medical device.

18. The system of claim 16, wherein the second magnetic axis is parallel to an axis of the invasive medical device.

19. The system of any of claims 10 to 18, wherein the at least one beacon comprises an electric source or a physical object detectable by photons.International Application Attorney Docket No.: 123178.8008.WO00 20. The system of any of claims 10 to 19, wherein the invasive medical device comprises an intravascular ultrasound catheter, a plasma ablation catheter, an obstruction removing catheter, or a percutaneous treatment catheter.

21. The system of any of claims 10 to 20, wherein tracking the location of the invasive medical device is further based on at least one signal associated with a biological process of the patient.

22. The system of claim 21, wherein the biological process comprises a heart beat of the patient, and wherein the at least one signal comprises an electrical signal of an electrocardiogram (ECG), an output signal from magnetically sensing a heart’s current, or a pulse signal generated by an optical sensor or a pressure sensor.

23. The system of claim 22, wherein an interval of the heart beat is monitored using the electrical signal of the ECG or a magnetocardiogram.

24. The system of claim 21, wherein the biological process comprises breathing by the patient, and wherein the at least one signal comprises: an electrical signal indicative of a motion of a chest of the patient, a measurement signal corresponding to an acceleration, an orientation, or a magnetic field strength, or an optical signal.

25. The system of claim 24, comprising: an inertial measurement unit (IMU) configured to generate the measurement signal; or an optical means configured to detect the optical signal, wherein the IMU comprises at least one of a magnetometer, an accelerometer, or a gyroscope, and wherein the breathing is monitored using a band sensitive to mechanical strain from the breathing, the IMU, or the optical means.

26. The system of claim 25, wherein the optical means comprises a camera or a laser range- finding device.International Application Attorney Docket No.: 123178.8008.WO00 27. The system of any of claims 10 to 26, wherein the previously obtained representation of the vasculature structure comprises at least one of a three-dimensional (3D) computed tomography (CT) scan, a collection of 2D X-ray images captured from multiple viewing angles, a single 2D X-ray image captured from a viewing angle of the live X-ray device, or another representation obtained from an imaging modality that excludes X-rays.

28. The system of any of claims 10 to 27, wherein a source or a sensor associated with the live X-ray device is monitored using: an analog signal or a digital signal generated by the source or the sensor, or an external monitoring system that reports on observed motion of the at least one beacon or at least one of the array of magnetic sensors.

29. A system comprising one or more processors that are configured to implement the method recited in one or more of claims 1 to 9.

30. A method for determining a representation of a vasculature structure of a patient that uses the system recited in one or more of claims 10 to 28.

Images