Magnetometry-based three-dimensional reconstruction of objects relative to an invasive medical device

Abstract

Systems and methods for generating a three-dimensional (3D) reconstruction to aid in the in situ navigation of invasive medical devices incorporating positioning beacons are described. An example method includes receiving multiple images corresponding to an object of interest being imaged from multiple perspectives by an imaging modality, and determining, using an array of magnetic sensors, a location and orientation of a positioning beacon affixed to an invasive medical device inside the body of the patient. The method then includes performing, based on magnetometry information and without using positions of the imaging modality emitter or the body of the patient as reference points, a calibration process for the imaging modality at each of the multiple perspectives. Based on the multiple images and an output of the calibration process, the method estimates the 3D representation of the object of interest from at least one of the multiple perspectives.

Description

MAGNETOMETRY-BASED THREE-DIMENSIONAL RECONSTRUCTION OF OBJECTS RELATIVE TO AN INVASIVE MEDICAL DEVICECROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 656,352, filed on lune 5, 2024. The entire contents of the before-mentioned patent application are incorporated by reference as pail of the disclosure of this application.TECHNICAL FIELD

[0002] This document generally relates to magnetic positioning, and more particularly, to enabling three-dimensional reconstruction of objects using magnetometers and beacons.BACKGROUND

[0003] Three-dimensional (3D) reconstruction from multiple images is the creation of 3D models (of a 3D object) from a set of two-dimensional (2D) images of the 3D object. The task of converting multiple 2D images into a 3D model consists of a series of processing steps that include (1) camera calibration (that consists of determining intrinsic and extrinsic parameters), (2) depth determination (that generates a depth map that contains information relating to the distance of the surfaces of scene objects from a viewpoint), and (3) registration (that combines multiple depth maps to create a final mesh representative of the 3D object.SUMMARY

[0004] Devices, systems, and methods for magnetometry-based 3D reconstruction of objects relative to an invasive medical device that include positioning beacons are described. The disclosed embodiments advantageously allow, among other features and benefits, the physician and medical team to navigate the invasive medical device while reducing the amount of time objects of interest must be imaged under resolvable contrast.

[0005] In an example aspect, a system for navigating an invasive medical device within a body of a patient includes at least one beacon, an array of magnetic sensors (external to the patient) configured to sense the at least one beacon affixed to the invasive medical device, and at least one processor. The at least one processor is configured to receive multiple images corresponding to one or more objects of interest, within the body of the patient, being imagedfrom multiple perspectives by an imaging modality. The at least one processor is further configured to determine, using the array of magnetic sensors, a location and an orientation of the at least one beacon, and perform, based on magnetometry information, a calibration process for the imaging modality at each of the multiple perspectives. When performing the calibration process, the at least one processor excludes using a position or an orientation of an emitter of the imaging modality or a position of the body of the patient as a reference point, and uses the magnetometry information that includes (a) the location and the orientation of the at least one beacon and (b) two-dimensional position information for the at least one beacon determined based on the multiple images that were captured from multiple perspectives. Herein, each perspective is associated with the reference point corresponding to a respective emitter. The at least one processor is then configured to estimate, based on the multiple images and an output of the calibration process, a three-dimensional (3D) representation of the one or more objects of interest from at least one of the plurality of perspectives, and navigate, based on the 3D representation of the one or more objects of interest, the invasive medical device within the body of the patient.

[0006] In another aspect, a method of estimating a three-dimensional (3D) representation of one or more objects of interest within a body of a patient includes receiving multiple images corresponding to the one or more objects of interest being imaged from multiple perspectives by an imaging modality; determining, using an array of magnetic sensors, a location and an orientation of at least one beacon affixed to an invasive medical device inside the body of the patient; and performing, based on magnetometry information, a calibration process for the imaging modality at each of the multiple perspectives. In this method, the magnetometry information includes (a) the location and the orientation of the at least one beacon and (b) two- dimensional position information for the at least one beacon determined based on the multiple images, but excludes using a position or an orientation of an emitter of the imaging modality or a position of the body of the patient as a reference point. The method further includes estimating, based on the multiple images and an output of the calibration process, the 3D representation of the one or more objects of interest from at least one of the multiple perspectives. Herein, each perspective is associated with the reference point corresponding to a respective emitter.

[0007] In yet another example aspect, the above-described method may be implemented by an apparatus or device that includes a processor and / or memory.

[0008] In yet another example aspect, this method may be embodied in the form of processor-executable instructions and stored on a computer-readable program medium.

[0009] The subject matter described in this patent document can be implemented in specific ways that provide one or more of the following features.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 illustrates an example of a cardiac vasculature, imaged under X-ray, using a contrast agent.

[0011] FIG. 2 is a flow diagram describing an example method of estimating a three- dimensional (3D) representation of one or more objects of interest within a body of a patient.

[0012] FIG. 3 illustrates an example embodiment for determining the three-dimensional reconstruction of objects of interest from a series of camera poses and tracking the object’s motion in a two-dimensional field of view.

[0013] FIG. 4 illustrates an example of the placement of beacons on an invasive medical device within a patient’s heart.

[0014] FIG. 5A illustrates an example embodiment of a beacon, which includes a permanent magnet, integrated into an invasive medical device.

[0015] FIG. 5B illustrates an example embodiment of a beacon, which includes an electromagnet, integrated into an invasive medical device.

[0016] FIG. 5C illustrates an example embodiment of a beacon, which includes both an electromagnet and a permanent magnet, integrated into an invasive medical device.

[0017] FIG. 6 illustrates an example embodiment of an array of sensors capable of sensing beacons incorporated into an invasive medical device.

[0018] FIG. 7 is a flow diagram for an example method for single-magnet localization.

[0019] FIG. 8 is a flow diagram for an example refinement process in magnet localization.

[0020] FIGS. 9A-9C illustrate examples of X-axis, Y-axis, and Z-axis motion, respectively, for a beacon.

[0021] FIGS. 10A and 10B illustrate examples of azimuthal angle motion and elevation angle motion, respectively, for a beacon.

[0022] FIG. 11 illustrates an example of a heartbeat signal with peaks of interest and phases of the quasi-periodic function labeled.

[0023] FIGS. 12A and 12B illustrate an example of a pinhole camera imaging geometry and the projection of a point in three dimensions onto the two-dimensional XY-plane and the two- dimensional YZ-plane, respectively.

[0024] FIGS. 13A-13C illustrate an example of three beacons imaged (a) under X-ray, with (b) their midpoints identified as key points, and (c) their two endpoints identified as key points, respectively.

[0025] FIG. 14 is a flow diagram for an example method of a random sample consensus pruning of a calibration correspondence set (or calibration correspondence point set).

[0026] FIGS. 15 A and 15B illustrate an example of cardiac vasculature imaged under X-ray (a) before an object of interest is to be selected, and (b) after selection and segmentation.

[0027] FIG. 16 illustrates an example of determining the maximum likelihood estimate reconstruction of a three-dimensional point from a set of two camera views.

[0028] FIG. 17 illustrates an example of the motion of the heart due to beating, imaged at two points in the beat cycle (with correspondences between key points highlighted).

[0029] FIGS. 18A-18C illustrate an example of a camera imaging an object, (b) with the object rotated, and (c) with an unrotated object and the camera being rotated instead.

[0030] FIG. 19 illustrates an example of cardiac X-ray images obtained, under contrast, at different phases of a heartbeat cycle.

[0031] FIG. 20 illustrates an example of a cardiac X-ray image with a two-dimensional rendering of an object of interest overlaid.

[0032] FIG. 21 illustrates a flowchart of an example method of estimating a three- dimensional (3D) representation of one or more objects of interest within a body of a patient.

[0033] FIG. 22 is a block diagram illustrating an example system configured to implement embodiments of the disclosed technology.DETAILED DESCRIPTION

[0034] During invasive medical diagnostic and therapeutic procedures, it is often necessary or desirable to determine the locations of objects of interest relative to an invasive medical device. For example, percutaneous procedures require a device such as a catheter to be navigated through the patient’s body to a specific location. Traditionally, this is accomplished using a two- dimensional imaging modality, possibly using a contrast agent. However, there are significantlimitations to navigation with these traditional methods.

[0035] Embodiments of the disclosed technology relate to methods, systems, and devices for estimating a three-dimensional reconstruction of one or more objects of interest within a patient’s body and determining its motion dynamically, using the position and orientation of positioning beacons incorporated into an invasive medical device. The methods and devices described in the present document advantageously, among other features and benefits, allow a medical team to navigate the invasive medical device while reducing the amount of time objects of interest must be imaged under resolvable contrast, and to provide accurate position and orientation information to co-render a three-dimensional reconstruction of one or more objects of interest, along with rendering its motion, with live images for real-time guidance and visualization.

[0036] Section headings are used in the present document to improve readability of the description and do not in any way limit the discussion or the embodiments (and / or implementations) to the respective sections only.

[0037] 1 Introduction

[0038] Percutaneous procedures, where the physician accesses a patient’s inner organs via a needle-puncture in the skin, are more favorable with the medical community compared to open procedures where organs are exposed. Often, an invasive medical device such as a catheter is inserted into a patient’s blood vessel or other lumen and guided to the treatment site. Common examples of percutaneous procedures include, but are not limited to, coronary procedures including angioplasty and stenting.

[0039] The benefit of the percutaneous approach is the ease of introducing devices into the patient without large incisions, which can be painful and may introduce complications due to bleeding and infection. Since percutaneous access only requires a small hole through the skin, the insertion site seals easily and heals quickly compared to other types of procedures.

[0040] To aid in navigation to the treatment site, fluoroscopy and other live imaging modalities are often used to guide the invasive medical device through the luminary structure. However, live imaging modalities are typically two-dimensional projective methods. The three- dimensional structure, which may be important for navigational purposes, is lost and the imaging technique must be augmented to recover this information. For example, computed tomography (CT) scanning is an imaging technique which permits three-dimensional reconstructions to begenerated using a set of two-dimensional X-ray images. However, live CT scanning is computationally difficult and its usage during a procedure is usually prohibitive due to the cost and restrictions on physician access to the patient.

[0041] Another difficulty in imaging during percutaneous procedures is that modalities such as fluoroscopy rely on ionizing radiation to produce the image. The harmful effects of repeated exposure to ionizing radiation are well known. Therefore, for the health of the patient and medical team, radiation dosages are heavily monitored and minimized to all extents possible. On the other hand, higher intensity, and therefore higher dosage, radiation sources produce better contrast images which make it easier for the physician to navigate the invasive medical device. Thus, there exists a tradeoff between reduced radiation exposure and ease of navigation.

[0042] When using X-ray based imaging modalities, contrast agents are often used to enhance the visibility of objects of interest during procedures. These compounds contain radioopaque material, highlights objects against the surrounding tissue, as shown in FIG. 1. However, care must be taken when using contrast agents as high dosages can lead to serious complications such as contrast-induced acute kidney injury. Therefore, the physician must balance the need to use more contrast agent to aid in navigation and to use less to avoid patient injury.

[0043] To aid in navigation during percutaneous procedures, systems have been developed to create three-dimensional models of objects of interest which can be isolated from the surrounding tissue or overlaid on live imaging data.

[0044] In an existing implementation, a system generates a three-dimensional representation of objects of interest using a biplane fluoroscope method where two simultaneous images are obtained. When the fluoroscope views are correctly calibrated, it is possible to generate a three- dimensional reconstruction of objects in the field of view through a projective geometry based method. However, this method relies on at least two simultaneous two-dimensional captures to perform the reconstruction.

[0045] Other implementations obtain images using a single fluoroscope view swept through a series of perspectives. Each perspective is calibrated using a set of positional markers with known relative distances in three-dimensional space. However, these methods require external calibration markers to be within the field of view and for these markers to be in a static position for every scan (and for each perspective of the series of perspectives).

[0046] In yet another existing implementation, the position and trajectory of a catheter aretracked using the combined information from a sweep of fluoroscope images fused together with three-dimensional data describing the position and orientation of the distal tip of the catheter. The catheter’s position is tracked using a sensor placed on the distal end of the catheter which senses an external electromagnetic field. However, the location of the catheter itself is tracked rather than other objects of interest within the images.

[0047] Embodiments of the disclosed technology, among other features and benefits, overcome at least the following limitations of current methods and systems:

[0048] - Simultaneous imaging of objects of interest to form the model representation requires additional, often costly, imaging equipment which also interferes with access to the patient. This is the case with biplane fluoroscopy which requires two expensive X-ray sources.

[0049] - External calibration markers and sources often interfere with patient access

[0050] - This is the case with external field generators, which are large, expensive, and typically surround the patient, and that require a dedicated surgical suite. They also must remain static, as moving them during the procedure will invalidate their calibration, further restricting patient access.

[0051] - External calibration markers must remain static during all imaging, which can interfere with physician access. These markers will also be visible in the image which can obscure the physician’s view of important features.

[0052] - A small form factor for the invasive medical device-mounted components is critical. The larger the device, the more limited are the physician’s options regarding where to position the device within the body. This is a significant problem for sensors mounted on the invasive medical device where the footprint can be much larger than a source measured by external sensors.

[0053] - Obtaining measurements of position data directly at the treatment site is not always possible. This is problematic for systems that track the distal tip of a catheter as for some existing implementations, space near’ an object of interest may be limited either because the size of the sensing or source element is too large or because the area is already occupied by another device required for the procedure. Using measurements from other locations to infer motion at the position of interest allows the physician to perform the specific procedure in a standard manner without concern for obstructions or hinderances caused by elements used for positioning.

[0054] - Minimizing ionizing radiation and contrast dosages is critical for both the healthof the patient and the medical team.

[0055] - Biplane fluoroscopy requires the use of two X-ray sources which, under the same imaging conditions, requires twice the radiation dosage for the same procedure compared to a single-view fluoroscopy.

[0056] - Forming the reconstruction using a fluoroscope sweep images at many vantage points (or many perspectives) can result in a larger integrated dose compared to imaging at only a few discrete vantage points.

[0057] Embodiments of the disclosed technology provide, amongst other features and benefits, the following:

[0058] - A small footprint on-board the invasive medical device;

[0059] - Allowing the physician and medical team unrestricted access to the patient without physical blockages due to additional imaging or localization systems;

[0060] - Complete three-dimensional position and orientation information of points within the body;

[0061] - Reducing the inherently harmful dose of ionizing radiation received by the patient and medical team during medical procedures;

[0062] - Three-dimensional reconstruction of objects of interest from two-dimensional views at only a few, possibly only two, vantage points;

[0063] - Self-calibration of the imaging views using the known three-dimensional positions of the source objects corresponding to their presentation in the imaging view;

[0064] - Motion tracking of objects of interest without requiring positional information directly from the site of interest; and

[0065] - Motion tracking of objects of interest without requiring constant imaging of the object with resolvable contrast.

[0066] According to some embodiments of the disclosed technology, images of one or more objects of interest, such as a blood vessel, are first obtained at multiple vantage points using a single image source and detector before the start of the medical procedure. The imaging planes are self-calibrated, using the positions of the sources within the image and their corresponding three-dimensional positions and orientations at the time of imaging, as determined by the system to perform the calibration. Objects of interest are segmented from the surrounding image and a three-dimensional reconstruction of the objects is generated from multiple calibrated images by aprocessing unit. When the procedure is started and the imaging device is positioned according to the needs of the medical team, a rendering of the one or more objects of interest in the imaging field of view is generated and overlaid on the live imaging view. Motion of the one or more objects of interest is estimated from time-series position and orientation data of the beacons and used to update the reconstructed three-dimensional model to match new position, orientation and deformation of objects of interest. The representation of the one or more object models in the field of view is updated according to the estimated motion. An example flow diagram of such a process is shown in FIG. 2.

[0067] FIG. 3 shows an example system that can implement the disclosed methods. As shown therein, an invasive medical device containing a source beacon is inserted into the patient percutaneously. The position and orientation of the beacon is monitored by a sensor array external to the patient’s body. A live X-ray device simultaneously images the patient and is positioned at multiple vantage points, and obtains one or more images of the one or more objects of interest at each vantage point. A processing unit computes the image calibrations and reconstruction model, determines dynamic updates to the model according to the observed motion, and renders an appropriate overlay of the model combined with the live X-ray image.

[0068] 2 Examples of beacon positioning and tracking

[0069] Embodiments of the disclosed technology include one or more position references, referred to as “beacons,” which are used to track the position and movement of points of interest, particularly within the body of the patient. Embodiments of these beacons include, but are not limited to, magnetic sources tracked with a magnetic positioning system, or electric sources tracked with impedance measurements.

[0070] Examples of magnetic source embodiments include electromagnets, permanent bar magnets, or both. These beacons may be placed on invasive medical devices which may carry other treatment devices and payloads, or on separate devices that are placed within the patient’s body. FIG. 4 shows an example of such an arrangement, wherein two beacons have been integrated into the distal end of two invasive guidewires and place in coronary arteries.

[0071] In some embodiments, as shown in FIG. 5A, the beacon is a permanent magnet, e.g., rare earth magnets and magnetized sections of catheter wire. In other embodiments, as shown in FIG. 5B, the beacon can be an electromagnet. In embodiments containing electromagnets, power is delivered from outside the body, along wires embedded in the catheter structure, to the coil.These electromagnets emit a constant magnetic field or a time-varying, alternating current (AC) field, e.g., a sinusoid oscillating a single frequency, spread spectrum signals such as Gold codes or Zadoff-Chu sequences, and the like.

[0072] In yet other embodiments, both permanent magnets and electromagnets can be incorporated into the invasive medical device. For example, two magnets arranged along and perpendicular to the catheter axis, referred to as the “cross axis” and “long axis” respectively, can be used. In some examples, a permanent magnet with an axis pointing along the long axis with an AC magnet along the cross axis are used. In other examples, the AC magnet points along the long axis with the DC magnet along the cross axis. An example of the latter configuration is illustrated in FIG. 5C.

[0073] Embodiments using magnetic sources are localized as described below, providing location and orientation information for the beacons as well as information regarding the motion of the surrounding tissue. This enables tracking of the surrounding tissue location even as it moves within the imaging field of view. Examples of this type of movement during treatments include, but are not limited to, beating of the patient’ s heart during a cardiac procedure, motion from respiration during cardiac and pulmonary procedures, and movement of the patient’s limb when treating vasculature in the patient’s peripherals such as their leg.

[0074] Embodiments of the disclosed technology can be configured to track magnetic beacons within the body. These are located by means of a plurality of sensors placed external to the patient, as illustrated in FIG. 6.

[0075] In some embodiments, the beacon’s magnetic field approximates that of a magnetic dipole source, with the field obeying the equation:

[0076] Since the magnetic dipole strength |m| is known, the above equation is solved for the position of the beacon r and its orientation m. In some examples, the vector r is stored as a set of three coordinates x, y, and z. In other examples, the unit vector m is stored as the azimuthal and elevation angles 0 and 4> respectively.

[0077] FIG. 7 is a flow diagram of an example method for localization of a magnetic beacon. As illustrated therein, the localization procedure begins with initializing the sensor array state, and loading the magnetic model constraints and initial parameter estimates. In an example, thedata collection rate is every millisecond. In some embodiments, each datum that is collected may be processed. In other embodiments, the collected data may be downsampled (or upsampled) prior to processing. In the case of embodiments with time-varying magnetic signals, processing may include a demodulation step to obtain DC magnetic field levels.

[0078] As shown in FIG. 7, if the most recently computed convergence metric is less than a predetermined threshold, then a refinement procedure (further detailed in FIG. 8) is performed. If not, then the following series of operations (which constitute the geometric feature extraction process) are performed prior to performing the refinement procedure:

[0079] (a) an initial pole axis search is performed with a constrained orientation range;

[0080] (b) the residual error minimization along the target axis is performed.

[0081] The refinement procedure is followed by estimating the model parameters (e.g., x, y, z, 0, and 4>) and the convergence metric. The estimated parameters are used to update the predicted sensor measurements in the feedback path illustrated in FIG. 7.

[0082] In some embodiments, the operations in the flow diagrams illustrated in FIG. 7 and FIG. 8 include:1) Magnetic field measurements are taken at each sensor within the sensor array.2) Earth and environmental magnetic fields are removed from each set of sensor measurements.In an example, for improved accuracy, each magnetometer can be individually calibrated with 6- or 10-parameter calibration for hard and soft iron impairments.3) An approximate initial solution, based on the measurement of geometric parameters of the system, is generated. a) In some embodiments, specific features corresponding to system geometry are found within the sensor array data, leading to initial position and orientation estimates. This approach applies to systems with one or more magnetic beacons. b) In some embodiments, a best fit for the one or more magnetic beacons is determined from a look-up table (LUT). c) In some embodiments, (a) the geometric features and (b) the position and orientation estimates determined using the table lookup method are blended.4) A parameterized model representing estimated magnetic beacon position parameters is generated. In some embodiments, a model is configured with 5 parameters, representing x, y, and z position of a single cylindrical dipole magnet plus azimuth and elevation (e.g., asillustrated in FIG. 7). In the case of a magnetic dipole, the rotation about the magnet axis does not change the resulting magnetic field, so this parameter does not need to be modeled. By the convention chosen, for zero theta (yaw rotation about z-axis), the magnetic poles are aligned in the sensor reference frame x-direction, with x position of the positive magnetic pole being less than the x position of the negative magnetic pole. ) Candidate model parameters can be used to predict the corresponding locations of each magnetic pole and the resulting magnetic field measurements at each sensor in the receive sensor array. ) A convergence metric, representing the difference between the set of field measurements at each element of the sensor array and the predicted field measurement for each element of the sensor array, is calculated. The convergence metric is a function of the set of differences between the measured values and the predicted values. a) In some embodiments, the convergence metric is a nonlinear function. b) In some embodiments, the initial location estimates of position and orientation are a blend of the geometric feature analysis and the table lookup method, with the relative weights of each method determined by the convergence metric. ) After initial position and orientation estimates have been made, precise estimates are made using the refinement stage (e.g., illustrated in FIG. 8) which successively updates the best candidate model parameters using the set of partial derivatives of the convergence metric relative to changes in each parameter, such that the system attempts to drive the convergence metric to war’d zero. a) In some embodiments, the absolute magnetic field strengths of the magnetic beacon or magnets are used in the convergence metric. b) In some embodiments, the measured and predicted magnetic fields are normalized relative to each other such that the total energy in the set of measured field value matches the total energy in the set of predicted field values. ) In the case where the amount of motion from one snapshot in time to the next is small, the initial geometric estimator can be skipped, and the system can proceed directly to the refinement stage. This is analogous to the continuous tracking mode following initial acquisition for a GPS navigation system.a) In some embodiments, the choice of whether to perform full position acquisition or continue with tracking mode is made by calculating the convergence metric between the last estimated location and the new set of measurements. The tracking mode is used if this metric is below a threshold.

[0083] FIGS. 9A-9C and 10A-10B show an example of the five beacon parameters. In FIGS. 9A-9C, the x, y, and z positions, respectively, of a magnetic beacon incorporated into an invasive medical device placed within a porcine heart are shown. The beating of the heart and breathing can be seen. FIG. 10A shows the azimuthal angle of the beacon orientation, and FIG. 10B shows the corresponding elevation angle.

[0084] 3 Examples of data augmentation

[0085] In some embodiments, additional input signals are used to further constrain the reconstruction and motion update estimates. Examples of such signals include, but are not limited to, measurements of the patient’s bodily movements, e.g., heartbeats and breathing, and the motion of medical imaging equipment.

[0086] In some embodiments, where a cardiac lumen is an object of interest, additional monitoring may provide an estimate of the phase of the heartbeat for which the beacon positions and images are obtained. FIG. 11 illustrates an example phase assignment scheme; therein, the phase of the heartbeat varies between 0 and 2n over a single R-peak to R-peak interval. While the heart rate may vary, the positions and shape of the heart are approximately equivalent at the same phase across heartbeats. The position and shape may differ by a rotation or scaling between individual beats due to other processes, such as breathing or patient motion, but these differences can be accounted for.

[0087] When choosing images from different views for reconstruction, knowledge of the heartbeat phase can provide additional constraints on estimates of the change in position, or local deformation of, an object of interest within a given image. As such, using images obtained at near or equal phases in this cycle can reduce reconstruction errors.

[0088] In some embodiments, heartbeat data can be generated from electrical sensing through an electrocardiogram (ECG) system. In other embodiments, and based on the heart operating by circulating a current through the tissue, it is possible to instead track the heart beats magnetically with a sensor that is sufficiently sensitive to detect its signals, e.g., with a magnetic flux density on the order of 10s or 100s of pT. These signals may be measured magnetically,thereby monitoring the heartrate through a magnetocardiogram. In yet other embodiments, the heart motion may also be inferred from the patient’s pulse using devices readily available in a medical setting such as an optical pulse monitor or a cuff monitor placed on the patient’s limb.

[0089] In some embodiments, a phase assignment scheme may be applied to the respiration cycle, and the breathing motion is tracked through additional signals and sensors. This is analogous to the heartrate phase, and here, breathing is modeled as a periodic process, and a phase between 0 and 2 is assigned based on the estimated phase within a breathing cycle.

[0090] In some embodiments, breathing can be tracked by using a band sensor wrapped around the patient’s chest that is sensitive to expansion and contraction. In other embodiments, breathing can be tracked using an inertial measurement unit (IMU), which includes one or more of an accelerometer, a gyroscope, or a magnetometer. In yet other embodiments, breathing motion may be tracked optically using a camera system or a laser range-finding device, which tracks the movement of the patient’ s chest.

[0091] In some embodiments, the phase is monitored directly from the beacons by identifying a periodic component in some, or all, of their position and orientation data, or derived quantities. These components are then associated with the periodic body movement, and a phase value is assigned to the cycles of the given motion component.

[0092] In some embodiments, the motion of a particular part of the patient’s body, such as a limb, may be tracked. Examples include, but are not limited to, using an IMU or optical camera system to track the particular part’s motion.

[0093] In some embodiments, estimates are further constrained by physical calibration targets. Elements with known spatial variations made of materials which are resolvable with sufficient contrast for a given imaging modality may be placed within the imaging field of view temporarily to act as a stationary reference positions. Examples embodiments include, but are not limited to, grids of metal spheres and checkerboard patterns.

[0094] In some embodiments, the movement and positions of the surgical equipment can be tracked using signals directly obtained from the equipment. For example, encoder positions of the C-arm or the timing of the X-ray pulses generated by the fluoroscope may be transmitted to the system as analog or digital signals.

[0095] In some embodiments, surgical equipment is tracked indirectly using auxiliary sensors and signals. For example, the C-arm position can be tracked using devices including, butnot limited to, IMUs, additional magnetic sensors and beacons, or optical camera systems.

[0096] 4 Examples of projective mapping, pose estimation, and camera calibration

[0097] In some embodiments, the pinhole camera model is used. An example of this model is illustrated in FIGS. 12A and 12B. The camera input is assumed to be localized to a point C in the world frame. The point at the three-dimensional co-ordinate Vis imaged on the imaging plane at the two-dimensional point v. As shown in FIG. 12A, the camera center passes through a point P on the image plane, when moved along the principal axis, i.e., the axis perpendicular to the imaging plane in the world frame.

[0098] In the pinhole camera model, points in three-dimensional space are mapped to the two-dimensional image plane of a camera through the following transformation: x = PX = [ / |C]

[0099] Herein, P is the 3x4 camera matrix that maps a homogeneous vector X in R3to a homogeneous vector x in R2. The matrix P is described by the 3x3 intrinsic calibration matrix K, the 3x3 rotation matrix R, and the 3x4 matrix that horizontally concatenates a 3x3 identity matrix / and a 3x1 inhomogeneous translation vector in R3, C, which corresponds to the camera pinhole’s position in space. The notation [ / 1 C] denotes the horizontal concatenation of the identity matrix and translation vector into a combined 3x4 matrix.

[0100] In some embodiments, determining the coefficients of the camera matrix can be viewed as two separate processes: camera calibration and pose estimation. Camera calibration refers to correcting the intrinsic camera parameters. In example embodiments where the imaging modality has no skew, the intrinsic calibration has the form

[0101] The above form describes the camera focus f, the shift between the intersection of the principal axis and the imaging plane (point P in FIGS. 12A and 12B, with coordinates (px, pyY), and the zero of the image co-ordinate system. On the other hand, pose estimation refers to the calibration of extrinsic camera parameters, describing the orientation of the camera to the position or object in the world frame. In some mathematical representations of the camera matrix, the pose determines the rotation matrix R and inhomogeneous translation vector t.

[0102] In some embodiments, the calibration and pose estimation are performedsimultaneously by correlating a set of known three-dimensional points to their corresponding positions observed in the camera image. For each correspondence between 3D and 2D space positions, Xtx;one forms a set of linear equations given by:

[0103] This set of linear equations relates Pl, the 4 elements of each row of the camera matrix P, to the 3D point XLand the elements of the homogenous 2D vector xt= (x^y^, Wi)T. The null space of this equation corresponds to the desired solution for the coefficients of P. This least- squares estimate of the parameters provides the minimum total error between the measured 2D positions xtand those determined from the model xt, i.e., the estimate minimizes the total error for all d(xz, ,).

[0104] In some embodiments, the camera is calibrated by determining correspondences between a plurality of beacon locations as measured by an external sensor array and their observed locations in one or more acquired images. An example of an acquired image is shown in FIG. 13A where a fluoroscope image of three beacons located within a porcine heart is shown.

[0105] In some examples, the midpoints of the beacons are localized by the system, providing a three-dimensional position of each beacon in world space. The midpoint of each beacon is then detected in the image, as shown in FIG. 13B, where the midpoint locations are denoted by circles and emphasized by arrows. The camera is calibrated with these correspondences. In other examples, the endpoints of the beacons are used instead, as illustrated in FIG. 13C, where the endpoints are likewise circled and emphasized with arrows.

[0106] In some embodiments, the intrinsic camera properties remain fixed and must be determined only once. For example, in fluoroscopy, the focal length f and imaging plane center px, py^ are determined by the location of the X-ray source to the imaging sensor and remain fixed as the C-arm rotates and the table moves.

[0107] In some embodiments, where an existing intrinsic camera calibration is valid, only the camera pose can be determined. In other embodiments, a set of correspondence points is used to iteratively determine the extrinsic parameters by minimizing a cost function.

[0108] 5 Examples of updating pose estimation and camera calibration

[0109] In some embodiments, the extrinsic and intrinsic calibration parameters are updatedas additional data is received by the system and fused into the calibration correspondence point set. Additional correspondences between the one or more beacon positions and their presented positions in received images can be acquired during system operation. These additional correspondences permit fusion with the original calibration correspondence set. The system may be recalibrated after this fusion process, improving on the accuracy and / or the precision of the original camera calibration.

[0110] In some embodiments, the set is fused by including the additional correspondence points in the set. In some examples, additional correspondences are used to identify points in the existing correspondence set that have large errors relative to the remainder of the set. In these examples, the additional data is used to detect these high error points and designate them as outliers to be pruned from the calibration correspondence set.

[0111] Fusing additional points into the correspondence set provides, among other benefits, the following:

[0112] - As the system is in operation, the one or more beacons may be repositioned by the medical team to positions which may not have been included in the initial calibration. This introduces correspondence points at additional spatial positions which were not included in a previous calibration.

[0113] - Including points with errors less than or up to the errors of those already in the correspondence set, assuming a similar spatial distribution, has the effect of reducing the error of the calibration parameters due to the increased number of samples.

[0114] Fusing additional correspondence points into the dataset may also permit the pruning of previously acquired data points which, considering the additional information provided by the new correspondences, can be seen to have higher errors d( j,t) relative to the remainder of the dataset. Removing them from the correspondence set has the effect of reducing the overall error of the fitting.

[0115] In some embodiments, outliers arc identified using a random sample consensus (RANSAC) method. FIG. 14 is a flow diagram of an example method for performing RANSAC pruning. As illustrated therein, the method iteratively acquires random subsets of the correspondence points and determines the calibration using only the subset. The remainder of the points are then compared against the subset to determine which samples fit the model and whichare outliers. This process is iteratively repeated to improve the model.

[0116] In some embodiments, the operations in the flow diagram illustrated in FIG. 14 include:1) The set of correspondence points is loaded, including additional correspondences obtained since the previous calibration procedure.2) Parameters configuring the RANSAC pruning are loaded including the maximum number of iterations (max_iter), the maximum fit error permitted (max_error), the number of points used for a fitting subset (n_pts), and the number of points in a subset required to be inliers (fits_reqd). a) In some embodiments, the values of the parameters are dynamically configured. Factors determining this configuration may include, but are not limited to, the error of the previous calibration, the number of points in the correspondence set, the number of points added to the set, and the physical area covered by the correspondence set.3) The system initializes parameters tracking the number of iterations performed, the current best error observed and the model corresponding to this error.4) The algorithm performs an additional iteration, including operations (5) through (7), if the maximum number of iterations has not been reached.5) A random subset of the correspondence set S, containing a number of points equal to n_pts, is chosen.6) A calibration is performed using the selected subset of points. The error associated with each point is determined based on the computed calibration.7) The points in S whose error are below the threshold max_error are selected. If the number of points remaining is greater than fits_reqd, then a calibration is performed with the selected points. The error of this new fit is compared to the lowest observed error so far, best_error. If the error is lower than this value, best_error is replaced with this value and the current calibration is chosen as the best model.8) After a maximum number of iterations of the operations (5) through (7), the best model has been chosen. The error of all points in the entire correspondence set is computed using the best model. Those points whose error exceeds a threshold are deemed outliers. a) In some embodiments, the error threshold is different from that used in operation (2). b) In some embodiments, the outliers are removed from the correspondence set.

[0117] In other embodiments, outliers are determined using statistical variance of the new correspondence point set including the additional points. The calibration parameters are estimated using the entire set and the new errors between each point and their modelled position d(Xj, xt) are used to determine the variance o2and standard deviation o. Those points where the distance exceeds some threshold, e.g., three times the standard deviation, arc deemed outliers and are removed from the correspondence set.

[0118] 6 Examples of segmentation and object representation

[0119] Segmentation is the process of dividing an image into regions with similar properties, e.g., anatomical structure. Embodiments of the disclosed technology perform pixel-level segmentation in multiple images and form a representation of the segmented objects for three- dimensional reconstruction.

[0120] An example of segmentation is illustrated in FIGS. 15A and 15B where a cardiac lumen is the object of interest. An image of the vasculature under contrast is shown in FIG. 15 A. In FIG. 15B, the lumen of interest has been segmented from the rest of the image. The segmentation mask is emphasized with an arrow in the illustration.

[0121] Examples of segmentation methods include, but are not limited to, edge detection followed by segment boundary formation, pixel intensity grouping using a waterfall or clustering method, and neural network segmentation with architectures like U-Nets and Fast Fully Convolutional Networks (FastFCN).

[0122] In some embodiments, where an object of interest is a particular lumen, the contrast between the lumen and the surrounding tissue may be enhanced for image capture. Examples of enhancement techniques include, but are not limited to, injecting a contrast agent into the lumen, or placing a radio-opaque device into the lumen such as a catheter. In some embodiments, where a contrast agent is injected, the injection may be synchronized with the image acquisition time by electrical, mechanical, or other means. In other embodiments, where a contrast agent is injected, the presence of contrast is detected when the images are processed.

[0123] In some embodiments, appropriate segmentation of an object of interest is determined with a pixel intensity analysis method using edge detection and / or a region-based analysis.

[0124] In some embodiments, user input may be incorporated into the segmentation process. Examples include, but are not limited to, the physician (and / or member of the medical team) highlighting regions of interest in the image using a GUI and human-machine interface device,e.g., a mouse or a stylus.

[0125] In some embodiments, the segmentation process includes the use of a neural network, e.g., a convolutional neural network with a U-net architecture.

[0126] In the described embodiments, segmented pixels are processed to form a model representation of an object of interest. In some examples, a lumen object is represented as an interpolated curve which follows the middle of the lumen through the segmented pixels region, which can be encoded as the set of control points of a Bezier curve. In other examples, the curve through the middle of the lumen is represented using a cubic B-spline function. In yet other examples, the two-dimensional segmentation region is represented using non-uniform ration basis splines (NURBS). In yet other examples, the segmented pixels are represented as a collection of two-dimensional points.

[0127] In some embodiments, the segmentation and model representation processes may be implemented simultaneously. In some examples, a pose estimation convolutional neural network is trained to segment lumens within a particular anatomical structure and assign a set of wireframe points to it.

[0128] 7 Examples of projective reconstruction

[0129] Projective reconstruction refers to the process of determining the three-dimensional structure of an object from multiple two-dimensional views of the object, e.g., multiple 2D images obtained by an imaging modality. Methods of projective reconstruction are used to generate three-dimensional representations of objects of interest from the multiple images obtained by the imaging modality.

[0130] In some embodiments, reconstruction of the 3D structure from the multiple 2D images is implemented using a maximum likelihood estimator (MLE), which minimizes the distances between the measured 2D points and those which would be measured from the estimated 3D structure by means of a cost function.

[0131] FIG. 16 illustrates an example of determining the MLE reconstruction of a 3D point from a set of two camera views. As shown therein, a point in 3D space X is imaged by two cameras with pinholes located at C and C . The 2D images measure points corresponding to X at x and x' respectively, but assumes there is some error between these points and their true positions due to errors. The MLE estimates points x and x' which correspond to the most likelytrue projective positions that constitute a valid reconstruction while simultaneously minimizing the distances between the pair of points x and x and the pair x' andThe distances are labelled d and d' respectively in FIG. 16. The 3D position X which produces the MLE points x and x' is the most likely reconstruction point.

[0132] In some embodiments, a trust metric is used to evaluate the quality of the reconstruction. In some examples, the distances d and d', as illustrated in FIG. 16, can be used to determine a trust metric, e.g.,T = Vrf2+ d'2

[0133] The trust metric expresses how confident the system is that a particular reconstruction point is reliable. In some embodiments, a high (or low) trust metric causes the system to issue a notification or warning to the user that the reconstruction should be performed again. In other embodiments, the trust metric for reconstruction is used by the system to estimate the confidence or trust metric for other estimated quantities.

[0134] In applications where there is appreciable motion of an object of interest between images taken from the multiple vantage points (or multiple perspectives), it may be necessary to compensate for this motion when performing the reconstruction. This situation may occur, for example, during a procedure targ eting a patient’s heart where imaging is performed with a fluoroscopy. Since there is only a single imaging source, a first set of images is obtained from one vantage point before the physician moves the imaging source to a different location. During this time, the heart may beat one or more times and breathing may also occur. When the next set of images are taken (from the different vantage point), the relative rotation and translation of an object of interest within the heart, relative to the center of the new perspective, may change.

[0135] FIG. 17 shows an example of cardiac vasculature images, under contrast, taken at two different phases of a heartbeat. Both images have been taken from the same vantage point. Nevertheless, motion of the vasculature is observed due to beating. Correspondences are drawn between different parts of the vasculature at each phase where the location and shape of the vasculature has changed due to the expansion and contraction of different heart muscles.

[0136] Other situations where this may occur include, but are not limited to, pulmonary applications where the patient’s respiration causes the lungs to move between image capture at different vantage points, or peripheral applications where the patient moves their limb.

[0137] In some embodiments, motion can be modeled as a global coordinate shift of all objects in the image, and local deformation of the one or more objects of interest. Examples of local deformation include, but are not limited to, expansion and contraction of the heart during beating, and the inflation and deflation of the lungs during respiration.

[0138] In some embodiments, global coordinate shifts are treated as a change in the relative position and orientation of the camera with respect to the one or more objects of interest, which is compensated for by re-estimating the camera pose. The change in beacon locations and other constraints applied by additional input signals of the embodiment are used to perform the pose re-estimation and update the camera matrix accordingly.

[0139] This principle is shown in FIGS. 18A-18C. In FIG. 18A, a camera located at the black dot with orientation given by the arrows views an object. In FIG. I8B, the object is rotated by some angle relative to the camera creating a rotated perspective. This perspective can be recreated in FIG. 18C by rotating the camera rather than the object. However, in the case of FIG. 18C, the rotation is the inverse rotation of that applied to the object in FIG. 18B. In this same manner, global changes to the object’s translation and rotation can be negated by applying an inverse rotation and translation to the camera pose, thereby changing its extrinsic parameters.

[0140] In some embodiments, local deformations caused by periodic motions such as heartbeats and respiration are minimized by performing the reconstruction using multiple images, where the phases of motion assigned to each image most closely match. In some examples, using the heartbeat phases of FIG. 11, the multiple images from different camera poses where the phase is closest to 71 are selected. Reconstruction is performed on the selected images, which significantly reduces the error introduced from the motion artifacts.

[0141] In some embodiments, the 3D reconstruction is estimated while assuming that there is motion occurring between when images are captured at the two vantage points. In some examples, the points are assumed to belong to a time-varying shape whose reconstruction is determined with a Probabilistic Principal Component Analysis. In other examples, a database of training examples is used to learn a set of template deformations, learned on examples representative of objects of interest, which are used to predict the most likely reconstruction.

[0142] In some embodiments, previously obtained representations of one or more objects of interest may be used to further constrain the reconstruction estimate. Examples of such representations include, but are not limited to, medical images of the object of interest such as aCT scan or an MRI. The structure of the objects of interest in these representations are used to estimate the likelihood that the current representation is accurate. In some examples, the following term can be included in the MLE reconstruction, thereby maximizing the probability:

[0143] Herein, the reconstruction points maximize a combined metric minimizing the distance between the point pair x and x and the pair x' and x' along with the probability of the reconstruction point being X given that the previous representation forms an image / .

[0144] In some embodiments, previously obtained representations of the objects of interest may be used when performing image registration for the newly obtained 3D representations. In these examples, image registration enables the alignment of images taken at different times, from various viewpoints, or using different imaging modalities (such as CT, MRI, or PET). This alignment advantageously improves tracking the medical device (e.g., as discussed in Section 8), and may include performing preprocessing steps, detecting key anatomical landmarks or distinctive image features (e.g., using algorithms like a scale-invariant feature transform (SIFT) or speeded-up robust feature (SURF), or neural network architectures like U-Nets and FastFCN), applying statistical methods (e.g., RANSAC) to filter out incorrect matches, and estimate the optimal transformation.

[0145] 8 Examples of motion tracking and compensation

[0146] After the initial reconstruction from multiple vantage points (or perspectives), the imaging device may be repositioned to continue the particular medical procedure. The new device location may have been previously used in the multiple views to generate the reconstruction, or it may be entirely new. It may remain at the new location for the remainder of the procedure, or it may be repositioned again depending on the needs of the physician.

[0147] Regardless, it may be desirable that an object of interest be rendered in a manner that shows its perceived projective representation at the new location, in its expected position and orientation if it were visible. It is also desirable that in such a representation, the object model be updated to account for relative motion between the camera and object of interest that occurs during the procedure, and an updated rendering of the object be displayed for the medical team.

[0148] In some embodiments, the relative motion is divided into a global motion and one or more local deformations. Global motion is the movement of all points of an object of interestrelative to the camera. Examples of global motion include, but are not limited to, relative rotations and translations between the object and camera. A local deformation is a motion of one or more portions of an object of interest which are not matched, either by other parts of the object or the surrounding tissue.

[0149] Global motion due to changes in the relative translation and orientation of the object and camera are used to update the extrinsic camera parameters which projects the 3D reconstruction into the image plane. In some embodiments, these global motion changes, as measured by the beacons, are directly applied to update the pose estimate. In other embodiments, the new beacon positions are used as input to a Kalman filter that models the change in extrinsic camera parameters, which is then used to update the estimate. In yet other embodiments, additional constraints on the motion estimates are applied from additional input signals (e.g., as described in Section 3).

[0150] Model updates to account for local deformations are also used in some embodiments. In some examples, the deformation is learned by regressing a function over observed changes in an object of interest to form a template. In other examples, an MLE minimizing the deformation energy can be leveraged.

[0151] In some embodiments, where multiple images are obtained in each pose during the reconstruction phase, local deformations are estimated by using the previously observed deformation as an example to form a template. Images taken from multiple poses, but associated with the same position in a deformation movement, are reconstructed together producing a 3D estimation of the deformation movement.

[0152] In some embodiments, phase assignment data is used to constrain the template and time align the deformation estimation. An example is illustrated in FIG. 19 where a heart is imaged under contrast at various phases of the heartbeat. Reconstruction from multiple poses is performed at different phases of the heartbeat and the 3D reconstruction is used to form the template. In some examples, a template may be formed for each of the assigned phases.

[0153] In some embodiments, a set of control points is assigned to positions on the model and the motion of the individual points is regressed over the entire motion sequence. The motion parameters seed a Kalman filter, which tracks and estimates the motion of each control point. For a given reference frame, the positions of the control points are used to update the model representation to most closely match the deformation.

[0154] In some embodiments, a discriminating function is used to estimate when the motion deviates significantly from the template so that the system identifies when the estimated deformation and global motion are likely to deviate from the estimated model motion. Examples where this may occur include, but are not limited to, cardiac arrythmia events where the heart beats irregularly, and sudden motions of the patient which deviate significantly from those expected by the model.

[0155] In some examples, the discriminating function includes an outlier detector that monitors the position of select model points and identifies (or detects) time instances when outliers deviate from the observed statistical distribution of the model point positions over time.

[0156] In other examples, the model point positions are assumed to have a certain distribution, i.e., the distribution of model point positions over a set time window are modeled as a probability density (e.g., a Gaussian probability density and distribution). When appreciable third and fourth order moments are measured in the probability density, the discriminating function identifies these as periods in time in which the system believes there is insufficient statistical evidence to validate the results of the model motion tracking.

[0157] 9 Examples of rendering

[0158] In some embodiments, the changes to the 3D reconstruction model are rendered and displayed to the physician and medical team on a display. In an example, a 3D rendering of the object is provided in its own display. In another example, a 2D rendering can be overlaid on other medical images.

[0159] An example of a 2D rendering being overlaid on another image is illustrated in FIG. 20 where the object of interest is a portion of the vasculature in an X-ray image of the heart. Without contrast, the vasculature is difficult to visualize, unlike FIG. 1 where the contrast highlights it. Instead, the 3D reconstruction of an object of interest is rendered and overlaid on the image, in the estimated position, orientation and deformation configurations. The 2D representation of the 3D model is generated by projecting the points of the model into the image frame using the camera matrix corresponding to the imaging device’s current configuration, as discussed above.

[0160] In some embodiments, the physician (and / or the medical team) is provided feedback regarding where the trust metric (as discussed in Section 7) is favorable and where it is unfavorable. In some examples, the user is presented with a green or red status symbol indicatinga favorable or unfavorable metric, respectively. In other examples, the user is presented a color- coded reconstruction where the color varies based on the favorability of the trust metric determined for the local area of the reconstruction.

[0161] In some embodiments, the physician is provided feedback indicating that the trust metric is sufficiently low, either globally or locally, and that the estimation of the reconstruction or motion tracking needs to be improved before proceeding.

[0162] In some embodiments, a previously obtained representations of one or more objects of interest are co-rendered with the system representation of the objects. In an example, a previously obtained MRI image of an anatomical structure is scaled, translated, and rotated in a manner that the image most closely matches the system’s representation of the same structure. The MRI image is then overlaid on the live medical image.

[0163] 10 Examples and implementations of the disclosed technology

[0164] FIG. 21 shows a flowchart of an example method of estimating a three-dimensional (3D) representation of one or more objects of interest within a body of a patient. The method 2100 includes, at operation 2110, receiving a plurality of images corresponding to the one or more objects of interest being imaged from a plurality of perspectives by an imaging modality. In some examples, the imaging modality is discussed in at least Section 1. In these and other examples, the plurality of perspectives is discussed in at least Sections 1 and 7.

[0165] The method 2100 includes, at operation 2120, determining, using an array of magnetic sensors, a location and an orientation of at least one beacon affixed to an invasive medical device inside the body of the patient. In some examples, the beacons and determining the location and orientation of the at least one beacon is discussed in at least Section 2.

[0166] The method 2100 includes, at operation 2130, performing, based on magnetometry information and without using a position or an orientation of an emitter of the imaging modality or a position of the body of the patient as a reference point, a calibration process for the imaging modality at each of the plurality of perspectives. In some examples, performing the calibration process is discussed in at least Sections 4 and 5. Herein, each perspective is associated with the reference point corresponding to a respective emitter, and the magnetometry information comprises (a) the location and the orientation of the at least one beacon and (b) two-dimensional position information for the at least one beacon determined based on the plurality of images. In these and other examples, the plurality of perspectives is discussed in at least Sections 1 and 7.

[0167] The method 2100 includes, at operation 2140, estimating, based on the plurality of images and an output of the calibration process, the 3D representation of the one or more objects of interest from at least one of the plurality of perspectives. In some examples, estimating the 3D representation is discussed in at least Section 7.

[0168] The described features can be implemented to further provide one or more of the following technical solutions:

[0169] S 1. A method of estimating a three-dimensional (3D) representation of one or more objects of interest within a body of a patient, comprising: receiving a plurality of images corresponding to the one or more objects of interest being imaged from a plurality of perspectives by an imaging modality; determining, using an array of magnetic sensors, a location and an orientation of at least one beacon affixed to an invasive medical device inside the body of the patient; performing, based on magnetometry information, a calibration process for the imaging modality at each of the plurality of perspectives, wherein the magnetometry information comprises (a) the location and the orientation of the at least one beacon and (b) two-dimensional position information for the at least one beacon determined based on the plurality of images, wherein performing the calibration process excludes using a position or an orientation of an emitter of the imaging modality or a position of the body of the patient as a reference point, and wherein each perspective is associated with the reference point corresponding to a respective emitter; and estimating, based on the plurality of images and an output of the calibration process, the 3D representation of the one or more objects of interest from at least one of the plurality of perspectives. In some technical solutions, the imaging modality is discussed in at least Section 1. In these and other technical solutions, the plurality of perspectives is discussed in at least Sections 1 and 7. In these and other technical solutions, the beacons and determining the location and orientation of the at least one beacon is discussed in at least Section 2. In these and other technical solutions, estimating the 3D representation is discussed in at least Section 7.

[0170] S2. The method of solution SI, wherein performing the calibration process comprises: performing a segmentation operation on the plurality of images to generate a plurality of segmented images, each segmented image comprising a distinct region corresponding to at least one object of interest; and performing, based on the plurality of segmented images, a projective reconstruction operation to generate the 3D representation of the at least one object of interest. In some technical solutions, segmentation operations are discussed in at least Section 6.

[0171] S3. The method of solution S2, wherein performing the segmentation operation comprises a pixel intensity analysis method, a neural network, and / or feedback from the patient.

[0172] S4. The method of solution S2, wherein the segmentation operation and the projective reconstruction operation are performed concurrently using a neural network.

[0173] S5. The method of any of solutions SI to S4, comprising: navigating, based on the 3D representation of the one or more objects of interest, the invasive medical device within the body of the patient. In some technical solutions, navigating the invasive medical device is discussed in at least Section 1.

[0174] S6. The method of any of solutions SI to S5, comprising: estimating a pose or a deformation in the 3D representation of the one or more objects of interest within the body of the patient. In some technical solutions, estimating a pose or a deformation is discussed in at least Section 4 and 5.

[0175] S7. The method of solution S6, wherein estimating the pose, the deformation, or the3D representation of the one or more objects of interest is constrained based on pixel intensities of the plurality of images, an observed motion of the at least one beacon determined from the plurality of images, an observed motion of the one or more objects of interest determined from the plurality of images, or feedback from the patient. In some technical solutions, performing a constrained estimation operation is discussed in at least Section 3.

[0176] S8. The method of solution S7, wherein the observed motion of the one or more objects of interest is determined based on using changes in the location and the orientation of the at least one beacon as an input to a Kalman filter.

[0177] S9. The method of solution S6, wherein estimating the pose, the deformation, or the3D representation of the one or more objects of interest is constrained based on at least one signal associated with a biological process of the patient. In some technical solutions, constraints being based on a biological process is discussed in at least Sections 3 and 8.

[0178] S10. The method of solution S9, wherein the biological process comprises a heart beat of the patient, and wherein the at least one signal comprises an electrical signal generated by an electrocardiogram (ECG) system, an output signal from magnetically sensing a heart’s current, or a pulse signal generated by an optical sensor or a pressure sensor.

[0179] Si l. The method of solution S10, wherein an interval of the heart beat is monitored using the ECG system or a magnetocardiogram system.

[0180] S 12. The method of solution S9, wherein the biological process comprises breathing by the patient, and wherein the at least one signal comprises: an electrical signal indicative of a motion of a chest of the patient, a signal from an inertial measurement unit (1MU) comprising at least one of a magnetometer, an accelerometer, or a gyroscope, or a signal detected by an optical means.

[0181] SI 3. The method of solution SI 2, wherein the breathing is monitored using a band sensitive to mechanical strain from the breathing, the IMU, or the optical means.

[0182] S14. The method of solution S12 or 13, wherein the optical means comprises a camera or a laser range-finding device.

[0183] S15. The method of any of solutions SI to S14, further comprising: performing, subsequent to initially performing the calibration process, the calibration process based on (a) an additional plurality of images corresponding to the one or more objects of interest and (b) the location and the orientation of the at least one beacon affixed to the invasive medical device. In some technical solutions, using the additional images is discussed in at least Sections 3 and 5.

[0184] S16. The method of solution S15, comprising: performing a pruning process on the plurality of images and / or the additional plurality of images by removing each image identified as having an error greater than a threshold.

[0185] S17. The method of solution SI 6, wherein removing each image comprises: removing a corresponding location and orientation of the at least one beacon from the magnetometry information.

[0186] S18. The method of solution S16 or 17, wherein the pruning process uses a random sample consensus method to identify each image with a corresponding error greater than the threshold.

[0187] S19. The method of solution S16 or 17, wherein the pruning process uses a statistical variance metric to identify each image with a corresponding error greater than the threshold.

[0188] S20. The method of any of solutions SI to SI 9, wherein performing the calibration process is based on a physical calibration target.

[0189] S21. The method of any of solutions SI to S20, comprising: determining, using a discriminating function, a confidence level for the 3D representation of the one or more objects of interest; and displaying, on a graphical user interface, the confidence level. In some technical solutions, deteimining a confidence level is discussed in Section 7.

[0190] S22. The method of solution S21, wherein the discriminating function is configured to detect statistical outliers in the 3D representation of the one or more objects of interest.

[0191] S23. The method of any of solutions SI to S22, wherein estimating the 3D representation of the one or more objects of interest is constrained based on a previously obtained representation of at least one of the one or more objects of interest.

[0192] S24. The method of any of solutions SI to S23, comprising: performing image registration using the 3D representation of the one or more objects of interest and a previously obtained representation of the one or more objects of interest. In some technical solutions, image registration is discussed in at least Section 7.

[0193] S25. The method of any of solutions SI to S24, wherein the at least one beacon is removable so as to allow repositioning of the at least one beacon at a different location or at a different orientation on the invasive medical device.

[0194] S26. The method of any of solutions SI to S24, wherein the at least one beacon is integrated as part of the invasive medical device, e.g., as shown in FIGS. 5A-5C.

[0195] S27. The method of any of solutions SI to S26, wherein reference points, which correspond to the position or the orientation of the emitter and the position of the body of the patient, define a frame of reference or a coordinate system associated with the imaging modality.

[0196] In some embodiments, technical solutions S28 through S43 describe a system for navigating an invasive medical device within a body of a patient, which is used in conjunction with the methods described in technical solutions SI to S27 and S44 to S51.

[0197] S28. A system for navigating an invasive medical device within a body of a patient, comprising: at least one beacon; an array of magnetic sensors, external to the patient, configured to sense the at least one beacon affixed to the invasive medical device; at least one processor configured to: receive a plurality of images corresponding to one or more objects of interest, within the body of the patient, being imaged from a plurality of perspectives by an imaging modality, determine, using the array of magnetic sensors, a location and an orientation of the at least one beacon, perform, based on magnetometry information, a calibration process for the imaging modality at each of the plurality of perspectives, wherein performing the calibration process excludes using a position or an orientation of an emitter of the imaging modality or a position of the body of the patient as a reference point, wherein each perspective is associated with the reference point corresponding to a respective emitter, and wherein the magnetometryinformation comprises (a) the location and the orientation of the at least one beacon and (b) two- dimensional position information for the at least one beacon determined based on the plurality of images, estimate, based on the plurality of images and an output of the calibration process, a three-dimensional (3D) representation of the one or more objects of interest from at least one of the plurality of perspectives, and navigate, based on the 3D representation of the one or more objects of interest, the invasive medical device within the body of the patient.

[0198] S29. The system of solution S28, wherein the at least one beacon comprises a permanent magnet and / or an electromagnet.

[0199] S30. The system of solution S29, wherein an electromagnetic field associated with the at least one beacon comprises a constant electromagnetic field.

[0200] S31. The system of solution S29, wherein an electromagnetic field associated with the at least one beacon comprises a time-varying electromagnetic field.

[0201] S32. The system of solution S31, wherein the time- varying electromagnetic field is generated using a single-frequency signal that is static or adjusted.

[0202] S33. The system of solution S31, wherein the time-varying electromagnetic field is generated using a multi-frequency spread- spectrum signal.

[0203] S34. The system of solution S28, wherein the at least one beacon comprises a permanent magnet with a first magnetic axis and an electromagnet with a second magnetic axis that is perpendicular to the first magnetic axis.

[0204] S35. The system of solution S34, wherein the first magnetic axis is parallel to an axis of the invasive medical device.

[0205] S36. The system of solution S34, wherein the second magnetic axis is parallel to an axis of the invasive medical device.

[0206] S37. The system of any of solutions S28 to S36, wherein the invasive medical device comprises an intravascular ultrasound catheter, a plasma ablation catheter, an obstruction removing catheter, or a percutaneous treatment catheter.

[0207] S38. The system of any of solutions S28 to S37, wherein the calibration process is based on a contrast level in at least one of the plurality of images.

[0208] S39. The system of solution S38, wherein the contrast level is based on introducing a contrast agent into at least one of the one or more objects of interest.

[0209] S40. The system of solution S38, wherein the contrast level is based on placing aradio-opaque device within or in close proximity of at least one of the one or more objects of interest.

[0210] S41. The system of any of solutions S28 to S40, wherein the at least one beacon is removable so as to allow repositioning of the at least one beacon at a different location or at a different orientation on the invasive medical device.

[0211] S42. The system of any of solutions S28 to S40, wherein the at least one beacon is integrated as part of the invasive medical device.

[0212] S43. The system of any of solutions S28 to S42, wherein reference points, which correspond to the position or the orientation of the emitter and the position of the body of the patient, define a frame of reference or a coordinate system associated with the imaging modality.

[0213] In some embodiments, the method described in technical solutions S44 through S51 is configured to incorporate aspects of the methods described in technical solutions SI to S27.

[0214] S44. A method of estimating a three-dimensional (3D) representation of one or more objects of interest within a body of a patient, comprising: receiving a plurality of images corresponding to the one or more objects of interest being imaged from a plurality of perspectives by an imaging modality, each perspective corresponding to an emitter of the imaging modality; determining, using an array of magnetic sensors, a location and an orientation of at least one beacon affixed to an invasive medical device inside the body of the patient; performing, based on magnetometry information, a calibration process for the imaging modality at each of the plurality of perspectives, wherein the magnetometry information comprises (a) the location and the orientation of the at least one beacon and (b) two-dimensional position information for the at least one beacon determined based on the plurality of images, and wherein performing the calibration process excludes using a position or an orientation of the emitter of the imaging modality and a position of the body of the patient as reference points; and estimating, based on the plurality of images and an output of the calibration process, the 3D representation of the one or more objects of interest from at least one of the plurality of perspectives.

[0215] S45. The method of solution S44, wherein a perspective of an image corresponding to the emitter of the imaging modality comprises the emitter being at a point of view when capturing the image with the perspective.

[0216] S46. A method comprising: determining a three-dimensional (3D) reconstruction ofan object of interest within a patient’ s body based on (a) a plurality of two-dimensional (2D) images of the object of interest and (b) 3D position information of at least one beacon associated with an invasive medical device that is in close proximity to the object of interest.

[0217] S47. The method of solution S46, wherein the 3D position information is used to determine dynamic motion updates to the 3D reconstruction of the object of interest.

[0218] S48. The method of solution S47, wherein an estimation accuracy associated with determining the 3D reconstruction of the object of interest and / or the dynamic motion updates is generated.

[0219] S49. The method of solution S47, wherein a visual representation of the 3D reconstruction of the object of interest and / or the dynamic motion updates are displayed on a graphical user interface.

[0220] S50. The method of any of solutions S46 to S49, wherein the 3D reconstruction of the object of interest is determined using only the plurality of 2D images and the 3D position information.

[0221] S51. The method of any of solutions S46 to S50, wherein the plurality of 2D images are obtained using an imaging modality, and wherein the 3D reconstruction of the object of interest is determined without using a reference frame associated with the imaging modality.

[0222] S52. A system comprising: a positioning beacon integrated into an invasive medical device; a plurality of sensors configured to obtain a plurality of measurements of a magnetic field of the positioning beacon; and one or more processors configured to implement the method of any of solutions SI to S27 and solutions S44 to S51.

[0223] S53. A method of navigating a medical device within a patient’s body that uses the system of one or more solutions S28 to S43.

[0224] FIG. 22 shows an example of a hardware platform 2200 that can be used to implement some of the techniques described in the present document. For example, the hardware platform 2200 can implement method 2200, or implement the various modules and algorithms described herein. The hardware platform 2200 includes a processor 2202 that can execute code to implement a method. The hardware platform 2200 includes a memory 2204 that is used to store processor-executable code and / or store data. The hardware platform 2200 further includes magnets 2206 and magnetometers 2208, which can communicate with the processor 2202 via leads or a wireless protocol. The processor 2202 is configured to implement localization, fusion,and registration algorithms. In some embodiments, the memory 2204 comprises multiple memories, some of which are exclusively used by the processor 2202 when implementing the fusion, localization, or registration algorithms.

[0225] Implementations of the subject matter and the functional operations described in this patent document can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine- readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing unit” or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0226] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0227] The processes and logic flows described in this specification can be performed by oneor more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and devices can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0228] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, these are optional. Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0229] While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0230] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

[0231] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.

Claims

WHAT IS CLAIMED IS:

1. A method of estimating a three-dimensional (3D) representation of one or more objects of interest within a body of a patient, comprising: receiving a plurality of images corresponding to the one or more objects of interest being imaged from a plurality of perspectives by an imaging modality; determining, using an array of magnetic sensors, a location and an orientation of at least one beacon affixed to an invasive medical device inside the body of the patient; performing, based on magnetometry information, a calibration process for the imaging modality at each of the plurality of perspectives, wherein the magnetometry information comprises (a) the location and the orientation of the at least one beacon and (b) two-dimensional position information for the at least one beacon determined based on the plurality of images, wherein performing the calibration process excludes using a position or an orientation of an emitter of the imaging modality or a position of the body of the patient as a reference point, and wherein each perspective is associated with the reference point corresponding to a respective emitter; and estimating, based on the plurality of images and an output of the calibration process, the 3D representation of the one or more objects of interest from at least one of the plurality of perspectives.

2. The method of claim 1, wherein performing the calibration process comprises: performing a segmentation operation on the plurality of images to generate a plurality of segmented images, each segmented image comprising a distinct region corresponding to at least one object of interest; and performing, based on the plurality of segmented images, a projective reconstruction operation to generate the 3D representation of the at least one object of interest.

3. The method of claim 2, wherein performing the segmentation operation comprises a pixel intensity analysis method, a neural network, and / or feedback from the patient.

4. The method of claim 2, wherein the segmentation operation and the projective reconstruction operation are performed concurrently using a neural network.

5. The method of claim 1, comprising: navigating, based on the 3D representation of the one or more objects of interest, the invasive medical device within the body of the patient.

6. The method of claim 1, comprising: estimating a pose or a deformation in the 3D representation of the one or more objects of interest within the body of the patient.

7. The method of claim 6, wherein estimating the pose, the deformation, or the 3D representation of the one or more objects of interest is constrained based on pixel intensities of the plurality of images, an observed motion of the at least one beacon determined from the plurality of images, an observed motion of the one or more objects of interest determined from the plurality of images, or feedback from the patient.

8. The method of claim 7, wherein the observed motion of the one or more objects of interest is determined based on using changes in the location and the orientation of the at least one beacon as an input to a Kalman filter.

9. The method of claim 6, wherein estimating the pose, the deformation, or the 3D representation of the one or more objects of interest is constrained based on at least one signal associated with a biological process of the patient.

10. The method of claim 9, wherein the biological process comprises a heart beat of the patient, and wherein the at least one signal comprises an electrical signal generated by an electrocardiogram (ECG) system, an output signal from magnetically sensing a heart’s current, or a pulse signal generated by an optical sensor or a pressure sensor.

11. The method of claim 10, wherein an interval of the heart beat is monitored using the ECG system or a magnetocardiogram system.

12. The method of claim 9, wherein the biological process comprises breathing by the patient, and wherein the at least one signal comprises: an electrical signal indicative of a motion of a chest of the patient, a signal from an inertial measurement unit (IMU) comprising at least one of a magnetometer, an accelerometer, or a gyroscope, or a signal detected by an optical means.

13. The method of claim 12, wherein the breathing is monitored using a band sensitive to mechanical strain from the breathing, the IMU, or the optical means.

14. The method of claim 12, wherein the optical means comprises a camera or a laser rangefinding device.

15. The method of claim 1, further comprising: performing, subsequent to initially performing the calibration process, the calibration process based on (a) an additional plurality of images corresponding to the one or more objects of interest and (b) the location and the orientation of the at least one beacon affixed to the invasive medical device.

16. The method of claim 15, comprising: performing a pruning process on the plurality of images and / or the additional plurality of images by removing each image identified as having an error greater than a threshold.

17. The method of claim 16, wherein removing each image comprises: removing a corresponding location and orientation of the at least one beacon from the magnetometry information.

18. The method of claim 16, wherein the pruning process uses a random sample consensus method to identify each image with a corresponding error greater than the threshold.

19. The method of claim 16, wherein the pruning process uses a statistical variance metric to identify each image with a corresponding error greater than the threshold.

20. The method of any of claims 1 to 19, wherein performing the calibration process is based on a physical calibration target.

21. The method of any of claims 1 to 19, comprising: determining, using a discriminating function, a confidence level for the 3D representation of the one or more objects of interest; and displaying, on a graphical user interface, the confidence level.

22. The method of claim 21, wherein the discriminating function is configured to detect statistical outliers in the 3D representation of the one or more objects of interest.

23. The method of any of claims 1 to 19, wherein estimating the 3D representation of the one or more objects of interest is constrained based on a previously obtained representation of at least one of the one or more objects of interest.

24. The method of any of claims 1 to 19, comprising: performing image registration using the 3D representation of the one or more objects of interest and a previously obtained representation of the one or more objects of interest.

25. The method of any of claims 1 to 19, wherein the at least one beacon is removable so as to allow repositioning of the at least one beacon at a different location or at a different orientation on the invasive medical device.

26. The method of any of claims 1 to 19, wherein the at least one beacon is integrated as pail of the invasive medical device.

27. The method of any of claims 1 to 19, wherein reference points, which correspond to the position or the orientation of the emitter and the position of the body of the patient, define a frame of reference or a coordinate system associated with the imaging modality.

28. A system for navigating an invasive medical device within a body of a patient, comprising: at least one beacon;an array of magnetic sensors, external to the patient, configured to sense the at least one beacon affixed to the invasive medical device; at least one processor configured to: receive a plurality of images corresponding to one or more objects of interest, within the body of the patient, being imaged from a plurality of perspectives by an imaging modality, determine, using the array of magnetic sensors, a location and an orientation of the at least one beacon, perform, based on magnetometry information, a calibration process for the imaging modality at each of the plurality of perspectives, wherein performing the calibration process excludes using a position or an orientation of an emitter of the imaging modality or a position of the body of the patient as a reference point, wherein each perspective is associated with the reference point corresponding to a respective emitter, and wherein the magnetometry information comprises (a) the location and the orientation of the at least one beacon and (b) two-dimensional position information for the at least one beacon determined based on the plurality of images, estimate, based on the plurality of images and an output of the calibration process, a three-dimensional (3D) representation of the one or more objects of interest from at least one of the plurality of perspectives, and navigate, based on the 3D representation of the one or more objects of interest, the invasive medical device within the body of the patient.

29. The system of claim 28, wherein the at least one beacon comprises a permanent magnet and / or an electromagnet.

30. The system of claim 29, wherein an electromagnetic field associated with the at least one beacon comprises a constant electromagnetic field.

31. The system of claim 29, wherein an electromagnetic field associated with the at least one beacon comprises a time-varying electromagnetic field.

32. The system of claim 31, wherein the time-varying electromagnetic field is generated using a single-frequency signal that is static or adjusted.

33. The system of claim 31, wherein the time-varying electromagnetic field is generated using a multi-frequency spread- spectrum signal.

34. The system of claim 28, wherein the at least one beacon comprises a permanent magnet with a first magnetic axis and an electromagnet with a second magnetic axis that is perpendicular to the first magnetic axis.

35. The system of claim 34, wherein the first magnetic axis is parallel to an axis of the invasive medical device.

36. The system of claim 34, wherein the second magnetic axis is parallel to an axis of the invasive medical device.

37. The system of any of claims 28 to 36, wherein the invasive medical device comprises an intravascular ultrasound catheter, a plasma ablation catheter, an obstruction removing catheter, or a percutaneous treatment catheter.

38. The system of any of claims 28 to 36, wherein the calibration process is based on a contrast level in at least one of the plurality of images.

39. The system of claim 38, wherein the contrast level is based on introducing a contrast agent into at least one of the one or more objects of interest.

40. The system of claim 38, wherein the contrast level is based on placing a radio-opaque device within or in close proximity of at least one of the one or more objects of interest.

41. The system of any of claims 28 to 36, wherein the at least one beacon is removable so as to allow repositioning of the at least one beacon at a different location or at a different orientation on the invasive medical device.

42. The system of any of claims 28 to 36, wherein the at least one beacon is integrated as part of the invasive medical device.

43. The system of any of claims 28 to 36, wherein reference points, which correspond to the position or the orientation of the emitter and the position of the body of the patient, define a frame of reference or a coordinate system associated with the imaging modality.

44. A method of estimating a three-dimensional (3D) representation of one or more objects of interest within a body of a patient, comprising: receiving a plurality of images corresponding to the one or more objects of interest being imaged from a plurality of perspectives by an imaging modality, each perspective corresponding to an emitter of the imaging modality; determining, using an array of magnetic sensors, a location and an orientation of at least one beacon affixed to an invasive medical device inside the body of the patient; performing, based on magnetometry information, a calibration process for the imaging modality at each of the plurality of perspectives, wherein the magnetometry information comprises (a) the location and the orientation of the at least one beacon and (b) two-dimensional position information for the at least one beacon determined based on the plurality of images, and wherein performing the calibration process excludes using a position or an orientation of the emitter of the imaging modality and a position of the body of the patient as reference points; and estimating, based on the plurality of images and an output of the calibration process, the 3D representation of the one or more objects of interest from at least one of the plurality of perspectives.

45. The method of claim 44, wherein a perspective of an image corresponding to the emitter of the imaging modality comprises the emitter being at a point of view when capturing the image with the perspective.

46. A method comprising: determining a three-dimensional (3D) reconstruction of an object of interest within a patient’s body based on (a) a plurality of two-dimensional (2D) images of the object of interest and (b) 3D position information of at least one beacon associated with an invasive medical device that is in close proximity to the object of interest.

47. The method of claim 46, wherein the 3D position information is used to determine dynamic motion updates to the 3D reconstruction of the object of interest.

48. The method of claim 47, wherein an estimation accuracy associated with determining the 3D reconstruction of the object of interest and / or the dynamic motion updates is generated.

49. The method of claim 47, wherein a visual representation of the 3D reconstruction of the object of interest and / or the dynamic motion updates are displayed on a graphical user interface.

50. The method of any of claims 46 to 49, wherein the 3D reconstruction of the object of interest is determined using only the plurality of 2D images and the 3D position information.

51. The method of any of claims 46 to 49, wherein the plurality of 2D images are obtained using an imaging modality, and wherein the 3D reconstruction of the object of interest is determined without using a reference frame associated with the imaging modality.

52. A system comprising: a positioning beacon integrated into an invasive medical device; a plurality of sensors configured to obtain a plurality of measurements of a magnetic field of the positioning beacon; and one or more processors configured to implement the method of any of claims 1 to 27 and claims 44 to 51.

53. A method of navigating a medical device within a patient’s body that uses the system of one or more claims 28 to 43.

Images