Methods of annotating ultrasound views and catheter maneuvers in three dimensions, and providing the user with visual aids and alerts to safely navigate in cardiovascular anatomy

Abstract

An example method includes receiving ultrasound (US) data from an echocardiogram probe for a region of interest for of the intracardiac transcatheter procedure. The method also includes constructing a three-dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The method additionally includes displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker. In some examples, the marker tracks a stationary position such as a rapid pacing position associated with a feature, including while the feature is in motion.

Description

TITLEMETHODS OF ANNOTATING ULTRASOUND VIEWS AND CATHETER MANEUVERS IN THREE DIMENSIONS, AND PROVIDING THE USER WITH VISUAL AIDS AND ALERTS TO SAFELY NAVIGATE IN CARDIOVASCULAR ANATOMYRELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 661 ,922, filed June 20, 2024, and U.S. Provisional Patent Application Serial No. 63 / 725,000, filed November 26, 2024, the entirety of each is hereby incorporated by reference for all purposes.FIELD

[0002] The present technology is generally related to a user interface for medical imaging and imaging systems employing such a user interface.BACKGROUND

[0003] Ultrasonography is used for imaging in a variety of fields, including medical imaging. An ultrasound (US) transducer emits US waves which are reflected by echogenic or echoic materials (for example bone or materials with physical properties similar to bone) and absorbed or allowed to pass through by non-echogenic or anechoic materials (for example purified water or materials with physical properties similar to purified water). The echo or reflection from echoic materials as opposed to the absorption or non-reflection of anechoic materials is used to determine imaging data, which is provided to a user such as a surgeon or US technician via a user interface.SUMMARY

[0004] The techniques of this disclosure generally relate to providing a user interface for an ultrasound imaging system, such as an echocardiogram (e.g., transesophageal echocardiogram (TEE), transthoracic echocardiogram (TTE), intracardiac echocardiogram (ICE), etc.) imaging system. Various examples facilitate adding markers of annotation(s), edge(s), hazard(s), target(s), and / or other point(s) on ultrasound view(s) or plane(s) to a three-dimensional (3D) model of a region of interest via a currently displayed view or plane. In other views or planes, marker(s) added to the 3D model are displayed based onthe 3D position(s) of the marker(s) on the 3D model. Additionally, example(s) track marker(s) (e.g., annotation(s), edge(s), hazard(s)) based on movement of anatomical features in the region of interest based on image processing, updating the position of the annotated or marked anatomy in real-time. Further, various examples use known geometry of an instrument or a device to track an avatar of the instrument / device in the 3D model, based on location(s) of tracked datum(s) of the device / instrument (e.g., associated with various locations on the device / instrument). Various examples calculate distance(s) between location(s) of 3D marker(s) and tracked datum(s) of a device / instrument to monitor the marker(s) relative to the position(s) of the device / instrument. Additionally, examples generate user notification(s) (e.g., visual aid(s) and / or alert(s), etc.) based on the tracked device / instrument avatar(s), 3D marker(s), and calculated distance(s). Additionally, while a device avatar is used in many examples, some examples maintain and track annotation(s) and / or other marker(s) without also tracking a device avatar (e.g., in examples wherein there is not an instrument with a tracked position / orientation, such as involving instruments without embedded tracker(s), for pre-procedural planning, etc.).

[0005] In various examples, the position(s) of feature(s) (e.g., a valve annulus such as a mitral valve annulus, sub-valvular feature(s), etc.) during rapid pacing is determined and tracked in the 3D model. In some examples, the rapid pacing position(s) of feature(s) are determined based on US imaging captured in connection with pre-procedural rapid pacing (e.g., testing the effectiveness of temporary pacing prior to the procedure, etc.). In the same or other examples, US imaging is gated to a cardiac cycle (e.g., based on an electrocardiogram, etc.) such that US imaging data (e.g., still image(s), recording(s), etc.) is captured of portion(s) of the cardiac cycle (e.g., the same PQR phase / position, etc.) where the feature(s) occupy the same position as during rapid pacing. Based on the US imaging of the rapid pacing position(s) of the feature(s), markers are added to the 3D model to track the rapid pacing position(s) of the feature(s). The rapid pacing position(s) of the marker(s) of feature(s) tracked in the 3D model are able to be displayed via an output image at various points during a procedure (e.g., based on user selection, etc.). Additionally, simulated rapid pacing of feature(s) is able to be displayed based on live US imaging of normal cardiac pacing, by outputting displayed image(s) showing US imaging from portion(s) of the cardiac cycle (e.g., the same PQR phase / position, etc.) where the feature(s) occupy the same position as during rapid pacing. In various examples, inresponse to the tracked marker(s) of feature(s) not being aligned with the associated feature(s) (e.g., based on movement of the heart or other region of interest, based on movement of the entire patient, etc.), the position(s) of the marker(s) are updated manually or automatically to align with the feature(s) tracked by the marker(s). In various automatic examples, an algorithm is implemented where visible portions of the feature(s) (e.g., valve annulus, etc.) are detected and used to update the position(s) of the marker(s) of the feature(s).

[0006] In one aspect, the present disclosure provides a user interface system for imaging a region of interest for a procedure that includes a memory for storing machine- readable instructions and a processor core for accessing the machine-readable instructions and executing the machine-readable instructions as operations. The operations include receiving ultrasound (US) data for the region of interest from a US probe. The operations also include constructing a three-dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The operations additionally include displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0007] In another aspect, the present disclosure provides a method for providing imaging data of a region of interest for a procedure. The method includes receiving ultrasound (US) data for the region of interest from a US probe. The method also includes constructing a three-dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The method additionally includes displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0008] In an additional aspect, the present disclosure includes a non-transitory machine-readable medium having machine executable instructions for a user interface system that causes a processor core to execute operations. The operations include receiving ultrasound (US) data from a US probe for a region of interest for a procedure. The operations also include constructing a three-dimensional (3D) model of the region ofinterest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The operations additionally include displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0009] In a further aspect, the present disclosure includes a user interface system for an intracardiac transcatheter procedure that includes a memory for storing machine- readable instructions and a processor core for accessing the machine-readable instructions and executing the machine-readable instructions as operations. The operations include receiving ultrasound (US) data from an echocardiogram probe for a region of interest for the intracardiac transcatheter procedure. The operations also include constructing a three- dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The operations additionally include displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0010] In additional aspects, the present disclosure includes a method for providing imaging data of a region of interest of an intracardiac transcatheter procedure. The method includes receiving ultrasound (US) data from an echocardiogram probe for a region of interest for the intracardiac transcatheter procedure. The method also includes constructing a three-dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The method additionally includes displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0011] Other aspects of the present disclosure include a non- transitory machine- readable medium having machine executable instructions for a user interface system that causes a processor core to execute operations. The operations include receiving ultrasound (US) data from an echocardiogram probe for a region of interest for the intracardiac transcatheter procedure. The operations also include constructing a three-dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The operations additionally include displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0012] Further aspects of the present disclosure include an ultrasound (US) system for imaging a region of interest for a procedure. The US system includes a US probe comprising a US transducer configured to obtain US data for the region of interest. The US system additionally includes a memory for storing machine-readable instructions and a processor core for accessing the machine-readable instructions and executing the machine- readable instructions as operations. The operations include receiving the US data for the region of interest from the US probe. The operations also include constructing a three- dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The operations additionally include displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0013] Still further aspects of the present disclosure include an ultrasound (US) system for imaging a region of interest for an intracardiac transcatheter procedure. The US system includes an echocardiogram probe comprising a US transducer configured to obtain US data for the region of interest. The system further includes a memory for storing machine-readable instructions and a processor core for accessing the machine- readable instructions and executing the machine-readable instructions as operations. The operations include receiving the US data for the region of interest from the echocardiogram probe. The operations also include constructing a three-dimensional (3D) model of the region of interest based on the US data. The 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model. The operations additionally include displaying an output image. The output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3Dmodel corresponding to the portion of the region of interest. The portion of the 3D model comprises the marker.

[0014] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0015] FIG. 1 is a schematic perspective view of one example of a treatment system useable in connection with various examples discussed herein.

[0016] FIG. 2 is a schematic block diagram illustrating an example system of hardware components capable of implementing examples of the systems and methods disclosed herein.

[0017] FIG. 3 is a series of images showing stages of a transcatheter mitral valve replacement procedure.

[0018] FIG. 4 is a pair of images showing adding markers to a 3D model of a region of interest for later use.

[0019] FIG. 5 is two images of a marked portion of anatomy with movement tracked based on electrocardiography (ECG) data.

[0020] FIG. 6 shows images of a transcatheter device in the left atrium along with tracked markers.

[0021] FIG. 7 shows an ultrasound (US) image of tracked markers for an inter-atrial septum and an open distal edge of a transfemoral sheath.

[0022] FIG. 8 is two images showing commissural, anterior, and posterior points marked along with a shape of the mitral valve annulus and a center axis of the annulus.

[0023] FIG. 9 is three images with the commissural, anterior and posterior points marked along with additional marked points around the mitral valve annulus.

[0024] FIG. 10 shows five example images of static annotation of the annulus of a mitral valve.

[0025] FIG. 11 shows five example images of annotation of the annulus of a mitral valve that tracks motion of the mitral valve.

[0026] FIG. 12A is two images of an effective orifice area (EOA) with automatically determined markers placed around the perimeter of the EOA.

[0027] FIG. 12B is an example image of a device avatar representing the position of a delivery system within a determined perimeter of an implant frame.

[0028] FIG. 13 shows two images of anatomical features of the left ventricle that present potential hazards during transcatheter mitral valve replacement (TMVR), along with an echo view showing the difficulty in visualizing sub-ventricular hazards.

[0029] FIG. 14 shows images of annotation of papillary muscle mobility zones on computed tomography (CT) and echocardiography along with display of annotations in connection with a device avatar.

[0030] FIG. 15 is an image of a mitral valve and left ventricle annotated with the mitral valve plane and ventricular distance along with an ultrasound image of a device avatar with mitral valve plane and ventricular distance annotations.

[0031] FIG. 16 is an image of a device avatar with sub- valvular feature annotations following the cardiac cycle to represent the expected movement of the features.

[0032] FIG. 17 A is an image showing example datum points, reference planes, and rings useable in connection with various device avatars.

[0033] FIG. 17B is three images showing examples of displaying distances from a defined point of an avatar.

[0034] FIG. 17C is an image showing examples of displaying distances from an anatomical feature tracked in the 3D model.

[0035] FIG. 18 is an example image showing a widget (lower right) that facilitates navigation assistance, displayed in connection with an imaging panel (left).

[0036] FIG. 19 shows three images of thresholds used in connection with smart device avatars.

[0037] FIG. 20 is a pair of images showing an expected deployment depth for an avatar.

[0038] FIG. 21 illustrates a pair of images showing the position of the mitral valve at various phases of cardiac cycles.

[0039] FIG. 22 is a flowchart of an example method for providing US imaging data for a region of interest via a user interface.

[0040] FIG. 23 is a flowchart of an example method for displaying the rapid pacing position(s) of feature(s) based on US imaging data for a region of interest.

[0041] FIG. 24 is a flowchart of an example method for displaying the rapid pacing position of a valve annulus based on US imaging data in connection with a valve replacement procedure.DETAILED DESCRIPTION

[0042] Various examples provide a user interface for an ultrasound (US) imaging system, such as a transesophageal echocardiogram (TEE) imaging system, as well as imaging systems employing such a user interface. Examples generate a three-dimensional (3D) model that tracks US imaging data of a region of interest.

[0043] Various examples determine stationary (e.g., or substantially stationary, etc.) position(s) (e.g., a rapid pacing position) associated with feature(s) (e.g., mitral valve annulus, sub-valvular hazards, etc.) based on US imaging of those features. The stationary (e.g., rapid pacing) position(s) associated with feature(s) is tracked via marker(s) in the 3D model in various examples and is able to be output via a display, including when those feature(s) are in periodic motion over a cycle (e.g., a normal cardiac cycle, etc.). In various examples, simulated stationary position(s) (e.g., simulated rapid pacing position, etc.) of feature(s) is able to be displayed during periodic motion (e.g., normal cardiac pacing) of a cycle (e.g., a normal cardiac cycle, etc.) based on displaying US imaging corresponding to a portion of the cycle (e.g., cardiac cycle phase, position on a PQRST complex, etc.) where the feature(s) occupy the stationary position(s) (e.g., rapid pacing position(s), etc.) associated with those feature(s). The marker(s) tracked in the 3D model are able to be updated manually and / or automatically during a procedure in various examples, such as via comparison of the live imaging (or portions of the available live imaging) with the simulated stationary position(s) (e.g., simulated rapid pacing position(s), etc.) of those feature(s).

[0044] Current clinical methods of imaging the intracardiac space using TEE have demonstrated feasibility in connection with transfemoral mitral valve replacement, and these methods are also useable in tricuspid valve replacement. However, clinical experience has identified challenges with conventional TEE imaging, which requiresignificant and constant monitoring by multiple physicians and multiple Medtronic clinical specialists in every implant procedure.

[0045] In general, these issues add significant cognitive burden and stress during the procedure because of the high level of vigilance required to avoid contact between the delivery system and anatomical features that are inherently difficult to see with typical echocardiogram (“echo”) views, cropped out of view, or shadowed by the delivery system itself. Patient outcomes are highly dependent on the vigilance of multiple specialists and effective communication between them.

[0046] In contrast to conventional techniques, various examples generate and maintain a 3D model of a region of interest (e.g., a heart or portion thereof involved in a transcatheter procedure, etc.) based on US data (e.g., two-dimensional (2D), 3D, fourdimensional (4D)) obtained from a US transducer (e.g., a US transducer of a TEE probe or other US probe), including portions of the region of interest not viewable based on the current position of the US probe and instrument(s). In various examples, a set of markers is stored on the 3D model. In various examples, the set of markers includes one or more of annotation(s), edge(s), hazard(s), etc. Depending on the example, a marker is generated based on user input and / or automatically generated (e.g., in some examples with automatically generated markers or shapes, those markers / shapes are subject to user review, modification, or approval, etc.). In various examples, a marker is connected to structure or another potentially mobile portion of the region of interest, and the 3D model tracks motion (e.g., periodic motion over a cycle, such as motion of a heart valve over the cardiac cycle, etc.) in the region of interest, including the marker tracking motion of the marked portion of the region of interest. In various examples, the tracked markers are tracked based on known periodic motion (e.g., from an echo loop over a cardiac cycle), providing location information on the marked portion of the region even when not currently visible with US or other imaging (e.g., by combining motion over the cardiac cycle with a known portion of the cardiac cycle from an electrocardiogram (ECG), etc.). Additionally or alternatively, stationary position(s) (e.g., rapid pacing position(s), etc.) associated with feature(s) are tracked in the 3D model, including during periodic motion (e.g., of a normal cardiac cycle, etc.) of the feature(s). The tracked stationary position(s) in the 3D model are able to be output via a display during a procedure, and are able to beupdated (e.g., manually and / or automatically) based on simulated stationary position(s) of the feature(s) that are determined from live US imaging data.

[0047] In various examples, the position and orientation of an instrument (e.g., transcatheter device, etc.) is tracked via a set of EM trackers at different positions on the instrument, and the 3D model includes a device avatar that represents the position and orientation of the instrument. The instrument in various examples is associated with a procedure, and in many such examples the shape of the instrument (e.g., transcatheter device) has a geometry that changes based on the stage of the procedure (e.g., a replacement heart valve, with different shapes at one or more of pre-deployment, middeployment, full-deployment, post-deployment, etc.). In such examples, the appearance (e.g., dimensions, alignment feature(s) based on current and / or future stage(s) of a procedure, etc.) of the instrument / device avatar in the 3D model are based on the geometry of the instrument / device at the current stage of the procedure (e.g., and potentially additional information, such as alignment features generated based on a detected procedure and stage(s) of the procedure, etc.).

[0048] Based on the tracked marker(s) and instrument(s) in the 3D model, various examples monitor distance(s) between marker(s), shape(s) (e.g., user and / or automatically generated shape(s) indicating structure(s) such as valve annulus(es), etc.), guideline(s) (e.g., user and / or automatically generated annotations related to a procedure or stage(s) thereof, such as a minimum height for a septal puncture, etc.) and / or instrument(s), and generate user notification(s) (e.g., indication of the distance(s), alert(s), etc.). In various example(s), notification(s) are generated based on comparing a distance (e.g., between two markers / shapes, between a marker / shape and the device or a portion thereof, etc.) to a threshold, such as whether the distance is less than a first threshold distance from a marker (e.g., a transcatheter device being too close to the bottom of a ventricle, etc.), greater than a second threshold distance from a marker (e.g., a rear of a transcatheter device is sufficiently past a septal puncture to turn toward a mitral valve, etc.), equal to a third threshold value, or combinations thereof (e.g., a replacement valve is within a range of acceptable locations for deployment, etc.). In other examples, notifications are generated based on alignment of angles (e.g., comparison of an angle between shape(s), guideline(s), or instrument(s) to one or more threshold values, such as whether it is greater than, equal to, and / or less than threshold value(s), etc.). In further examples, notifications aregenerated based on the intersection or non-intersection of shapes, guidelines, or instrument(s) (e.g., generating a notification when an alignment feature for a future deployment stage of a procedure (e.g., indicating an expected deployment location based on instrument position / orientation, etc.) intersects marked anatomy, etc.). In the same or other examples, notifications are generated based on an identity or other information associated with a marker / shape (e.g., an identity or category of a marker / shape can indicate whether to avoid, measure a distance / angle relative to, contact, etc. the marker / shape).

[0049] In some examples, marker(s) / shape(s) / guideline(s) on the 3D model are displayed on output images (e.g., as annotation overlay ed on output US images) for some procedure stages and not displayed for other procedure stages

[0050] Additionally, two or more markers are useable to define shapes that are tracked on the 3D model, such as valve hinge points from which a valve structure is determined automatically and / or based on user input, etc.

[0051] In some examples, marker(s) are useable to track stationary position(s) (e.g., rapid pacing position(s), etc.) of feature(s) (e.g., valve annulus, sub-valvular feature(s), etc.). In various examples, marker(s) to track stationary position(s) of feature(s) are generated based on preprocedural US imaging (e.g., US imaging in connection with a rapid pacing test of the effectiveness of temporary pacing for use during the procedure, etc.) and / or intraprocedural imaging. In various examples, marker(s) tracking stationary position(s) of feature(s) are useable during a procedure in various ways. In some examples, marker(s) tracking stationary position(s) of feature(s) are displayed (e.g., in connection with a still image associated with the marker(s), overlaying live US imaging, etc.) to offer user guidance, such as for navigating a tool (e.g., transcatheter delivery system, etc.) into position for a stage of the procedure. In the same or other examples, marker(s) tracking stationary position(s) of feature(s) are used to generate alerts to a user (e.g., including when not displayed), such as to indicate when a tool is in a correct position for a next stage of a procedure (e.g., a replacement valve delivery system aligned with a valve annulus at an appropriate depth for deployment, etc.) or an incorrect / potentially hazardous position (e.g., a replacement valve delivery system is misaligned with a valve annulus, would potentially impact papillary muscles or other sub-valvular hazards if deployed, etc.).

[0052] The stationary position(s) of feature(s) are able to be simulated based on live US imaging of the features during periodic motion (e.g., of a normal cardiac cycle, etc.) at various stages of the procedure, by displaying imaging frames where the feature(s) occupy the stationary position(s), based on the timing of the imaging frames relative to the periodic motion (e.g., with the US imaging gated to an electrocardiography signal, imaging frames corresponding to portions of a PQRST complex where the feature(s) occupy their rapid pacing position(s), etc.). In various examples, the simulated stationary position(s) of feature(s) is used to update marker(s) tracking the stationary position(s) of the feature(s) in response to the marker(s) being misaligned with the simulated stationary position(s) of the feature(s) (e.g., based on physical movement of a patient or the heart of the patient during a procedure, etc.). Depending on the example, updating the position(s) of the marker(s) is performed one of manually, automatically, automatically subject to user review and / or approval, etc.

[0053] Referring to FIG. 1, illustrated a schematic perspective view of one example of a treatment system 10 useable in connection with various examples. Treatment system 10 includes a computing device 100, a display 110, a table 120, an instrument 130 (e.g., a transcatheter device, etc.), an ultrasound (US) imager 140 that includes a US probe (e.g., a transesophageal echocardiogram (TEE) probe), and a US workstation 150. In various examples, computing device 100 is, for example, a laptop computer, desktop computer, tablet computer, smart phone, or other similar device. In examples, computing device 100 is configured to control an electrosurgical generator, a peristaltic pump, a power supply, and / or any other accessories and peripheral devices relating to, or forming part of, system 10. Display 110 is configured to output an image based on an ultrasound (US) image, which in various examples is output along with additional information, such as by overlaying a portion of a 3D model (e.g., saved marker(s), an avatar of an instrument or device, tracked distance(s), user notification(s) / alert(s), US imaging parameters, etc.). In various examples, table 120 is, for example, an operating table or other table suitable for use during a surgical procedure that includes an electromagnetic (EM) field generator 121. EM field generator 121 is used to generate an EM field during the procedure and forms part of an EM tracking system that is used to track the position and orientation of the US probe (e.g., TEE probe, etc.) of the US imager 140 within a region of interest (e.g., an anatomical region associated with a procedure within the body of a patient, such as a heartor portion thereof, etc.) and / or the position and orientation of the instrument 130. EM field generator 121 may include various components, such as a specially designed pad to be placed under, or integrated into, an operating table or patient bed. While the present disclosure describes the use of system 10 in a surgical environment, it is also envisioned that some or all of the components of system 10 may be used in alternative settings, for example, an imaging laboratory and / or an office setting.

[0054] In addition to the EM tracking system, the instrument 130 may also be visualized by using ultrasound imaging. US imager 140, which includes the US probe, is useable to image the patient's body during the procedure to visualize the position of the surgical instruments, such as instrument 130, inside the patient's body. In various examples, the US imager 140 has an EM tracking sensor embedded within or attached to the US probe, for example, a clip-on sensor, or a sticker sensor. As described further herein, the position / orientation of the US probe determined by the EM tracking system, in combination with the US image data obtained by the US imager 140, allows for the ability to construct a 3D model of a region of interest based on the US image data along with the position / orientation of the US probe relative to the region of interest. The position / orientation of the instrument 130 determined by the EM tracking system (e.g., based on a set of EM trackers at various locations on the instrument 130) and / or by the US imager 140 are used in various examples to register the position / orientation of the instrument 130 to the 3D model for tracking. In various examples, one or more US sensors 140 are positioned near a region of interest, such as placed inside the body of the patient (e.g., a TEE probe placed inside the esophagus of the patient) or outside the body of the patient (e.g., a transthoracic echocardiogram (TTE) probe placed on the chest or abdomen. The EM tracking system tracks the position of the instrument 130 (and optionally the US imager 140) inside the body of the patient.

[0055] The position of the instrument 130 within the body of the patient may be tracked during the surgical procedure. An example method of tracking the position of the instrument 130 includes using the EM tracking system, which tracks the position of the instrument by tracking sensors attached to or incorporated in the instrument. Various types of sensors are useable, such as a printed sensor, the construction and use of which is more fully described in U.S. Patent Publication No. 2016 / 0174873, entitled MEDICAL INSTRUMENT WITH SENSOR FOR USE IN A SYSTEM AND METHOD FORELECTROMAGNETIC NAVIGATION, the entirety of which is incorporated by reference herein. Prior to starting the procedure, the clinician can verify the accuracy of the tracking system and / or US imager 140 using any suitable technique or techniques, for example, as described in U.S. Patent 8,811,662, entitled METHOD AND APPARATUS FOR CALIBRATING AND RE- ALIGNING AN ULTRASOUND IMAGE PLANE TO A NAVIGATION TRACKER, the entirety of which is incorporated by reference herein.

[0056] For ease of illustration, specific examples are discussed in connection with a transcatheter device as the instrument 130, in connection with a transcatheter mitral valve replacement (TMVR) procedure. However, in various examples, any suitable instrument or device 130 can be utilized with the system 10, e.g., one or more implantable devices, implant delivery devices, therapy delivery devices, surgical devices, mechanical circulatory support (e.g. LVAD) devices, coronary stent devices, heart valve devices, heart valve repair devices, cardiac ablation devices, cardiac lead devices, drug delivery devices, catheter delivery devices, and endoscopic delivery devices.

[0057] FIG. 2 is a schematic block diagram illustrating an example system 200 of hardware components capable of implementing examples of the systems and methods disclosed herein. The system 200 can include various systems and subsystems, and in some examples is employable as the computing device 100. The system 200 can be a personal computer, a laptop computer, a workstation, a computer system, an appliance, an application-specific integrated circuit (ASIC), a server, a server BladeCenter, a server farm, etc.

[0058] The system 200 can include a system bus 202, a processing unit 204, a system memory 206, memory devices 208 and 210, a communication interface 212 (e.g., a network interface), a communication link 214, a display 216 (e.g., a video screen), and an input device 218 (e.g., a keyboard, touch screen, and / or a mouse). The system bus 202 can be in communication with the processing unit 204 and the system memory 206. The additional memory devices 208 and 210, such as a hard disk drive, server, standalone database, or other non-volatile memory, can also be in communication with the system bus 202. The system bus 202 interconnects the processing unit 204, the memory devices 206- 210, the communication interface 212, the display 216, and the input device 218. In some examples, the system bus 202 also interconnects an additional port (not shown), such as a universal serial bus (USB) port.

[0059] The processing unit 204 can be a computing device and can include an application-specific integrated circuit (ASIC) and / or include one or more processing cores (e.g., single-core, multi-core), which in various examples include CPU(s), GPU(s), etc. The processing unit 204 executes a set of instructions to implement the operations of examples disclosed herein. The processing unit can include a processing core.

[0060] The additional memory devices 206, 208, and 210 can store data, programs, instructions, database queries in text or compiled form, and any other information that may be needed to operate a computer. The memories 206, 208 and 210 can be implemented as computer-readable media (integrated or removable), such as a memory card, disk drive, compact disk (CD), or server accessible over a network. In certain examples, the memories 206, 208 and 210 can comprise text, images, video, and / or audio, portions of which can be available in formats comprehensible to human beings. Additionally, or alternatively, the system 200 can access an external data source or query source through the communication interface 212, which can communicate with the system bus 202 and the communication link 214.

[0061] In operation, the system 200 can be used to implement one or more parts of a system in accordance with examples discussed herein. Computer executable logic for implementing the user interface system and / or US imaging system resides on one or more of the system memory 206, and the memory devices 208 and 210 in accordance with certain examples. The processing unit 204 executes one or more computer executable instructions originating from the system memory 206 and the memory devices 208 and 210. The terms “computer readable medium” or “machine readable medium” as used herein includes a medium that participates in providing instructions to the processing unit 204 for execution and in various examples includes non-transitory, volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD- ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 100. In one or more embodiments, computer-readable storage media can be stored in the cloudor remote storage and accessed using any suitable technique or techniques through at least one of a wired or wireless connection.

[0062] In various examples, one or more software programs stored in at least one of the system memory 206, the memory device 208, or the memory device 210 include instructions that are executed by the processing unit 204 to perform operations associated with the examples.

[0063] Various examples are employable in connection with a range of procedures. One example use case is a clinical procedure for transcatheter mitral valve replacement (TMVR), but examples are also employable in connection with a range of other scenarios, including other intracardiac transcatheter procedures such as transcatheter edge-to-edge repair (TEER), annuloplasty, left atrial appendage occlusion (LAAO), etc.

[0064] One use case of examples discussed herein is TMVR, which involves multiple stages. In a transfemoral stage, access to the femoral vein is achieved via cutdown or percutaneous and a puncture is made in the atrial septum and dilated. In a transseptal stage, a sheath is introduced and advanced to the vena cava and the sheath is installed in the septal puncture. In a mitral valve replacement stage, a delivery system is passed through the sheath and into the left atrium (LA), the delivery system is manipulated into the mitral valve annulus, and the replacement valve is deployed. In a system removal stage, the delivery system is recombined and manipulated out of the implant and the delivery system is retracted into the sheath and removed from the patient.

[0065] Referring to FIG. 3, illustrated are a series of images showing stages of a TMVR procedure. At 310, the sheath is advanced into the LA. At 320, the catheter is advanced into the LA. At 330, the catheter is advanced across the mitral valve (MV) annulus. At 340, the brim of the replacement valve is expanded. At 350, the replacement valve is advanced to a target location. At 360, the valve fixation ring is expanded. At 370, the replacement valve is fully deployed. During deployment, rapid pacing is applied to maintain the valve annulus in a fixed position. At 380, the delivery system is closed and retracted.

[0066] As can be seen in FIG. 3, TMVR is a complex procedure involving a number of stages, many of which involve actions being performed at precise locations. Many of these actions occur at locations that are not co-planar with one another, but which are performed at distinct locations in the 3D cardiac environment. Additionally, cardiacanatomy continues to move during the cardiac cycle. However, conventional imaging systems do not track portions of a region of interest that are not shown on a current view, including portions that are not currently viewed because of a focus on other portions but which may present hazards, as well as portions not currently viewable (e.g., because they are shadowed by an instrument 130, etc. Additionally, with conventional systems users frequently rely solely on their memory of still image(s) or alternate view(s) (e.g., an en face view) when performing portions of a procedure that involve both a given view and information from the still image(s) or alternate view(s). These complications of conventional systems add significant cognitive burden and stress during a procedure because of the high level of vigilance required to avoid contact between an instrument (e.g., instrument 130, such as a transcatheter delivery system, etc.) and anatomical and other features (e.g., existing implants, etc.) that are inherently difficult to see with typical echo views, cropped out of view, or shadowed by the instrument itself.

[0067] To provide specific examples, a cardiac region of interest is discussed, along with examples applying to specific cardiac anatomy.

[0068] In various examples, marker(s) such as annotation(s), hazard(s), edge(s), stationary (e.g., rapid pacing, etc.) position(s) of feature(s), etc. are saved in connection with the 3D model of the region of interest, representing the 3D position of the marked portion(s) of the region of interest.

[0069] A first set of example use cases involve visualization of the left atrium (LA). In a TMVR procedure, atrium visualization is relevant during multiple procedure stages, such as the delivery system exiting a sheath in the left atrium, retracting the sheath to the inferior vena cava (IVC), and the delivery system crossing the annulus of the mitral valve (MV).

[0070] In conventional systems, with a large patient LA, the whole LA cannot fit in one view, such that the inter-atrial septum and LA lateral wall are not visible at the same time. Because of this, with conventional systems, such scenarios involve multiple probe positions and measurements to monitor the sheath / dilator tip / capsule distance past the septum and avoid hitting the anatomy. Additionally, conventional systems involve crosschecks of fluoroscopy and echo (e.g., TEE, TTE, etc.) imaging to know when the capsule fully crosses the septum.

[0071] In various examples, relevant or critical features are marked (e.g., via a user input based on a current 2D US image on the display 110, etc.). For example, when the septal puncture has been performed, marking is placed in-plane with the septal plane. As another example, a marking highlighting the location of the left atrial appendage osteum (LAAO) is added to the 3D model. A further example includes placing a marking on a tissue boundary such as the LA wall.

[0072] In various examples, these markers, and distances measured from them, are useable in future steps in the procedure. In various examples, the avatar of the delivery system (e.g., as instrument 130) includes built in measurement calculations relevant to the procedure. Examples of distance measurements used include: a capsule tip to marked feature(s), other delivery system datum(s) / threshold(s) relative to marked feature(s), distance between two tracked targets or marked features, etc. FIG. 4 illustrates a pair of images associated with adding markers to a 3D model of a region of interest for later use. The left image shows a first marking added at a location of a septal puncture and a second marking added at a location of a LAAO. The right image shows a later procedure stage, with a delivery system (e.g., as one example instrument 130) having an avatar with a ring near the rear of the delivery system avatar useable to determine whether the delivery system has advanced sufficiently far past the septal puncture, while the front of the avatar is close to the marked LAAO, with a distance monitored for potential future alert(s).

[0073] In various examples, one or more markers are locked onto a moving edge or feature of the region of interest, which is determined via user selection, edge detection, machine learning employed on color doppler edge(s), black and white echo edge(s), etc. Additionally, markers generated automatically are able to be adjusted by users. As the edge / feature the marker is locked onto moves (e.g., via a cardiac cycle, etc.), the marker moves with the edge / feature, such as based on edge detection, ECG data, etc.

[0074] Various examples use edge detection on US for dynamic tracking of a marker (e.g., a selected point of cardiac anatomy), including for visualizing anatomy not currently visible (e.g., on display 110) in the US view (e.g., from US imager 140). FIG. 5 shows two images of a marked portion of anatomy with movement tracked based on ECG data. The top image of FIG. 5 shows a marked target (the oval with the x), which can be marked on a still image of an echo loop. As the marked anatomy moves, the marked target moveswith the anatomy as shown in the bottom image of FIG. 5, based on edge detection and / or matching data from an echo loop with ECG data.

[0075] One example method of dynamic tracking involves capturing an echo loop (e.g., via US imager 140) of anatomy of interest and pausing the echo loop at a cardiac phase where the particular anatomy presents greatest risk. Using the selected still image on the 2D echo (e.g., shown on display 110), the point of interest is identified and a marker / target is added (e.g., via US workstation 150). In various examples, the target / marker includes a direction vector used to calculate positive / negative distances from the target. FIG. 6 shows images of a transcatheter device in the left atrium along with tracked markers. The left image of FIG. 6 shows a fossa identified at a point of minimum LA width (the oval ring, which can visually indicate user notifications, such as being blue to indicate to a user to keep advancing the instrument across the septum). Next, using image processing to detect an edge / boundary of contrast between tissue and fluid, the position of the placed target is tracked as it moves during the cardiac cycle. As the user returns to live echo (e.g., via computing device 100 and / or US workstation 150), the marked target remains visible in the 3D space and tracking with the ECG to maintain a visual representation (e.g., on display 110) of the position of the marked target during the live cardiac cycle, even if not currently visible on the US view. While the position of the implanting device is tracked (e.g., via computing device 100 and / or US workstation 150) the distance between the device and the target is calculated. Depending on the example and / or scenario, a measurement (e.g., distance to the tracked LAAO, etc.) is displayed on the screen (e.g., display 110) and / or an alert is generated to the user when the proximity reaches a threshold. The right image of FIG. 6 shows a scenario wherein the rear of the delivery system has crossed the septum but the front of the delivery system is within a threshold distance of the marked LAAO, triggering an alert (e.g., changing the color of the marked target to red, etc.). Although color change alerts are discussed in connection with FIG. 6, other alerts or notifications include distance measurements, arrows, etc.

[0076] In various examples, the method also includes allowing proximity tracking of multiple edge(s), marker(s), or instrument(s). Referring to FIG. 7, illustrated is a US image showing tracked markers for an inter-atrial septum and an open distal edge of a transfemoral sheath. User input identifying and marking edge(s) to be tracked is received (e.g., via computing device 100, US workstation 150, etc.), placing the markers in the 3Dmodel. Image processing is employed to detect an edge / boundary of contrast between solid and fluid, and the edge is continually tracked with the position of the marker(s) updated as the device and / or tissue moves. The distance between the marked edge(s) is monitored (e.g., via computing device 100, etc.), and user notifications are output (e.g., via display 110) as appropriate (e.g., measurement(s), alert(s) based on distance(s) being above and / or below threshold(s), etc.).

[0077] In some examples, markers are added as part of pre-procedure planning (e.g., via computing device 100, etc.). For example, pre-procedure computed tomography (CT) or echo imaging is obtained, and used to add markers (e.g., via US workstation 150, computing device 100, etc.) for stages of a procedure, such as a planned location or range of locations for a septal puncture (e.g., a line indicating a minimum puncture height, etc.). During or after the relevant stage of the procedure (e.g., when the septal puncture is made, etc.), the marker is updated (e.g., via computing device 100, etc.) to track the actual location used in the procedure.

[0078] The first set of example use cases also includes visualization of relevant anatomy, such as annulus visualization. Accurate determination of the annulus is relevant in multiple TMVR stages, such as the delivery system crossing the annulus, alignment (e.g., axial alignment) of the implant with the annulus plane, and advancement of depth toward the annulus.

[0079] In conventional systems, the annulus / leaflet insertion are marked on a still image captured during rapid pacing. The user relies on their memory of the still image when they switch back to live echo to continue the procedure. Some disease states can “trick” the eye to a false leaflet hinge point (e.g., primary disease or annular disjunction), making it difficult to remember the “true” annulus. Additionally, only two planes are visible on biplane imaging during deployment, therefore only four points of the annulus perimeter are used to cross the annulus and to make a judgment call on the positioning. Having a full picture of each patient’s unique annulus shape curtails unwanted contact with the edge of the annulus during crossing and to make a more informed, individualized decision on implant placement including irregularities in the annulus.

[0080] Various examples overlay markers and / or a shape with the echo view for visualization of the MV annulus, such as an outline or points along an annulus saddle, and / or differentiated markers for commissural and anterior / posterior points for the peaksand valleys of the saddle shape. FIG. 8 shows two images with the commissural, anterior, and posterior points marked along with a shape of the MV annulus and a center axis of the annulus. FIG. 9 shows three images with the commissural, anterior and posterior points marked along with additional marked points around the MV annulus.

[0081] Continuing with the first set of example use cases, various examples facilitate 3D annotation of a shape of an anatomical structure (e.g., 3D annulus annotation, etc.) for display on biplane view (e.g., via display 110, etc.), such as the annular plane and / or perimeter. One example 3D annulus annotation method includes capturing biplane images and a four-dimensional (4D) echo loop of the annulus and / or target osteum (e.g., via US imager 140). In various scenarios, this loop is taken during rapid pacing or a normal cardiac cycle. Using the 2D echo planes, a user defines points in at least four quadrants of interest around annulus (e.g., via computing device 100, US workstation 150, etc.). In some examples, the user places the four points individually. In other examples the user draws two lines, one on each respective echo plane (e.g., from the medial (M) to lateral (L) commissure and from the anterior (A) to posterior (P) hinge points), the endpoints of the lines remain marked on the 3D model for four total points, and the connecting lines are useable as visual aids. In further examples, the hinge points are automatically identified (e.g., via computing device 100, etc.), and are able to be modified, rejected, or approved by a user. On the 4D echo loop and / or by exploring other planes in 2D (e.g., via computing device 100, US workstation 150, etc.), a user manually selects the remainder of the annulus perimeter (e.g., as points or a perimeter line). One example interface for point selection allows a cursor (e.g., on display 110) controlled by a user (e.g., via computing device 100) to track on the surface of the tissue as seen in 4D, and when marking the point, the cursor determines the X-Y position (parallel with the screen) and the depth into the screen is automatically determined (e.g., via computing device 100). 4D echo colorization is based on distance from the probe, so this depth data is able to be fetched (e.g., by computing device 100 from US workstation 150, etc.) as input to the point placement. In other examples, the remainder of annulus perimeter (pictured as points or a perimeter line) is automatically determined (e.g., by computing device 100) as the boundary edges of contrast between the closed and the open mitral valve or between static and dynamic tissue (as the leaflets are more mobile than atrium wall). In various examples, the detection algorithm is run on the echo loop or on the live echo. Theexample annulus annotation method also includes generating a visual aid representing the deployment target (e.g., output via display 110) based on the perimeter points or connecting lines. In various examples, the visual aid includes line(s) and / or spline(s) that are auto-generated (e.g., by computing device 100) between the points to simulate the annular plane, an “average” plane, or another visual aid shape to aid in mating up the shape of the implant / device (e.g., of instrument 130). The example annulus annotation method further includes returning to the live echo, where the markers (e.g., targets and visual aids) remain visible (e.g., via display 110) in the 3D space representing the target implantation position as seen and selected from the echo loop.

[0082] FIG. 10 shows five example images of static annotation of the annulus of a mitral valve. The top two images are a biplane view showing the mitral valve closed (left) and open (right). The middle images show a surgeon’s view of the mitral valve closed (left) and open (right). The bottom image shows static markers (e.g., targets) representing the rapid-paced annulus location and overlaid with the live echo view in 3D space.

[0083] The first set of example use cases also include using background analysis of the 2D and / or 4D echo to dynamically auto-update the position of annulus annotation for display on a biplane view. One example method of dynamic auto-update of annulus annotation includes receiving (e.g., by computing device 100, based on user selections of echo imaging on display 110) user-defined points of interest (e.g., via a biplane view), similarly to static annulus annotations. Additionally, a surface / edge of contrast analysis algorithm is employed (e.g., by computing device 100, etc.) to track the defined points as they move throughout the cardiac cycle. The annulus perimeter annotation is completed by a user or automatically (e.g., via computing device 100), similarly to with static annotation. On returning to live echo, the targets remain visible in the 3D space and are moved to continuously track the annulus position based on the live analysis (e.g., by computing device 100) of the annulus edge on the 4D echo. In various examples, analysis of the 4D echo may occur in the background while viewing 2D echo, or on the current live view. In scenarios where only the annotation of four quadrants is desired, the 2D echo view(s) are useable for the analysis of dynamic motion (e.g., by computing device 100, etc.). In scenarios where a portion of the annulus is shadowed by the device used during the procedure (e.g., the instrument 130, such as a transcatheter delivery system or other transcatheter device, etc.), the live annulus tracking is combined (e.g., by computingdevice 100 and / or US workstation 150, etc.) with a previously-captured echo loop (e.g., via US imager 140 and US workstation 150) that preceded insertion of the device. The gaps that may be shadowed during the live cardiac cycle are fillable (e.g., by computing device 100, for output via display 110, etc.) by tracking the previously-capture portion of the annulus to the ECG. In various examples, non-live points are indicated differently (e.g. color or pattern) to differentiate assumed positions as opposed to live points.

[0084] FIG. 11 shows five example images of annotation of the annulus of a mitral valve that tracks motion of the mitral valve. The top two images are a biplane view showing the mitral valve closed (left) and open (right). The middle images show a surgeon’s view of the mitral valve closed (left) and open (right). The bottom image shows markers (e.g., targets) representing the rapid-paced annulus location and overlaid with the live echo view in 3D space. The pulsatility of the markers / targets is matched to the cardiac cycle from the ECG in 3D space and / or continuously tracking based on the live analysis of the annulus edge on 4D echo.

[0085] Another example from the first set of example use cases involves identification and marking of an orifice (e.g., an effective orifice area, an annulus, a septal defect, an appendage, etc.), which is relevant in procedures during delivery system removal postdeployment. In examples wherein the instrument or transcatheter device (e.g., instrument 140) is a delivery system (e.g., for a replacement valve for TMVR), the delivery system is EM tracked (e.g., at multiple points to determine multiple datums, etc.), but the implant is not EM tracked. Additionally, the delivery system capsule tip is not easily visible on echo due to shadowing. In conventional systems, contact between the delivery system and implant frame is visualized on fluoroscopy and typically limited to only one plane.However, depth perception is not feasible on fluoroscopy.

[0086] One example method of identification and marking of an orifice involves uses color doppler echo imaging (e.g., from US imager 140 and US workstation 150) to detect boundary edges (e.g., via computing device 100) such as the perimeter of an effective orifice area (EOA) and automatically apply targets (e.g., markers) to the boundary (e.g., via computing device 100). FIG. 12A shows two images of an effective orifice area (EOA) with automatically determined markers placed around the perimeter of the EOA. The perimeter is recognized (e.g., by computing device 100) by the boundary of contrast between open prosthetic valve and the fluid flow colorized on the doppler (in variousexamples, the detection algorithm is run on the echo loop or on the live echo). In various examples, the determined markers are subject to user review, modification, etc. After targets are determined and / or approved, upon returning to live echo the targets remain visible (e.g., via display 110) in the 3D space (based on the 3D model tracked by computing device 100) and are continuously tracked with motion throughout the cardiac cycle (e.g., by computing device 100 for output via display 110, etc.), which in various examples is based at least in part on locking onto the inflow and / or outflow crowns of the implant frame (or equivalently visible features of implanted device), which are high contrast and more easily detected by the algorithm (e.g., as implemented via computing device 100). In some examples, machine-based identification and / or marking of the location of a device is additionally or alternatively based on locking onto the inflow and / or outflow crowns of the implant frame is employed in addition to or instead of color doppler imaging. In the same or other examples, the identification and / or marking is performed by a user (e.g., optionally based on machine-generated suggestions), for example, based on color doppler imaging, etc. Based on the identified and / or marked location, in some examples, the algorithm aligns a virtual representation of a deployed implant with the identified inflow and / or outflow crowns of the implant frame. FIG. 12B illustrates an example image of a device avatar representing the position of a delivery system within a determined perimeter of an implant frame. The markers are useable as a visual aid for manual navigation and / or generation of notifications (e.g., distance between delivery system and implant frame) and / or alerts based on the device avatar being within a threshold distance from the implant frame, curtailing unintended contact between the delivery system and implant frame while withdrawing the capsule post. Similar techniques are employable in connection with other orifices with a clearly defined flow through the orifice.

[0087] Additionally, the first example set of use cases includes applications to visualization of sub-valvular features or other features that are difficult to image during a procedure. These aspects are relevant to procedure steps that include the delivery system crossing the annulus, assessing the delivery system trajectory, and deploying the implant.

[0088] For example, papillary muscles in the left ventricle are difficult to image on TEE for multiple reasons. Because only two planes are in view at one time, the view may be shadowed, and the papillaries are completely out of view when the view is truncated.

[0089] In various examples, anatomical features such as sub- valvular hazards (e.g., structures that an instrument 130 such as a transcatheter delivery system might potentially impact) are able to marked in 3D space for tracking on the 3D model (e.g., by computing device 100) prior to a procedure stage when the instrument (e.g., delivery system) is inserted (at which point the features are difficult to view), and the markers (e.g., output via display 110) are useable for reference relative to the device avatar. Features are markable based on imaging during an earlier procedure stage or based on pre-procedure imaging (e.g., CT, magnetic resonance imaging (MRI), US, etc.) co-registered to the 3D model. FIG. 13 shows two images of anatomical features of the left ventricle that present potential hazards during TMVR, which are not readily viewed during relevant procedure stages, along with an echo view showing the difficulty in visualizing sub-ventricular hazards.

[0090] In various embodiments, imaging from multiple probes, imaging modalities, etc. (pre-procedural imaging and / or intra-procedural imaging) are able to be registered to the same 3D model, and to be displayed or not displayed (e.g., on a current output image as an overlay on an output US image, etc.) based on user selection, automatically (e.g., based on procedure stage, etc.), automatically subject to user confirmation, etc. Marker(s), shape(s), etc. added via interaction with any imaging data co-registered to the 3D model are able to be added to the 3D model, such as by importing / co-registering the imaging data and marker(s) / shape(s). In such examples, imported marker(s) / shape(s) are able to be manually or automatically (e.g., which can be modified via user feedback, etc.) matched to other marker(s) in the model and / or to other imaging data (e.g., to US data). Additionally, in some examples, shape(s) and / or relative locations of markers are able to be manually or automatically (e.g., which can be modified via user feedback, etc.) transformed to fit other markers and / or imaging data (e.g., to account for structural changes between pre- procedural and intra-procedural imaging, etc.).

[0091] The first set of example use cases also includes an example method of marking ventricular hazard zones for static reference when not visible on live echo, such as papillary muscle mobility zones, pre-existing implants (e.g., pacemaker lead / ICD anchor, Micra), moderator band, irregular tissues, etc. The example method includes, prior to introducing the delivery system, identifying ventricular hazards (e.g., and / or “no-fly zones” to avoid with the delivery system), determining motion during the cardiac cycle of the identified hazards, and highlighting the region of movement and / or the point at whichthe hazards (e.g., papillary head, etc.) presents the greatest risk (e.g., the point when the hazard is closest to the annulus or other implant location). In various examples, this annotation, etc. is done manually by a user, using an artificial intelligence (Al) and / or machine learning (ML) algorithm (e.g., employed by computing device 100) to detect edges and recognize the expected shape of anatomy of interest and / or identify items that are recognized as not part of the native anatomy (e.g., an implant anchored in the ventricle wall). Multiple modalities are useable for the annotation, such as: pre-procedure CT to observe all phases of the cardiac cycle and locate the movement of the structure, either through importing annotations of the CT imaging into the 3D model (e.g., via computing device 100) or importing the CT scan into the 3D model followed by annotation on the 3D model (e.g., via computing device 100); TEE / TTE (e.g., via US imager 140 and US workstation 150) prior to the relevant stage(s) of the procedure when the annotation is to be employed. In some such examples, TEE / TTE is used as the primary method to identify structures, while in other examples imported annotations from CT analysis are the primary method to identify structures and TEE / TTE is used to confirm, overlay, and / or edit the annotation (e.g., via computing device 100) for accuracy on the live image (e.g., output via display 110). Following annotation and returning to the live echo view, the markers / targets remain visible in the 3D space. Based on the tracked device avatar and the marked hazards, the trajectory ahead of the device is analyzed to determine if the path is clear of annotations (except for any annotation(s) where the intention is to interact with those markers / targets). FIG. 14 shows annotation of papillary muscle mobility zones on CT (top left) and echo (top right) along with display of annotations in connection with a device avatar. While FIG. 14 shows the papillary muscle locations as an obstruction, similar techniques are employable with other hazards not viewable on live echo.

[0092] The first set of example use cases further includes an example method of marking ventricular ventricle orthogonal depth for static reference when not visible on live echo. The example method includes, prior to introducing the delivery system, identifying (e.g., manually or automatically, via US workstation 150 and / or computing device 100, etc.) the annular plane and the axis orthogonal to the annular plane, such as via techniques discussed herein. In some examples, the annotation is done using an Al and / or ML algorithm (e.g., employed by computing device 100, etc.) to detect edges and recognize the expected shape of the anatomy of interest. Based on observation of the cardiac cycle,the phase at which to annotate the ventricle depth is determined (e.g., manually or automatically, via US workstation 150 and / or computing device 100, etc.). As examples, the user selects a phase at which the ventricle is the “shortest,” that is, when there is the highest risk of interaction with the device, or the user views a loop of rapid pacing to determine the ventricle depth during rapid pacing, which may be representative of the conditions during an implant placement. The example method also includes annotating (e.g., manually or automatically, via US workstation 150 and / or computing device 100, etc.) the location at which the orthogonal axis intersects with the ventricle. In various scenarios, the axis is central to the valve or off-center, based on user determination of the axis at highest risk (e.g., the shortest distance) to the ventricle tissue. In various examples, the annotation includes directionality (e.g., a direction vector) to determine positive and negative direction from other objects such as the capsule. Depending on the example, annotation is done using one or more of multiple modalities, such as: (1) CT analysis prior to the procedure to observe all phases of the cardiac cycle and locate the orthogonal ventricle depth (e.g., manually or automatically, via US workstation 150 and / or computing device 100, etc.); (2) importing annotations from the CT scan into the 3D model; (3) importing the CT scan into the 3D model for annotation (e.g., manually or automatically, via US workstation 150 and / or computing device 100, etc.); (4) TEE / TTE prior to the relevant stage (e.g., at the start of the procedure, etc.) as the primary method for determining the ventricular depth; or (5) TEE / TTE prior to the relevant stage to edit annotations imported from CT analysis for accuracy. Upon returning to the live echo view, the markers / targets remain visible (e.g., via display 110) in the 3D space (as tracked by computing device 100 via the 3D model). Additionally, the trajectory ahead of device is analyzed (e.g., by computing device 100) to determine if the path is clear of the annotations (except for any annotation(s) where the intention is to interact with those markers / targets). In some examples, the ventricular depth is used to determine (e.g., by computing device 100) whether the delivery system or other instrument 130 is at a correct depth or within a range of correct depths for deployment. FIG. 15 illustrates an image of a mitral valve and left ventricle annotated with the mitral valve plane and ventricular distance (top image) and an US image of a device avatar with the MV plane and ventricular distance annotations (bottom image) for visual aids and / or generation of user notification(s).

[0093] Similarly, the first set of example use cases also includes an example method of employing cardiac cycle synchronization to view sub-valvular anatomy for dynamic reference when not visible on live echo. The example method includes annotating the papillary muscle heads and / or ventricle depth prior to introducing the delivery system (e.g., manually or automatically, via US workstation 150 and / or computing device 100, etc.), according to various techniques discussed herein. Additionally, “lock-on” techniques are employed using a surface / edge of contrast analysis algorithm (e.g., employed by computing device 100, etc.) to track the marked points to move with the anatomy throughout the cardiac cycle. The movement of the targeted anatomy is linked to the ECG phase (e.g., by computing device 100), based on a received ECG signal. Upon returning to live echo, the targets remain visible (e.g., via display 110) in the 3D space (e.g., based on the 3D model tracked by computing device 100). In scenarios where part of the anatomy is not visible in the cropped ultrasound image, the location of the targets is predicted (e.g., by computing device 100) based on the patient’s live ECG. Because the targets are linked to the previously-captured ECG data, the previous ECG is matched (e.g., by computing device 100) to the live ECG received and the targets are moved in accordance with their expected locations during the cardiac cycle. Additionally, the trajectory ahead of the device is able to be analyzed to determine if the path is clear of the annotations (except for any annotation(s) where the intention is to interact with those markers / targets). In some examples, the ventricular depth is used to determine (e.g., by computing device 100) whether the delivery system or other instrument 130 is at a correct depth or within a range of correct depths for deployment. FIG. 16 shows an image of a device avatar with sub-valvular feature annotations following the cardiac cycle to represent the expected movement of the features.

[0094] A second set of example use cases involve a smart device avatar to facilitate safe maneuvering of devices (e.g., instrument 130, such as a TMVR delivery system, etc.), which include employing device geometry datums, calculations, and / or user notifications / alerts. Various examples employ a smart avatar for device(s) / instrument(s) that includes measurement aids. Such aspects are relevant to procedure stages that include the delivery system exiting the sheath in the LA, the delivery system crossing the annulus, and delivery system trajectory assessment.

[0095] With conventional systems, current practice involves cross-checking fluoroscopy and echo to know when the capsule (e.g., of the transcatheter device for TMVR) fully crosses the septum. However, due to shadowing and / or the TEE probe location, the capsule tip may not be visible after beginning to add flex to the delivery system (e.g., to turn toward the MV). While it is easier to adjust the trajectory of the capsule prior to expanding the implant brim (because there is more room in the atrium when the brim is still crimped), the trajectory is easier to visualize after expanding the implant brim. Examples discussed herein simplify the procedure by improving trajectory visualization methods independent of the implant brim. Additionally, the ventricle is not visible when deploying the implant. As a result, it is difficult to predict if the capsule will contact anatomy.

[0096] The second set of example use cases involve employing a dynamic avatar that includes geometric aids that change (e.g., in the 3D model tracked by computing device 100, etc.) depending on the procedure step. In some examples, the geometric aids are not visible to the user (e.g., via display 110) at all times, but are used as datums for calculations (e.g., by computing device 100) that alert the user (e.g., via display 110) of measured distances and / or alignments relative to the avatar during procedure steps. While the specific design of the avatar varies in some examples or use cases (e.g., different procedures), the use of datum points, reference planes, and measurements that can be taken by including these references on the avatar is employable in connection with various different devices.

[0097] FIG. 17A is an image showing example datum points, reference planes, rings, etc. useable in connection with various device avatars. FIG. 17A shows rings that are useable near a rear of a device, such as for determining if the device is sufficiently past a septal puncture to begin flexing; rings around the device avatar to provide sufficient clearance relative to anatomy, implants, etc. and / or positioning for implants; distances in front of the device avatar for determining clearance relative to anatomy and / or implant placement, etc. FIG. 17B illustrates three images 1700, 1710, and 1720 showing examples of displaying distances from a defined point of an avatar. Various examples provide measurements and / or distances from selected points on avatars. In various examples, the measurements include one or more linear distances from the selected point(s) (e.g., as shown at image 1700, etc.), one or more spherical distances from the selected point(s)(e.g., as shown at image 1710, etc.), one or more hemispherical (or other portions of a sphere, etc.) distances from the selected point(s) (e.g., as shown at image 1720, etc.), one or more surfaces of constant distance from the surface of the avatar (e.g., every point that is a given distance from the surface of the avatar, etc.), etc. In some examples, the measurements / distances are provided (e.g., subject to user selection / preference, etc.) passively (e.g., visually displayed, etc.). In the same or other examples, alerts (e.g., visual, auditory, etc.) are generated based on the measurements / distances (e.g., based on a comparison to a threshold distance, etc.). Some example avatars for TMVR include monitoring of the capsule / device position / orientation in real-time (e.g., based on EM tracker(s) included on the capsule / device), along with one or more thresholds for distances such as a distance for crossing the septum, etc.

[0098] Similarly, in various examples, measurements and / or distances from selected points on anatomy (e.g., one or more anatomical points, regions, etc. tracked in the 3D model, etc.). FIG. 17C shows an image showing an example providing distances from a anatomical feature (e.g., an annulus in FIG. 17C) tracked in the 3D model. Similarly to measurements / distances from avatars, in various examples, the measurements include one or more linear distances from the selected point(s) (e.g., similarly to image 1700, etc.), one or more spherical distances from the selected point(s) (e.g., similarly to image 1710, etc.), one or more hemispherical (or other portions of a sphere, etc.) distances from the selected point(s) (e.g., as shown in FIG. 17C, etc.), one or more surfaces of constant distance from the surface of the avatar (e.g., every point that is a given distance from the surface of the avatar, etc.), etc. In some examples, the measurements / distances are provided (e.g., subject to user selection / preference, etc.) passively (e.g., visually displayed, etc.). In the same or other examples, alerts (e.g., visual, auditory, etc.) are generated based on the measurements / distances (e.g., based on a comparison to a threshold distance, etc.).

[0099] In various examples, the device is associated with a specific procedure (e.g., which in some examples is encoded in a plug-in of the delivery device), such that the system (e.g., computing device 100) is aware of the procedure step being performed (e.g., in response to connecting the plug-in of the delivery device). In some such examples, a user is prompted to complete or indicate completion of each procedure step in turn. In some examples, when the avatar passes a certain checkpoint (e.g., as determined by computing device 100 based on the 3D model, etc.), the user is prompted to acknowledgethe step is completed. When the procedure step changes, the visual aids and avatars are automatically advanced to those relevant to the next step(s). Additionally, in some examples, a path is defined (e.g., by a user, automatically subject to user review / revision / approval, etc.) in connection with one or more steps of the procedure. In various examples, the path is one or more of: a visual aid for a user, a potential trigger for alerts (e.g., based on an avatar being greater than a threshold distance from the path, etc.), a trigger to prevent motion of the device (e.g., based on an avatar being greater than a threshold distance from the path, etc.), and / or used in robotic examples to control movement of the device along the path.

[0100] The second set of example use cases also includes integrating determination of threshold crossing(s) with the device avatar. In various example methods, a threshold plane(s), ring(s), distance(s), and / or measurement(s) are included with the avatar (e.g., for at least the relevant procedure step(s)), such as those shown in connection with FIGS. 17A-B. The 3D position and directional vector of the device avatar is actively monitored and tracked in the 3D model (e.g., by computing device 100). One example device includes a threshold set at around 5.5 cm (e.g., 5.5cm ±10%, etc.) from the capsule tip. However, depending on use cases and scenarios, other examples are employed. In various examples, the avatar is sized according to the implant size indicated for the delivery system so that it matches the actual capsule length. In some examples, when the delivery system is connected to the system (e.g., system 10), the device size is encoded to automatically register the appropriate avatar (e.g., with computing device 100). While in some scenarios device threshold(s) are displayed (e.g., via display 110), in other scenarios, one or more thresholds associated with the device avatar are hidden from the user (or able to be hidden by the user) but are tracked in the 3D model for the purpose of calculating distance. In various examples, distance(s) are calculated in the background (e.g., by computing device 100 in connection with the 3D model) between the threshold and a tissue / marker to be crossed. In some scenarios, the measurement is displayed or optionally displayed (e.g., on display 110) and / or alert(s) are generated when the threshold is crossed (e.g., when a distance reaches 0, etc.). In some scenarios, the threshold crossing involves the entire ring passing across a plane, a certain percent of points around the ring passing across a plane, etc.. In some examples, during one or more steps of a procedure (e.g., one, some, or all steps) a widget is displayed (e.g., based on user preferences, etc.) inconnection with an imaging panel (e.g., adjacent to, inset, etc.), showing information that aids in navigation, such as a simplified alignment trajectory, placement of an avatar relative to one or more selected annotations, distances, angles, etc. Referring to FIG. 18, illustrated is an example image showing a widget (lower right) that facilitates navigation assistance, displayed in connection with an imaging panel (left). The widget in FIG. 18 shows a simplified alignment trajectory from an en face perspective, with the inner circle corresponding to the cross section of the device avatar and the outer dotted circle corresponding to the annotated mitral valve annulus. In various examples, widgets are provided as passive visual aids and / or are used to generate alerts (e.g., visual, auditory, etc.).

[0101] FIG. 19 illustrates images showing thresholds used in connection with smart device avatars. The top image of FIG. 19 shows a measurement threshold near the rear of a device avatar to determine whether the device has sufficiently crossed a septal puncture to begin flexing toward the MV. The bottom two images of FIG. 19 show how distance measurements of the device avatar or portions thereof relative to markers on tissue are used to determine proper device placement and / or generate alerts.

[0102] The second set of example use cases further includes integrating safe depth projection with a device avatar. One example method of integrating safe depth projection includes incorporating a deployment depth projection for a device (e.g., as determined by computing device 100 based on information obtained at device plug-in) into the device avatar. In various examples, the deployment depth projection indicates the maximum extension of the capsule during deployment and can vary depending on the instructions for use for the implant type. In various examples the extension is sized according to the implant size indicated for the delivery system so that it matches the actual capsule length. In various examples, when the delivery system is connected, the device size is encoded to automatically register the appropriate avatar. In various examples, the distance is calculated (e.g., in the background and / or output to the user via display 110) between the most distal surface of the projection and the ventricle orthogonal depth marker and / or other annotated hazards. When the capsule is expected to extend past the marker, various examples generate an alert for the user. Additionally, in some examples with electronic control of the device deployment mechanics, the system gates deployment until the depth is corrected.

[0103] Referring to FIG. 20, illustrated are a pair of images showing an expected deployment depth for an avatar. In the left image of FIG. 20, the expected deployment depth will not contact the marked ventricular depth and a user notification indicating deployment can proceed is generated. In the right image of FIG. 20, the expected deployment depth will contact the marked ventricular depth and a user notification indicating not to deploy the device at that position is generated.

[0104] A third set of example use cases include methods of tracking the rapid pacing position(s) of feature(s) (e.g., mitral valve annulus, sub-valvular feature(s) or hazard(s), etc.) in a 3D model. Referring to FIG. 21, illustrated are a pair of images 2100 and 2110 showing the position of the mitral valve at various phases of cardiac cycles. Image 2100 shows a cardiac cycle with normal pacing, with the position of the mitral valve moving between a maximum upper position (labeled position A) and a maximum lower position (labeled position B). Positions A and B occur at specific phases of the cardiac cycle (e.g., specific positions relative to a PQRST complex, etc.), such as the example phases / positions indicated on the associated electrocardiogram. During rapid pacing, as shown in image 2110, the mitral valve remains at a fixed position (labeled position C). During a variety of procedures, such as TMVR, rapid pacing is applied to curtail vertical movement of the mitral valve, so that the replacement valve is placed at a suitable depth (e.g., and / or to assist in other relevant procedure steps, for other procedures, etc.).

[0105] The positions A, B, and C along with the associated timing of positions A, B, and C shown in FIG. 21 show that positions A, B, and C occur at specific phases of the cardiac cycle, although those phases may differ from the phases shown in FIG. 21. The vertical position of the mitral valve during rapid pacing (labeled position C) corresponds to a position that the mitral valve passes through at specific phase(s) of the cardiac cycle that are able to be known in advance of applying rapid pacing during a procedure (e.g., for deployment of a replacement valve, etc.). As a result, during US imaging of a normal cardiac cycle, the position the mitral valve would be in during rapid pacing corresponds to the position of the mitral valve during the phase(s) of the electrocardiogram that correspond to position C. For example, in image 2110 position C is shown as occurring at the start of the T wave, thus in the example of FIG. 21 , the position of the mitral valve at the start of the T wave during normal pacing corresponds to the position of the mitral valve during rapid pacing.

[0106] The third set of example use cases includes a method of tracking markers that correspond to the rapid pacing position of a valve (e.g., mitral valve, etc.) annulus and / or other feature(s). In various examples, the markers that track the rapid pacing position(s) of feature(s) (also referred to herein as “rapid pacing marker(s)”) are defined at the beginning of a procedure, and are then useable throughout the procedure (e.g., for display, alert generation, etc.). In procedures involving rapid pacing, the effectiveness of the rapid pacing is tested at the beginning of the procedure.

[0107] In various examples, rapid pacing marker(s) are defined (e.g., manually, automatically, automatically based on user review / approval, etc.) in the 3D model (e.g., maintained by processing unit 204 and / or computing device 100, etc.) to track the rapid pacing position(s) of feature(s) based on US imaging captured while rapid pacing is tested. In the same or other examples, rapid pacing marker(s) are defined (e.g., manually, automatically, automatically based on user review / approval, etc.) based on US imaging of those feature(s) during normal pacing at or near the beginning of the procedure (e.g., before stages of a procedure when those features are shadowed by an instrument, etc.).US imaging of feature(s) during normal cardiac pacing that is gated to an electrocardiography signal includes recordings or still images wherein the feature(s) occupy the rapid pacing position(s) of the feature(s) based on the cardiac phase of those recordings / images (e.g., phase / position relative to PQRST complex, etc.) corresponding to the rapid pacing position(s) of those feature(s).

[0108] Based on the rapid pacing marker(s) tracked in the 3D model, the position(s) feature(s) (e.g., valve annulus, etc.) will occupy during rapid pacing are available (e.g., for display to a user, generation of alerts, etc.) throughout the procedure. This allows a user to confirm whether anatomical landmarks are still accurate to initial, planned location(s) before rapid pacing is begun for deployment, rather than having to verify the landmarks as part of the deployment workflow while rapid pacing. This provides advantages in terms of reducing the steps involved in deployment and reduced strain on the patient by curtailing the time during which rapid pacing is applied.

[0109] Various examples determine a rapid pacing position of the mitral valve based on US imaging of the mitral valve during normal pacing and display an output image that simulates rapid pacing by showing the position of the mitral valve during phases of the normal cardiac cycle that correspond to the position of the mitral valve during rapidpacing. In various examples, simulated rapid pacing is displayed in response to user selection at one or more points during a procedure, such as to verify alignment between a tool (e.g., a delivery capsule for a replacement valve, etc.) and the mitral valve.

[0110] The third set of example use cases also includes methods for updating the position(s) of the rapid pacing marker(s) tracked in the 3D model. In some scenarios, the position(s) of the feature(s) within the region of interest move during the procedure. Example scenarios include the entire patient moving during the procedure, or the heart or portions thereof moving during stages of the procedure (e.g., as a result of introducing the catheter to the heart, septal puncture, etc.).

[0111] In various examples, the tracked rapid pacing marker(s) of feature(s) are updated (e.g., manually, automatically, automatically based on user review / approval, etc.) based on US imaging of the normal cardiac cycle motion of the feature(s). In some examples, an algorithm is employed to detect visible portions of the annulus in relevant US imaging. Based on the detected portions, the position(s) of the rapid pacing marker(s) of the feature(s) (e.g., anatomical landmarks, etc.) are updated if the feature(s) have moved move and / or as the feature(s) move. This ensures the user has accurate and current data for the procedure (e.g., for valve deployment, etc.), even if the relevant feature(s) have moved (e.g., based on introduction of the catheter causing the whole heart including the annulus to move, etc.). Various examples perform automatic update of rapid pacing marker(s) of feature(s) based on US imaging gated to an electrocardiography signal (e.g., using image frames / recordings of the feature(s) during the normal cardiac cycle at phases / positions relative to the PQRST complex corresponding to the rapid pacing position(s) of the feature(s), etc.). In some such examples, accurate simulated rapid pacing position(s) of feature(s) (e.g., based on US imaging frames / recordings of phases / positions relative to the PQRST complex corresponding to the rapid pacing position(s) of the feature(s), etc.) is used to update rapid pacing marker(s) based on only gross position changes (e.g., movement of the entire heart, entire patient, etc.), but is not updated due to motion of the normal heartbeat. In various examples, based on known geometry of a set of markers relative to each other (e.g., based on the shape(s) and / or relative position(s) anatomical structure(s) tracked by marker(s) of the set of markers, etc.), position(s) of marker(s) from the set of markers that are not visible on live imaging are updated based on updated position(s) of marker(s) from the set of markers that are visible on live imaging.As one example, an annulus or a portion thereof tracked by first marker(s) is not visible on live imaging (e.g., due to shadowing, etc.) but other feature(s) (e.g., septal wall, a second portion of the annulus, etc.) tracked by second marker(s) are visible on live imaging. In the example, in response to position(s) of the second marker(s) being updated to align with the feature(s) tracked by the second marker(s), position(s) of the first marker(s) are updated based on the updated position(s) of the second marker(s) and the known geometry of the feature(s) tracked by the second marker(s) relative to the annulus or portion thereof.

[0112] In view of the foregoing structural and functional features described above, example methods will be better appreciated with reference to FIGS. 22-24. While, for purposes of simplicity of explanation, the example method of FIGS. 22-24 is shown and described as executing serially, it is to be understood and appreciated that the present examples are not limited by the illustrated order, as some actions could in other examples occur in different orders, multiple times and / or concurrently from that shown and described herein. Moreover, it is not necessary that all described actions be performed to implement a method.

[0113] FIG. 22 illustrates a flowchart of an example method 2200 for providing US imaging data for a region of interest via a user interface. In other examples, the blocks of example method 2200 are a set of machine-readable instructions on a non-transitory machine-readable medium or are a set of operations performed by a processor (e.g., processing unit 204 and / or a processor of computing device 100, etc.) executing machine- readable instructions as the operations.

[0114] At block 2210, method 2200 includes receiving US data (e.g., 2D, 3D, 4D) for the region of interest from a US probe (e.g., US imager 140, such as a TEE probe, a TTE probe, etc.).

[0115] At block 2220, method 2200 includes receiving tracking data indicating a position / orientation of an instrument (e.g., instrument 130, such as a transcatheter device, etc.) relative to the region of interest. In some examples, block 2220 is omitted, such as when there is no instrument within the region of interest with a tracked position / orientation (e.g., instrument(s) without embedded / attached tracker(s), no instruments ), etc.).

[0116] At block 2230, method 2200 includes constructing a 3D model of the region of interest based on the US data. In various examples, the 3D model is also based on one ormore of additional US data (e.g., pre-procedure and / or from an additional US probe), other imaging data (e.g., pre-procedure CT data), a set of markers (e.g., manually or automatically generated, etc.) added to the 3D model and / or other imaging data imported (e.g., and co-registered) to the 3D model, etc.

[0117] At block 2240, method 2200 includes adding an avatar of the instrument to the 3D model, with the position / orientation of the avatar based on the position / orientation of the instrument as tracked. In various examples, the avatar is based on information imported on plug-in of the instrument. Additionally, the appearance of the avatar in various examples are based on the dimensions of the instrument and / or current / future stage(s) of a procedure. In some examples (e.g., examples in which block 2220 is omitted, etc.), block 2240 is omitted, such as when there is no instrument within the region of interest with a tracked position / orientation (e.g., instrument(s) without embedded / attached tracker(s), no instrument(s), etc.).

[0118] At block 2250, method 2200 includes displaying an output image (e.g., via display 110) that includes an output US image of a portion of the region of interest overlaid with a portion of the 3D model corresponding to the portion of the region of interest (e.g., based on the 3D model and the position / orientation of marker(s), the device avatar, etc.). Additionally, in various examples the output image includes one or more markers and / or a shape determined based on one or more markers of the set of markers. In some examples (e.g., examples in which blocks 2220 and / or 2240 are omitted, etc.), the output image displays the portion of the region of interest and the position / orientation of the marker(s) without a device avatar.

[0119] At block 2260, method 2200 includes generating a user notification based on the device avatar, marker(s) of the set of markers on the 3D model, and / or shape(s) on the 3D model. In various examples, user notifications include any of a variety of notifications discussed herein. In a first example, the notification includes a distance / angle (e.g., constantly displayed on the screen during one or more steps of the procedure, displayed based on user preference, displayed or displayed differently (e.g., different colors, size, within a box, etc.) based on a comparison between the distance / angle and threshold value(s), etc), an alert that a distance / angle (e.g., between the device and marker(s), etc.) is above, below, or equal to a threshold value, in or outside of a range of distances / angles, etc. In a second example, the notification includes an indication that the device and ashape intersect or do not intersect. In some examples, the visual representation of the device avatar, annotation(s), marker(s), etc. are provided with or without additional notification / alert, to allow a user to visually determine the relative position(s) of the avatar, annotation(s), marker(s), etc.

[0120] FIG. 23 illustrates a flowchart of an example method 2300 for displaying the rapid pacing position(s) of feature(s) based on US imaging data for a region of interest. In other examples, the blocks of example method 2300 are a set of machine-readable instructions on a non-transitory machine-readable medium or are a set of operations performed by a processor (e.g., processing unit 204 and / or a processor of computing device 100, etc.) executing machine-readable instructions as the operations.

[0121] At block 2310, method 2300 includes capturing US imaging (e.g., via US imager 140, etc.) that shows the rapid pacing position(s) of one or more features (e.g., valve annulus, sub-valvular feature(s), etc.). In some examples, the captured US imaging includes US imaging of the features during rapid pacing, such as during testing of rapid pacing at the beginning of the procedure. In the same or other examples, the captured US imaging includes US imaging of the features during normal cardiac pacing, and the US imaging is gated to an electrocardiography signal. Based on the electrocardiography signal, portion(s) of the captured US imaging showing the feature(s) in the rapid pacing position(s) of the feature(s) are identified (e.g., based on the phase / position relative to a PQRST complex, etc.).

[0122] At block 2320, method 2300 includes creating virtual feature(s) in a 3D model (e.g., maintained by processing unit 204 and / or computing device 100, etc.) by setting markers (e.g., manually, automatically, etc.) tracked in the 3D model based on the rapid pacing position(s) of the feature(s), which are determined from the US imaging captured at 2310.

[0123] At block 2330, method 2300 includes beginning a procedure (e.g., TMVR, etc.) within the region of interest.

[0124] At block 2340, method 2300 includes displaying the simulated rapid pacing position(s) of feature(s) (e.g., based on US imaging corresponding to phase(s) of a PQRST complex when the feature(s) occupy the rapid pacing position(s), etc.) and / or the marker(s) tracking the rapid pacing position(s) of the feature(s) in the 3D model at one ormore points during the procedure (e.g., based on user selection and / or relevant procedure stage(s), etc.).

[0125] At block 2350, method 2300 includes determining if the tracked rapid pacing marker(s) in the 3D model correctly track the rapid pacing position(s) of the feature(s) as shown via simulated rapid pacing at 2340. In some examples, an algorithm implemented by a processor (e.g., processing unit 204 and / or a processor of computing device 100, etc.) determines whether the tracked marker(s) correctly track the associated feature(s). In other examples, a user manually determines whether the tracked marker(s) are aligned with the simulated rapid pacing position(s) of the feature(s).

[0126] At block 2360, method 2300 includes updating (e.g., via the processing unit 204 and / or the computing device 100, etc.) the rapid pacing marker(s) associated with the feature(s) in response to a determination that the rapid pacing marker(s) are not aligned with the simulated rapid pacing position(s) of the feature(s).

[0127] FIG. 24 illustrates a flowchart of an example method 2400 for displaying the rapid pacing position of a valve annulus based on US imaging data in connection with a valve replacement procedure. In other examples, the blocks of example method 2400 are a set of machine-readable instructions on a non-transitory machine-readable medium or are a set of operations performed by a processor (e.g., processing unit 204 and / or a processor of computing device 100, etc.) executing machine-readable instructions as the operations.

[0128] At block 2402, method 2400 includes capturing US imaging data (e.g., via US workstation 140, etc.) of the valve annulus during rapid pacing (e.g., during testing of temporary pacing prior to or at the start of the procedure, etc.). Additionally or alternatively (e.g., in response to a failure to capture US imaging during rapid pacing, etc.), US imaging data showing the rapid pacing position of the valve annulus (e.g., based on the phase / position of the US imaging relative to the PQRST complex, etc.) is captured (e.g., via US workstation 140, etc.) during normal cardiac pacing.

[0129] At block 2404, method 2400 includes creating a virtual annulus in a 3D model (e.g., tracked by processing unit 204 and / or computing device 100, etc.) of a region of interest for the procedure by setting markers in the 3D model based on the rapid pacing position of the annulus. In various examples, virtual feature(s) based on other feature(s) (e.g., sub-valvular hazards, etc.) are also created in the 3D model by setting marker(s) inthe 3D model based on the rapid pacing position(s) of the other feature(s). The virtual annulus (e.g., and other virtual feature(s), etc.) tracked in the 3D model are available throughout the procedure, for example, to be displayed to a user, to generate alerts based on the position of the virtual annulus relative to an avatar of a transcatheter device, etc.

[0130] At block 2406, method 2400 includes inserting the catheter into the patient and advancing the transcatheter delivery capsule into position for the capsule to be aligned with the annulus. The specific position for the capsule to be aligned with the annulus depends on the procedure and associated valve (e.g., mitral, tricuspid, aortic, etc.), for example, a position along an axis of the valve annulus there is an unobstructed path between the position and valve annulus where the capsule can be aligned to be coaxial with the annulus (e.g., past the septal wall for left side therapies such as TMVR, etc.). The capsule being in position to be aligned with the annulus is provided as one example of a procedure stage where knowing the rapid pacing position of the annulus is important for a user, although the information is available to the user throughout the procedure. In contrast to conventional techniques, examples provide users information indicating the rapid pacing position of features such as the valve annulus without inducing additional rapid pacing and associated stress on the heart of the patient.

[0131] At block 2408, method 2400 includes activating simulated rapid pacing of the valve annulus (e.g., and other feature(s)) based on cardiac gated US imaging (e.g., via US imager 140, etc.) showing image frames of live US imaging that correspond to portions of the normal cardiac cycle where the annulus (e.g., and other feature(s)) is in the rapid pacing position (e.g., based on the image frames corresponding to relevant electrocardiography signal phases based on the timing of the image frames relative to the PQRST complex, etc.).

[0132] At block 2410, method 2400 includes determining whether the tracked rapid pacing markers of the annulus (e.g., and other feature(s)) are aligned with the position(s) of the annulus (e.g., the position the annulus occupies during rapid pacing, etc.). In various examples, determining whether the markers are aligned with the annulus manually, automatically (e.g., via processing unit 204 and / or computing device 100, etc.), automatically subject to user review / approval, etc. In various examples, the simulated rapid pacing position(s) of the annulus / feature(s) is used in the determination, such that thedetermination is based on gross movement of the heart or patient instead of motion from the normal cardiac cycle.

[0133] At block 2412, method 2400 includes updating (e.g., via processing unit 204 and / or computing device 100, etc.) the marker locations in the 3D model of the virtual annulus (e.g., and of other feature(s)) in response to a determination at block 2410 that the tracked rapid pacing marker(s) are not aligned with the position(s) of the annulus (and other feature(s)).

[0134] At block 2414, method 2400 includes aligning the capsule with the virtual annulus of the 3D model (e.g., generated via processing unit 204 and / or computing device 100, etc.) in preparation for deployment of the replacement valve.

[0135] At block 2416, method 2400 includes adjusting the capsule to a correct depth for deployment of the replacement valve.

[0136] At block 2418, method 2400 includes advancing the replacement valve into position for deployment. The replacement valve being in position for deployment is provided as a second example of a procedure stage where knowing the rapid pacing position of the annulus is important for a user and is provided by examples throughout the procedure. The positioning of the replacement valve for deployment can depend on the type of device, and could include, for example, a brim of the replacement valve being in position for deployment. In contrast, conventional techniques either rely on the memory of the user or induce additional rapid pacing during the procedure.

[0137] At block 2420, method 2400 includes activating simulated rapid pacing of the valve annulus (e.g., and other feature(s)) based on cardiac gated US imaging (e.g., via US imager 140, etc.) showing image frames of live US imaging that correspond to portions of the normal cardiac cycle where the annulus is in the rapid pacing position. Block 2420 is similar to block 2408, as another example of a procedure stage for activating simulated rapid pacing.

[0138] At block 2422, method 2400 includes determining (e.g., manually, automatically, etc.) whether the tracked rapid pacing markers of the annulus (etc.) are aligned with the position(s) of the annulus (e.g., the position the annulus occupies during rapid pacing, etc.). Block 2422 is similar to block 2410, as another example of a procedure stage for determining whether tracked markers are aligned with the associated features (e.g., annulus, etc.).

[0139] At block 2424, method 2400 includes updating (e.g., via processing unit 204 and / or computing device 100, etc.) the marker locations in the 3D model of the virtual annulus (e.g., and of other feature(s)) in response to a determination at block 2422 that the tracked rapid pacing marker(s) are not aligned with the position(s) of the annulus (and other feature(s)). Block 2424 is similar to block 2422, as another example of a procedure stage for updating markers that are not aligned with the annulus or other feature(s).

[0140] At block 2426, method 2400 includes entering rapid pacing. Based on user guidance from the tracked markers in the 3D model and simulated rapid pacing, rapid pacing can be shorter than in conventional techniques where it is also used for determining correct alignment, depth, etc.

[0141] At block 2428, method 2400 includes confirming that the replacement valve is accurately positioned for deployment. In some examples, a first alert is generated in response to tracked marker(s) associated with the replacement valve (e.g., an avatar of the deployment device, etc.) being properly aligned with tracked marker(s) of the valve annulus and / or a second alert is generated in response to tracked marker(s) associated with the replacement valve (e.g., an avatar of the deployment device, etc.) being misaligned with tracked marker(s) of the valve annulus.

[0142] At block 2430, method 2400 includes deploying the implant. Following deployment, the deployment device is removed, and in various examples, relevant marker(s) are tracked and / or alert(s) generated when appropriate after deployment.

[0143] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the techniques). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a medical device.

[0144] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software,the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0145] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0146] The following additional examples are provided in connection with various aspects.1. A user interface system for imaging a region of interest for a procedure, the system comprising: a memory for storing machine-readable instructions; and a processor core for accessing the machine-readable instructions and executing the machine-readable instructions as operations, the operations comprising: receiving ultrasound (US) data for the region of interest from a US probe; constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.2. The user interface system of example 1, wherein the operations further comprise: receiving tracking data indicating a position of an instrument relative to the region of interest and an orientation of the instrument relative to the region of interest; andadding an avatar of the instrument to the 3D model, wherein the avatar tracks the position of the instrument relative to the region of interest and the orientation of the instrument relative to the region of interest, and an appearance of the avatar is based at least in part on the geometry of the instrument for a current stage of the procedure.3. The user interface system of example 2, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the procedure or a future stage of the procedure.4. The user interface system of any of examples 1-3, wherein the 3D model tracks a periodic motion of the marker over a cycle.5. The user interface system of example 4, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cycle.6. The user interface system of any of examples 4-5, wherein the periodic motion is tracked based on edge detection performed on the output US image.7. The user interface system of any of examples 4-6, wherein the periodic motion is tracked based on a tracked electrocardiography signal.8. The user interface system of any of examples 4-7, wherein the output image shows a range of the periodic motion of the marker over the cycle.9. The user interface system of any of examples 4-8, wherein the output image shows the periodic motion of the marker over the cycle.10. The user interface system of any of examples 2-9, wherein the operations further comprise generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of an instrument.11. The user interface system of example 10, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.12. The user interface system of any of examples 10-1 1 , wherein the user notification comprises an indication of the distance.13. The user interface system of any of examples 10-12, wherein the user notification is generated based on an identity of the marker.14. The user interface system of any of examples 2-13, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.15. The user interface system of example 14, wherein the position of the marker is further determined based on user feedback to the automatic detection.16. The user interface system of any of examples 2-13, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and the position of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.17. The user interface system of any of examples 2-16, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.18. The user interface system of example 17, wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.19. The user interface system of any of examples 2-16, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one guideline associated with the procedure.20. The user interface system of any of examples 17-19, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.21. The user interface system of any of examples 17-20, wherein the operations further comprise generating a user notification based on one of: a distance between an avatar of an instrument and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.22. The user interface system of any of examples 2-21, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.23. The user interface system of example 22, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.24. The user interface system of any of examples 22-23, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.25. The user interface system of any of examples 1-24, wherein the procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.26. The user interface system of any of examples 1-25, wherein the marker tracks a stationary position associated with a feature.27. The user interface system of example 26, wherein the output image is displayed during a periodic motion of the feature over a cycle.28. The user interface system of any of examples 1-27, wherein the output US image shows a feature in a stationary position associated with the feature.29. The user interface system of example 28, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an image frame associated with a phase of the electrocardiography signal where the feature is located in the stationary position.30. The user interface system of example 29, the operations further comprising: determining whether the marker is aligned with the stationary position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the stationary position associated with the feature.31. The user interface system of example 30, wherein determining whether the marker is aligned with the stationary position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.32. The user interface system of any of examples 30-31, the operations further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.33. A method for providing imaging data of a region of interest for a procedure, comprising: receiving ultrasound (US) data for the region of interest from a US probe; constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; anddisplaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.34. The method of example 33, further comprising: receiving tracking data indicating a position of an instrument relative to the region of interest and an orientation of the instrument relative to the region of interest; and adding an avatar of the instrument to the 3D model, wherein the avatar tracks the position of the instrument relative to the region of interest and the orientation of the instrument relative to the region of interest, and an appearance of the avatar is based at least in part on the geometry of the instrument for a current stage of the procedure.35. The method of example 34, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the procedure or a future stage of the procedure.36. The method of any of examples 33-35, wherein the 3D model tracks a periodic motion of the marker over a cycle.37. The method of example 36, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cycle.38. The method of any of examples 36-37, wherein the periodic motion is tracked based on edge detection performed on the output US image.39. The method of any of examples 36-38, wherein the periodic motion is tracked based on a tracked electrocardiography signal.40. The method of any of examples 36-39, wherein the output image shows a range of the periodic motion of the marker over the cycle.41 . The method of any of examples 36-40, wherein the output image shows the periodic motion of the marker over the cycle.42. The method of any of examples 34-41, further comprising generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of an instrument.43. The method of example 42, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.44. The method of any of examples 42-43, wherein the user notification comprises an indication of the distance.45. The method of any of examples 42-44, wherein the user notification is generated based on an identity of the marker.46. The method of any of examples 34-45, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.47. The method of example 46, wherein the position of the marker is further determined based on user feedback to the automatic detection.48. The method of any of examples 34-45, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and the position of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.49. The method of any of examples 34-48, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.50. The method of example 49, wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.51. The method of any of examples 34-48, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one guideline associated with the procedure.52. The method of any of examples 49-51, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.53. The method of any of examples 49-52, further comprising generating a user notification based on one of: a distance between an avatar of an instrument and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.54. The method of any of examples 34-53, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.55. The method of example 54, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.56. The method of any of examples 54-55, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.57. The method of any of examples 33-56, wherein the procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.58. The method of any of examples 33-57, wherein the marker tracks a stationary position associated with a feature.59. The method of example 58, wherein the output image is displayed during a periodic motion of the feature over a cycle.60. The method of any of examples 33-59, wherein the output US image shows a feature in a stationary position associated with the feature.61. The method of example 60, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an image frame associated with a phase of the electrocardiography signal where the feature is located in the stationary position.62. The method of example 61, further comprising: determining whether the marker is aligned with the stationary position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the stationary position associated with the feature.63. The method of example 62, wherein determining whether the marker is aligned with the stationary position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.64. The method of any of examples 62-63, further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.65. A non-transitory machine-readable medium having machine executable instructions for a user interface system that causes a processor core to execute operations, the operations comprising:receiving ultrasound (US) data from a US probe for a region of interest for a procedure; constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.66. The non-transitory machine-readable medium of example 65, wherein the operations further comprise: receiving tracking data indicating a position of an instrument relative to the region of interest and an orientation of the instrument relative to the region of interest; and adding an avatar of the instrument to the 3D model, wherein the avatar tracks the position of the instrument relative to the region of interest and the orientation of the instrument relative to the region of interest, and an appearance of the avatar is based at least in part on the geometry of the instrument for a current stage of the procedure.67. The non-transitory machine-readable medium of example 66, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the procedure or a future stage of the procedure.68. The non-transitory machine-readable medium of any of examples 65-67, wherein the 3D model tracks a periodic motion of the marker over a cycle.69. The non-transitory machine-readable medium of example 68, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cycle.70. The non-transitory machine-readable medium of any of examples 68-69, wherein the periodic motion is tracked based on edge detection performed on the output US image.71. The non-transitory machine-readable medium of any of examples 68-70, wherein the periodic motion is tracked based on a tracked electrocardiography signal.72. The non-transitory machine-readable medium of any of examples 68-71 , wherein the output image shows a range of the periodic motion of the marker over the cycle.73. The non-transitory machine-readable medium of any of examples 68-72, wherein the output image shows the periodic motion of the marker over the cycle.74. The non-transitory machine-readable medium of any of examples 66-73, wherein the operations further comprise generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of an instrument.75. The non-transitory machine-readable medium of example 74, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.76. The non-transitory machine-readable medium of any of examples 74-75, wherein the user notification comprises an indication of the distance.77. The non-transitory machine-readable medium of any of examples 74-76, wherein the user notification is generated based on an identity of the marker.78. The non-transitory machine-readable medium of any of examples 66-77, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.79. The non-transitory machine-readable medium of example 78, wherein the position of the marker is further determined based on user feedback to the automatic detection.80. The non-transitory machine-readable medium of any of examples 66-77, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and the position of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.81. The non-transitory machine-readable medium of any of examples 66-80, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.82. The non-transitory machine-readable medium of example 81 , wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.83. The non-transitory machine-readable medium of any of examples 66-82, wherein the output image comprises a representation of a target for the procedure, wherein therepresentation of the target is generated based on the marker and at least one guideline associated with the procedure.84. The non-transitory machine-readable medium of any of examples 81-83, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.85. The non-transitory machine-readable medium of any of examples 81-84, wherein the operations further comprise generating a user notification based on one of: a distance between an avatar of an instrument and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.86. The non-transitory machine-readable medium of any of examples 66-85, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.87. The non-transitory machine-readable medium of example 86, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.88. The non-transitory machine-readable medium of any of examples 86-87, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.89. The non-transitory machine-readable medium of any of examples 65-88, wherein the procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.90. The non-transitory machine-readable medium of any of examples 65-89, wherein the marker tracks a stationary position associated with a feature.91. The non-transitory machine-readable medium of example 90, wherein the output image is displayed during a periodic motion of the feature over a cycle.92. The non-transitory machine-readable medium of any of examples 65-91, wherein the output US image shows a feature in a stationary position associated with the feature.93. The non-transitory machine-readable medium of example 92, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an imageframe associated with a phase of the electrocardiography signal where the feature is located in the stationary position.94. The non-transitory machine-readable medium of example 93, the operations further comprising: determining whether the marker is aligned with the stationary position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the stationary position associated with the feature.95. The non-transitory machine-readable medium of example 94, wherein determining whether the marker is aligned with the stationary position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.96. The non-transitory machine-readable medium of any of examples 94-95, the operations further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.97. A user interface system for an intracardiac transcatheter procedure, the system comprising: a memory for storing machine-readable instructions; and a processor core for accessing the machine-readable instructions and executing the machine-readable instructions as operations, the operations comprising: receiving ultrasound (US) data from an echocardiogram probe for a region of interest for the intracardiac transcatheter procedure; constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.98. The user interface system of example 97, wherein the operations further comprise:receiving tracking data indicating a position of a transcatheter device relative to the region of interest and an orientation of the transcatheter device relative to the region of interest; and adding an avatar of the transcatheter device to the 3D model, wherein the avatar tracks the position of the transcatheter device relative to the region of interest and the orientation of the transcatheter device relative to the region of interest, and the appearance of the avatar are based on the geometry of the transcatheter device for a current stage of the intracardiac transcatheter procedure.99. The user interface system of example 98, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the intracardiac transcatheter procedure or a future stage of the intracardiac transcatheter procedure.100. The user interface system of any of examples 97-99, wherein the 3D model tracks a periodic motion of the marker over a cardiac cycle.101. The user interface system of example 100, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cardiac cycle.102. The user interface system of any of examples 100-101, wherein the periodic motion is tracked based on edge detection performed on the output US image.103. The user interface system of any of examples 100-102, wherein the periodic motion is tracked based on a tracked electrocardiography signal.104. The user interface system of any of examples 100-103, wherein the output image shows a range of the periodic motion of the marker over the cardiac cycle.105. The user interface system of any of examples 100-104, wherein the output image shows the periodic motion of the marker over the cardiac cycle.106. The user interface system of any of examples 98- 105, wherein the operations further comprise generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of a transcatheter device.107. The user interface system of example 106, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.108. The user interface system of any of examples 106-107, wherein the user notification comprises an indication of the distance.109. The user interface system of any of examples 106-108, wherein the user notification is generated based on an identity of the marker.110. The user interface system of any of examples 98- 109, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.111. The user interface system of example 110, wherein the position of the marker is further determined based on user feedback to the automatic detection.112. The user interface system of any of examples 98- 109, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and the position of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.113. The user interface system of any of examples 98- 112, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.114. The user interface system of example 113, wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.115. The user interface system of any of examples 98-112, wherein the output image comprises a representation of a target for the intracardiac transcatheter procedure, wherein the representation of the target is generated based on the marker and at least one guideline associated with the intracardiac transcatheter procedure.116. The user interface system of any of examples 113-115, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.117. The user interface system of any of examples 113-116, wherein the operations further comprise generating a user notification based on one of: a distance between an avatar of a transcatheter device and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.118. The user interface system of any of examples 98- 117, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.119. The user interface system of example 118, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.120. The user interface system of any of examples 118-119, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.121. The user interface system of any of examples 97-120, wherein the intracardiac transcatheter procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.122. The user interface system of any of examples 97-121, wherein the echocardiogram probe is one of a transthoracic echocardiogram probe, a transesophageal echocardiogram probe, or an intracardiac echocardiogram probe.123. The user interface system of any of examples 97-122, wherein the marker tracks a rapid pacing position associated with a feature.124. The user interface system of example 123, wherein the output image is displayed during a periodic motion of the feature over a normal cardiac cycle.125. The user interface system of any of examples 97-124, wherein the output US image shows a feature in a rapid pacing position associated with the feature.126. The user interface system of example 125, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an image frame associated with a phase of the electrocardiography signal where the feature is located in the rapid pacing position.127. The user interface system of example 126, the operations further comprising: determining whether the marker is aligned with the rapid pacing position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the rapid pacing position associated with the feature.128. The user interface system of example 127, wherein determining whether the marker is aligned with the rapid pacing position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.129. The user interface system of any of examples 127-128, the operations further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.130. The user interface system of any of examples 123-129, wherein the feature is a valve annulus.131. The user interface system of any of examples 123-130, wherein the marker is created based on determining the rapid pacing position of the feature during rapid pacing.132. The user interface system of any of examples 123-130, wherein the marker is created based on determining the rapid pacing position of the feature during normal cardiac pacing.133. A method for providing imaging data of a region of interest of an intracardiac transcatheter procedure, comprising: receiving ultrasound (US) data from an echocardiogram probe for the region of interest; constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.134. The method of example 133, further comprising: receiving tracking data indicating a position of a transcatheter device relative to the region of interest and an orientation of the transcatheter device relative to the region of interest; and adding an avatar of the transcatheter device to the 3D model, wherein the avatar tracks the position of the transcatheter device relative to the region of interest and the orientation of the transcatheter device relative to the region of interest, and the appearance of the avatar are based on the geometry of the transcatheter device for a current stage of the intracardiac transcatheter procedure.135. The method of example 134, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the intracardiac transcatheter procedure or a future stage of the intracardiac transcatheter procedure.136. The method of any of examples 133-135, wherein the 3D model tracks a periodic motion of the marker over a cardiac cycle.137. The method of example 136, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cardiac cycle.138. The method of any of examples 136-137, wherein the periodic motion is tracked based on edge detection performed on the output US image.139. The method of any of examples 136-138, wherein the periodic motion is tracked based on a tracked electrocardiography signal.140. The method of any of examples 136-139, wherein the output image shows a range of the periodic motion of the marker over the cardiac cycle.141. The method of any of examples 136-140, wherein the output image shows the periodic motion of the marker over the cardiac cycle.142. The method of any of examples 136-141, further comprising generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of a transcatheter device.143. The method of example 142, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.144. The method of any of examples 142-143, wherein the user notification comprises an indication of the distance.145. The method of any of examples 142- 144, wherein the user notification is generated based on an identity of the marker.146. The method of any of examples 134-145, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.147. The method of example 146, wherein the position of the marker is further determined based on user feedback to the automatic detection.148. The method of any of examples 134-145, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and theposition of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.149. The method of any of examples 134- 148, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.150. The method of example 149, wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.151. The method of any of examples 134-148, wherein the output image comprises a representation of a target for the intracardiac transcatheter procedure, wherein the representation of the target is generated based on the marker and at least one guideline associated with the intracardiac transcatheter procedure.152. The method of any of examples 149-151, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.153. The method of any of examples 149-152, wherein the operations further comprise generating a user notification based on one of: a distance between an avatar of a transcatheter device and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.154. The method of any of examples 134-153, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.155. The method of example 154, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.156. The method of any of examples 154-155, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.157. The method of any of examples 133-156, wherein the intracardiac transcatheter procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.158. The method of any of examples 133-157, wherein the echocardiogram probe is one of a transthoracic echocardiogram probe, a transesophageal echocardiogram probe, or an intracardiac echocardiogram probe.159. The method of any of examples 133-158, wherein the marker tracks a rapid pacing position associated with a feature.160. The method of example 159, wherein the output image is displayed during a periodic motion of the feature over a normal cardiac cycle.161. The method of any of examples 133-160, wherein the output US image shows a feature in a rapid pacing position associated with the feature.162. The method of example 161, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an image frame associated with a phase of the electrocardiography signal where the feature is located in the rapid pacing position.163. The method of example 162, the operations further comprising: determining whether the marker is aligned with the rapid pacing position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the rapid pacing position associated with the feature.164. The method of example 163, wherein determining whether the marker is aligned with the rapid pacing position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.165. The method of any of examples 163-164, further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.166. The method of any of examples 159-165, wherein the feature is a valve annulus.167. The method of any of examples 159-166, wherein the marker is created based on determining the rapid pacing position of the feature during rapid pacing.168. The method of any of examples 159- 166, wherein the marker is created based on determining the rapid pacing position of the feature during normal cardiac pacing.169. A non-transitory machine-readable medium having machine executable instructions for a user interface system that causes a processor core to execute operations, the operations comprising: receiving ultrasound (US) data from an echocardiogram probe for a region of interest for the intracardiac transcatheter procedure;constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.170. The non-transitory machine-readable medium of example 169, wherein the operations further comprise: receiving the tracking data indicating a position of a transcatheter device relative to the region of interest and an orientation of the transcatheter device relative to the region of interest; and adding an avatar of the transcatheter device to the 3D model, wherein the avatar tracks the position of the transcatheter device relative to the region of interest and the orientation of the transcatheter device relative to the region of interest, and the appearance of the avatar are based on the geometry of the transcatheter device for a current stage of the intracardiac transcatheter procedure.171. The non-transitory machine-readable medium of example 170, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the intracardiac transcatheter procedure or a future stage of the intracardiac transcatheter procedure.172. The non-transitory machine-readable medium of any of examples 169-171, wherein the 3D model tracks a periodic motion of the marker over a cardiac cycle.173. The non-transitory machine-readable medium of example 172, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cardiac cycle.174. The non-transitory machine-readable medium of any of examples 172-173, wherein the periodic motion is tracked based on edge detection performed on the output US image.175. The non-transitory machine-readable medium of any of examples 172-173, wherein the periodic motion is tracked based on a tracked electrocardiography signal.176. The non-transitory machine-readable medium of any of examples 172-174, wherein the output image shows a range of the periodic motion of the marker over the cardiac cycle.177. The non-transitory machine-readable medium of any of examples 172-176, wherein the output image shows the periodic motion of the marker over the cardiac cycle.178. The non-transitory machine-readable medium of any of examples 170-177, wherein the operations further comprise generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of a transcatheter device.179. The non-transitory machine-readable medium of example 178, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.180. The non-transitory machine-readable medium of any of examples 178-179, wherein the user notification comprises an indication of the distance.181. The non-transitory machine-readable medium of any of examples 178-180, wherein the user notification is generated based on an identity of the marker.182. The non-transitory machine-readable medium of any of examples 170-181, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.183. The non-transitory machine-readable medium of example 182, wherein the position of the marker is further determined based on user feedback to the automatic detection.184. The non-transitory machine-readable medium of any of examples 170-181, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and the position of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.185. The non-transitory machine-readable medium of any of examples 170-184, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.186. The non-transitory machine-readable medium of example 185, wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.187. The non-transitory machine-readable medium of any of examples 170-184, wherein the output image comprises a representation of a target for the intracardiac transcatheter procedure, wherein the representation of the target is generated based on the marker and at least one guideline associated with the intracardiac transcatheter procedure.188. The non-transitory machine-readable medium of any of examples 185-187, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.189. The non-transitory machine-readable medium of any of examples 185-188, wherein the operations further comprise generating a user notification based on one of: a distance between an avatar of a transcatheter device and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.190. The non-transitory machine-readable medium of any of examples 170-189, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.191. The non-transitory machine-readable medium of example 190, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.192. The non-transitory machine-readable medium of any of examples 190-191, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.193. The non-transitory machine-readable medium of any of examples 169-192, wherein the intracardiac transcatheter procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.194. The non-transitory machine-readable medium of any of examples 169-193, wherein the echocardiogram probe is one of a transthoracic echocardiogram probe, a transesophageal echocardiogram probe, or an intracardiac echocardiogram probe.195. The non-transitory machine-readable medium of any of examples 169-194, wherein the marker tracks a rapid pacing position associated with a feature.196. The non-transitory machine-readable medium of example 195, wherein the output image is displayed during a periodic motion of the feature over a normal cardiac cycle.197. The non-transitory machine-readable medium of any of examples 169-196, wherein the output US image shows a feature in a rapid pacing position associated with the feature.198. The non-transitory machine-readable medium of example 197, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an image frame associated with a phase of the electrocardiography signal where the feature is located in the rapid pacing position.199. The non-transitory machine-readable medium of example 198, the operations further comprising: determining whether the marker is aligned with the rapid pacing position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the rapid pacing position associated with the feature.200. The non-transitory machine-readable medium of example 199, wherein determining whether the marker is aligned with the rapid pacing position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.201. The non-transitory machine-readable medium of any of examples 198-199, further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.202. The non-transitory machine-readable medium of any of examples 195-201, wherein the feature is a valve annulus.203. The non-transitory machine-readable medium of any of examples 195-202, wherein the marker is created based on determining the rapid pacing position of the feature during rapid pacing.204. The non-transitory machine-readable medium of any of examples 195-202, wherein the marker is created based on determining the rapid pacing position of the feature during normal cardiac pacing.205. An ultrasound (US) system for imaging a region of interest for a procedure, the system comprising: a US probe comprising a US transducer configured to obtain US data for the region of interest; a memory for storing machine-readable instructions; and a processor core for accessing the machine-readable instructions and executing the machine-readable instructions as operations, the operations comprising: receiving the US data for the region of interest from the US probe; constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.206. The user interface system of example 205, further comprising: an instrument; and a tracking system that determines tracking data indicating a position of the instrument relative to the region of interest and an orientation of the instrument relative to the region of interest, wherein the operations further comprise: receiving the tracking data; and adding an avatar of the instrument to the 3D model, wherein the avatar tracks the position of the instrument relative to the region of interest and the orientation of the instrument relative to the region of interest, and an appearance ofthe avatar is based at least in part on the geometry of the instrument for a current stage of the procedure.207. The US system of example 206, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the procedure or a future stage of the procedure.208. The US system of any of examples 205-207, wherein the 3D model tracks a periodic motion of the marker over a cycle.209. The US system of example 208, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cycle.210. The US system of any of examples 208-209, wherein the periodic motion is tracked based on edge detection performed on the output US image.211 . The US system of any of examples 208-210, wherein the periodic motion is tracked based on a tracked electrocardiography signal.212. The US system of any of examples 208-211, wherein the output image shows a range of the periodic motion of the marker over the cycle.213. The US system of any of examples 208-212, wherein the output image shows the periodic motion of the marker over the cycle.214. The US system of any of examples 206-213, wherein the operations further comprise generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of an instrument.215. The US system of example 214, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.216. The US system of any of examples 214-215, wherein the user notification comprises an indication of the distance.217. The US system of any of examples 214-216, wherein the user notification is generated based on an identity of the marker.218. The US system of any of examples 206-217, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.219. The US system of example 218, wherein the position of the marker is further determined based on user feedback to the automatic detection.220. The US system of any of examples 206-217, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and the position of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.221. The US system of any of examples 206-220, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.222. The US system of example 221 , wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.223. The US system of any of examples 206-220, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one guideline associated with the procedure.224. The US system of any of examples 221 -223, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.225. The US system of any of examples 221-224, wherein the operations further comprise generating a user notification based on one of: a distance between an avatar of an instrument and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.226. The US system of any of examples 206-225, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.227. The US system of example 226, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.228. The US system of any of examples 226-227, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.229. The US system of any of examples 205-228, wherein the procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.230. The US system of any of examples 205-229, wherein the marker tracks a stationary position associated with a feature.231. The US system of example 230, wherein the output image is displayed during a periodic motion of the feature over a cycle.232. The US system of any of examples 205-231, wherein the output US image shows a feature in a stationary position associated with the feature.233. The US system of example 232, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an image frame associated with a phase of the electrocardiography signal where the feature is located in the stationary position.234. The US system of example 233, the operations further comprising: determining whether the marker is aligned with the stationary position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the stationary position associated with the feature.235. The US system of example 234, wherein determining whether the marker is aligned with the stationary position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.236. The US system of any of examples 234-235, the operations further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.237. An ultrasound (US) system for imaging a region of interest for an intracardiac transcatheter procedure, the system comprising: an echocardiogram probe comprising a US transducer configured to obtain US data for the region of interest; a memory for storing machine-readable instructions; and a processor core for accessing the machine-readable instructions and executing the machine-readable instructions as operations, the operations comprising: receiving the US data for the region of interest from the echocardiogram probe;constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.238. The US system of example 237, further comprising: a transcatheter device comprising a set of transcatheter device trackers, wherein a shape of the transcatheter device is based on a current stage in an intracardiac transcatheter procedure; a tracking system that determines, based on the set of transcatheter device trackers, tracking data indicating a position of the transcatheter device relative to the region of interest and an orientation of the transcatheter device relative to the region of interest, wherein the operations further comprise: receiving the tracking data; adding an avatar of the transcatheter device to the 3D model, wherein the avatar tracks the position of the transcatheter device relative to the region of interest and the orientation of the transcatheter device relative to the region of interest, and the appearance of the avatar are based on the geometry of the transcatheter device for the current stage of the intracardiac transcatheter procedure.239. The US system of example 238, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the intracardiac transcatheter procedure or a future stage of the intracardiac transcatheter procedure.240. The US system of any of examples 237-239, wherein the 3D model tracks a periodic motion of the marker over a cardiac cycle.241. The US system of example 240, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cardiac cycle.242. The US system of any of examples 240-241, wherein the periodic motion is tracked based on edge detection performed on the output US image.243. The US system of any of examples 240-242, wherein the periodic motion is tracked based on a tracked electrocardiography signal.244. The US system of any of examples 240-243, wherein the output image shows a range of the periodic motion of the marker over the cardiac cycle.245. The US system of any of examples 240-244, wherein the output image shows the periodic motion of the marker over the cardiac cycle.246. The US system of any of examples 238-245, wherein the operations further comprise generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of a transcatheter device.247. The US system of example 246, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.248. The US system of any of examples 246-247, wherein the user notification comprises an indication of the distance.249. The US system of any of examples 246-248, wherein the user notification is generated based on an identity of the marker.250. The US system of any of examples 238-249, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.251. The US system of example 250, wherein the position of the marker is further determined based on user feedback to the automatic detection.252. The US system of any of examples 238-249, wherein the marker is added to the set of markers based on a user input indicating a portion of the output US image, and the position of the marker on the 3D model is based on the portion of the output US image and a depth associated with the portion of the output US image.253. The US system of any of examples 238-252, wherein the output image comprises a representation of a target for the procedure, wherein the representation of the target is generated based on the marker and at least one additional marker of the set of markers.254. The US system of example 253, wherein a shape of the representation of the target is automatically generated based on the marker and the at least one additional marker.255. The US system of any of examples 238-252, wherein the output image comprises a representation of a target for the intracardiac transcatheter procedure, wherein therepresentation of the target is generated based on the marker and at least one guideline associated with the intracardiac transcatheter procedure.256. The US system of any of examples 253-255, wherein a shape of the representation of the target is automatically generated based on analysis of the US data.257. The US system of any of examples 253-256, wherein at least one of a shape or a marked location of a representation of a selected device of a set of devices is automatically generated based on analysis of the US data, wherein the set of devices comprises the transcatheter device.258. The US system of any of examples 253-257, wherein the operations further comprise generating a user notification based on one of: a distance between an avatar of a transcatheter device and the shape, an angle between an axis of the avatar and the shape, or whether the axis of the avatar intersects the shape.259. The US system of any of examples 238-258, wherein the marker or a shape associated with the marker is generated based on additional imaging data of the region of interest, wherein the additional imaging data is co-registered to the US data.260. The US system of example 259, wherein the shape associated with the marker is one of: aligned automatically with the marker and an additional marker of the set of markers, aligned manually with the marker and the additional marker, or aligned with the US data.261. The US system of any of examples 259-260, wherein the shape is aligned based on transforming dimensions of the shape one of manually or automatically.262. The US system of any of examples 237-261, wherein the intracardiac transcatheter procedure comprises a set of stages, and at least one of the presence of the marker in the output image or the appearance of the marker in the output image is based on a determination that the marker is associated with a current stage of the set of stages.263. The US system of any of examples 237-262, wherein the echocardiogram probe is one of a transthoracic echocardiogram probe, a transesophageal echocardiogram probe, or an intracardiac echocardiogram probe.264. The US system of any of examples 237-263, wherein the marker tracks a rapid pacing position associated with a feature.265. The US system of example 264, wherein the output image is displayed during a periodic motion of the feature over a normal cardiac cycle.266. The US system of any of examples 237-265, wherein the output US image shows a feature in a rapid pacing position associated with the feature.267. The US system of example 266, wherein the US data is gated to an electrocardiography signal, and the output US image comprises an image frame associated with a phase of the electrocardiography signal where the feature is located in the rapid pacing position.268. The US system of example 267, the operations further comprising: determining whether the marker is aligned with the rapid pacing position associated with the feature; and updating a tracked position of the marker in response to a determination that the marker is not aligned with the rapid pacing position associated with the feature.269. The US system of example 268, wherein determining whether the marker is aligned with the rapid pacing position associated with the feature is based on a comparison between the image frame and the tracked position of the marker.270. The US system of any of examples 267-269, the operations further comprising updating an additional tracked position of an additional marker of the set of markers based on the tracked position of the marker and a known position of the additional marker relative to the marker.271. The US system of any of examples 263-270, wherein the feature is a valve annulus.272. The US system of any of examples 263-271, wherein the marker is created based on determining the rapid pacing position of the feature during rapid pacing.273. The US system of any of examples 263-271, wherein the marker is created based on determining the rapid pacing position of the feature during normal cardiac pacing.

Claims

WHAT IS CLAIMED IS:

1. A user interface system for imaging a region of interest for a procedure, the system comprising: a memory for storing machine-readable instructions; and a processor core for accessing the machine-readable instructions and executing the machine-readable instructions as operations, the operations comprising: receiving ultrasound (US) data for the region of interest from a US probe; constructing a three-dimensional (3D) model of the region of interest based on the US data, wherein the 3D model tracks a set of markers, and a marker of the set of markers has a position on the 3D model; and displaying an output image, wherein the output image comprises an output US image of a portion of the region of interest, overlaid with a portion of the 3D model corresponding to the portion of the region of interest, wherein the portion of the 3D model comprises the marker.

2. The user interface system of claim 1, wherein the operations further comprise: receiving tracking data indicating a position of an instrument relative to the region of interest and an orientation of the instrument relative to the region of interest; and adding an avatar of the instrument to the 3D model, wherein the avatar tracks the position of the instrument relative to the region of interest and the orientation of the instrument relative to the region of interest, and an appearance of the avatar is based at least in part on the geometry of the instrument for a current stage of the procedure.

3. The user interface system of claim 2, wherein the appearance of the avatar comprises an alignment feature associated with the current stage of the procedure or a future stage of the procedure.

4. The user interface system of any of claims 1-3, wherein the 3D model tracks a periodic motion of the marker over a cycle.

5. The user interface system of claim 4, wherein the position of the marker is fixed to a position of a feature in the output image and moves with a periodic motion of the feature over the cycle.

6. The user interface system of any of claims 4-5, wherein the periodic motion is tracked based on edge detection performed on the output US image.

7. The user interface system of any of claims 4-6, wherein the periodic motion is tracked based on a tracked electrocardiography signal.

8. The user interface system of any of claims 4-7, wherein the output image shows a range of the periodic motion of the marker over the cycle.

9. The user interface system of any of claims 4-8, wherein the output image shows the periodic motion of the marker over the cycle.

10. The user interface system of any of claims 2-9, wherein the operations further comprise generating a user notification based on a distance between the marker and one of an additional marker of the set of markers or a tracked position of an instrument.

11. The user interface system of claim 10, wherein the user notification comprises an alert generated based on a comparison between the distance and a threshold value.

12. The user interface system of any of claims 10-11, wherein the user notification comprises an indication of the distance.

13. The user interface system of any of claims 10-12, wherein the user notification is generated based on an identity of the marker.

14. The user interface system of any of claims 2-13, wherein the marker is added to the set of markers automatically, and the position of the marker is determined based on an automatic detection of an edge or a feature from the US data.

15. The user interface system of claim 14, wherein the position of the marker is further determined based on user feedback to the automatic detection.

Images