Autonomous TEE probe with graph model generation and landmark-based navigation

Abstract

Systems and methods for an autonomous TEE probe with graph model generation and landmark-based navigation. An example system is configured to obtain, via a TEE probe, ultrasound images of a portion of a heart. The system is configured to compute a forward pass through a machine learning model trained to output a patient mesh based on an average heart mesh. The system is configured to extract hierarchical features and align the average heart mesh based on a confidence map associated with the hierarchical features. The system is configured to output the patient mesh based on deformation of the aligned average heart mesh. An interactive user interface presents a graphical representation of the patient mesh.

Description

AUTONOMOUS TEE PROBE WITH GRAPH MODEL GENERATION AND LANDMARKBASED NAVIGATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Prov. Patent App. No. 63 / 656044 titled “AUTONOMOUSTEE PROBE WITH LANDMARK-BASED NAVIGATION” and filed on June 4, 2024. This application further claims priority to U.S. Prov. Patent App. No. 63 / 699512 titled “AUTONOMOUS TEE PROBE WITH LANDMARK-BASED NAVIGATION” and filed on September 26, 2024. Each of the above-recited applications is hereby incorporated herein by reference in its entirety for all purposes.BACKGROUNDTECHNICAL FIELD

[0002] The present disclosure relates to transesophageal echocardiograms (TEE), and more particularly, to software for automated control of TEE probes and analysis of TEE probe sensor information.DESCRIPTION OF RELATED ART

[0003] A transesophageal echocardiogram (TEE) is used to analyze cardiac health of a patient using a semi-invasive procedure. For example, a TEE probe is passed into the patient’s esophagus while an ultrasound transducer on the TEE probe obtains sensor information. The TEE probe may be manipulated in position and / or orientation within the esophagus to obtain specific cardiac views of interest to a medical professional. TEE may be relied upon for relatively clear images of a patient’s heart as compared to other semi-invasive or non-invasive techniques (e.g., a transthoracic echocardiogram (TTE)). Additionally, and as known by those skilled in the art, TTE is limited by the available locations through the chest wall which allow for views of the patient’s heart.

[0004] While TEE provides technical benefits over other techniques, at present use of a TEE probe requires multiple specialized medical professionals. For example, a specialized medical professional may be required to control the TEE probe system whileanother specialized medical professional analyzes images. In this example, it may take substantial time to obtain clear images of specific cardiac views of interest. Additionally, once a cardiac view is obtained, and the TEE probe subsequently moved, it may take substantial time to later adjust the TEE probe to again obtain images of the cardiac view.SUMMARY

[0005] An example embodiment includes a system, method and computer readable media. The system comprises a transesophageal echocardiogram (TEE) probe, the TEE probe being robotically controlled by the system and the TEE probe being adjustable with respect to a plurality of degrees of freedom; and one or more processors and non- transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to: obtain, via TEE probe, one or more ultrasound images of at least a portion of a heart, the one or more ultrasound images comprising one or more three-dimensional (3D) ultrasounds; compute a forward pass through a machine learning model trained to output a patient mesh reflecting a plurality of vertices forming a contour associated with the heart, wherein the machine learning model is configured to deform an average heart mesh reflecting corresponding vertices that form a contour associated with an average heart, and wherein the instructions that cause the one or more processors to compute a forward pass cause the processors to: extract a hierarchical feature map associated with a first ultrasound image of the one or more ultrasound images, wherein the hierarchical features encode features for individual voxels forming the first ultrasound image, wherein a confidence map associated with the hierarchical features is obtained, align the average heart mesh based on the first ultrasound image, wherein one or more transformations are applied to the average heart mesh based, at least in part, on the confidence map, and deform the aligned average heart mesh based on the hierarchical features to form the patient mesh; and cause presentation of an interactive user interface, wherein the interactive user interface presents a graphical representation of the patient mesh.

[0006] An example embodiment includes a method implemented by a system of one or more processors, the method comprising: causing presentation of an interactive user interface, wherein the interactive user interface includes a two-dimensional (2D)ultrasound image obtained via a transesophageal echocardiogram (TEE) probe and a graphical representation of a patient mesh, the patient mesh including a plurality of vertices reflecting a contour associated with a heart of the patient, and the TEE probe being controllable with respect to a plurality of degrees of freedom; receiving user input selecting a landmark reflecting a physical feature of interest, the landmark being selected on the graphical representation; and updating the interactive user interface to present additional 2D ultrasound images obtained via the TEE probe based on adjusted degrees of freedom, and wherein the additional 2D ultrasound images depict the landmark.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A is a block diagram illustrating an example automated inspection system in communication with a transesophageal echocardiogram (TEE) tool.

[0008] Figure 1 B is a block diagram illustrating the example automated inspection system obtaining sensor information based on control of the TEE tool using different station coordinates.

[0009] Figure 1 C is a block diagram illustrating the example automated inspection system performing cardiac view registration.

[0010] Figure 1 D is a block diagram illustrating the example automated inspection system obtaining a real-time image of a requested cardiac view based on the cardiac view registration.

[0011] Figure 2A is a flowchart of an example process for cardiac view registration using a TEE tool.

[0012] Figure 2B is an example user interface enabling selection of information relevant to the cardiac view registration.

[0013] Figure 2C is an example user interface that includes images determined to be associated with specific cardiac views based on the cardiac view registration.

[0014] Figure 3A is a flowchart of an example process for updating station coordinates of a TEE tool that result in images associated with a particular cardiac view.

[0015] Figure 3B is an example user interface that enables selection of a cardiac view to be adjusted.

[0016] Figure 3C is an example user interface depicting manual adjustments to station coordinates of the TEE tool.

[0017] Figure 4A is a flowchart of an example process for determining updates to station coordinates in response to one or more triggers.

[0018] Figure 4B is a flowchart of an example process for determining adjustments to station coordinates.

[0019] Figure 5 is a flowchart of an example process for cardiac view registration based on use of cardiac model data.

[0020] Figure 6 illustrates an example cardiac model along with an imaging plane.

[0021] Figure 7 illustrates an example user interface.

[0022] Figures 8A-8C illustrate example techniques to generate a patient mesh based on segmented ultrasound images.

[0023] Figure 9 illustrates another example technique to generate a patient mesh.

[0024] Figure 10A is a flowchart of an example process for training a patient mesh and inference ultrasound data from a TEE tool.

[0025] Figure 10B is a flowchart of an example process illustrating detail of inferencing ultrasound data from a TEE tool.

[0026] Figure 11A is a flowchart of an example process for presenting a clinically preferred cardiac view based on a patient mesh.

[0027] Figure 11 B is a flowchart of an example process for navigating the TEE tool based on the patient mesh.

[0028] Figure 11C is an example user interface of an ultrasound image proximate to a graphical representation of a patient mesh.

[0029] Figure 11 D is an example user interface associated with landmark navigation.

[0030] Figure 11 E is an example user interface associated with bookmarking aportion of an ultrasound image and / or a patient mesh.

[0031] Figures 12A-12E illustrate examples user interfaces associated with navigation techniques.

[0032] Figure 13 is a block diagram illustrating techniques to determine tissue pressure associated with a TEE probe.

[0033] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0034] This application describes techniques to simplify the operation, and increase the effectiveness, of transesophageal echocardiograms (TEE). As will be described, a TEE tool may be controlled by a system in an automated or semi-automated fashion to reliably perform disparate TEE procedures. For example, the TEE tool, such as a TEE probe, may be controlled in a multitude of degrees of freedom while inserted into a patient. Example degrees of freedom may include translations, rotations of the TEE tool (also referred to herein as pose), electronic or mechanical rotations of an ultrasound sensor to adjust the 2-D echo plane, and so on. As another example, the system may control the TEE tool to rapidly determine combinations of degrees of freedom (referred to herein as station coordinates) which result in specific cardiac views of interest. Example cardiac views may include a two-chamber view, a four-chamber view, a five- chamber view, a mid-esophageal long axis view, and so on. As will be described, the disclosed technology may allow a medical professional to quickly view images corresponding to specific cardiac views and reliably cycle between such cardiac views.

[0035] At present, TEE tools are generically designed for disparate ultrasound-based procedures. For example, a TEE tool may typically require interaction with complex software applications to adjust operation of the TEE tool for use in cardiac analyses. In this example, the TEE tool may be used for analyses of other portions of a patient’s body,such as the kidney. Thus, hardware and / or software adjustments may be required prior to use of the TEE tool for cardiac analyses. Furthermore, specialized training is typically required to use these TEE tools. For example, current software to control, and use, TEE tools require substantial interaction with complex user interfaces. Such interactions increase the amount of time that the TEE tool is inserted into a patient.

[0036] Furthermore, the above-described TEE tools require manualcontrol of the TEE tool while inserted into a patient. Thus, a medical professional may manually adjust the station coordinates (e.g., translation, flexion, rotation, and so on) of the TEE tool to obtain images associated with a specific cardiac view. As known by those skilled in the art, specific cardiac views may be preferred by medical professionals. Thus, the medical professional may traverse the tool through the patient finding these specific cardiac views. During a TEE procedure, the medical professional may prefer to return to a particular cardiac view to obtain images associated with the particular cardiac view. Due to such manual control, it may be impractical to reliably return to the same station coordinates. Thus, the medical professional may review images which are distinct in view from the earlier-reviewed images.

[0037] As may be appreciated, since TEE is a semi-invasive technique such manual controls may introduce safety concerns. For example, incorrect adjustment of the TEE tool along one or more degrees of freedom may result in injury of a patient. As an example, incorrect control may cause application of pressure to the interior of the esophagus which may result in a safety concern.

[0038] The disclosed technology, in contrast, relies upon modern software techniques to automate, or semi-automate, control of a TEE tool. For example, and with respect to automation, the disclosed technology may control a robotic TEE tool to automatically obtain a multitude of cardiac views. As another example, and with respect to semi-automation, the disclosed technology may inform a medical professional’s movement within a patient’s esophagus. For example, a user interface may provide recommendations regarding movement of the TEE tool to obtain a specific, or set of, cardiac views.

[0039] As will be described, a system (e.g., the automated inspection system 100) may control a TEE tool to cause insertion of a TEE probe into a patient. Once inserted,the system may perform a cardiac view registration process to identify station coordinates associated with specific cardiac views. As described herein, station coordinates may reflect control parameters associated with the TEE tool. For example, station coordinates may include a translation of the TEE probe from a resting position. In this example, the translation may be indicative of a length within the patient. As another example, station coordinates may include a rotation of the TEE probe about an axis. As another example, station coordinates may include an electronic rotation of an ultrasound sensor (e.g., ultrasound transducer) about an axis. In some embodiments, the ultrasound sensor may use beamforming techniques, or mechanical adjustment, to enable scanning about the axis.

[0040] With respect to cardiac view registration, the system may therefore obtain station coordinates which result in images which depict, or are otherwise associated with, the specific cardiac views. As will be described, to determine that an image is associated with a cardiac view the system may leverage one or more trained classifiers or regression models. For example, a convolutional or transformer network may be used to assign a label, or score or value, indicative of the cardiac view (e.g., a score or value indicative of a likelihood of current station coordinates corresponding to a particular cardiac view). While a classifier is described herein, other techniques may be used and fall within the scope of the disclosure. For example, more generally a regression model may be used to approximate a real valued number.

[0041] The system may automatically navigate the TEE tool to obtain station coordinates for the specific cardiac views. In some embodiments, the TEE tool may cycle through different station coordinates and compute forward passes of images obtained at the station coordinates through the one or more classifiers. In this way, the system may cause automatic traversal through known station coordinates. In some embodiments, the system may use artificial intelligence techniques, such as deep learning techniques, to automatically navigate the TEE probe. For example, and as will be described in Figure 5, the system may analyze received images and determine the images’ association to a cardiac model. As an example, the system may map image information to a vector space or tensor space associated with the cardiac model.

[0042] In an effort to ensure that previously-determined station coordinates remainaccurate, the system may optionally trigger a redetermination of one or more station coordinates. For example, a patient may be bumped or moved slightly while under anesthesia. As another example, the TEE tool may be bumped or moved slightly. In yet another example the esophagus and organs constraint on the TEE may not be sufficient to maintain the relation between the TEE's perceived position and its real position relative to an initial global coordinate system

[0043] Thus, the specific station coordinates associated with a cardiac view may be inaccurate due to this movement. As will be described below, for example in Figures 4A- 4B, the system may update station coordinates. In this way, a medical professional may reliably cycle between cardiac views and corresponding images which accurately depict the cardiac views.

[0044] Additional disclosure related to an example TEE tool, and associated software, is included in International Publication WO 2023 / 147544 which is incorporated herein in its entirety.

[0045] The disclosed technology will now be described in more detail.

[0046] Figure 1A is a block diagram illustrating an example automated inspection system 100 in communication with a transesophageal echocardiogram (TEE) tool 110. As will be described, the automated inspection system 100 may provide control instructions 102 to cause movement of a TEE probe 114 within a patient. The TEE probe 114 may include one or more ultrasound sensors (e.g., ultrasound transducers) to perform ultrasound imaging. The automated inspection system 100 may therefore receive sensor information 112, such as images, from the TEE tool 112 and include information in a user interface 120 accessible to an end-user. Images may include, for example, ultrasound images, images or sensor information which includes a 3D volume over an increment of time, and so on.

[0047] The automated inspection system 100 may represent a system of one or more processors orone or more computers. In some embodiments, the automated inspection system 100 may be a computer system which is positioned proximate to a patient and the TEE tool 110. For example, the computer system may be a mobile device (e.g., a tablet, a smart phone), a laptop, a computer, and so on. In these embodiments, theautomated inspection system 100 may be in wired or wireless communication with the TEE tool 110. Additionally, the automated inspection system 100 may be in wired or wireless communication with a display configured to present user interface 120.

[0048] With respect to the automated inspection system 100 being proximate to a patient, the system 100 may execute software such as an application. The application may optionally be a mobile application which communicates with the TEE tool 110. Thus, and as an example, a tablet may execute the mobile application to control the TEE tool 110 and present user interface 120 via its touch-screen display.

[0049] In some embodiments, the automated inspection system 100 may represent a cloud-based system or software executing on a cloud-based system. For example, the automated inspection system 100 may be a web application (e.g., a dockerized application) which is in communication with a multitude of TEE tools. In this example, the TEE tool may be connected to a network (e.g., the internet, a private cloud or networked system) and controlled by the system 100. Similarly, an end-user device may be connected to the network and used to render user interface 120. For example, the end-user device may execute an application, or web browser, which renders user interface 120.

[0050] In the illustrated example, the automated inspection system 100 is providing control instructions 102 to the TEE tool 110. The TEE tool 110 may automatically adjust the probe 114 in a multitude of dimensions. For example, the TEE tool 110 may cause the TEE probe 114 to extend along a dimension and be adjustable in two or more dimensions. In this example, the TEE tool 110 may cause rotation of the TEE probe 114 (e.g., via flexion, retroflexion, flexion tilts). The TEE tool 110 may additionally cause rotation or adjustment of an ultrasound transducer (e.g., field of view), for example via beamforming techniques. Additional disclosure related to the TEE tool 110 is included in International Publication WO 2023 / 147544 which forms part of this disclosure as if set forth herein.

[0051] The control instructions 102 may thus cause adjustment of the TEE probe 114 within a patient. For example, the control instructions 102 may cause the TEE probe 114 to extend through the esophagus of the patient. In this example, the control instructions 102 may be associated with station coordinates which describe the TEEprobe’s 114 values in the multitude of dimensions (e.g., values for a multitude of degrees of freedom). As described above, example values may reflect the TEE probe’s 114 distance (e.g., from a consistent origin) into the patient, a pose or rotation of the probe, an electronic rotation, or other adjustment (e.g., focal distance), of the ultrasound sensor, and so on. Thus, the automated inspection system 100 may accurately navigate the TEE probe 114 to specific station coordinates via control instructions 102.

[0052] In some embodiments, the automated inspection system 100 may ensure that theTEE probe 114 safely navigates th rough the esophagus of a patient. For example, the automated inspection system 100 may determine a path from first station coordinates to second station coordinates. In this example, the first station coordinates may be associated with a first cardiac view and a medical professional may preferto view images associated with a second cardiac view. The automated inspection system 100 may determine control instructions 102 which ensure that pressure applied to the interior of the esophagus is below a threshold. As an example, the system 100 may cause a straightening out of the TEE probe 112 as it is translated towards the second station coordinates. In some embodiments, the automated inspection system 100 may access a geometrical model indicative of the interior of the esophagus. In some embodiments, the model may represent an average or normal model of a person’s esophagus. In some embodiments, the average or normal model may be adjusted based on deviations detected through imaging or through movement of the probe (e.g., via a force sensor). The model may then be used to inform the specific path towards the second station coordinates (e.g., an automated navigation via the specific path). In some embodiments, the information may be presented as instructions or recommendations to a user who is controlling the TEE tool.

[0053] The TEE probe 114 may additionally be configured with a force or pressure sensor. In this way, the TEE probe 114 may determine the extent to which the TEE probe 114 is applying pressure to the interior of the esophagus. If the pressure exceeds a threshold, the system 100 may adjust rotation of the TEE probe 114 along one or more dimensions to reduce the applied pressure (e.g., via flexion, retroflexion, flexion tilts, and so on). In some embodiments, an electrical impedance sensor may be used (e.g., single or array).

[0054] As the TEE tool 110 is controlled via the automated inspection system 100, the TEE probe 114 may obtain ultrasound images of the heart of a patient. These ultrasound images may be obtained, for example, at 30 Hz, 50 Hz, 60 Hz, 120 Hz, and so on. The TEE tool 110 may thus provide sensor information 112, which includes the ultrasound images, to the system 100.

[0055] The automated inspection system 100 may analyze the sensor information 112 to determine station coordinates which are associated with specific cardiac views. As will be described below, the automated inspection system 100 may execute one or more classifiers or regression models to effectuate the determination. Example classifiers or regression models may include deep-learning models, support vector machines, clustering techniques, and so on. With respect to deep-learning models, the automated inspection system 100 may compute forward passes through convolutional neural networks, transformer networks, fully-connected networks, and so on.

[0056] In this way, the automated inspection system 100 may present images which depict, or are otherwise associated with, specific cardiac views. In the illustrated example, user interface 120 is presenting an image depicting a two-chamber view of a patient’s heart. The user interface 120 further includes a confidence value indicative of a likelihood that the image depicts the two-chamber view. For example, the confidence value indicates an 87% certainty. As may be appreciated, a neural network may be trained to assign confidence or regression values to images as being associated with a particular label (e.g., via a sigmoid or sigmoid activation function).

[0057] As will be described below, with respect to Figures 2A-2C, the automated inspection system 100 may enable an end-user (e.g., a medical professional) to update station coordinates associated with a specific cardiac view. For example, if the confidence value is below a threshold the system 100 may update the user interface 120 to identify, or otherwise call out, the low confidence value. The end-user may then update the station coordinates to better conform to the specific cardiac view. In some embodiments, the system 100 may re-scan the local space to find a better conformal view. Optionally, the re-scan may be prioritized by electronic rotation of the ultrasound sensorthen motion (e.g., translation, rotation).

[0058] The user interface 120 may thus depict images associated with different cardiac views. An end-user may provide user input to traverse to different cardiac views to view real-time images from the TEE tool 110. Example user input may include verbal input, touch-based input, mouse / keyboard input, and so on. Advantageously, the user interface 120 may depict the last received images, or video sequences, associated with cardiac views. In this way, the end-user may view up-to-date images of different cardiac views. At the end-user’s discretion, the end-user may then cause navigation of the TEE probe 114 to a specific cardiac view to view real-time images (e.g., real-time ultrasound data) of that specific cardiac view.

[0059] Figure 1 B is a block diagram illustrating the example automated inspection system 100 obtaining sensor information 112 based on control of the TEE tool 110 using different station coordinates. In the illustrated example, an inspection volume 130 is included. As may be appreciated, the inspection volume 130 may represent a patient undergoing a TEE procedure. For example, the inspection volume 130 may include the patient’s esophagus. As another example, the inspection volume 130 may include the patient’s stomach. The inspection volume 130 may additionally include the organ being imaged (e.g., the heart).

[0060] In Figure 1 B, the automated inspection system 100 is providing control instructions 102 to the TEE tool 110. As described in Figure 1A, the control instructions 102 may cause the TEE probe to navigate to specific station coordinates.

[0061] In some embodiments, the automated inspection system 100 may perform cardiac view registration by traversing through pre-defined station coordinates and obtaining sensor information 112. For example, the system 100 may traverse through an ordered list of station coordinates. In some embodiments, the ordered list may vary based on a specific procedure being performed. As an example, a septal crossing procedure may be associated with specific station coordinates.

[0062] Initial station coordinates may cause, as an example, the TEE probe to translate by a threshold distance into the inspection volume 130. At the threshold distance, the automated inspection system 100 may then cause the TEE probe to maintain a particular initial rotation. At the distance, and initial rotation, the TEE probe may then obtain images at different electronic rotations and / or focal distances of theultrasound transducer. Once the TEE probe obtains images at the electronic rotations and / or focal distances, the system 100 may cause an adjustment to the rotation of the TEE probe (e.g., physical rotation).

[0063] At individual station coordinates, the TEE tool 110 may cease movement and then obtain sensor information 112 for a threshold amount of time. In some embodiments, the threshold amount of time may be one second, two seconds, three seconds, and so on. Optionally, the amount of time may relate to capturing sensor information 112 for a threshold number of cardiac cycles. Thus, in some embodiments a patient’s heart rate may be used to inform how the amount of time. In some embodiments, the automated inspection system 100 may receive electrocardiogram (ECG) data to identifya time at which the threshold number of cardiac cycles is reached.

[0064] Thus, in some embodiments the TEE probe may obtain images at each of the station coordinates for a threshold amount of time. In some embodiments, the TEE probe may be configured to adjust electronic rotation substantially quickly. For example, the TEE probe may be capable of adjusting the electronic rotation (e.g., adjustments may be at hundreds or thousands of hertz). In some embodiments, the system 100 may cause the ultrasound transducer to sweep through the different electronic rotations at a particular translation and rotation of the TEE probe. For example, since the adjustment may be substantially fasterthan the cardiac cycle an initial electronic rotation and a final electronic rotation in a sweep may be associated with a similar cardiac cycle. Thus, the system 100 may reduce an amount of time associated with cardiac view registration. As an example, the system 100 may avoid pausing at each electronic rotation value for a threshold amount of time and instead pause at different combinations of TEE probe translation and rotation values.

[0065] As described above, cardiac view registration may allow for an association of individual station coordinates with individual cardiac views. As will be described in Figure 1C, the automated inspection system 100 may thus analyze images obtained at individual station coordinates using one or more classifiers and / or regression models. Since the TEE tool 110 may obtain images at a particular frequency (e.g., 30 Hz, 50 Hz, 60 Hz, and so on) over a threshold amount of time or cardiac cycles, there may be a substantial number of images. In some embodiments, the system 100 may analyze asubset of the images which are associated with a specific cardiac phase.

[0066] An example specific cardiac phase may include an end-diastolic phase. As may be appreciated, the end-diastolic phase may cause the heart to be the most expanded such that images depict features of the heart more clearly. Thus, in some embodiments the automated inspection system 100 may identify end-diastolic images obtained at specific station coordinates. For example, the system 100 may use a classifier and / or regression model (e.g., a machine learning model) trained to identify end-diastolic images. In this example, an image may be input into the classifier. Optionally, one or more images before the image and one or more images after the image (e.g., in time) may be concatenated. Optionally, the classifier may input a set of images spanning one or more heartbeats (e.g., including both systole and diastole). The classifier may then determine a value indicative of the image being end-diastolic. Optionally, the system 100 may use ECG information to identify end-diastolic images. For example, the system 100 may determine a time at which the ECG information indicates the heart is an end-diastolic phase.

[0067] In the illustrated example, images are included for different station coordinates 132. For example, station coordinates A 132A may represent a combination of values associated with different degrees of freedom. As described above, the TEE tool 110 may maintain the TEE probe at station coordinates A 132A and obtain a multitude of images (e.g., Images A-N). Similarly, images associated with station coordinates N 132 are included in Figure 1 B.

[0068] Figure 1 C is a block diagram illustrating the example automated inspection system 100 performing cardiac view registration. As described in Figure 1 B, the system 100 may cause a multitude of images to be obtained from the TEE tool for individual station coordinates. As will be described, the system 100 may identify which station coordinates correspond to which cardiac views.

[0069] The automated inspection system 100 includes a control and classifier engine 150 to assign confidence values to images 152. For example, the control and classifier engine 150 may provide control instructions to cause the TEE tool to navigate to particular station coordinates. The engine 150 may then receive sensor information 112 that includes images 152 obtained at the particular station coordinates.As described above, the engine 150 may execute one or more classifiers and / or regression models to assign confidence values associated with different cardiac views. In some embodiments, the confidence values may depend on, or be weighted relative to, a particular procedure.

[0070] In some embodiments, a classifier and / or regression model may be trained to assign confidence values or scores for a multitude of cardiac views. For example, the classifier may output confidence values for a multitude of labels. In some embodiments, individual classifiers may be trained for individual cardiac views. In these embodiments, the engine 100 may optionally select a subset of classifiers to receive the images 152 as input. As an example, the subset may be based on the station coordinates associated with the TEE probe’s position. For example, certain station coordinates may be expected to be able to view a subset of cardiac views.

[0071] As described in Figure 1 B, the automated inspection system 100 may filter the sensor information 112 such that images 152 represent images associated with a particular cardiac phase (e.g., end-diastolic images). In this way, the system 100 reduce an extent to which forward passes through the classifier engine 150 are required to be computed.

[0072] The automated inspection system 100 further includes a registration engine 160 which stores station coordinates for specific cardiac views. For example, the registration engine 160 may store station coordinates for a two-chamber view based on confidence values for images 152 exceeding a threshold. As may be appreciated, as the TEE probe adjusts in translation and / or rotation the system 100 may obtain images with greater confidence values for the two-chamber view. Thus, the registration engine 160 may update the station coordinates associated with the two-chamber view during cardiac view registration.

[0073] In this way, the registration engine 160 may store station coordinates for different cardiac views. The registration engine 160 may optionally generate a lookup table which may be relied upon to navigate backto specific cardiac views. For example, an end-user may use a user interface (e.g., user interface 120) to request real-time images corresponding to a specific cardiac view.

[0074] Figure 1 D is a block diagram illustrating the example automated inspection system 100 obtaining real-time image(s) of a requested cardiac view 170 based on the cardiac view registration. As described above, the automated inspection system 100 may generate a lookup table which associates station coordinates with cardiac views.

[0075] In the illustrated example, an end-user has provided a cardiac view request 170. The request 170 may indicate the cardiac view being requested and be provided, for example, via a user interface (e.g., user interface 120). As an example, the end-user may select a particular cardiac view (e.g., from a list, from a drop-down menu, from presented textual identifiers, verbally saying the particular cardiac view, and so on).

[0076] The registration engine 160 may receive the request 170 and determine station coordinates associated with the cardiac view. For example, in Figure 1 D the end-user has requested a two-chamber view. In this example, the registration engine 160 has determined (e.g., using a lookup table) that the two-chamberview is viewable at specific station coordinates 162. The control and classifier engine 150 may then cause the TEE tool to navigate to the specific station coordinates 162.

[0077] The user interface may then be updated to present a real-time image 172, or real-time images 172, of the two-chamber view.

[0078] In some embodiments, the automated inspection system 100 may trigger a new cardiac view registration. For example, the real-time image(s) 172 may be analyzed the control and classifier engine 150. In this example, a classifier and / or regression model may assign a confidence value or label associated with the two-chamberview. If the confidence value is less than a threshold, or greaterthan a threshold reduction from a prior confidence value, the system 100 may trigger a redetermination of station coordinates associated with the two-chamber view. Additional description related to triggering a determination is described in more detail below, with respect to Figures 4A- 4B.

[0079] The description above focused on use of station coordinates, for example in a look-up table. However, in some embodiments the control and classifier engine 150 may determine a path or trajectory to specific station coordinates using received sensor information. For example, the control and classifier engine 150 may use a cardiac modelto determine a path towards the station coordinates. In this example, the engine 150 may generate data indicative of the cardiac model during cardiac view registration. As an example, as sensor information is received the engine 150 may map image features into positions of a consistent vector space. The engine 150 may optionally be trained to associate such image features with the vector space to form a cardiac model. In some embodiments, the engine 150 may use certain markers, such as the apex of the left ventricle, to inform positioning of the image features.

[0080] In some embodiments, a standard or generalized cardiac model may be used by the system 100. For example, the standard cardiac model may be based on average heart features of patients. In this example, as sensor information 112 is received the system 100 may update the standard cardiac model to conform to the sensor information 112. For example, a marker (e.g., the apex of the left ventricle) may be used to adjust the standard cardiac model in orientation (e.g., different patients may have slightly differently oriented hearts). As another example, the geometry of the cardiac heart model may be updated based on the sensor information 112. As an example, a length and / or size associated with portions of the cardiac model may be updated based on the sensor information 112. With respect to length, the system may determine length based on the station coordinates of the TEE tool 110 in combination with received images. For example, stereo vision techniques, or other computer vision techniques, may be leveraged to estimate length and / or size.

[0081] With respect to the standard cardiac model, the system may determine a path towards a particular cardiac view (e.g., as described in Figure 1 D). Since the standard cardiac model may not correspond precisely to the patient’s anatomy, the system may navigate the TEE tool to station coordinates which are expected to be proximate to station coordinates associated with the particular cardiac view. The system may then cause refinements to the TEE tool’s station coordinates to obtain images at the refined station coordinates. Via analyzing the images, for example using a classifier, the system may determine station coordinates most closely matching the requested cardiac view.

[0082] Figure 2A is a flowchart of an example process 200 for cardiac view registration usinga TEEtool. Forconvenience. the process 200 will be described as being performed by a system of one or more processors (e.g., the automated inspectionsystem 100).

[0083] At block 202, the system causes presentation of an interactive user interface associated cardiac analysis. The interactive user interface may be presented via a display of the system, or in which is in communication with the system. Example user interfaces are illustrated in Figures 2B-2C and described below.

[0084] At block 204, the system initiates cardiac view registration. As described above, the system may control a TEE tool to obtain images at different station coordinates within a patient. Cardiac view registration may be initiated based on receipt of user input from an end-user (e.g., a medical professional). For example, the end-user may provide user input to the user interface indicative of initiating the registration.

[0085] At block 206, the system causes adjustment of the TEE tool. In some embodiments, and as described in Figure 1 B, the system may cause traversal through particular station coordinates. For example, the particular station coordinates may be pre-stored. In some embodiments, the particular station coordinates may be adjusted based on the patient. For example, the system may reduce an extent to which the TEE tool extends into a shorter patient as compared to a taller patient.

[0086] At block 208, the system determines station coordinates for individual cardiac views. As the TEE tool navigates to station coordinates, the system analyzes received images from the TEE tool. The system may use, for example, a classifier and / or regression model which is trained to output confidence values associated with cardiac views. Thus, the system may determine which station coordinates correspond to images depicting which cardiac views.

[0087] At block 210, the system updates the user interface based on the cardiac view registration. The system may include an example image for each cardiac view in the user interface. Optionally, the system may indicate a confidence value for each example image. In this way, the end-user may quickly identify how responsive each image is to an associated cardiac view. As will be described in Figures 3A-3C, the end-user may optionally update (e.g., manually update) station coordinates. For example, the enduser may manually control the TEE tool.

[0088] Figure 2B is an example user interface 220 enabling selection of informationrelevant to the cardiac view registration. User interface 220 may be presented to an end-user near the start of a TEE procedure. The end-user may select from different preset types of procedures. Example procedures may include MitralClip, Mitral Valve, atrial appendage closure, and so on as known by those skilled in the art. As described above, these selections may inform, at least, the different cardiac views which may be of interest to the end-user.

[0089] Figure 2C is an example user interface 230 that includes images determined to be associated with specific cardiac views based on the cardiac view registration. In the illustrated example, four cardiac views are included along with ultrasound images depicting the four cardiac views. For example, an image corresponding to a two-chamber view is included in the upper left. As another example, an image corresponding to four-chamber view is included in the upper right. As another example, an image corresponding to a five-chamber view is included in the lower left. As another example, an image correspondingto an MC view is included in the lower right.

[0090] The user interface 230 further includes confidence values for each of the cardiac views. These confidence values, in the example, range from 98% certainty to 89% certainty. Thus, an end-user may review the confidence values to determine whether to manually update station coordinates which are associated with the cardiac views.

[0091] The user interface 230 may respond to selection of a particular cardiac view. For example, selection of the particular cardiac view may trigger the system 100 to navigate the TEE tool to station coordinates associated with the particular cardiac view. The user interface 230 may then update with real-time image(s), or specific image(s) (e.g., end-diastolic images), from the TEE probe while at the station coordinates.

[0092] Figure 3A is a flowchart of an example process 300 for updating station coordinates of a TEE tool that result in images associated with a particular cardiac view. For convenience, the process 300 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0093] At block 302, the system determines confidence values associated with cardiac views during cardiac view registration. As described above, the system causesthe TEE tool to traverse through station coordinates to obtain images at the station coordinates. The system analyzes the images and determines station coordinates which are associated with the cardiac views. In addition, the system determines confidence values indicative of respective confidences that the station coordinates are associated with the cardiac views.

[0094] At block 304, the system responds to user input to update station coordinates fora particular cardiac view. A user interface, such as included in Figure 2C and 3B, may be presented which includes individual images that correspond to individual cardiac views. The user interface may additionally presentthe confidence values for the cardiac views.

[0095] As described herein, an end-user may provide user input indicative of a request to update station coordinates (e.g., for the particular cardiac view). For example, the particular cardiac view may have a confidence value less than a threshold. In this example, the user interface may identify, or otherwise visually or auditorily call out, the low confidence value. As another example, the end-user may view the image which corresponds to the particular cardiac view and prefer to change the station coordinates.

[0096] At block 306, the system determines updated station coordinates for the particular cardiac view. The system may receive user input which manually controls the TEE tool. For example, the user interface may include controls to adjust the station coordinates. In this example, the end-user may cause translation of the TEE tool, rotation of the tool, electronic rotation of the ultrasound transducer, and so on.

[0097] The system may update the user interface to include real-time images from the TEE tool during manual control. In this way, the end-user may cease manual control based on finding one or more real-time images to satisfactorily correspond to the particular cardiac view. The system then stores the updated station coordinates as being associated with the particular cardiac view.

[0098] At block 308, the system optionally updates station coordinates for other cardiac views. The system may determine that some, or all, remaining cardiac views may have associated station coordinates adjusted. For example, the updated station coordinates determined in block 306 may be used as a mapping from the expectedstation coordinates determined by the system to the user-refined station coordinates. As another example in which a cardiac model is used, the updated station coordinates may be used to model three-dimensional positions associated with remaining cardiac views. For example, if the particular cardiac view in block 306 is a two-chamber view then the cardiac model may be used to inform a refinement to station coordinates associated with the four-chamber view.

[0099] As described above, in some embodiments the system may traverse through station coordinates and analyze images obtained at those station coordinates. Thus, there may be respective threshold differences in the degrees of freedom during traversal between station coordinates. For example, first station coordinates and second station coordinates may differ by a threshold translation distance or threshold rotation degree. In this way, the system may analyze images obtained at discrete steps through an inspection volume (e.g., an esophagus as described in Figure 1 B).

[0100] During cardiac view registration, for example as described at least in Figure 2A, the system may therefore associate one of the above-described discrete steps (e.g., particular station coordinates) with a cardiac view. In some embodiments, the system may update station coordinates for other cardiac views to correspond with station coordinates which were not initially searched. For example, if the particular view in block 306 is a two-chamber view then the system may update station coordinates for the four- chambre viewto correspond with discrete values which were not initially traversed. Thus, the updated station coordinates may reflect a refinement to the discrete steps through the inspection volume. Forexample, the updated station coordinates maybe in between the discrete steps.

[0101] For any updated station coordinates, the system may optionally navigate to those updated station coordinates and present one or more images to the end-user. The end-user can then validate that the updated station coordinates provide more accurate views of the associated cardiac views. The system may optionally input the images to the classifier and / or regression model described herein to determine confidence scores.

[0102] Figure 3B is an example user interface 310 that enables selection of a cardiac view 312 to be adjusted. As described in Figure 2C, a user interface may be presented with example images which correspond to cardiac views. Proximate to the images, theuser interface may reflect confidence values associated with the cardiac views. In this way, an end-user may rapidly identify whether the station coordinates are determined to confidently reflect a particular cardiac view.

[0103] In the illustrated example, user interface 310 includes an image associated with a five-chamber cardiac view 312. The user interface 310 indicates that a confidence value that the image reflects the five-chamber cardiac view 312 is “60%”. In some embodiments, the system may cause the user interface 310 to reflect that the confidence value is below a threshold. For example, a portion of the user interface that includes the image may be highlighted or otherwise called out. As another example, the user interface 310 may include a marker 314 which is of a particular color (e.g., red, a user-selectable color).

[0104] The user interface 310 may include a selectable option 316 to adjust the station coordinates associated with the five-chamber view. For example, the user interface 310 may respond to user input from the end-user to cause presentation of selectable option 316. Example user input may include touching the presented image (e.g., a long press, a short press, press with greater than a threshold force or pressure, and so on). Example user input may also include verbal input, mouse / keyboard input, and so on.

[0105] Figure 3C is an example user interface 320 depicting manual adjustments to station coordinates of the TEE tool. User interface 320 may be presented, in some embodiments, in response to selection of selectable option 316 described in Figure 3B. In the example of Figure 3C, the user interface 320 includes manual controls associated with the TEE tool described herein. For example, the manual controls are included in user interface portion 322. As illustrated, the end-user may adjust the translation 324, rotation 326, and electronic rotation 328 via interactions with portion 322.

[0106] The user interface 320 additionally includes user interface portion 324 which may present information relevant to the end-user determining updated station coordinates. For example, user interface portion 324 may include real-time images from the TEE tool (e.g., from the ultrasound transducer included at the terminal end of the TEE probe). In this example, the real-time images may be adjusted by the system 100. For example, automated gain adjustments, filtering, and so on may be performed. Inembodiments in which a cardiac model is used, the user interface portion 324 may present a graphical representation of the cardiac model along with an estimated location of the TEE probe (e.g., the ultrasound transducer).

[0107] The user interface 320 may respond to user input which causes updated station coordinates to be saved. For example, the updated station coordinates may be registered as being associated with the five-chamber view. In implementations in which a look-up table is used, the updated station coordinates may be included in the look-up table as being associated with the five-chamber view.

[0108] The user interface 320 includes selectable option 330 which enables bookmarking of station coordinates. In some embodiments, the end-user of user interface 320 may create a custom cardiac view by interacting with selectable option 330. For example, the end-user may identify an interesting cardiac view during manual control ofthe TEE tool. In response, the system 100 may store information identify! ng the custom cardiac view along with associated station coordinates. The custom cardiac view may be included in a user interface, such as user interface Figure 2C, optionally alongwith an identifier assigned by the end-user.

[0109] Figure 4A is a flowchart of an example process 400 for determining updates to station coordinates in response to one or more triggers. For convenience, the process 400 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0110] Process 400, as will be described, may be performed to update the station coordinates during a TEE procedure. As described above, the patient may move under her / his own power while anesthetized or may be inadvertently bumped or moved slightly during the procedure. Additionally, the TEE tool (e.g., tool 110) may be inadvertently bumped or moved. Thus, the system may trigger a redetermination of station coordinates for at least some of the cardiac views. As noted above, in some embodiments the physiological constraints on the probe may not be sufficient to allow the probe to return to the same physical location relative to a physiological ordinate such as a specific view - e.g. the esophagus shifts or the heart shifts.

[0111] At block 402, the system obtains sensor information the TEE tool. During theTEE procedure, the TEE tool may provide sensor information to the system for analysis and / or presentation in a user interface. For example, a medical professional may view real-time images from the TEE tool (e.g., from the ultrasound sensor in the TEE probe).

[0112] In some embodiments, the TEE tool may have a force sensor (e.g., a force transducer, force sensing resistor, and so on) positioned proximate to the ultrasound sensor. The system may utilize the force sensor, for example, to determine whether the TEE probe is applying greater than a threshold pressure or force to the patient. The system may also utilize the force sensor to determine if the TEE probe has inadvertently moved. For example, the system may determine that the registered force sensor readings are indicative of movement within the patient.

[0113] At block 404, the system triggers updates to station coordinates for one or more cardiac views based on the sensor information. As described above, the system may trigger the updates based on inadvertent movement of the patient and / or TEE tool.

[0114] An example technique to detect movement includes analyzing the real-time images from the TEE tool. For example, a medical professional may have requested that the TEE tool navigate to station coordinates associated with a particular cardiac view. In this example, the system may receive images from the TEE tool at the station coordinates. The system may perform comparisons between the received images and one or more previously stored images associated with the particular cardiac view. The previously stored images may have been obtained, for example, during cardiac view registration as described at least in Figure 2A above. The comparisons may be performed between images which are associated with a same, or similar, cardiac phase. For example, the system may compare a received image which is end-diastolic with a prior image which is end-diastolic.

[0115] An example comparison may be based on mean squared error between a received image and a previous image. Another example comparison may be based on a structural similarity index. Another example comparison may be based on use of local descriptors, such as use of SIFT, SURF, and so on. Another example comparison may be based on deep learning techniques (e.g., Siamese networks). Additional comparisons, such as use of similarity metrics, may be used and fall within the scope of the disclosure herein.

[0116] Another example technique to detect movement is based on use of one or more force sensors. For example, the TEE tool may have a force sensor, an accelerometer, an inertial measurement unit (IMU), and so on, which may be used to detect a bump or adjustment to the TEE tool. Similarly, the patient, table on which the patient is lying, and so on, may have a force sensor, accelerometer, IMU, and so on. TEE probe may have a force sensor and may be used to determine whether the probe has moved within the patient.

[0117] Another example technique to detect movement is based on analyzing images received from the TEE tool. While the TEE tool is at particular station coordinates, the images received from the TEE tool may be expected to have a same viewpoint. As may be appreciated, the heart is in continual movement such that the images may form a video sequence of the movement. The system may analyze the received images to determine whether the TEE tool is being adjusted in viewpoint. For example, the system may determine motion vectors which are associated with certain image features in the received images. In this example, certain image features (e.g., a portion of the heart) may be expected to move due to movement of the heart. However, such movements may be monitored and if their position and / or orientation in the images changes then the system may determine that the TEE tool has moved.

[0118] In some embodiments, the system may average, or add, images which form a video sequence of a cardiac cycle or a portion thereof. The average, or addition, may be expected to be similar during a fixed period of time (e.g., a cardiac cycle, such as determined using image processing or ECG data). Thus, if the average, or addition, changes the system may determine that the TEE tool has moved.

[0119] At block 406, the system determines updates to station coordinates. The system may update the station coordinates for all, or a subset, of the cardiac views. For example, the system may perform a new cardiac view registration process to re-determine the station coordinates. The system may also refine the previously determined station coordinates. For example, the system may navigate to each, or some, of the previously determined station coordinates and perform a fine search around the station coordinates. In this example, the system may analyze images using a classifier and / or regression model (e.g., as described above) to determine updated stationcoordinates.

[0120] An example of adjusting station coordinates will now be described with reference to Figure 4B.

[0121] Figure 4B is a flowchart of an example process 410 for determining adjustments to station coordinates. For convenience, the process 410 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0122] At block 412, the system causes adjustment of the TEE tool to station coordinates associated with a particular cardiac view. The particular cardiac view may represent, for example, a cardiac view requested by an end-user. As described above, the system may cause the TEE tool to navigate to the particular cardiac view. For example, the system may control the TEE tool based on station coordinates associated with the particular cardiac view. In implementations in which a cardiac model is used, the system may navigate based on the cardiac model.

[0123] Additional description related to navigation using a cardiac model is included below with respect to, at least, Figure 6.

[0124] At block 414, the system obtains images from the TEE tool at the station coordinates.

[0125] At block 416, the system compares the images with images stored during cardiac view registration. As described in Figure 4A, the system may compare real-time images from the TEE tool with previously stored images which are associated with the station coordinates.

[0126] At block 418, the system determines adjustments to the station coordinates. Based on the comparisons indicating that the TEE tool has moved since cardiac view registration, for example as described in Figure 4A, the system may determine updated station coordinates. These updated station coordinates may then be stored by the system, for example included in the previously-described look-up table which associates station coordinates to cardiac views. In some embodiments, a global coordinate shift may be used such that the original station coordinates are valid.

[0127] In some embodiments, the system may identify previously stored images which are closest (e.g., based on the comparisons described above) to the real-time images. As described in Figure 1 B and 2A, during cardiac view registration the system may cause the TEE tool to navigate to discrete station coordinates. Images obtained at these discrete station coordinates may then be analyzed via one or more classifiers. Thus, the system may have access to prior images taken at a multitude of station coordinates to which the TEE tool navigated during cardiac view registration.

[0128] The system may therefore compare the real-time images to these prior images. For example, the system may compare the real-time images to prior images taken at station coordinates which are proximate to the station coordinates of block 412. In this example, the system may identify that prior images taken at particular station coordinates are the closest to the real-time images. Thus, the system may associate the particular station coordinates with the particular cardiac view of block 412. In some embodiments, the system may perform a search around the particular station coordinates to refine the coordinates.

[0129] As described in Figure 4A, the system may compare real-time images with prior images using a particular cardiac phase. For example, the system may compare prior images which are end-diastolic to real-time images which are end-diastolic. Since the system has access to the prior images, the system may optionally compute a multitude of comparisons in parallel. For example, the system may obtain real-time images over one or more cardiac cycles. In this example, the system may identify end-diastolic images and compare them with previously-obtained end-diastolic images from a multitude of station coordinates. Optionally, the system may sum, or average, the obtained real-time images and compare them with summed, or averaged, prior images from the multitude of station coordinates.

[0130] Optionally, the system may update station coordinates associated with other cardiac views. For example, the system may determine a mapping (e.g., linear mapping) between the station coordinates determined in block418 with the station coordinates of block 412. The mapping may be applied to other station coordinates to translate them to the new orientation of the TEE probe after movement.

[0131] In some embodiments, the updated station coordinates may inform updatesto the other cardiac views. For example, rather than applying a linear mapping the system may adjust station coordinates for the other cardiac views based on a difference between the translation (e.g., length) determined in block 418 is different from the translation (e.g., length) of block 412. The difference may be assumed to reflect that the TEE tool extended further into, or further out of, the patient. This difference may then be applied to remaining cardiac views. Similar to the above, the system may optionally navigate to the updated station coordinates for remaining cardiac views and perform local searches to refine the coordinates.

[0132] In some embodiments, the system may update station coordinates of a single cardiac view. The system may then navigate the TEE tool to those updated station coordinates and obtain real-time images. The system may use a classifier and / or regression model, along with a local search about the updated station coordinates, to refine the coordinates for the single cardiac view. The system may then update the mapping and repeat the process for another cardiac view. In this way, the system may determine a more complete picture regarding updates to cardiac views caused due to movement of the TEE tool.

[0133] Figure 5 is a flowchart of an example process 500 for cardiac view registration based on use of cardiac model data. Forconvenience, the process 500 will be described as being performed by a system of one or more processors (e.g., the automated inspection system 100).

[0134] At block 502, the system initiates cardiac view registration associated with cardiac views. As described above, for example in Figure 2A, the system may perform cardiac view registration to associate station coordinates with cardiac views.

[0135] At block 504, the system causes a first adjustment of a TEE tool. The system may navigate the TEE tool, such as extending and / or rotating a TEE probe at the terminal end of the TEE tool, within a patient. As the TEE tool is maneuvered within the patient, the system receives images from the TEE tool for analysis.

[0136] At block 506, the system associates received images with portions of a cardiac model. The system may execute a machine learning model, such as a convolutional or transformer neural network, to detect image features. The machine learning model mayadditionally be trained to position the image features within a cardiac model. For example, the machine learning model may be trained using image data with ground truth data indicating image features and associated positions within a heart. As the TEE tool is controlled to extend laterally in the patient, and to be rotated in position, the system may map detected image features into a consistent vector space. In this example, the image features maybe represented as voxels or other data which identifies position, size, and so on, of the image features.

[0137] In some embodiments, the cardiac model may represent a model which corresponds to an average, or common, heart. The system may detect certain markers or features which may be used to adjust the cardiac model. For example, an orientation of the patient’s heart within her / his body may be determined via detection of the apex of the left ventricle. As another example, size information associated with portions of the heart may be determined via the received images. For example, three-dimensional size information may be extracted via correlating same image features as depicted at different station coordinates. Thus, the cardiac model may be refined to better correspond to the patient’s heart.

[0138] In this way, an estimated position of the TEE tool may be determined as it navigates through the patient’s esophagus. For example, real-time images may be analyzed to compare the viewpoint of the heart as depicted in the images with an expected viewpoint based on the cardiac model and the current station coordinates of the TEE tool.

[0139] At block 508, the system determines second adjustment of the TEE tool. The system may use the cardiac model to more rapidly navigate to cardiac views. For example, the system may estimate station coordinates for a particular cardiac view during cardiac view registration. In this example, the estimate may be based on the cardiac model as described in block 506. Thus, the system may cause the TEE tool to navigate to the estimated station coordinates. The system may then perform a local search about the estimated station coordinates and analyze received images using the above-described classifier. In this way, the system may identify station coordinates which are to be associated with the particular cardiac view.

[0140] The above-described identification may be used to update and / or refine thecardiac model described in block 506. For example, the system may adjust dimensions and / or orientation of the cardiac model. As an example, the cardiac model may be updated to correspond with station coordinates which have been determined to be associated with specific cardiac views during cardiac view registration.Cardiac Model Navigation

[0141] As described above, a cardiac model may be used, in part, to inform navigation of the TEE tool. For example, the cardiac model may be generated based on sensor information (e.g., ultrasound data) received from the TEE tool as it moves within a portion of a patient (e.g., the esophagus). As another example, the cardiac model may be based on an average, or common, heart model of patients. The above-described system (e.g., automated inspection system 100) may adjust the cardiac model to customize it based on a specific patient. To obtain a particular view, such as a four- chamber view, the system may identify the view based on analyzing the model.

[0142] In some embodiments, the system may access a trained machine learning model which outputs information indicative of the views described herein. The machine learning model may represent a neural network (e.g., a dense or fully-connected network), a convolutional neural network, a transformer-based neural network, and so on. As will be described, the system may input the cardiac model of the patient (e.g., information defining the cardiac model) and obtain information indicative of the views. In this way, the machine learning model may be relied upon to output, for example all at once or sequentially, the specific views which are of interest to a medical professional.

[0143] The TEE probe, as described herein, may include an ultrasound sensor which obtains ultrasound sensor data of a patient’s heart. The TEE probe may be positioned such that it moves within the patient while pointing substantially at the heart. As described above, the TEE probe may have a multitude of degrees of freedom to obtain ultrasound sensor data with different field of views. For example, the TEE probe may be able to translate within the patient, rotate within the patient, have an electronic focus adjuster, have the ultrasound electronically or physical rotate, and so on as described herein. In some embodiments, the TEE probe may represent a three-dimensional or four-dimensional echocardiography probe. Thus, the TEE probe may obtain information which is able to determine a three-dimensional representation of the heart.

[0144] As an example, the TEE probe may obtain a plane of sensordata which extends substantially from an apex, or region, (collectively the apex) associated with the TEE probe. The plane, as may be appreciated, may be substantially triangular. TheTEE probe may therefore obtain sensor data while at particular station coordinates with the field of view relating to the apex.

[0145] The TEE probe may be navigated within the patient while obtaining ultrasound sensor data at different station coordinates. In some embodiments, the ultrasound sensor data may represent ultrasound images which image along the above-described planes. In some embodiments, the ultrasound sensor data may represent, or be transformed into, a point cloud. For example, the TEE probe may obtain ultrasound data at different depths within a volume of space. TheTEE probe may optionally adjust up and down such that the plane sweeps up and down to extend the volume of space up and down along with increasing in depth. In this example, a point cloud may be generated that represents points orfeatures within the volume of space.

[0146] The system may thus generate a three-dimensional model of the patient’s heart based on the ultrasound sensor data received via the TEE probe. As an example, the system may use a machine learning model to combine (e.g., stitch together) features in the ultrasound sensor data to form a consistent three-dimensional model. As another example, computer vision techniques (e.g., point cloud registration techniques) may be used to form the three-dimensional model. For exam pie, the system may use ultrasound images to form the three-dimensional model. As another example, the system may use point clouds to form the three-dimensional model.

[0147] Without being constrained by way of theory or example, the system may use ultrasound sensor data associated with a particular cardiac phase of the heart. In this way, the ultrasound sensor data may reflect features of the heart at substantially similar positions. These features may then form the three-dimensional model of the patient’s heart. In some embodiments, the system may obtain ultrasound sensor data while at particular station coordinates. The system may then rapidly adjust the ultrasound sensor up and down and optionally in depth to obtain sensor data. The ultrasoundsensor, via this electronic adjustment, may therefore obtain ultrasound sensor data of the heart while at a substantially similar cardiac phase. As the TEE probe sensor moves, the system may similarly obtain ultrasound sensor data at the cardiac phase. In some embodiments, the system may use machine learning techniques to form a three- dimensional model using ultrasound sensor data from arbitrary cardiac phases.

[0148] While obtaining ultrasound sensor data to generate the three-dimensional mode, the system may store information describing the portion of the patient in which the TEE probe is maneuvering. For example, the system may identify how far the probe extends into the patient. In this example, the system may use this information to, in part, define each apex at which the TEE probe obtains ultrasound sensor data. In some embodiments, the TEE probe may include a force or other sensor to detect when the TEE probe is touching, and optionally amount of pressure or force being applied to, the patient’s body. This information may be used to inform navigation as described herein. The information may additionally be used to inform a contour of the patient’s interior. Thus, the system may have, for a given apex, the station coordinates of the TEE probe along with some information describing the patient’s interior at those station coordinates.

[0149] In some embodiments, the system may additionally access other types of medical data which may define geometry information associated with the patient’s interior (e.g., throat, esophagus). For example, a computed tomography (CT) scan may be performed and used to define a volume in space in which the TEE probe is able to operate. The system may use this information to determine where in the patient the TEE probe is relative to the volume of space.

[0150] As described above, the three-dimensional model may be used to inform navigation of the TEE probe. For example, a medical professional may prefer to view a particular medical view of the patient’s heart. In this example, the system may cause the TEE probe to adjust in station coordinates such that the TEE probe is able to obtain a realtime (e.g., live) view of the particular medical view. Specifically, a machine learning model may be trained to output information indicative of different medical views.

[0151] For example, the machine learning model may be trained to translate between a three-dimensional cardiac model and specific views. As described above, the TEEprobe may obtain ultrasound data from a particular apex with the ultrasound sensor data corresponding to a plane extending from the apex. Each view may thus represent one of these planes.

[0152] The above-described machine learning model may thus output information identifying specific apexes at which respective cardiac views may be obtained. Each apex may optionally be defined according to particular station coordinates. Thus, the machine learning model may output station coordinates which correspond to different cardiac views. Optionally, the machine learning model may receive information describing geometry of the patient’s esophagus or the station coordinates used to generate the cardiac model. Using this information, the model may directly output station coordinates which correspond to medical views.

[0153] Optionally, the machine learning model may output information which is adjusted based on the patient. For example, the model may output information which is relative to the input cardiac model. In this example, the information may indicate apexes at which the TEE probe will extend imaging planes which are associated with cardiac views. As an example, the model may identify that a four-chamber view is associated with a specific orientation, position, and so on, relative to the cardiac model. That is, the four-chamber view is associated with a plane that extends from an apex at the specific orientation, position, and so on. The system may then use this output to determine the specific station coordinates for the patient. For example, the system may determine a length of translation by the TEE probe when obtaining sensor data to form the cardiac model. The system may also use information from the force or touch sensor to inform the contour of the esophagus. The system may also use CT data. Thus, the system may determine the station coordinates which correspond to output of the machine learning model.

[0154] Figure 6 illustrates an example cardiac model along with an imaging plane. In the illustrated example, a green and red line define a plane which is being imaged by the TEE probe. A user interface may present the cardiac model (e.g., a dynamic / beating modelor a substantially fixed model). The user interface may additionally present a realtime view of a cardiac view. In some embodiments, Figure 6 may be presented to illustrate which plane corresponds to the cardiac view.

[0155] In some embodiments, the system may present a dynamic view of the patient’s heart. For example, the system may obtain ultrasound sensor data while at particular station coordinates and use that to inform updating of the cardiac model. In this example, the ultrasound sensor data may have limited visibility into the full heart. However, the movement of the heart as seen by the TEE probe may be monitored and used to update the cardiac model. For example, the movement may be used to cause or simulate motion of the cardiac model. In this example, the system may track specific landmarks or features which may be used to inform motion of the overall heart. In some embodiments, when the TEE probe is moving to navigate to station coordinates for a cardiac view, the TEE probe may traverse at a particular speed such that the TEE probe is substantially in phase with the cardiac phase of the heart.

[0156] To control the TEE probe, the system described herein may allow for fine-grain and course-grain controls. In some embodiments, the system may execute a machine learning model which maps input of simplified user input to complex controls of the TEE probe. As known by those skilled in the art, there may be a plethora of controls for an ultrasound sensor. Thus, a medical professional using a user interface to control the TEE probe may provide simplified input regarding where the TEE probe is to navigate or an intended goal (e.g., obtain a clearer picture, move to see a particular feature in more detail, and so on). This information may be analyzed, for example via a machine learning model, to output the specific fine-grain controls which would be required to implement the navigation or intended goal.

[0157] Figure 7 illustrates an example user interface. In this user interface, an indication of force being applied by the TEE probe is shown (e.g., 19 grams). As described above, this may be determined using a force sensor. In addition, the user interface includes a review option at the upper left. The review option may provide summary information associated with the cardiac views, cardiac model, and so on.

[0158] In some embodiments, the system may include functionality which may be organized into disparate layers. For example, the functionality may be organized similar to that of the open systems interconnection (OSI model). In this example, lower-level control information may represent a lower-level layer. Upper-level layers may include higher level data, such as a layerto extract a cardiac model, a layerto perform movement,commands which cause translation into fine-grain controls, and so on.Patient Mesh Generation / Landmark-based Navigation

[0159] As described above, a system (e.g., the automated inspection system 100) may cause substantially autonomous navigation about a patient’s esophagus. For example, the system may control degrees of freedom associated with a TEE tool (e.g., the TEE probe described herein) to obtain distinct cardiac views of interest to medical professionals. Example degrees of freedom, which are referred to herein as station coordinates, may include translation, flexion, rotation, and so on.

[0160] In an effort to rapidly present the cardiac views, as described above the system may leverage classifier(s) which are trained to detect specific cardiac views based on input of ultrasound images. For example, Figure 2C illustrates ultrasound images as being associated with specific cardiac views. In this example, confidence measures are included to indicate a degree to which the classifier(s) are confident in the cardiac views.

[0161] Such techniques, as an example, may thus rely upon analyses of real-time received ultrasound images. The system may record particular station coordinates that reflect particular cardiac views. While these techniques enable substantially autonomous determination as to cardiac views, they may, as an example, require iterative analysis of ultrasound images while the TEE tool adjusts station coordinates. This may lead to increased time for the system to obtain the relevant cardiac views. Furthermore, this iterative analysis may be required for each use and there may be difficulties with generalizing station coordinates across patients. As an example, station coordinates corresponding to a four-chamber view in a first patient may be distinct from the station coordinates in a second patient.

[0162] As will be described, in some embodiments the system may determine a patient mesh reflecting an individualized geometry associated with a patient’s heart. For example, the mesh may reflect a contour of a surface associated with the patient’s heart. The mesh may, in some embodiments, be formed from a multitude of triangles or other elements (e.g., polygons, implicit surfaces). In some embodiments, the mesh may begenerated based on input of ultrasound image(s) of the patient’s heart. For example, the TEE tool described herein may obtain ultrasound images, such as three-dimensional (‘3D’) ultrasounds. In this example, the system may compute a forward pass through one or more machine learning models to output the patient mesh.

[0163] Advantageously, the system may rapidly generate the patient mesh based on a threshold number of ultrasound images (e.g., 3D ultrasounds). In some embodiments, a single 3D ultrasound may be used to generate the patient mesh. As will be described, the system may leverage the patient mesh to more directly navigate the TEE tool to specific cardiac views. For example, a plane (e.g., an imaging plane) may be identified as corresponding to a particular cardiac view. In this example, the plane may intersect the patient mesh similar to how an ultrasound image (e.g., a 2D ultrasound image) would image the patient’s heart. The system may therefore navigate the TEE tool to obtain an ultrasound image associated with the identified plane. In this way, the system may intelligently navigate the TEE tool to cause the TEE probe’s orientation to match specific planes.

[0164] Advantageously, the patient mesh may be rapidly generalizable with other patient meshes. In embodiments in which the patient mesh is formed from a multitude of triangles (e.g., 1 thousand triangles, 10 thousand triangles, 50 thousand triangles, 1 million triangles, and so on), vertices, vertices / edges, same, or similar, physical portions of a human heart may correspond to same, or similar, vertices of the patient meshes. For example, a medical professional may define a customized cardiac view. In this example, the medical professional may be viewing real-time ultrasound images from the TEE tool and decide that the specific view is to be used for other patients. As will be described, the system may identify a plane through vertices of the patient mesh that result in the specific view. The system may then fit the plane to other patient meshes, such that the same, or similar, customized cardiac view may be presented. In this way, the medical professional may customize operation of the system and reduce a workflow burden associated with imaging patients.

[0165] In some embodiments, the patient mesh may be formed as output from a machine learning model, such as a graph convolutional network. As will be described at least in Figures 8A-9, the system may deform an average heart mesh based onultrasound images from the TEE tool. The average heart mesh may reflect an average heart geometry associated with humans or subsets thereof (e.g., adults, males, females, and so on). For example, Figures 8A-8C describe use an encoder / decoder that uses a trained latent space to output distance information to individual segmented regions of a heart. This distance information may be used by the machine learning model to adjust corresponding segmented regions of the average heart mesh. As another example, Figure 9 describes use of a feature extractor, such as a U-NET, that outputs hierarchical features (e.g., a feature pyramid). For this example, the hierarchical features may be projected onto, or otherwise mapped to, vertices of the average heart mesh. The machine learning model may then deform the average heart mesh, such as iteratively deform, usingthe hierarchical features.

[0166] As known by those skilled in the art, training machine learning models using medical information presents technological challenges. For example, ground truth training data is scarce and technically challenging to obtain. As another example, the ground truth training data may lack diversity such that resulting machine learning models do not generalize well. To address this, the system may perform innovative data augmentation techniques to increase the extent to which training data is available.

[0167] For example, in some embodiments the training data may include 3D ultrasounds. In this example, the 3D ultrasounds may have ground truth labels informing physical portions or features of the heart. As one example, individual voxels may be annotated, or otherwise associated, with an individual label of a set of labels. Example labels may include a left atrium, left ventricle, left atrial appendage, right atrium, right ventricle, background, and so on. Such training data may, as an example, reflect 3D ultrasounds obtained from TEE probes at similar positions within a patient’s esophagus (e.g., similar translations down the esophagus). For example, the 3D ultrasounds may have been obtained by medical professionals to depict a particular cardiac view (e.g., a four-chamber view). Thus, the training data may reflect similar viewpoints of TEE probes. In this way, the machine learning model(s) may learn to rely upon these viewpoints such that, at inference time, otherviewpoints may cause inaccurate output of the model(s).

[0168] The system may therefore address the inherent shortcomings of obtainable ground truth information. For example, the system may perform a spherical dataaugmentation that fits a sphere within a 3D ultrasound volume (e.g., a pyramid). In this example, portions outside of the sphere, such as the apex of the 3D ultrasound, are discarded such that the resulting augmented training data does not have a pyramid extends to the same, or similar, TEE probe position. The sphere may further be rotated, or otherwise adjusted, to further generate diverse training data.

[0169] In addition to training data, the system may reduce a computational burden, and increase an accuracy, associated with forming a patient mesh at inference time. For example, and with respect to Figure 9, a U-NET may be used to output hierarchical features. In this example, a head (e.g., an output branch) downstream of the U-NET may output a confidence map that indicates confidences associated with feature extractions for the voxels. Thus, the confidence map reflects degrees of certainty or confidences associated with the output from U-NET. The system may use this confidence map to better align the average heart mesh to the geometry of the patient’s heart. For example, a transformation (e.g., an affine transformation, rigid transformation, and so on) may be applied to the average heart mesh or individual portions or regions thereof.

[0170] As described herein, the graph model may be used to inform, or otherwise effectuate, navigation of the TEE tool. Thus, the system may cause the TEE tool to navigate about a patient’s esophagus based on an understanding of the patient’s physiology / anatomy.

[0171] As an example of navigation illustrated in Figures 11 C-11 D, a user interface may allow a medical professional to select a particular landmark (e.g., a mitral annulus or other physical cardiac landmark of interest to medical professionals). In this example, the system may adjust the TEE probe to remain focused on the particular landmark. As an example, the system may adjust the probe’s station coordinates to view the particular landmark from a different point of view (e.g., due to shadowing, blood, a bone in the way, and so on). For this example, the system may adjust the probe’s degrees of freedom such that the probe flexes, translates, rotates, and so on, while preserving the focus on the particular landmark. As another example, the system may automatically focus on landmarks depending on a particular cardiac view selected or specified by the medical professional. As an example, a particular view (e.g., a four-chamber view) may have one or more physical landmarks known to be of interest to medical professionals.

[0172] Thus, the system may cause a landmark to be in focus during the particular view. The system may adjust the probe for disparate reasons while preserving the landmark’s focus. For example, the system may adjust the probe to reduce local heating which may be caused by the ultrasound transducer while stationary. The system may also adjust the probe to reduce pressure being applied to an interior of the esophagus. The system may also adjust the probe in response to shadowing, blood pooling, a bone, and so on, which may be interfering with proper imaging of the landmark.

[0173] The system may additionally leverage, in some embodiments, near field ultrasound images reflecting a surface of a patient’s esophagus. For example, and as described in Figure 13, the system may obtain near field images to ascertain tissue interaction with the patient’s esophagus (e.g., pressure applied to the esophagus). In this example, the system may determine a heat map, or other information, reflecting an extent to which individual portions of the patient’s esophagus had pressure applied. As may be appreciated, too much pressure may cause bruising or other negative side effects of the TEE procedure. Thus, the system may monitor the surface of the patient’s esophagus.

[0174] In addition to monitoring for pressure or bruising, the system may leverage the near field images for navigation. For example, the system may determine, based on the heat map, that first station coordinates should be temporarily or entirely avoided during aTEE procedure. In this example, the system may determine second station coordinates with reduced tissue interaction that provide a similar viewpoint as the first station coordinates. The system, in some embodiments, may leverage the patient mesh to determine the second station coordinates.

[0175] The above, and additional features, will now be described in more detail.

[0176] Figures 8A-8C illustrate example techniques to generate a patient mesh. For example, the system may segment ultrasound images into heart chambers and great vessels. The system may then robotically navigate a TEE probe (e.g., probe 114) based on the ultrasound images.

[0177] Figure 8A illustrates example autoencoders, with each autoencoder being formed from an encoder engine and a decoder engine. The autoencoders may beimplemented by the system 100 described herein. As known by those skilled in the art, the encoder engine is trained to map input information into a vector associated with a latent space (e.g., a latent feature representation). The decoder engine is trained to regenerate the input using the latent space vector.

[0178] In the illustrated example, ultrasound images 802 are used as in put to encoder engine A 804A. The ultrasound images 802 may represent training data that includes cardiac images. For example, the cardiac images may be obtained from a TEE probe, such as the TEE probe described herein. In some embodiments, the cardiac images may be obtained from other TEE probes such as TEE probes which are not autonomously controlled. In some embodiments, the cardiac images may be obtained from other imaging modalities such as TTE, CT, or MRI. The autoencoder A (e.g., encoder engine A 804A and decoder engine A 806A) may be trained to reconstruct the cardiac images. The latent feature representation A 808A may reflect squashed, or otherwise dimensionally reduced, information. For example, a latent feature space may be trained to understand ultrasound images.

[0179] Similarly, distance fields 812 may be input into autoencoder B (e.g., encoder engine B 804B and decoder engine B 806B). The distance fields 812 may represent a set of signed distance fields (SDFs), with each signed distance field reflecting the shortest distance to a boundary of a particular type of labeled region of interest. For example, an ultrasound image may be obtained from a TEE probe. In this example, each pixel or element of the ultrasound image may be associated with the set of SDFs. As an example, the set of SDFs for the pixel may reflect the distance from that pixel to each of a set of labeled regions of interest. Example regions may include different parts of the heart which may serve to segment the heart or otherwise which serve as landmarks to understand the probe’s orientation relative to the heart.

[0180] Autoencoder B therefore is trained to reconstruct distance fields 812 using latent feature representation B808B. In some embodiments, the distance fields 812 may be paired with the ultrasound images 802 input into autoencoder A. For example, individual pixels or elements of the ultrasound images 802 may be associated with distance fields 812. In some embodiments, the distance fields 812 may be obtained, or derived, using computed tomography (CT) images. For example, the CT images may beused to accurately determine distances, for each pixel, to the different labeled regions of a heart. In this way, autoencoder B may learn geometry information about hearts. That is, autoencoder B may learn how different labeled regions of the heart are physically connected or proximate to each other.

[0181] As may be appreciated, distance fields 812 may be allow for an understanding of how far it is to the different labeled regions along with a direction to a particular portion of the heart. For example, the distance fields 812 may function as signs to a particular portion of the heart which is of interest to a medical professional, or which is associated with a cardiac view as described herein.

[0182] Since the autoencoders A-B learn to reconstruct their inputs, a technique may be used to allow for inputting of an ultrasound image and outputting distance fields (e.g., SDFs per pixel or element of the ultrasound image). In some embodiments, a discriminator (e.g., discriminator engine 810) may be used to force the latent vectors to be similar for both the ultrasound image and for the SDF. As an example, the discriminator, such as a discriminator similar to that of a generative adversarial network, may cause the latent feature spaces A-B 808A-808B to be similar during training.

[0183] In Figure 8B, an ultrasound image 814 is being received via encoder engine A 804A. This illustration may reflect, for example, the techniques described in Figure 8A at inference time. As described above, encoder engine A 804A may be trained to map an ultrasound image into a latent feature space A808A. In the illustrated example, the encoder engine A 804A has mapped the ultrasound image into a vector associated with latent feature space A808A. This vector is then similarly mapped into latent feature space B 808B. Decoder engine B 806B reconstructs distance fields 816 (e.g., SDFs) per pixel or element, or per subset of pixels or elements, of the ultrasound image 814. In this way the autoencoders collectively enable mapping between ultrasound images and distance fields.

[0184] The distance fields may thus reflect segmentation information associated with the ultrasound images. For example, the distance fields may be converted into pixel masks associated with labeled regions. As another example, the distance fields may be maintained as regression vectors associated with pixels of the ultrasound images. Asdescribed above, the distance fields may be used to inform global and / or local navigation of the TEE probe.

[0185] Figure 8C illustrates example navigation techniques. In the illustrated embodiment, a graph convolutional network model may be used to determine, or otherwise convert, a segmented image 824-826 into a mesh of triangles or tetrahedra. For example, a model of a heart (e.g., an average heart 820 or a heart formed from basic shapes, such as spheres or ellipses) may be used by the system (e.g., system 100). In this example, the segmented image may be used to adjust the model 820 to correspond with a patient’s actual heart geometry. As an example, a multitude of segmented images may be input into the graph model (e.g., graph neural network). The graph model may then adjust the model to correspond with the patient.

[0186] Certain techniques may output meshes of triangles which vary based on how big objects or heart features are. In some embodiments, the graph model may use a template mesh with set number of vertices which map to every patient’s heart.

[0187] In some embodiments, the model 820 may represent four spheres with each sphere corresponding to each chamber. The four spheres may then be stretched, or otherwise fit, to the patient’s heart based on the segmentation information (e.g., the SDFs described above). A graph neural network may be used to morph the chambers, for example as illustrated.

[0188] In some embodiments, an average heart 820 may be defined. For example, a threshold amount of training data may be obtained which reflects heart models determined based on deformation of the above-described four spheres. The training data may represent, in some embodiments, CT images. In this example, the average heart 820 may represent an average of these heart models. At inference time of a patient, the patient mesh 828 may represent a deformation of the average heart model 820, such that the heart model may be more accurate to the patient.

[0189] Views for the average heart 820 may be determined. For example, the cardiac views described herein may be determined for the average heart 820. An example view is reflected in Figure 8C as a plane (e.g., the line 824) through the shown heart (e.g., the four spheres). To navigate, for example to perform global navigation between cardiacviews, the system may determine where the plane maps to the patient mesh 828 (e.g., as adjusted via the graph model). This mapping may be used to inform the orientation (e.g., the station coordinates described above) of the TEE probe to correspond with the plane.

[0190] The patient mesh 828 may be generated at different points in time (e.g., a threshold frequency). In this way, the system can track points on the patient mesh 828 over time (e.g., over cardiac cycles). Motion may therefore be determined for these points.

[0191] In some embodiments, to perform local navigation (e.g., navigation between cardiac views, or locally around or about a view) the system may use reinforcement learning (RL) techniques. The system may leverage a model which is trained based on simulations, and with the training updating a policy associated with the reinforcement learning. Thus, the system may learn to navigate about the patient’s esophagus. The reinforcement learning may, in some embodiments, ensure that the TEE probe navigates safely. For example, the TEE probe may navigate without putting undue, unsafe, or unnecessary, pressure on the tissue of the esophagus. With respect to training, an example may include training the TEE probe to navigate to a four-chamber view.

[0192] In some embodiments, the graph model may determine uncertainty associated with different portions of the graph model (e.g., vertices). In some embodiments, thresholding may be used to update presentation, or use, of the model. As an example, for portions bellowing a threshold a wireframe may be presented while for portions above the threshold a smooth model may be presented.

[0193] Figure 9 illustrates another example technique to generate a patient mesh. As described in Figures 8A-8C, the system (e.g., system 100) may use a discriminator to force two latent spaces to be substantially similar. Each latent space may be trained based on a different input, for example ultrasound images and distance fields, such that the resulting trained model may map ultrasound images to distance fields. The distance fields may be used as segmentation information to deform portions of an average heart mesh 1016 to correspond to a patient’s heart geometry.

[0194] The average heart mesh 1016, as described herein, may be generated based on training data that includes 3D ultrasounds. For example, individual patient heartmeshes may be generated from labeled 3D ultrasounds. In this example, the generation may leverage techniques to extract a mesh from the 3D ultrasound (e.g., the 3D ultrasound may be converted into an isosurface). The individual patient heart meshes may be aligned to a common coordinate system such that a fixed correspondence may be established between vertices of one mesh to another. The average heart mesh 1016 may be generated based on measures of central tendency (e.g., mean) of the vertices of the meshes. For example, a particular vertex of the average heart mesh 1016 may reflect an average of the correspondingvertices of the patient heart meshes.

[0195] Advantageously, individual vertices of the average heart mesh 1016 may be assigned labels corresponding to regions or features of the heart, such as the labels described above (e.g., left atrium, right atrium). As will be described below, during inference a 3D ultrasound may be used to generate a patient mesh 1020 based on the average heart mesh 1016. Due to the inherent labeling of the vertices, the resulting patient mesh 1020 may similarly be segmented or labeled.

[0196] As described in Figure 9, in some embodiments the system may generate the patient mesh based on ultrasound images 1000, such as 3D ultrasounds, using one or more machine learning model(s). Engines 1002, 1008, 1014, and 1018 may be executed, for example, by the system.

[0197] In the illustrated embodiment, a data augmentation engine 1002 obtains ultrasound images 1000 (e.g., 3D ultrasounds, such as ultrasound pyramids). As may be appreciated, this engine 1002 may be used during training only. The system may access training data that includes ground truth labeled 3D ultrasounds. These 3D ultrasounds may have been obtained using the TEE tool described herein (e.g., tool 110) or may have been obtained using non-autonomous navigated TEE probes. The ultrasound images 1000 may therefore reflect 3D ultrasounds of different patients’ hearts.

[0198] As described above, data augmentation may be performed to increase the reliability, and generalizability, of the machine learning model(s). For example, it may be common for medical professionals to obtain 3D ultrasounds at specific positions and / or orientations within patients. As one example, medical professionals may maneuver TEE probes to image the heart fro m a certain vantage point (e.g., behind the left atrium). Thus, absent innovative data augmentation techniques which are specific to TEE-based 3Dultrasounds, the resulting machine learning model(s) may suffer from shortcut learning. For example, the model(s) may be unable to reliably generate patient meshes based on input of 3D ultrasounds from different vantage points.

[0199] In the illustrated embodiment, a 3D ultrasound volume 1004 is depicted as extending from a probe. This volume 1004 may represent an example of training data, and may, as an example, represent voxels. The data augmentation engine 1002 may supplement this example through spherical augmentation. For example, the engine 1002 mayfit a sphere within the 3d ultrasound volume 1004. In this example, a spherical region may be randomly defined within the ultrasound volume 1004. The sphere’s center and radius may be sampled randomly for each training instance, ensuring variability in the location and extent of the augmented region. Additionally, random rotations about the sphere (e.g., the center of the sphere) may be performed.

[0200] The data augmentation engine 1004 may retain voxels that fall inside the sphere, while voxels outside the sphere may be masked out, set to zero, or otherwise discarded. Spherical augmentation may effectively crop the central region of interest (e.g., the heart) while discarding peripheral or artifact-prone areas near the apex or base. As a result, the machine learning model(s) may learn features within the anatomically relevant zone, making the model(s) more robust to variations in probe position, field of view, and so on.

[0201] The data augmentation engine 1002 may additionally perform probe-centric augmentation. For this augmentation, the engine 1002 may apply random rotations of the 3D ultrasound volume 1004 around the probe position. As an example, this random rotation may help simulate different viewing angles and probe positioning. The engine 1002 may additionally apply random translations to simulate probe movement.

[0202] The data augmentation engine 1002 may perform affine transformations. For example, the engine 1002 may perform random shifts along each axis. As another example, the engine 1002 may perform random rotations about each axis, random scaling along each axis, and / or random shearing along each axis. These transformations may be applied to the ultrasound images 1000 and the average heart mesh 1016 to ensure consistency for these training samples. In some embodiments, the engine 1002may apply random intensity scaling, forexample adjust intensity of 3D ultrasounds while preserving the volume 1004.

[0203] A feature extraction engine 1008 may be used to output hierarchical features 1010. In some embodiments, the feature extraction engine 1008 may represent a machine learning model, such as a U-NET, Swin-UNET, and so on. As known by those skilled in the art, U-NET may output a feature pyramid which includes a threshold number of feature maps at different scales. These hierarchical features 1010, as will be described, may be used to deform the average heart mesh 1016 described herein to form the patient mesh 1020.

[0204] With respect to U-NET, hierarchical features 1010 may be obtained from the encoder or decoder portion of U-NET. In some embodiments, the features 1010 may represent features output via the encoder. However, in some embodiments the features 1010 may represent features output via the decoder and / or a concatenation of the encoder and decoder features.

[0205] As illustrated, the engine 1008 may determine a confidence map 1012 associated with the hierarchical features 1010. For example, features from the hierarchical features 1010 may be sampled at individual voxel positions. The engine 1008 may include a head that outputs individual confidence values or measures for features sampled at individual voxel positions. These confidence values or measures may indicate a confidence that the features accurately characterize a 3D ultrasound.

[0206] The above-described confidence map 1012 may be used, in some embodiments, to perform pre-alignment of the average heart mesh 1016. For example, an alignment engine 1014 may use the confidence map 1012 optionally along with a correspondence field reflecting, for each vertex of the average heart mesh 1016, a predicted 3D position within an input 3D ultrasound. The predicted 3D positions forms a correspondence field that maps each mesh vertex to a target anatomical location in the 3D ultrasound.

[0207] During pre-alignment, the engine 1014 may determine a transformation between the average heart mesh 1016 and the predicted 3D positions. Example transformations may include a rigid transformation, similarity transformation, affinetransformation, and so on. These transformations may be weighted according to the confidence map 1012, such that more trustworthy or confident predicted 3D positions are given greater influence in aligning corresponding vertices of the average heart mesh 1016.

[0208] A graph model engine 1018 may then output a patient mesh 1020 based on the hierarchical features 1010 and aligned average heart mesh 1016. In some embodiments, the features 1010 may be projected onto the average heart mesh 1016. For example, interpolation may be used (e.g., trilinear interpolation). In this example, based on spatial locations of each vertex of the average heart mesh 1016, the engine 1018 may sample the hierarchical features 1010 at the corresponding spatial locations. In this way, each vertex may be assigned features (e.g., a feature vector). In some embodiments, the engine 1018 may use an attention-based technique to project features. For example, features for each vertex may be generated using attention with respect to the features 1010.

[0209] The graph model engine 1018 may represent a machine learning model, for example a graph convolutional network. In some embodiments, Chebyshev convolutions maybe used. In some embodiments, the graph convolutional network may reflect a DiffusionNet. To output the patient mesh 1020, the engine 1018 may include a threshold number of graph blocks that iteratively cause deformation ofthe average heart mesh 1016.

[0210] For example, a first block may deform the average heart mesh into a first intermediate mesh. In this example, a second block may deform the first intermediate mesh into a second intermediate mesh. There may be 2, 3, 4, 5, and so on, iterative blocks. In some embodiments, each block may leverage particular hierarchical features. For example, lower resolution features may be used in the first block while higher resolution features may be used in the second block. At each block, the features may be projected onto the mesh so that the mesh may be further deformed towards the patient mesh.

[0211] Figure 10A is a flowchart of an example process 1000 for training a patient mesh and inferencing ultrasound data from a TEE tool. For convenience, the process 1000 will be described as being performed by a system of one or morecomputers (e.g., the automated inspection system 100).

[0212] At block 1002, the system obtains training data reflecting ground truth information. As described above, for example with respect to Figures 8A-9, the system may obtain 3D ultrasounds with labels reflecting different portions of the heart.

[0213] At block 1004, the system performs data augmentation. The system may perform disparate augmentation, for example spherical augmentation, to generate additionaltraining data.

[0214] At block 1006, the system trains machine learning model(s) to output a patient mesh. With respect to, for example, Figure 9, the system may train a U-NET and graph convolutional network to output a patient mesh. For example, different losses may be computed (e.g., mesh vertex position loss, segmentation loss, and so on) and used to adjust weights of the model(s).

[0215] At block 1008, the system inferences real-time ultrasound data. As described herein, the system may obtain ultrasound images, such as 3D ultrasounds, from a TEE probe (e.g., probe 114). The system may thus generate a patient mesh based on the obtained ultrasound images.

[0216] In some embodiments, as the TEE probe obtains ultrasound images, the system may generate a patient mesh based on a prior generated patient mesh instead of the average heart mesh. For example, the system may replace the average heart mesh with the last, or a recent, patient mesh. In this way, the system may refine the patient mesh as the system navigates the TEE probe. As may be appreciated, an obtained ultrasound image may include shadowing or portions of the heart which are blocked or have low fidelity. Thus, portions of a new patient mesh may be updated to reflect the obtained ultrasound image while other portions with shadowing or poor fidelity may be preserved.

[0217] Figure 10B is a flowchart of an example process 1010 illustrating detail of inferencing ultrasound data from a TEE tool (e.g., TEE tool 110 with probe 114). For convenience, the process 1010 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0218] At block 1012, the system obtains real-time ultrasound images. As described herein, the system may obtain ultrasound images, such as 3D ultrasounds, from the probe within the esophagus of a patient. In some embodiments, a medical professional may operate the TEE tool to cause the probe to be at an initial registration orientation, such as initial station coordinates, that cause result in a particular cardiac view (e.g., a four-chamber view). For example, 2D ultrasound images from the probe may correspond to the particular cardiac view. The system may then obtain a resulting 3D ultrasound for generation of a patient mesh.

[0219] At block 1014, the system extracts features via a machine learning model. As described in Figure 9, the system may compute a forward pass through the machine learning model to extract hierarchical features. In some embodiments, the machine learning model may be a U-NET.

[0220] At block 1016, the system performs pre-alignment of an average heart mesh. The system may obtain a confidence map reflecting confidences associated with the extracted features. The system may additionally obtain a correspondence field which associates portions of the 3D ultrasound with vertices of the average heart mesh. For example, the confidence map may predict a 3D position for an individual vertex of the average heart mesh. The system may then pre-align the average heart mesh to correspond to patient geometry, such that there is less delta between the average heart mesh and patient mesh. As described in Figure 9, the system may leverage transformations which may be weighted based on the confidence map.

[0221] At block 1018, the system projects extracted features onto vertices of the aligned average heart mesh. As described in Figure 9, the system may use interpolation or attention-based techniques.

[0222] At block 1020, the system deforms the average heart mesh to form the patient mesh. The system may use a machine learning model, such as a graph convolutional network, to deform (e.g., iteratively deform) the average heart mesh. In some embodiments, each iteration, or forward pass (e.g., block of the model), may leverage a different subset of the hierarchical features. For example, there may be 3, 4, 5, 6, and so on, hierarchies or levels of features (e.g., different resolutions). In this example, eachiteration may leverage one, or a subset, of these hierarchies. In some embodiments, each iteration may leverage the full hierarchy.

[0223] As described in Figure 9, the average heart mesh may have defined vertices with individual labels. For example, the labels may indicate physical regions or features of the heart. Thus, the resulting patient mesh may inherently maintain, or persist, these labels.

[0224] The system may present a graphical representation of the patient mesh. For example, Figures 11 C-11 E illustrate an example of a graphical representation. As may be appreciated, the underlying patient mesh may be graphical shown using different techniques. For example, the graphical representation may have textures. In some embodiments, the textures may reflect the underlying labels (e.g., different regions may have different colors or textures).

[0225] The average heart mesh and patient mesh may be leveraged to perform navigation and also enable customizations of the system by a medical professional. For example, the medical professional may define a custom cardiac view. In this example, the medical professional may be viewing real-time ultrasound images (e.g., 2D images) from the TEE probe (e.g., probe 114) within a patient. As one example, the medical professional may be manually adjustingthe probe (e.g., as described in Figure 3C) or may be refining a particular cardiac view through manual adjustments.

[0226] As described below, with respect to Figure 11 A, the system may determine a plane (e.g., an imaging plane) that corresponds to the custom cardiac view. In some embodiments, a medical professional may define standard cardiac views, such as the four-chamber view, based on the average heart mesh. The system may then use these definitions, such as respective planes that corresponds to the standard cardiac views, to present the cardiac views based on patient meshes. In this way, each standard cardiac view may be defined once based on the average heart mesh. Subsequently, the system may surface the standard cardiac view during operation of the TEE tooldescribed herein.

[0227] Figure 11 Aisa flowchart of an example process 1100 for presenting a clinically preferred cardiac view based on a patient mesh. For convenience, the process 1100 willbe described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0228] At block 1102, the system obtains an indication of a clinically preferred cardiac view. As described herein, the system may operate, or otherwise control, a TEE tool (e.g., tool 110) to obtain specific cardiac views. In the example of process 1100, the clinically preferred cardiac view may represent a standard cardiac view (e.g., the views described at least in Figure 3B) or it may represent a custom cardiac view.

[0229] The system may obtain the indication based on input from a medical professional. For example, a medical professional may be viewing a graphical representation of the average heart mesh via a user interface. In this example, simulated, or actually recorded, ultrasound images (e.g., 2D images) may be presented based on station coordinates associated with a TEE probe of the TEE tool. The medical professional may identify that a particular ultrasound image is associated with the clinically preferred cardiac view. For example, the medical professional may determine that a current view represents the four-chamber view.

[0230] In some embodiments, the medical professional may be viewing a graphical representation of a patient mesh. For example, the medical professional may be operating, or otherwise providing input to, the system. In this example, the TEE tool may be used during a TEE procedure for a patient, or the medical professional may be interacting with a previously generated patient mesh after a TEE procedure. Similar to the above, the medical professional may identify that a particular ultrasound image is associated with the clinically preferred cardiac view.

[0231] At block 1104, the system determines a plane associated with the average heart mesh or the patient mesh based on the clinically preferred view. The system identifies a plane that intersects the average heart mesh or patient mesh. As described above, the plane may represent an imaging plane associated with the TEE probe.

[0232] At block 1106, the system identifies vertices associated with the determined plane. The system determines vertices that fall within a threshold of the plane (e.g., plus, or minus, 1 mm, 2 mm, 3 mm, and so on).

[0233] At block 1108, the system fits the plane to at least one patient mesh. The system identifies vertices of the at least one patient mesh that correspond to the vertices identified in block 1106. Since these vertices may be non-coplanar, the system fits a plane to the vertices of the at least one patient mesh. This fit plane may thus represent the same, or substantially similar, clinically preferred cardiac view.

[0234] At block 1110, the system presents the clinically preferred cardiac view associated with the at least one patient mesh. The medical professional may thus define a cardiac view, such as a standard view or a custom view, and have that definition persist during operation of the system with respect to multitudes of patients. The system may thus present the clinically preferred cardiacviewdurin a workflow associated with a TEE procedure. For example, the system may cause the TEE tooldescribed herein to navigate within a patient to obtain clinically preferred cardiac view. An example of navigation based on a patient mesh is described below, with respect to Figure 11 B.

[0235] Figure 11 B is a flowchart of an example process 1120 for navigating the TEE tool based on the patient mesh. For convenience, the process 1120 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0236] At block 1122, the system receives an indication to navigate to a particular cardiac view. As described herein, in some embodiments the system may autonomously or semi-autonomous navigate to different cardiac views. For example, the specific cardiac views may be based on a workflow associated with a TEE procedure. The system may also receive user input instructing the system to navigate the TEE probe (e.g., probe 114) to obtain ultrasound images reflecting the particular cardiac view.

[0237] At block 1124, the system accesses a plane associated with the particular cardiac view. As described above, with respect to Figure 11 A, the system may have access to a plane (e.g., with respect to an average heart mesh) that defines the particular cardiac view. The system may determine a target plane specific to the patient mesh, for example as described in Figure 11 A.

[0238] At block 1126, the system compares a current plane (e.g., a current image plane) to the accessed plane. The current plane may reflect a plane that reflects acurrent 2D ultrasound image. Similar to Figure 11 A, the system may identify vertices that are associated with the current plane.

[0239] At block 1128, the system iteratively adjusts station coordinates based on one or more comparisons. The system may adjust station coordinates to iteratively adjust the TEE probe to navigate towards the target plane. While different techniques may be used, in some embodiments the system may define each plane based on a center point (e.g., a position in 3D space) along with a normal vector reflecting directionality associated with the plane.

[0240] The system may, for example, compute a difference between the current plane and the target plane. The difference may reflect a distance (e.g., translation) along with rotation. As may be appreciated, the difference may reflect an error which the system may iteratively reduce (e.g., via gradient descent). Thus, the system may determine a translational error that reflects a difference in 3D position between centers of the current plane and the target plane. The system may additionally determine an angular error that reflects a difference in orientation between the normal vectors defined forthe current plane and the target plane.

[0241] The system may map these errors into control actions that update station coordinates. In some embodiments, the system may constrain the particular station coordinates used. For example, the system may allow 3 degrees of freedom. The system may additionally allow adjustments which are below one or more thresholds. As an example, the system may cause the TEE probe to navigate up to a threshold translation within the patient’s esophagus. Similarly, the system may cause the TEE probe to flex or rotate less than threshold(s). Subsequent to updating of the station coordinates, the system may determine an updated current plane and repeat the process. The system may, in some embodiments, stop the iterations once the determine differences, orerrors, are less than a threshold.

[0242] Figure 11 C is an example user interface of an ultrasound image 1132 proximate to a graphical representation 1130 of a patient mesh. During operation of the system described herein, a user interface may be presented that succinctly indicates the currently viewed image 1132 along with the patient mesh. The graphical representation 1132 of the patient mesh may be adjustable via user input, for example to rotate orotherwise view the patient mesh. In some embodiments, a plane representing the ultrasound image 1132 may be presented as intersecting the representation 1130.

[0243] Figure 11 D is an example user interface associated with landmark navigation. In the illustrated embodiment, the graphical representation 1130 is illustrated along with an ultrasound image 1132 and a landmark 1134 of interest to a user. As described above, the system may navigate based on an understanding of the geometry of the heart. The system may additionally identify particular landmarks, such as a chamberof the heart, a mitral annulus, and so on. As described above, the system may navigate to maintain a landmark in focus. For example, the system may determine adjustments to station coordinates (e.g., degrees of freedom) that maintain the landmark 1134 on a current plane (e.g., an imaging plane).

[0244] In the user interface, degrees of freedom 1136 are illustrated which reflect the example degrees described herein. To keep the landmark in focus, the system may adjust these degrees of freedom to ensure that the landmark remains in focus. For example, the landmark may remain in focus using different combinations of the degrees of freedom (e.g., station coordinates).

[0245] In some embodiments, the system may autonomously adjust the station coordinates to maintain focus. For example, the system may determine that the ultrasound image 1132 has visual fidelity issues with respect to the landmark 1134. As an example, the landmark 1134 may be partially occluded or may have undesirable shadowing. The system may leverage classical or machine learning-based vision techniques to identify these issues. The system may then adjust the station coordinates to obtain a view of the landmark 1134 from a different vantage point.

[0246] Figure 11 E is an example user interface associated with bookmarking a portion of an ultrasound image 1132 and / or a graphical representation 1130 of a patient mesh. In some embodiments, a bookmark 1140 may be identified by a user (e.g., a medical professional). For example, the bookmark 1140 may be assigned to a portion of the ultrasound image 1132 or graphical representation 1132. In this example, the user may interact with the user interface, such as clicking on the portion, long pressing, and so on. In response, the user interface may update to present the bookmark 1140 which in the illustrated embodiment is a sphere of a particular color.

[0247] The system may identify one or more vertices associated with the bookmark 1140. For example, the user may have clicked on the representation 1130. In response, the system may access the patient mesh and identify a closest vertex. Similarly, the user may have clicked on the ultrasound image 1132. The system may thus map the ultrasound image 1132 onto the patient mesh (e.g., identify a current plane and associated vertices) and identify a closest vertex.

[0248] As may be appreciated, the user may have identified the bookmark 1140 within the volume of the representation 1130 (e.g., not on a surface of the patient mesh). Since, as described herein, the vertices may form the surface of contour of the heart, the system may store information reflecting the position of the bookmark with respect to vertices of the patient mesh. As one example, the system may identify a line that connects two vertices on which the bookmark is positioned. The system may thus store information identifying the vertices along with the bookmark’s position along the line.

[0249] The bookmark 1140 may be used, for example, to indicate features of interest to the user. The user may maintain multiple bookmarks and cause them to be presented as reminders of noted physical features. Advantageously, the bookmark may be positioned both on an ultrasound image (e.g., if visible in the image) and the representation 1130. The user may additionally cause the TEE probe to navigate to obtain views of the bookmark, such as via indicating it as a landmark to maintain in focus.

[0250] Figures 12A-12E illustrate examples user interfaces and navigation techniques.

[0251] Figure 12A illustrates a medical professional using a TEE probe as described herein. A user interface is rendered on a display proximate to the medical professional. The user interface includes a left-most portion in which ultrasound image(s) are presented (e.g., from the TEE probe). The right-most portion includes a cardiac model, such as a graphical representation of a patient mesh, which is generated based on the ultrasound image(s). The model may depict a particular cardiac view to which the TEE probe navigated. In some embodiments, the model may be substantially real-time such that it may show the heart beating in substantially real time.

[0252] As illustrated, the medical professional may interact with a user interfacepositioned proximate to the professional’s hand. As described herein, the medical professional may select an updated cardiac view to cause adjustment of the probe to the new view. The medical professional may also manually adjust the probe. Th medical professional may also save or register a new cardiac view, and so on.

[0253] Figure 12B illustrates detail of an example cardiac view (e.g., a four-chamber view). In the example, three views are presented in the right-most portion of the figure. Each of views corresponds to one of the three planes depicted in the lower-right portion (e.g., represented by green, red, and blue). The three images shown are ultrasound images which have been adjusted using computer vision techniques. For example, portions of the images are assigned colors with the portions representing aspects of interest to a medical professional. As an example, the colors may indicate different physical features. As another example, the colors may indicate different chambers within the four-chamber view. The computer vision adjusted images may be presented in a user interface, for example optionally along with a cardiac model.

[0254] Figures 12C-12D illustrate adjustment of the images. For example, the planes have been adjusted to depict different views.

[0255] Figure 12E illustrates example cardiac views which a user may view. For example, the user may select a two-chamber view. In this example, the system (e.g., system 100) may access a plane associated with the two-chamber view. The system may then iteratively, or in some embodiments smoothly (e.g., the system may analyze realtime ultrasound images), navigate towards the two-chamber view.

[0256] Figure 13 is a block diagram illustrating techniques to determine tissue pressure associated with a TEE probe. The system, such as the illustrated tissue pressure engine 1300, may guard against negative pressure which may be caused by the TEE probe.

[0257] As an example, ultrasound images may be of higherquality(e.g., higher clarity) if the ultrasound probe is pressing against tissue. For this example, the system may analyze ultrasound images which are being received to determine measures of clarity or other information indicative of pressure applied to tissue. In some embodiments, thesystem may cause the probe to adjust its orientation (e.g., adjust flexion) to reduce the pressure being applied based on the determined measures or information.

[0258] The system may additionally cause the TEE probe to obtain near field focused ultrasound images. For example, the ultrasound images may depict the tissue of the esophagus rather than deeper into the patient to depict the heart. These images may be analyzed to determine the effects of the TEE probe on the tissue. For example, a same portion of the tissue may be tracked over time to determine whether the TEE probe is causing any negative effects. As an example, strain may be measured. As another example, particles in a field of view may be mapped to generate vectors of movement of the particles. As another example, tissue deformation may be tracked. Similarly, blood flow, perfusion, and so on may be mapped. Change to morphology may be identified, such as tearing, ripping, inelastic deformation, breakdown inflammation, swelling, and sliding. Doppler measurements at the blood scattering level may be tracked to determine flow dynamic changes. This may be indicative of vessel breakage and bruising.

[0259] In some embodiments, elastographic tracking may be used (e.g., forces may be applied, such as physical applied, acoustically applied, ultrasonically applied). Elastographic calculations may be performed, for example to correlate displacement over a time series. Doppler ultrasound measurements may be made.

[0260] In some embodiments, a map of the esophagus may be built as the probe navigates (e.g., a heat map). The map may indicate hotspots associated with probe usage, probe pressure, and so on. For exam pie, the near-field images maybe associated with the station coordinates (e.g., position / orientation) of the probe. The system may monitor portions of the esophagus which experience pressure, forexample based on the station coordinates. As an example, the probe may be positioned within a patient’s esophagus. For this example, the map may record a measure of time associated with this positioning. As another example, the map may record an estimate of pressure applied to the esophagus (e.g., based on flexion). The system may identify pressure, bruising, tissue interaction, based on analyzing the near-field images. In this way, the system may generate an overall heat map reflecting tissue interaction which is determined based on the near-field images.

[0261] In some embodiments, the probe may navigate based on the map. For example, the probe may focus on a landmark and may adjust its orientation and / or position (e.g., station coordinates) to view the landmark from different vantage points. As an example, the probe may adjust station coordinates to reduce negative effects on the tissue. In some embodiments, the system may cause the probe to alternate, or interleave, images at a near field distance and images of the heart. In this way, the system may limit negative tissue effects.

[0262] The tissue pressure engine 1300 may thus be used to track near-filed tissue features. In some embodiments, speckle tracking, opticalflow, coherent difference, ML- based classification techniques, and so on, may be used to localize the TEE probe within the esophagus. The engine 1300 may detect sliding of the probe, for example based on comparisons of tracking features using near-field images, tearing, and so on.

[0263] While the tissue pressure engine 1300 may be used to inform pressure, bruising, and so on, in some embodiments the engine 1300 may be used, in part, to navigate the TEE probe. For example, clarity or speckle coherence can serve as proxies for probe pressure or contact quality. In this example, the system may adjust the probe (e.g., orientation of the probe) to avoid excessive force while retaining a particular viewpoint. The system may also adjust the probe to increase clarity of a resulting ultrasound image of a heart. In some embodiments, beamforming strategies can be optimized for higher axial / lateral resolution using higher frequencies and linear scan geometries. This may improve the precision of deformation measurements and enhances land mark visibility,Other Embodiments

[0264] All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computers or processors. The code modules may be stored in any type of non-transitory computer-readable medium or other computer storage device. Some or all the methods may be embodied in specialized computer hardware.

[0265] Many other variations than those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain acts, events, or functions of any of the algorithms described herein can be performed in a different sequence or can be added, merged, or left out altogether (for example, not all described acts or events are necessary for the practice of the algorithms). Moreover, in certain embodiments, acts or events can be performed concurrently, for example, through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially. In addition, different tasks or processes can be performed by different machines and / or computing systems that can function together.

[0266] The various illustrative logical blocks, modules, and engines described in connection with the embodiments disclosed herein can be implemented or performed by a machine, such as a processing unit or processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor can be a microprocessor, but in the alternative, the processor can be a controller, microcontroller, or state machine, combinations of the same, or the like. A processor can include electrical circuitry configured to process computer-executable instructions. In another embodiment, a processor includes an FPGA or other programmable device that performs logic operations without processing computerexecutable instructions. A processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, a processor may also include primarily analog components. For example, some or all of the signal processing algorithms described herein may be implemented in analog circuitry or mixed analog and digital circuitry. A computing environment can include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a mainframe computer, a digital signalprocessor, a portable computing device, a device controller, or a computational engine within an appliance, to name a few.

[0267] Conditional language such as, among others, “can,” “could,” “might” or “may,” unless specifically stated otherwise, are understood within the context as used in general to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps. Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in anyway required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and / or steps are included orare to be performed in any particular embodiment.

[0268] Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (for example, X, Y, and / or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present.

[0269] Any process descriptions, elements or blocks in the flow diagrams described herein and / or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or elements in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown, or discussed, including substantially concurrently or in reverse order, depending on the functionality involved as would be understood by those skilled in the art.

[0270] Unless otherwise explicitly stated, articles such as “a” or “an” should generally be interpreted to include one or more described items. Accordingly, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices can also be collectively configured to carry out the stated recitations. For example, “a processorconfigured to carryout recitations A, B andC” can include a first processor configured to carry out recitation A working in conjunction with a second processor configured to carry out recitations B and C.

[0271] It should be emphasized that many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure.

Claims

WHAT IS CLAIMED IS:

1. A system comprising: a transesophageal echocardiogram (TEE) probe, the TEE probe being robotically controlled by the system and the TEE probe being adjustable with respect to a plurality of degrees of freedom; and one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to: obtain, via TEE probe, one or more ultrasound images of at least a portion of a heart, the one or more ultrasound images comprising one or more three-dimensional (3D) ultrasounds; compute a forward pass through a machine learning model trained to output a patient mesh reflecting a plurality of vertices forming a contour associated with the heart, wherein the machine learning model is configured to deform an average heart mesh reflecting corresponding vertices that form a contour associated with an average heart, and wherein the instructions that cause the one or more processors to compute a forward pass cause the processors to: extract a hierarchical feature map associated with a first ultrasound image of the one or more ultrasound images, wherein the hierarchical features encode features for individual voxels forming the first ultrasound image, wherein a confidence map associated with the hierarchical features is obtained, align the average heart mesh based on the first ultrasound image, wherein one or more transformations are applied to the average heart mesh based, at least in part, on the confidence map, and deform the aligned average heart mesh based on the hierarchical features to form the patient mesh; and cause presentation of an interactive user interface, wherein the interactive user interface presents a graphical representation of the patient mesh.

2. The system of claim 1 , wherein extracting the hierarchical feature map is performed based on application of U-NET, and wherein the hierarchical feature map includes a plurality of features associated with individual resolutions.

3. The system of claim 2, wherein a head connected to the U-NET is configured to output the confidence map and a correspondence field, the correspondence field associating, for individual vertices of the average heart mesh, 3D locations based on the first ultrasound image, and wherein aligning is based on the confidence map and the correspondence field.

4. The system of claims 1-3, wherein the one or more transformations are applied to individual regions of the average heart mesh or globally to the average heart mesh.

5. The system of claims 1-4, wherein deforming the aligned average heart mesh as based on application of a graph convolutional network, and wherein the graph convolutional network iteratively performs deformations.

6. The system of claim 5, wherein each iteration uses a different subset of the hierarchical features.

7. The system 5, wherein the hierarchical features are projected onto the aligned average heart mesh, wherein individual vertices sample the hierarchical features, and wherein the hierarchical features are interpolated.

8. The system of claims 1-7, wherein the one or more processors are configured to generate a new patient mesh based on a new ultrasound image, and wherein the new patient mesh is generated based on the average heart mesh.

9. The system of claims 1-8, wherein the one or more processors are configured to generate a new patient mesh based on a new ultrasound image, and wherein the new patient mesh is generated based on the patient mesh.

10. The system of claims 1-9, wherein the interactive user interface is configured receive user input identifying a particular cardiac view, and wherein the one or more processors are configured to: access a first plane associated with the particular cardiac view, the plane reflecting an image plane associated with the TEE probe, and the plane being defined based on the average heart mesh; determine a second plane associated with the patient mesh; and update the interactive user interface to present the particular cardiac view.

11. The system of claim 10, wherein the first plane is defined based on a first subset of vertices of the average heart mesh which intersect the first plane, and wherein determining the second plane comprises identifying a second subset of the vertices of the average heart mesh, the second subset including the first subset and additional vertices within a threshold distance of the first subset, and fitting the second plane based on vertices of patient mesh corresponding to the second subset of the vertices.

12. The system of claim 10, wherein the instructions further cause the one or more processors to control the TEE probe to obtain the particular cardiac view.

13. The system of claim 12, wherein controlling the TEE probe comprises iteratively adjusting degrees of freedom of the TEE probe, and wherein for an individual iteration the one or more processors are configured to: obtain a current plane associated with the TEE probe; and based on translational and angular differences associated with the current view and the second plane, adjust at least a subset of the degrees of freedom to reduce the translational and angular differences.

14. The system of claims 1-13, wherein the machine learning model was trained based on ground truth ultrasound images, and wherein spherical data augmentation was applied to the ground truth ultrasound images.

15. The system of claims 1-14, wherein the instructions further cause the one or more processors to:obtain one or more near-field images associated with the esophagus, wherein near-field images are configured for use in tracking pressure applied to the esophagus via the TEE probe.

16. The system of claim 15, wherein the one or more processors are configured to generate a heat map associated with the esophagus that reflects tissue interaction via the TEE probe.

17. The system of claim 15, wherein the one or more processors are configured to perform speckle tracking to track a surface of the esophagus.

18. The system of claim 15, wherein based on a portion of the esophagus exhibiting increased pressure, the one or more processors are configured to adjust the TEE probe away from the portion while maintaining a same cardiac view.

19. The system of claims 1-18, wherein the interactive user interface is configured to receive user input reflecting a particular landmark to focus on, and wherein the TEE probe is controlled, based on the patient mesh, to maintain focus on the particular landmark while degrees of freedom of the TEE probe are adjusted.

20. The system of claims 1-19, wherein the interactive user interface is configured to receive user input reflecting selection of a bookmark, and wherein the bookmark is associated with one or more vertices of the patient mesh.

21. The system of claim 20, wherein the patient mesh maintains correspondence between vertices as new patient meshes are generated, such that the bookmark persists to the new patient meshes.

22. The system of claim 20, wherein the bookmark is between two or more vertices.

23. The system of claim 20, wherein the bookmark reflects a physical feature of interest.

24. The system of claim 20, wherein the graphical representation of the patient mesh is updated to depict the bookmark.

25. The system of claim 20, wherein the bookmark is selected in an interior of the graphical representation, and wherein the bookmark is associated with at least two vertices of the patient mesh and a line extending therebetween on which the bookmark is positioned.

26. A method implemented by a system of one or more processors, the system including a transesophageal echocardiogram (TEE) probe and one or more processors, the TEE probe being robotically controlled by the system and the TEE probe being adjustable with respect to a plurality of degrees of freedom, and the method comprising: obtaining, via TEE probe, one or more ultrasound images of at least a portion of a heart, the one or more ultrasound images comprising one or more three-dimensional (3D) ultrasounds; computing a forward pass through a machine learning model trained to output a patient mesh reflecting a plurality of vertices forming a contour associated with the heart, wherein the machine learning model is configured to deform an average heart mesh reflecting corresponding vertices that form a contour associated with an average heart, and wherein computing a forward pass comprises: extracting a hierarchical feature map associated with a first ultrasound image of the one or more ultrasound images, wherein the hierarchical features encode features for individual voxels forming the first ultrasound image, wherein a confidence map associated with the hierarchical features is obtained, aligning the average heart mesh based on the first ultrasound image, wherein one or more transformations are applied to the average heart mesh based, at least in part, on the confidence map, and deforming the aligned average heart mesh based on the hierarchical features to form the patient mesh; and causing presentation of an interactive user interface, wherein the interactive user interface presents a graphical representation of the patient mesh.

27. The method of claim 26, wherein extracting the hierarchical feature map is performed based on application of U-NET, and wherein the hierarchical feature map includes a plurality of features associated with individual resolutions.

28. The method of claim 27, wherein a head connected to the U-NET is configured to output the confidence map and a correspondence field, the correspondence field associating, for individual vertices of the average heart mesh, 3D locations based on the first ultrasound image, and wherein aligning is based on the confidence map and the correspondence field.

29. The method of claims 26-28, wherein the one or more transformations are applied to individual regions of the average heart mesh or globally to the average heart mesh.

30. The method of claims 26-29, wherein deforming the aligned average heart mesh as based on application of a graph convolutional network, and wherein the graph convolutional network iteratively performs deformations.

31. The method of claim 30, wherein each iteration uses a different subset of the hierarchical features.

32. The method of claim 30, wherein the hierarchical features are projected onto the aligned average heart mesh, wherein individual vertices sample the hierarchical features, and wherein the hierarchical features are interpolated.

33. The method of claims 26-32, wherein the method further comprises generating a new patient mesh based on a new ultrasound image, and wherein the new patient mesh is generated based on the average heart mesh.

34. The method of claims 26-32, wherein the further comprises generating a new patient mesh based on a new ultrasound image, and wherein the new patient mesh is generated based on the patient mesh.

35. The method of claims 26-32, wherein the interactive user interface is configured receive user input identifying a particular cardiac view, and wherein the method further com prises accessing a first plane associated with the particular cardiac view, the plane reflecting an image plane associated with the TEE probe, and the plane being defined based on the average heart mesh; determining a second plane associated with the patient mesh; and updating the interactive user interface to present the particular cardiac view.

36. The method of claim 35, wherein the first plane is defined based on a first subset of vertices of the average heart mesh which intersect the first plane, and wherein determining the second plane comprises identifying a second subset of the vertices of the average heart mesh, the second subset including the first subset and additional vertices within a threshold distance of the first subset, and fitting the second plane based on vertices of patient mesh corresponding to the second subset of the vertices.

37. The method of claim 35, wherein the method further comprises controlling the TEE probe to obtain the particular cardiac view.

38. The method of claim 37, wherein controlling the TEE probe comprises iteratively adjusting degrees of freedom of the TEE probe, and wherein for an individual iteration the method comprises: obtaining a current plane associated with the TEE probe; and based on translational and angular differences associated with the current view and the second plane, adjusting at least a subset of the degrees of freedom to reduce the translational and angular differences.

39. The method of claims 26-38, wherein the machine learning model was trained based on ground truth ultrasound images, and wherein spherical data augmentation was applied to the ground truth ultrasound images.

40. The method of claims 26-39, wherein the method further comprises:obtaining one or more near-field images associated with the esophagus, wherein near-field images are configured for use in tracking pressure applied to the esophagus via the TEE probe.

41. The method of claim 40, wherein the method further comprises generating a heat map associated with the esophagus that reflects tissue interaction via the TEE probe.

42. The method of claim 40, wherein the method further comprises performing speckle tracking to track a surface of the esophagus.

43. The method of claim 40, wherein based on a portion of the esophagus exhibiting increased pressure, the method further comprises adjusting the TEE probe away from the portion while maintaining a same cardiac view.

44. The method of claims 26-43, wherein the interactive user interface is configured to receive user input reflecting a particular landmark to focus on, and wherein the TEE probe is controlled, based on the patient mesh, to maintain focus on the particular landmark while degrees of freedom of the TEE probe are adjusted.

45. The method of claims 26-43, wherein the interactive user interface is configured to receive user input reflecting selection of a bookmark, and wherein the bookmark is associated with one or more vertices of the patient mesh.

46. The method of claim 45, wherein the bookmark reflects a physical feature of interest.

47. The method of claim 45, wherein the patient mesh maintains correspondence between vertices as new patient meshes are generated, such that the bookmark persists to the new patient meshes.

48. The s method of claim 45, wherein the bookmark is between two or more vertices.

49. The method of claim 45, wherein the graphical representation of the patient mesh is updated to depict the bookmark.

50. The method of claim 45, wherein the bookmark is selected in an interior of the graphical representation, and wherein the bookmark is associated with at least two vertices of the patient mesh and a line extending therebetween on which the bookmark is positioned.51 . Computer readable media storing instructions that when executed by a system of one or more processors, the system including a transesophageal echocardiogram (TEE) probe, the TEE probe being robotically controlled by the system and the TEE probe being adjustable with respect to a plurality of degrees of freedom, cause the one or more processors to perform the method of claims 26-50.

52. A method implemented by a system of one or more processors, the method comprising: causing presentation of an interactive user interface, wherein the interactive user interface includes a two-dimensional (2D) ultrasound image obtained via a transesophageal echocardiogram (TEE) probe and a graphical representation of a patient mesh, the patient mesh including a plurality of vertices reflecting a contour associated with a heart of the patient, and the TEE probe being controllable with respect to a plurality of degrees of freedom; receiving user input selecting a landmark reflecting a physical feature of interest, the landmark being selected on the graphical representation; and updating the interactive user interface to present additional 2D ultrasound images obtained via the TEE probe based on adjusted degrees of freedom, and wherein the additional 2D ultrasound images depict the landmark.

53. The method of claim 52, wherein the interactive user interface includes controls associated with adjusting the degrees of freedom, and wherein adjusting an individual control causes at least a subset of remaining controls to be adjusted.

54. The method of claims 52-53, wherein user input selecting a bookmark is received via the interactive user interface, and wherein the bookmark is associated with one or more vertices of the patient mesh.

55. The method of claim 46, wherein the bookmark is depicted as a graphical object proximate to the patient mesh.

56. The method of claim 47, wherein the bookmark is depicted on at least one 2D ultrasound image included in the interactive user interface.

57. The method of claims 52-56, wherein the patient mesh was generated based on deformation of an average heart mesh, and wherein generating the patient mesh comprises extracting hierarchical features and deforming the average heart mesh based on the hierarchical features.

58. The method of claims 52-57, wherein the patient mesh was generated based on deformation of an average heart mesh, and wherein generating the patient mesh comprises obtaining signed distance fields based on input of an ultrasound image, and wherein the average heart mesh is deformed based on the signed distance fields.

59. A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed, cause the one or more processors to perform the method of claims 52-58.

60. Computer readable media storing instructions that when executed by a system of one or more processors cause the one or more processors to perform the method of claims 52-58.

61. A method implemented by a system of one or more processors, the method comprising: causing presentation of an interactive user interface, wherein the interactive user interface includes a two-dimensional (2D) ultrasound image obtained via a transesophageal echocardiogram (TEE) probe and a graphical representation of a patient mesh, the patient mesh including a plurality of vertices reflecting a contour associated with a heart of the patient, and the TEE probe being controllable with respect to a plurality of degrees of freedom; receiving user input selecting a bookmark reflecting a physical feature of interest, the bookmark being selected on the graphical representation; andupdating the interactive user interface to present the bookmark as a graphical feature overlaid over the patient mesh.

62. The method of claim 61 , wherein the bookmark is associated with one or more vertices of the patient mesh.

63. The method of claims 61-62, wherein the bookmark is depicted proximate to the patient mesh.

64. The method of claims 61 -63, wherein the bookmark is depicted on at least one 2D ultrasound image included in the interactive user interface.

65. The method of claims 61-64, wherein the patient mesh was generated based on deformation of an average heart mesh, and wherein generating the patient mesh comprises extracting hierarchical features and deforming the average heart mesh based on the hierarchical features.

66. The method of claims 61-65, wherein the patient mesh was generated based on deformation of an average heart mesh, and wherein generating the patient mesh comprises obtaining signed distance fields based on input of an ultrasound image, and wherein the average heart mesh is deformed based on the signed distance fields.

67. A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed, cause the one or more processors to perform the method of claims 61-66.

68. Computer readable media storing instructions that when executed by a system of one or more processors cause the one or more processors to perform the method of claims 61 -66.

69. A system comprising: a transesophageal echocardiogram (TEE) probe, the TEE probe being robotically controlled by the system and the TEE probe being adjustable with respect to a plurality of degrees of freedom; andone or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to: obtain, via the TEE probe, near-field images associated with the esophagus, the near-field images reflecting tissue interaction with the esophagus; and form a heat map based, at least in part, on the near-field images, wherein the heat map reflects tissue interaction with respect to individual portions of the esophagus.

70. The system of claim 69, wherein the TEE probe is periodically transitioned into a state associated with obtaining near-field images.

71. The system of claims 69-70, wherein the one or more processors are configured to perform speckle tracking to track a surface of the esophagus.

72. The system of claims 69-71 , wherein the heat map is based on time and / or pressure associated with the TEE probe being at individual positions.

73. A method implemented by a system of one or more processors, the system comprising a transesophageal echocardiogram (TEE) probe and one or more processors, the TEE probe being robotically controlled by the system and the TEE probe being adjustable with respect to a plurality of degrees of freedom, wherein the method comprises: obtaining, via the TEE probe, near-field images associated with the esophagus, the near-field images reflecting tissue interaction with the esophagus; and forming a heat map based, at least in part, on the near-field images, wherein the heat map reflects tissue interaction with respect to individual portions of the esophagus.

74. The method of claim 73, wherein the TEE probe is periodically transitioned into a state associated with obtaining near-field images.

75. The method of claims 73-74, wherein the one or more processors are configured to perform speckle tracking to track a surface of the esophagus.

76. The method of claims 73-75, wherein the heat map is based on time and / or pressure associated with the TEE probe being at individual positions.

Images