Automated transesophageal echocardiogram control and sensor analysis system

Abstract

Systems and methods for automated transesophageal echocardiogram control and sensor analysis. An example method includes controlling the TEE tool within the esophagus to be at a plurality of station coordinates, wherein the station coordinates reflect unique combinations of values of degrees of freedom of the TEE tool; analyzing, using a classifier and / or regression model, sensor information obtained from the TEE tool while at the station coordinates; and determining a subset of the station coordinates which are associated with respective cardiac views, wherein station coordinates associated with a cardiac view indicates that sensor information obtained from the TEE tool at the station coordinates depicts the cardiac view.

Description

LAZA.020WO PCT APPLICATION AUTOMATED TRANSESOPHAGEAL ECHOCARDIOGRAM CONTROL AND SENSOR ANALYSIS SYSTEM CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Prov. Patent App No. 63 / 501608 titled “AUTOMATED TRANSESOPHAGEAL ECHOCARDIOGRAM CONTROL AND SENSOR ANALYSIS SYSTEM” and filed on May 11, 2023, and also to U.S. Prov. Patent App. No. 63 / 593514 titled “AUTOMATED TRANSESOPHAGEAL ECHOCARDIOGRAM CONTROL AND SENSOR ANALYSIS SYSTEM” and filed on October 26, 2023. The disclosures of the above-recited provisional applications are hereby incorporated herein by reference in their entirety.

[0002] This application further incorporates International Publication WO 2023 / 147544 in its entirety. BACKGROUND TECHNICAL FIELD

[0003] The present disclosure relates to transesophageal echocardiograms (TEE), and more particularly, to software for automated control of TEE probes and analysis of TEE probe sensor information. DESCRIPTION OF RELATED ART

[0004] A transesophageal echocardiogram (TEE) is used to analyze cardiac health of a patient using a semi-invasive procedure. For example, a TEE probe is passed into the patient’s esophagus while an ultrasound transducer on the TEE probe obtains sensor information. The TEE probe may be manipulated in position and / or orientation within the esophagus to obtain specific cardiac views of interest to a medical professional. TEE may be relied upon for relatively clear images of a patient’s heart as compared to other semi-invasive or non-invasive techniques (e.g., a transthoracic echocardiogram (TTE)). Additionally, and as known by those skilled in the art, TTE is limited by the available locations through the chest wall which allow for views of the patient’s heart.

[0005] While TEE provides technical benefits over other techniques, at present use of a TEE probe requires multiple specialized medical professionals. For example, a specialized medical professional may be required to control the TEE probe system while anotherspecialized medical professional analyzes images. In this example, it may take substantial time to obtain clear images of specific cardiac views of interest. Additionally, once a cardiac view is obtained, and the TEE probe subsequently moved, it may take substantial time to later adjust the TEE probe to again obtain images of the cardiac view. SUMMARY

[0006] An example embodiment includes a method implemented by a system of one or more processors, the system being in communication with a robotic transesophageal echocardiogram (TEE) tool configured for insertion into an esophagus of a patient. The method includes initiating, using the robotic TEE tool, cardiac view registration to associate station coordinates with respective cardiac views, wherein individual station coordinates are indicative of, at least, distance and pose of the robotic TEE tool within the esophagus; determining individual station coordinates associated with individual cardiac views, wherein the robotic TEE tool is adjusted within the esophagus and images obtained from the robotic TEE tool are input into a classifier and / or regression model trained to assign confidence values indicative of the images depicting the cardiac views; and causing presentation, via a user device, of an interactive user interface, wherein the interactive user interface includes an individual image determined to depict an individual cardiac view.

[0007] An example embodiment includes a method implemented by a system of one or more processors, the system being in communication with a robotic transesophageal echocardiogram (TEE) tool configured for insertion into an esophagus of a patient. The method includes controlling the TEE tool within the esophagus to be at a plurality of station coordinates, wherein the station coordinates reflect unique combinations of values of degrees of freedom of the TEE tool; analyzing, using a classifier and / or regression model, sensor information obtained from the TEE tool while at the station coordinates; and determining a subset of the station coordinates which are associated with respective cardiac views, wherein station coordinates associated with a cardiac view indicates that sensor information obtained from the TEE tool at the station coordinates depicts the cardiac view.

[0008] An example embodiment includes a method implemented by a system of one or more processors, wherein the system is configured to present an interactive user interface, and wherein the interactive user interface: presents a plurality of images which depict respective cardiac views, wherein the cardiac views reflect views of the patient’s heart and the images areobtained from a robotic transesophageal echocardiogram (TEE) tool, and wherein confidence values associated with the images depicting the respective cardiac views are included; responds to user input associated with selection of a first cardiac view of the cardiac views, wherein the selection is indicative of a request (1) to cause navigation of the TEE tool to station coordinates associated with the first cardiac view or (2) to update the station coordinates associated with the first cardiac view, wherein station coordinates are indicative of, at least, distance and pose of the robotic TEE tool within an esophagus of the patient; and updates based on the selection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A is a block diagram illustrating an example automated inspection system in communication with a transesophageal echocardiogram (TEE) tool.

[0010] Figure 1B is a block diagram illustrating the example automated inspection system obtaining sensor information based on control of the TEE tool using different station coordinates.

[0011] Figure 1C is a block diagram illustrating the example automated inspection system performing cardiac view registration.

[0012] Figure 1D is a block diagram illustrating the example automated inspection system obtaining a real-time image of a requested cardiac view based on the cardiac view registration.

[0013] Figure 2A is a flowchart of an example process for cardiac view registration using a TEE tool.

[0014] Figure 2B is an example user interface enabling selection of information relevant to the cardiac view registration.

[0015] Figure 2C is an example user interface that includes images determined to be associated with specific cardiac views based on the cardiac view registration.

[0016] Figure 3A is a flowchart of an example process for updating station coordinates of a TEE tool that result in images associated with a particular cardiac view.

[0017] Figure 3B is an example user interface that enables selection of a cardiac view to be adjusted.

[0018] Figure 3C is an example user interface depicting manual adjustments to station coordinates of the TEE tool.

[0019] Figure 4A is a flowchart of an example process for determining updates to station coordinates in response to one or more triggers.

[0020] Figure 4B is a flowchart of an example process for determining adjustments to station coordinates.

[0021] Figure 5 is a flowchart of an example process for cardiac view registration based on use of cardiac model data.

[0022] Figure 6 illustrates an example cardiac model along with an imaging plane.

[0023] Figure 7 illustrates an example user interface.

[0024] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same. DETAILED DESCRIPTION

[0025] This application describes techniques to simplify the operation, and increase the effectiveness, of transesophageal echocardiograms (TEE). As will be described, a TEE tool may be controlled by a system in an automated or semi-automated fashion to reliably perform disparate TEE procedures. For example, the TEE tool, such as a TEE probe, may be controlled in a multitude of degrees of freedom while inserted into a patient. Example degrees of freedom may include translations, rotations of the TEE tool (also referred to herein as pose), electronic or mechanical rotations of an ultrasound sensor to adjust the 2-D echo plane, and so on. As another example, the system may control the TEE tool to rapidly determine combinations of degrees of freedom (referred to herein as station coordinates) which result in specific cardiac views of interest. Example cardiac views may include a two-chamber view, a four-chamber view, a five-chamber view, a mid-esophageal long axis view, and so on. As will be described, the disclosed technology may allow a medical professional to quickly view images corresponding to specific cardiac views and reliably cycle between such cardiac views.

[0026] At present, TEE tools are generically designed for disparate ultrasound-based procedures. For example, a TEE tool may typically require interaction with complex software applications to adjust operation of the TEE tool for use in cardiac analyses. In this example,the TEE tool may be used for analyses of other portions of a patient’s body, such as the kidney. Thus, hardware and / or software adjustments may be required prior to use of the TEE tool for cardiac analyses. Furthermore, specialized training is typically required to use these TEE tools. For example, current software to control, and use, TEE tools require substantial interaction with complex user interfaces. Such interactions increase the amount of time that the TEE tool is inserted into a patient.

[0027] Furthermore, the above-described TEE tools require manual control of the TEE tool while inserted into a patient. Thus, a medical professional may manually adjust the station coordinates (e.g., translation, flexion, rotation, and so on) of the TEE tool to obtain images associated with a specific cardiac view. As known by those skilled in the art, specific cardiac views may be preferred by medical professionals. Thus, the medical professional may traverse the tool through the patient finding these specific cardiac views. During a TEE procedure, the medical professional may prefer to return to a particular cardiac view to obtain images associated with the particular cardiac view. Due to such manual control, it may be impractical to reliably return to the same station coordinates. Thus, the medical professional may review images which are distinct in view from the earlier-reviewed images.

[0028] As may be appreciated, since TEE is a semi-invasive technique such manual controls may introduce safety concerns. For example, incorrect adjustment of the TEE tool along one or more degrees of freedom may result in injury of a patient. As an example, incorrect control may cause application of pressure to the interior of the esophagus which may result in a safety concern.

[0029] The disclosed technology, in contrast, relies upon modern software techniques to automate, or semi-automate, control of a TEE tool. For example, and with respect to automation, the disclosed technology may control a robotic TEE tool to automatically obtain a multitude of cardiac views. As another example, and with respect to semi-automation, the disclosed technology may inform a medical professional’s movement within a patient’s esophagus. For example, a user interface may provide recommendations regarding movement of the TEE tool to obtain a specific, or set of, cardiac views.

[0030] As will be described, a system (e.g., the automated inspection system 100) may control a TEE tool to cause insertion of a TEE probe into a patient. Once inserted, the system may perform a cardiac view registration process to identify station coordinates associated with specific cardiac views. As described herein, station coordinates may reflect control parametersassociated with the TEE tool. For example, station coordinates may include a translation of the TEE probe from a resting position. In this example, the translation may be indicative of a length within the patient. As another example, station coordinates may include a rotation of the TEE probe about an axis. As another example, station coordinates may include an electronic rotation of an ultrasound sensor (e.g., ultrasound transducer) about an axis. In some embodiments, the ultrasound sensor may use beamforming techniques, or mechanical adjustment, to enable scanning about the axis.

[0031] With respect to cardiac view registration, the system may therefore obtain station coordinates which result in images which depict, or are otherwise associated with, the specific cardiac views. As will be described, to determine that an image is associated with a cardiac view the system may leverage one or more trained classifiers or regression models. For example, a convolutional or transformer network may be used to assign a label, or score or value, indicative of the cardiac view (e.g., a score or value indicative of a likelihood of current station coordinates corresponding to a particular cardiac view). While a classifier is described herein, other techniques may be used and fall within the scope of the disclosure. For example, more generally a regression model may be used to approximate a real valued number.

[0032] The system may automatically navigate the TEE tool to obtain station coordinates for the specific cardiac views. In some embodiments, the TEE tool may cycle through different station coordinates and compute forward passes of images obtained at the station coordinates through the one or more classifiers. In this way, the system may cause automatic traversal through known station coordinates. In some embodiments, the system may use artificial intelligence techniques, such as deep learning techniques, to automatically navigate the TEE probe. For example, and as will be described in Figure 5, the system may analyze received images and determine the images’ association to a cardiac model. As an example, the system may map image information to a vector space or tensor space associated with the cardiac model.

[0033] In an effort to ensure that previously-determined station coordinates remain accurate, the system may optionally trigger a redetermination of one or more station coordinates. For example, a patient may be bumped or moved slightly while under anesthesia. As another example, the TEE tool may be bumped or moved slightly. In yet another example the esophagus and organs constraint on the TEE may not be sufficient to maintain the relation between the TEE's perceived position and its real position relative to an initial global coordinate system

[0034] Thus, the specific station coordinates associated with a cardiac view may be inaccurate due to this movement. As will be described below, for example in Figures 4A-4B, the system may update station coordinates. In this way, a medical professional may reliably cycle between cardiac views and corresponding images which accurately depict the cardiac views.

[0035] Additional disclosure related to an example TEE tool, and associated software, is included in International Publication WO 2023 / 147544 which is incorporated herein in its entirety.

[0036] The disclosed technology will now be described in more detail.

[0037] Figure 1A is a block diagram illustrating an example automated inspection system 100 in communication with a transesophageal echocardiogram (TEE) tool 110. As will be described, the automated inspection system 100 may provide control instructions 102 to cause movement of a TEE probe 114 within a patient. The TEE probe 114 may include one or more ultrasound sensors (e.g., ultrasound transducers) to perform ultrasound imaging. The automated inspection system 100 may therefore receive sensor information 112, such as images, from the TEE tool 112 and include information in a user interface 120 accessible to an end- user. Images may include, for example, ultrasound images, images or sensor information which includes a 3D volume over an increment of time, and so on.

[0038] The automated inspection system 100 may represent a system of one or more processors or one or more computers. In some embodiments, the automated inspection system 100 may be a computer system which is positioned proximate to a patient and the TEE tool 110. For example, the computer system may be a mobile device (e.g., a tablet, a smart phone), a laptop, a computer, and so on. In these embodiments, the automated inspection system 100 may be in wired or wireless communication with the TEE tool 110. Additionally, the automated inspection system 100 may be in wired or wireless communication with a display configured to present user interface 120.

[0039] With respect to the automated inspection system 100 being proximate to a patient, the system 100 may execute software such as an application. The application may optionally be a mobile application which communicates with the TEE tool 110. Thus, and as an example, a tablet may execute the mobile application to control the TEE tool 110 and present user interface 120 via its touch-screen display.

[0040] In some embodiments, the automated inspection system 100 may represent a cloud-based system or software executing on a cloud-based system. For example, the automated inspection system 100 may be a web application (e.g., a dockerized application) which is in communication with a multitude of TEE tools. In this example, the TEE tool may be connected to a network (e.g., the internet, a private cloud or networked system) and controlled by the system 100. Similarly, an end-user device may be connected to the network and used to render user interface 120. For example, the end-user device may execute an application, or web browser, which renders user interface 120.

[0041] In the illustrated example, the automated inspection system 100 is providing control instructions 102 to the TEE tool 110. The TEE tool 110 may automatically adjust the probe 114 in a multitude of dimensions. For example, the TEE tool 110 may cause the TEE probe 114 to extend along a dimension and be adjustable in two or more dimensions. In this example, the TEE tool 110 may cause rotation of the TEE probe 114 (e.g., via flexion, retroflexion, flexion tilts). The TEE tool 110 may additionally cause rotation or adjustment of an ultrasound transducer (e.g., field of view), for example via beamforming techniques. Additional disclosure related to the TEE tool 110 is included in International Publication WO 2023 / 147544 which forms part of this disclosure as if set forth herein.

[0042] The control instructions 102 may thus cause adjustment of the TEE probe 114 within a patient. For example, the control instructions 102 may cause the TEE probe 114 to extend through the esophagus of the patient. In this example, the control instructions 102 may be associated with station coordinates which describe the TEE probe’s 114 values in the multitude of dimensions (e.g., values for a multitude of degrees of freedom). As described above, example values may reflect the TEE probe’s 114 distance (e.g., from a consistent origin) into the patient, a pose or rotation of the probe, an electronic rotation, or other adjustment (e.g., focal distance), of the ultrasound sensor, and so on. Thus, the automated inspection system 100 may accurately navigate the TEE probe 114 to specific station coordinates via control instructions 102.

[0043] In some embodiments, the automated inspection system 100 may ensure that the TEE probe 114 safely navigates through the esophagus of a patient. For example, the automated inspection system 100 may determine a path from first station coordinates to second station coordinates. In this example, the first station coordinates may be associated with a first cardiac view and a medical professional may prefer to view images associated with a second cardiac view. The automated inspection system 100 may determine control instructions 102 which ensure that pressure applied to the interior of the esophagus is below a threshold. As anexample, the system 100 may cause a straightening out of the TEE probe 112 as it is translated towards the second station coordinates. In some embodiments, the automated inspection system 100 may access a geometrical model indicative of the interior of the esophagus. In some embodiments, the model may represent an average or normal model of a person’s esophagus. In some embodiments, the average or normal model may be adjusted based on deviations detected through imaging or through movement of the probe (e.g., via a force sensor). The model may then be used to inform the specific path towards the second station coordinates (e.g., an automated navigation via the specific path). In some embodiments, the information may be presented as instructions or recommendations to a user who is controlling the TEE tool.

[0044] The TEE probe 114 may additionally be configured with a force or pressure sensor. In this way, the TEE probe 114 may determine the extent to which the TEE probe 114 is applying pressure to the interior of the esophagus. If the pressure exceeds a threshold, the system 100 may adjust rotation of the TEE probe 114 along one or more dimensions to reduce the applied pressure (e.g., via flexion, retroflexion, flexion tilts, and so on). In some embodiments, an electrical impedance sensor may be used (e.g., single or array).

[0045] As the TEE tool 110 is controlled via the automated inspection system 100, the TEE probe 114 may obtain ultrasound images of the heart of a patient. These ultrasound images may be obtained, for example, at 30 Hz, 50 Hz, 60 Hz, 120 Hz, and so on. The TEE tool 110 may thus provide sensor information 112, which includes the ultrasound images, to the system 100.

[0046] The automated inspection system 100 may analyze the sensor information 112 to determine station coordinates which are associated with specific cardiac views. As will be described below, the automated inspection system 100 may execute one or more classifiers or regression models to effectuate the determination. Example classifiers or regression models may include deep-learning models, support vector machines, clustering techniques, and so on. With respect to deep-learning models, the automated inspection system 100 may compute forward passes through convolutional neural networks, transformer networks, fully-connected networks, and so on.

[0047] In this way, the automated inspection system 100 may present images which depict, or are otherwise associated with, specific cardiac views. In the illustrated example, user interface 120 is presenting an image depicting a two-chamber view of a patient’s heart. The user interface 120 further includes a confidence value indicative of a likelihood that the imagedepicts the two-chamber view. For example, the confidence value indicates an 87% certainty. As may be appreciated, a neural network may be trained to assign confidence or regression values to images as being associated with a particular label (e.g., via a sigmoid or sigmoid activation function).

[0048] As will be described below, with respect to Figures 2A-2C, the automated inspection system 100 may enable an end-user (e.g., a medical professional) to update station coordinates associated with a specific cardiac view. For example, if the confidence value is below a threshold the system 100 may update the user interface 120 to identify, or otherwise call out, the low confidence value. The end-user may then update the station coordinates to better conform to the specific cardiac view. In some embodiments, the system 100 may re- scan the local space to find a better conformal view. Optionally, the re-scan may be prioritized by electronic rotation of the ultrasound sensor then motion (e.g., translation, rotation).

[0049] The user interface 120 may thus depict images associated with different cardiac views. An end-user may provide user input to traverse to different cardiac views to view real- time images from the TEE tool 110. Example user input may include verbal input, touch-based input, mouse / keyboard input, and so on. Advantageously, the user interface 120 may depict the last received images, or video sequences, associated with cardiac views. In this way, the end-user may view up-to-date images of different cardiac views. At the end-user’s discretion, the end-user may then cause navigation of the TEE probe 114 to a specific cardiac view to view real-time images (e.g., real-time ultrasound data) of that specific cardiac view.

[0050] Figure 1B is a block diagram illustrating the example automated inspection system 100 obtaining sensor information 112 based on control of the TEE tool 110 using different station coordinates. In the illustrated example, an inspection volume 130 is included. As may be appreciated, the inspection volume 130 may represent a patient undergoing a TEE procedure. For example, the inspection volume 130 may include the patient’s esophagus. As another example, the inspection volume 130 may include the patient’s stomach. The inspection volume 130 may additionally include the organ being imaged (e.g., the heart).

[0051] In Figure 1B, the automated inspection system 100 is providing control instructions 102 to the TEE tool 110. As described in Figure 1A, the control instructions 102 may cause the TEE probe to navigate to specific station coordinates.

[0052] In some embodiments, the automated inspection system 100 may perform cardiac view registration by traversing through pre-defined station coordinates and obtaining sensorinformation 112. For example, the system 100 may traverse through an ordered list of station coordinates. In some embodiments, the ordered list may vary based on a specific procedure being performed. As an example, a septal crossing procedure may be associated with specific station coordinates.

[0053] Initial station coordinates may cause, as an example, the TEE probe to translate by a threshold distance into the inspection volume 130. At the threshold distance, the automated inspection system 100 may then cause the TEE probe to maintain a particular initial rotation. At the distance, and initial rotation, the TEE probe may then obtain images at different electronic rotations and / or focal distances of the ultrasound transducer. Once the TEE probe obtains images at the electronic rotations and / or focal distances, the system 100 may cause an adjustment to the rotation of the TEE probe (e.g., physical rotation).

[0054] At individual station coordinates, the TEE tool 110 may cease movement and then obtain sensor information 112 for a threshold amount of time. In some embodiments, the threshold amount of time may be one second, two seconds, three seconds, and so on. Optionally, the amount of time may relate to capturing sensor information 112 for a threshold number of cardiac cycles. Thus, in some embodiments a patient’s heart rate may be used to inform how the amount of time. In some embodiments, the automated inspection system 100 may receive electrocardiogram (ECG) data to identify a time at which the threshold number of cardiac cycles is reached.

[0055] Thus, in some embodiments the TEE probe may obtain images at each of the station coordinates for a threshold amount of time. In some embodiments, the TEE probe may be configured to adjust electronic rotation substantially quickly. For example, the TEE probe may be capable of adjusting the electronic rotation at kilohertz, megahertz, and so on. In some embodiments, the system 100 may cause the ultrasound transducer to sweep through the different electronic rotations at a particular translation and rotation of the TEE probe. For example, since the adjustment may be substantially faster than the cardiac cycle an initial electronic rotation and a final electronic rotation in a sweep may be associated with a similar cardiac cycle. Thus, the system 100 may reduce an amount of time associated with cardiac view registration. As an example, the system 100 may avoid pausing at each electronic rotation value for a threshold amount of time and instead pause at different combinations of TEE probe translation and rotation values.

[0056] As described above, cardiac view registration may allow for an association ofindividual station coordinates with individual cardiac views. As will be described in Figure 1C, the automated inspection system 100 may thus analyze images obtained at individual station coordinates using one or more classifiers and / or regression models. Since the TEE tool 110 may obtain images at a particular frequency (e.g., 30 Hz, 50 Hz, 60 Hz, and so on) over a threshold amount of time or cardiac cycles, there may be a substantial number of images. In some embodiments, the system 100 may analyze a subset of the images which are associated with a specific cardiac phase.

[0057] An example specific cardiac phase may include an end-diastolic phase. As may be appreciated, the end-diastolic phase may cause the heart to be the most expanded such that images depict features of the heart more clearly. Thus, in some embodiments the automated inspection system 100 may identify end-diastolic images obtained at specific station coordinates. For example, the system 100 may use a classifier and / or regression model (e.g., a machine learning model) trained to identify end-diastolic images. In this example, an image may be input into the classifier. Optionally, one or more images before the image and one or more images after the image (e.g., in time) may be concatenated. Optionally, the classifier may input a set of images spanning one or more heartbeats (e.g., including both systole and diastole). The classifier may then determine a value indicative of the image being end-diastolic. Optionally, the system 100 may use ECG information to identify end-diastolic images. For example, the system 100 may determine a time at which the ECG information indicates the heart is an end-diastolic phase.

[0058] In the illustrated example, images are included for different station coordinates 132. For example, station coordinates A 132A may represent a combination of values associated with different degrees of freedom. As described above, the TEE tool 110 may maintain the TEE probe at station coordinates A 132A and obtain a multitude of images (e.g., Images A-N). Similarly, images associated with station coordinates N 132 are included in Figure 1B.

[0059] Figure 1C is a block diagram illustrating the example automated inspection system 100 performing cardiac view registration. As described in Figure 1B, the system 100 may cause a multitude of images to be obtained from the TEE tool for individual station coordinates. As will be described, the system 100 may identify which station coordinates correspond to which cardiac views.

[0060] The automated inspection system 100 includes a control and classifier engine 150 to assign confidence values to images 152. For example, the control and classifier engine 150may provide control instructions to cause the TEE tool to navigate to particular station coordinates. The engine 150 may then receive sensor information 112 that includes images 152 obtained at the particular station coordinates. As described above, the engine 150 may execute one or more classifiers and / or regression models to assign confidence values associated with different cardiac views. In some embodiments, the confidence values may depend on, or be weighted relative to, a particular procedure.

[0061] In some embodiments, a classifier and / or regression model may be trained to assign confidence values or scores for a multitude of cardiac views. For example, the classifier may output confidence values for a multitude of labels. In some embodiments, individual classifiers may be trained for individual cardiac views. In these embodiments, the engine 100 may optionally select a subset of classifiers to receive the images 152 as input. As an example, the subset may be based on the station coordinates associated with the TEE probe’s position. For example, certain station coordinates may be expected to be able to view a subset of cardiac views.

[0062] As described in Figure 1B, the automated inspection system 100 may filter the sensor information 112 such that images 152 represent images associated with a particular cardiac phase (e.g., end-diastolic images). In this way, the system 100 reduce an extent to which forward passes through the classifier engine 150 are required to be computed.

[0063] The automated inspection system 100 further includes a registration engine 160 which stores station coordinates for specific cardiac views. For example, the registration engine 160 may store station coordinates for a two-chamber view based on confidence values for images 152 exceeding a threshold. As may be appreciated, as the TEE probe adjusts in translation and / or rotation the system 100 may obtain images with greater confidence values for the two-chamber view. Thus, the registration engine 160 may update the station coordinates associated with the two-chamber view during cardiac view registration.

[0064] In this way, the registration engine 160 may store station coordinates for different cardiac views. The registration engine 160 may optionally generate a lookup table which may be relied upon to navigate back to specific cardiac views. For example, an end-user may use a user interface (e.g., user interface 120) to request real-time images corresponding to a specific cardiac view.

[0065] Figure 1D is a block diagram illustrating the example automated inspection system 100 obtaining real-time image(s) of a requested cardiac view 170 based on the cardiac viewregistration. As described above, the automated inspection system 100 may generate a lookup table which associates station coordinates with cardiac views.

[0066] In the illustrated example, an end-user has provided a cardiac view request 170. The request 170 may indicate the cardiac view being requested and be provided, for example, via a user interface (e.g., user interface 120). As an example, the end-user may select a particular cardiac view (e.g., from a list, from a drop-down menu, from presented textual identifiers, verbally saying the particular cardiac view, and so on).

[0067] The registration engine 160 may receive the request 170 and determine station coordinates associated with the cardiac view. For example, in Figure 1D the end-user has requested a two-chamber view. In this example, the registration engine 160 has determined (e.g., using a lookup table) that the two-chamber view is viewable at specific station coordinates 162. The control and classifier engine 150 may then cause the TEE tool to navigate to the specific station coordinates 162.

[0068] The user interface may then be updated to present a real-time image 172, or real- time images 172, of the two-chamber view.

[0069] In some embodiments, the automated inspection system 100 may trigger a new cardiac view registration. For example, the real-time image(s) 172 may be analyzed the control and classifier engine 150. In this example, a classifier and / or regression model may assign a confidence value or label associated with the two-chamber view. If the confidence value is less than a threshold, or greater than a threshold reduction from a prior confidence value, the system 100 may trigger a redetermination of station coordinates associated with the two- chamber view. Additional description related to triggering a determination is described in more detail below, with respect to Figures 4A-4B.

[0070] The description above focused on use of station coordinates, for example in a look- up table. However, in some embodiments the control and classifier engine 150 may determine a path or trajectory to specific station coordinates using received sensor information. For example, the control and classifier engine 150 may use a cardiac model to determine a path towards the station coordinates. In this example, the engine 150 may generate data indicative of the cardiac model during cardiac view registration. As an example, as sensor information is received the engine 150 may map image features into positions of a consistent vector space. The engine 150 may optionally be trained to associate such image features with the vector space to form a cardiac model. In some embodiments, the engine 150 may use certain markers,such as the apex of the left ventricle, to inform positioning of the image features.

[0071] In some embodiments, a standard or generalized cardiac model may be used by the system 100. For example, the standard cardiac model may be based on average heart features of patients. In this example, as sensor information 112 is received the system 100 may update the standard cardiac model to conform to the sensor information 112. For example, a marker (e.g., the apex of the left ventricle) may be used to adjust the standard cardiac model in orientation (e.g., different patients may have slightly differently oriented hearts). As another example, the geometry of the cardiac heart model may be updated based on the sensor information 112. As an example, a length and / or size associated with portions of the cardiac model may be updated based on the sensor information 112. With respect to length, the system may determine length based on the station coordinates of the TEE tool 110 in combination with received images. For example, stereo vision techniques, or other computer vision techniques, may be leveraged to estimate length and / or size.

[0072] With respect to the standard cardiac model, the system may determine a path towards a particular cardiac view (e.g., as described in Figure 1D). Since the standard cardiac model may not correspond precisely to the patient’s anatomy, the system may navigate the TEE tool to station coordinates which are expected to be proximate to station coordinates associated with the particular cardiac view. The system may then cause refinements to the TEE tool’s station coordinates to obtain images at the refined station coordinates. Via analyzing the images, for example using a classifier, the system may determine station coordinates most closely matching the requested cardiac view.

[0073] Figure 2A is a flowchart of an example process 200 for cardiac view registration using a TEE tool. For convenience, the process 200 will be described as being performed by a system of one or more processors (e.g., the automated inspection system 100).

[0074] At block 202, the system causes presentation of an interactive user interface associated cardiac analysis. The interactive user interface may be presented via a display of the system, or in which is in communication with the system. Example user interfaces are illustrated in Figures 2B-2C and described below.

[0075] At block 204, the system initiates cardiac view registration. As described above, the system may control a TEE tool to obtain images at different station coordinates within a patient. Cardiac view registration may be initiated based on receipt of user input from an end- user (e.g., a medical professional). For example, the end-user may provide user input to theuser interface indicative of initiating the registration.

[0076] At block 206, the system causes adjustment of the TEE tool. In some embodiments, and as described in Figure 1B, the system may cause traversal through particular station coordinates. For example, the particular station coordinates may be pre-stored. In some embodiments, the particular station coordinates may be adjusted based on the patient. For example, the system may reduce an extent to which the TEE tool extends into a shorter patient as compared to a taller patient.

[0077] At block 208, the system determines station coordinates for individual cardiac views. As the TEE tool navigates to station coordinates, the system analyzes received images from the TEE tool. The system may use, for example, a classifier and / or regression model which is trained to output confidence values associated with cardiac views. Thus, the system may determine which station coordinates correspond to images depicting which cardiac views.

[0078] At block 210, the system updates the user interface based on the cardiac view registration. The system may include an example image for each cardiac view in the user interface. Optionally, the system may indicate a confidence value for each example image. In this way, the end-user may quickly identify how responsive each image is to an associated cardiac view. As will be described in Figures 3A-3C, the end-user may optionally update (e.g., manually update) station coordinates. For example, the end-user may manually control the TEE tool.

[0079] Figure 2B is an example user interface 220 enabling selection of information relevant to the cardiac view registration. User interface 220 may be presented to an end-user near the start of a TEE procedure. The end-user may select from different preset types of procedures. Example procedures may include MitralClip, Mitral Valve, atrial appendage closure, and so on as known by those skilled in the art. As described above, these selections may inform, at least, the different cardiac views which may be of interest to the end-user.

[0080] Figure 2C is an example user interface 230 that includes images determined to be associated with specific cardiac views based on the cardiac view registration. In the illustrated example, four cardiac views are included along with ultrasound images depicting the four cardiac views. For example, an image corresponding to a two-chamber view is included in the upper left. As another example, an image corresponding to four-chamber view is included in the upper right. As another example, an image corresponding to a five-chamber view is included in the lower left. As another example, an image corresponding to an MC view isincluded in the lower right.

[0081] The user interface 230 further includes confidence values for each of the cardiac views. These confidence values, in the example, range from 98% certainty to 89% certainty. Thus, an end-user may review the confidence values to determine whether to manually update station coordinates which are associated with the cardiac views.

[0082] The user interface 230 may respond to selection of a particular cardiac view. For example, selection of the particular cardiac view may trigger the system 100 to navigate the TEE tool to station coordinates associated with the particular cardiac view. The user interface 230 may then update with real-time image(s), or specific image(s) (e.g., end-diastolic images), from the TEE probe while at the station coordinates.

[0083] Figure 3A is a flowchart of an example process 300 for updating station coordinates of a TEE tool that result in images associated with a particular cardiac view. For convenience, the process 300 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0084] At block 302, the system determines confidence values associated with cardiac views during cardiac view registration. As described above, the system causes the TEE tool to traverse through station coordinates to obtain images at the station coordinates. The system analyzes the images and determines station coordinates which are associated with the cardiac views. In addition, the system determines confidence values indicative of respective confidences that the station coordinates are associated with the cardiac views.

[0085] At block 304, the system responds to user input to update station coordinates for a particular cardiac view. A user interface, such as included in Figure 2C and 3B, may be presented which includes individual images that correspond to individual cardiac views. The user interface may additionally present the confidence values for the cardiac views.

[0086] As described herein, an end-user may provide user input indicative of a request to update station coordinates (e.g., for the particular cardiac view). For example, the particular cardiac view may have a confidence value less than a threshold. In this example, the user interface may identify, or otherwise visually or auditorily call out, the low confidence value. As another example, the end-user may view the image which corresponds to the particular cardiac view and prefer to change the station coordinates.

[0087] At block 306, the system determines updated station coordinates for the particular cardiac view. The system may receive user input which manually controls the TEE tool. Forexample, the user interface may include controls to adjust the station coordinates. In this example, the end-user may cause translation of the TEE tool, rotation of the tool, electronic rotation of the ultrasound transducer, and so on.

[0088] The system may update the user interface to include real-time images from the TEE tool during manual control. In this way, the end-user may cease manual control based on finding one or more real-time images to satisfactorily correspond to the particular cardiac view. The system then stores the updated station coordinates as being associated with the particular cardiac view.

[0089] At block 308, the system optionally updates station coordinates for other cardiac views. The system may determine that some, or all, remaining cardiac views may have associated station coordinates adjusted. For example, the updated station coordinates determined in block 306 may be used as a mapping from the expected station coordinates determined by the system to the user-refined station coordinates. As another example in which a cardiac model is used, the updated station coordinates may be used to model three- dimensional positions associated with remaining cardiac views. For example, if the particular cardiac view in block 306 is a two-chamber view then the cardiac model may be used to inform a refinement to station coordinates associated with the four-chamber view.

[0090] As described above, in some embodiments the system may traverse through station coordinates and analyze images obtained at those station coordinates. Thus, there may be respective threshold differences in the degrees of freedom during traversal between station coordinates. For example, first station coordinates and second station coordinates may differ by a threshold translation distance or threshold rotation degree. In this way, the system may analyze images obtained at discrete steps through an inspection volume (e.g., an esophagus as described in Figure 1B).

[0091] During cardiac view registration, for example as described at least in Figure 2A, the system may therefore associate one of the above-described discrete steps (e.g., particular station coordinates) with a cardiac view. In some embodiments, the system may update station coordinates for other cardiac views to correspond with station coordinates which were not initially searched. For example, if the particular view in block 306 is a two-chamber view then the system may update station coordinates for the four-chambre view to correspond with discrete values which were not initially traversed. Thus, the updated station coordinates may reflect a refinement to the discrete steps through the inspection volume. For example, theupdated station coordinates may be in between the discrete steps.

[0092] For any updated station coordinates, the system may optionally navigate to those updated station coordinates and present one or more images to the end-user. The end-user can then validate that the updated station coordinates provide more accurate views of the associated cardiac views. The system may optionally input the images to the classifier and / or regression model described herein to determine confidence scores.

[0093] Figure 3B is an example user interface 310 that enables selection of a cardiac view 312 to be adjusted. As described in Figure 2C, a user interface may be presented with example images which correspond to cardiac views. Proximate to the images, the user interface may reflect confidence values associated with the cardiac views. In this way, an end-user may rapidly identify whether the station coordinates are determined to confidently reflect a particular cardiac view.

[0094] In the illustrated example, user interface 310 includes an image associated with a five-chamber cardiac view 312. The user interface 310 indicates that a confidence value that the image reflects the five-chamber cardiac view 312 is “60%”. In some embodiments, the system may cause the user interface 310 to reflect that the confidence value is below a threshold. For example, a portion of the user interface that includes the image may be highlighted or otherwise called out. As another example, the user interface 310 may include a marker 314 which is of a particular color (e.g., red, a user-selectable color).

[0095] The user interface 310 may include a selectable option 316 to adjust the station coordinates associated with the five-chamber view. For example, the user interface 310 may respond to user input from the end-user to cause presentation of selectable option 316. Example user input may include touching the presented image (e.g., a long press, a short press, press with greater than a threshold force or pressure, and so on). Example user input may also include verbal input, mouse / keyboard input, and so on.

[0096] Figure 3C is an example user interface 320 depicting manual adjustments to station coordinates of the TEE tool. User interface 320 may be presented, in some embodiments, in response to selection of selectable option 316 described in Figure 3B. In the example of Figure 3C, the user interface 320 includes manual controls associated with the TEE tool described herein. For example, the manual controls are included in user interface portion 322. As illustrated, the end-user may adjust the translation 324, rotation 326, and electronic rotation 328 via interactions with portion 322.

[0097] The user interface 320 additionally includes user interface portion 324 which may present information relevant to the end-user determining updated station coordinates. For example, user interface portion 324 may include real-time images from the TEE tool (e.g., from the ultrasound transducer included at the terminal end of the TEE probe). In this example, the real-time images may be adjusted by the system 100. For example, automated gain adjustments, filtering, and so on may be performed. In embodiments in which a cardiac model is used, the user interface portion 324 may present a graphical representation of the cardiac model along with an estimated location of the TEE probe (e.g., the ultrasound transducer).

[0098] The user interface 320 may respond to user input which causes updated station coordinates to be saved. For example, the updated station coordinates may be registered as being associated with the five-chamber view. In implementations in which a look-up table is used, the updated station coordinates may be included in the look-up table as being associated with the five-chamber view.

[0099] The user interface 320 includes selectable option 330 which enables bookmarking of station coordinates. In some embodiments, the end-user of user interface 320 may create a custom cardiac view by interacting with selectable option 330. For example, the end-user may identify an interesting cardiac view during manual control of the TEE tool. In response, the system 100 may store information identifying the custom cardiac view along with associated station coordinates. The custom cardiac view may be included in a user interface, such as user interface Figure 2C, optionally along with an identifier assigned by the end-user.

[0100] Figure 4A is a flowchart of an example process 400 for determining updates to station coordinates in response to one or more triggers. For convenience, the process 400 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0101] Process 400, as will be described, may be performed to update the station coordinates during a TEE procedure. As described above, the patient may move under her / his own power while anesthetized or may be inadvertently bumped or moved slightly during the procedure. Additionally, the TEE tool (e.g., tool 110) may be inadvertently bumped or moved. Thus, the system may trigger a redetermination of station coordinates for at least some of the cardiac views. As noted above, in some embodiments the physiological constraints on the probe may not be sufficient to allow the probe to return to the same physical location relative to a physiological ordinate such as a specific view - e.g. the esophagus shifts or the heart shifts.

[0102] At block 402, the system obtains sensor information the TEE tool. During the TEE procedure, the TEE tool may provide sensor information to the system for analysis and / or presentation in a user interface. For example, a medical professional may view real-time images from the TEE tool (e.g., from the ultrasound sensor in the TEE probe).

[0103] In some embodiments, the TEE tool may have a force sensor (e.g., a force transducer, force sensing resistor, and so on) positioned proximate to the ultrasound sensor. The system may utilize the force sensor, for example, to determine whether the TEE probe is applying greater than a threshold pressure or force to the patient. The system may also utilize the force sensor to determine if the TEE probe has inadvertently moved. For example, the system may determine that the registered force sensor readings are indicative of movement within the patient.

[0104] At block 404, the system triggers updates to station coordinates for one or more cardiac views based on the sensor information. As described above, the system may trigger the updates based on inadvertent movement of the patient and / or TEE tool.

[0105] An example technique to detect movement includes analyzing the real-time images from the TEE tool. For example, a medical professional may have requested that the TEE tool navigate to station coordinates associated with a particular cardiac view. In this example, the system may receive images from the TEE tool at the station coordinates. The system may perform comparisons between the received images and one or more previously stored images associated with the particular cardiac view. The previously stored images may have been obtained, for example, during cardiac view registration as described at least in Figure 2A above. The comparisons may be performed between images which are associated with a same, or similar, cardiac phase. For example, the system may compare a received image which is end-diastolic with a prior image which is end-diastolic.

[0106] An example comparison may be based on mean squared error between a received image and a previous image. Another example comparison may be based on a structural similarity index. Another example comparison may be based on use of local descriptors, such as use of SIFT, SURF, and so on. Another example comparison may be based on deep learning techniques (e.g., Siamese networks). Additional comparisons, such as use of similarity metrics, may be used and fall within the scope of the disclosure herein.

[0107] Another example technique to detect movement is based on use of one or more force sensors. For example, the TEE tool may have a force sensor, an accelerometer, an inertialmeasurement unit (IMU), and so on, which may be used to detect a bump or adjustment to the TEE tool. Similarly, the patient, table on which the patient is lying, and so on, may have a force sensor, accelerometer, IMU, and so on. TEE probe may have a force sensor and may be used to determine whether the probe has moved within the patient.

[0108] Another example technique to detect movement is based on analyzing images received from the TEE tool. While the TEE tool is at particular station coordinates, the images received from the TEE tool may be expected to have a same viewpoint. As may be appreciated, the heart is in continual movement such that the images may form a video sequence of the movement. The system may analyze the received images to determine whether the TEE tool is being adjusted in viewpoint. For example, the system may determine motion vectors which are associated with certain image features in the received images. In this example, certain image features (e.g., a portion of the heart) may be expected to move due to movement of the heart. However, such movements may be monitored and if their position and / or orientation in the images changes then the system may determine that the TEE tool has moved.

[0109] In some embodiments, the system may average, or add, images which form a video sequence of a cardiac cycle or a portion thereof. The average, or addition, may be expected to be similar during a fixed period of time (e.g., a cardiac cycle, such as determined using image processing or ECG data). Thus, if the average, or addition, changes the system may determine that the TEE tool has moved.

[0110] At block 406, the system determines updates to station coordinates. The system may update the station coordinates for all, or a subset, of the cardiac views. For example, the system may perform a new cardiac view registration process to re-determine the station coordinates. The system may also refine the previously determined station coordinates. For example, the system may navigate to each, or some, of the previously determined station coordinates and perform a fine search around the station coordinates. In this example, the system may analyze images using a classifier and / or regression model (e.g., as described above) to determine updated station coordinates.

[0111] An example of adjusting station coordinates will now be described with reference to Figure 4B.

[0112] Figure 4B is a flowchart of an example process 410 for determining adjustments to station coordinates. For convenience, the process 410 will be described as being performed by a system of one or more computers (e.g., the automated inspection system 100).

[0113] At block 412, the system causes adjustment of the TEE tool to station coordinates associated with a particular cardiac view. The particular cardiac view may represent, for example, a cardiac view requested by an end-user. As described above, the system may cause the TEE tool to navigate to the particular cardiac view. For example, the system may control the TEE tool based on station coordinates associated with the particular cardiac view. In implementations in which a cardiac model is used, the system may navigate based on the cardiac model.

[0114] Additional description related to navigation using a cardiac model is included below with respect to, at least, Figure 6.

[0115] At block 414, the system obtains images from the TEE tool at the station coordinates.

[0116] At block 416, the system compares the images with images stored during cardiac view registration. As described in Figure 4A, the system may compare real-time images from the TEE tool with previously stored images which are associated with the station coordinates.

[0117] At block 418, the system determines adjustments to the station coordinates. Based on the comparisons indicating that the TEE tool has moved since cardiac view registration, for example as described in Figure 4A, the system may determine updated station coordinates. These updated station coordinates may then be stored by the system, for example included in the previously-described look-up table which associates station coordinates to cardiac views. In some embodiments, a global coordinate shift may be used such that the original station coordinates are valid.

[0118] In some embodiments, the system may identify previously stored images which are closest (e.g., based on the comparisons described above) to the real-time images. As described in Figure 1B and 2A, during cardiac view registration the system may cause the TEE tool to navigate to discrete station coordinates. Images obtained at these discrete station coordinates may then be analyzed via one or more classifiers. Thus, the system may have access to prior images taken at a multitude of station coordinates to which the TEE tool navigated during cardiac view registration.

[0119] The system may therefore compare the real-time images to these prior images. For example, the system may compare the real-time images to prior images taken at station coordinates which are proximate to the station coordinates of block 412. In this example, the system may identify that prior images taken at particular station coordinates are the closest to the real-time images. Thus, the system may associate the particular station coordinates withthe particular cardiac view of block 412. In some embodiments, the system may perform a search around the particular station coordinates to refine the coordinates.

[0120] As described in Figure 4A, the system may compare real-time images with prior images using a particular cardiac phase. For example, the system may compare prior images which are end-diastolic to real-time images which are end-diastolic. Since the system has access to the prior images, the system may optionally compute a multitude of comparisons in parallel. For example, the system may obtain real-time images over one or more cardiac cycles. In this example, the system may identify end-diastolic images and compare them with previously-obtained end-diastolic images from a multitude of station coordinates. Optionally, the system may sum, or average, the obtained real-time images and compare them with summed, or averaged, prior images from the multitude of station coordinates.

[0121] Optionally, the system may update station coordinates associated with other cardiac views. For example, the system may determine a mapping (e.g., linear mapping) between the station coordinates determined in block 418 with the station coordinates of block 412. The mapping may be applied to other station coordinates to translate them to the new orientation of the TEE probe after movement.

[0122] In some embodiments, the updated station coordinates may inform updates to the other cardiac views. For example, rather than applying a linear mapping the system may adjust station coordinates for the other cardiac views based on a difference between the translation (e.g., length) determined in block 418 is different from the translation (e.g., length) of block 412. The difference may be assumed to reflect that the TEE tool extended further into, or further out of, the patient. This difference may then be applied to remaining cardiac views. Similar to the above, the system may optionally navigate to the updated station coordinates for remaining cardiac views and perform local searches to refine the coordinates.

[0123] In some embodiments, the system may update station coordinates of a single cardiac view. The system may then navigate the TEE tool to those updated station coordinates and obtain real-time images. The system may use a classifier and / or regression model, along with a local search about the updated station coordinates, to refine the coordinates for the single cardiac view. The system may then update the mapping and repeat the process for another cardiac view. In this way, the system may determine a more complete picture regarding updates to cardiac views caused due to movement of the TEE tool.

[0124] Figure 5 is a flowchart of an example process 500 for cardiac view registrationbased on use of cardiac model data. For convenience, the process 500 will be described as being performed by a system of one or more processors (e.g., the automated inspection system 100).

[0125] At block 502, the system initiates cardiac view registration associated with cardiac views. As described above, for example in Figure 2A, the system may perform cardiac view registration to associate station coordinates with cardiac views.

[0126] At block 504, the system causes a first adjustment of a TEE tool. The system may navigate the TEE tool, such as extending and / or rotating a TEE probe at the terminal end of the TEE tool, within a patient. As the TEE tool is maneuvered within the patient, the system receives images from the TEE tool for analysis.

[0127] At block 506, the system associates received images with portions of a cardiac model. The system may execute a machine learning model, such as a convolutional or transformer neural network, to detect image features. The machine learning model may additionally be trained to position the image features within a cardiac model. For example, the machine learning model may be trained using image data with ground truth data indicating image features and associated positions within a heart. As the TEE tool is controlled to extend laterally in the patient, and to be rotated in position, the system may map detected image features into a consistent vector space. In this example, the image features may be represented as voxels or other data which identifies position, size, and so on, of the image features.

[0128] In some embodiments, the cardiac model may represent a model which corresponds to an average, or common, heart. The system may detect certain markers or features which may be used to adjust the cardiac model. For example, an orientation of the patient’s heart within her / his body may be determined via detection of the apex of the left ventricle. As another example, size information associated with portions of the heart may be determined via the received images. For example, three-dimensional size information may be extracted via correlating same image features as depicted at different station coordinates. Thus, the cardiac model may be refined to better correspond to the patient’s heart.

[0129] In this way, an estimated position of the TEE tool may be determined as it navigates through the patient’s esophagus. For example, real-time images may be analyzed to compare the viewpoint of the heart as depicted in the images with an expected viewpoint based on the cardiac model and the current station coordinates of the TEE tool.

[0130] At block 508, the system determines second adjustment of the TEE tool. Thesystem may use the cardiac model to more rapidly navigate to cardiac views. For example, the system may estimate station coordinates for a particular cardiac view during cardiac view registration. In this example, the estimate may be based on the cardiac model as described in block 506. Thus, the system may cause the TEE tool to navigate to the estimated station coordinates. The system may then perform a local search about the estimated station coordinates and analyze received images using the above-described classifier. In this way, the system may identify station coordinates which are to be associated with the particular cardiac view.

[0131] The above-described identification may be used to update and / or refine the cardiac model described in block 506. For example, the system may adjust dimensions and / or orientation of the cardiac model. As an example, the cardiac model may be updated to correspond with station coordinates which have been determined to be associated with specific cardiac views during cardiac view registration. Cardiac Model Navigation

[0132] As described above, a cardiac model may be used, in part, to inform navigation of the TEE tool. For example, the cardiac model may be generated based on sensor information (e.g., ultrasound data) received from the TEE tool as it moves within a portion of a patient (e.g., the esophagus). As another example, the cardiac model may be based on an average, or common, heart model of patients. The above-described system (e.g., automated inspection system 100) may adjust the cardiac model to customize it based on a specific patient. To obtain a particular view, such as a four-chamber view, the system may identify the view based on analyzing the model.

[0133] In some embodiments, the system may access a trained machine learning model which outputs information indicative of the views described herein. The machine learning model may represent a neural network (e.g., a dense or fully-connected network), a convolutional neural network, a transformer-based neural network, and so on. As will be described, the system may input the cardiac model of the patient (e.g., information defining the cardiac model) and obtain information indicative of the views. In this way, the machine learning model may be relied upon to output, for example all at once or sequentially, the specific views which are of interest to a medical professional.

[0134] The TEE probe, as described herein, may include an ultrasound sensor whichobtains ultrasound sensor data of a patient’s heart. The TEE probe may be positioned such that it moves within the patient while pointing substantially at the heart. As described above, the TEE probe may have a multitude of degrees of freedom to obtain ultrasound sensor data with different field of views. For example, the TEE probe may be able to translate within the patient, rotate within the patient, have an electronic focus adjuster, have the ultrasound electronically or physical rotate, and so on as described herein. In some embodiments, the TEE probe may represent a three-dimensional or four-dimensional echocardiography probe. Thus, the TEE probe may obtain information which is able to determine a three-dimensional representation of the heart.

[0135] As an example, the TEE probe may obtain a plane of sensor data which extends substantially from an apex, or region, (collectively the apex) associated with the TEE probe. The plane, as may be appreciated, may be substantially triangular. The TEE probe may therefore obtain sensor data while at particular station coordinates with the field of view relating to the apex.

[0136] The TEE probe may be navigated within the patient while obtaining ultrasound sensor data at different station coordinates. In some embodiments, the ultrasound sensor data may represent ultrasound images which image along the above-described planes. In some embodiments, the ultrasound sensor data may represent, or be transformed into, a point cloud. For example, the TEE probe may obtain ultrasound data at different depths within a volume of space. The TEE probe may optionally adjust up and down such that the plane sweeps up and down to extend the volume of space up and down along with increasing in depth. In this example, a point cloud may be generated that represents points or features within the volume of space.

[0137] The system may thus generate a three-dimensional model of the patient’s heart based on the ultrasound sensor data received via the TEE probe. As an example, the system may use a machine learning model to combine (e.g., stitch together) features in the ultrasound sensor data to form a consistent three-dimensional model. As another example, computer vision techniques (e.g., point cloud registration techniques) may be used to form the three- dimensional model. For example, the system may use ultrasound images to form the three- dimensional model. As another example, the system may use point clouds to form the three- dimensional model.

[0138] Without being constrained by way of theory or example, the system may useultrasound sensor data associated with a particular cardiac phase of the heart. In this way, the ultrasound sensor data may reflect features of the heart at substantially similar positions. These features may then form the three-dimensional model of the patient’s heart. In some embodiments, the system may obtain ultrasound sensor data while at particular station coordinates. The system may then rapidly adjust the ultrasound sensor up and down and optionally in depth to obtain sensor data. The ultrasound sensor, via this electronic adjustment, may therefore obtain ultrasound sensor data of the heart while at a substantially similar cardiac phase. As the TEE probe sensor moves, the system may similarly obtain ultrasound sensor data at the cardiac phase. In some embodiments, the system may use machine learning techniques to form a three-dimensional model using ultrasound sensor data from arbitrary cardiac phases.

[0139] While obtaining ultrasound sensor data to generate the three-dimensional mode, the system may store information describing the portion of the patient in which the TEE probe is maneuvering. For example, the system may identify how far the probe extends into the patient. In this example, the system may use this information to, in part, define each apex at which the TEE probe obtains ultrasound sensor data. In some embodiments, the TEE probe may include a force or other sensor to detect when the TEE probe is touching, and optionally amount of pressure or force being applied to, the patient’s body. This information may be used to inform navigation as described herein. The information may additionally be used to inform a contour of the patient’s interior. Thus, the system may have, for a given apex, the station coordinates of the TEE probe along with some information describing the patient’s interior at those station coordinates.

[0140] In some embodiments, the system may additionally access other types of medical data which may define geometry information associated with the patient’s interior (e.g., throat, esophagus). For example, a computed tomography (CT) scan may be performed and used to define a volume in space in which the TEE probe is able to operate. The system may use this information to determine where in the patient the TEE probe is relative to the volume of space.

[0141] As described above, the three-dimensional model may be used to inform navigation of the TEE probe. For example, a medical professional may prefer to view a particular medical view of the patient’s heart. In this example, the system may cause the TEE probe to adjust in station coordinates such that the TEE probe is able to obtain a real-time (e.g., live) view of the particular medical view. Specifically, a machine learning model may be trained to output information indicative of different medical views.

[0142] For example, the machine learning model may be trained to translate between a three-dimensional cardiac model and specific views. As described above, the TEE probe may obtain ultrasound data from a particular apex with the ultrasound sensor data corresponding to a plane extending from the apex. Each view may thus represent one of these planes.

[0143] The above-described machine learning model may thus output information identifying specific apexes at which respective cardiac views may be obtained. Each apex may optionally be defined according to particular station coordinates. Thus, the machine learning model may output station coordinates which correspond to different cardiac views. Optionally, the machine learning model may receive information describing geometry of the patient’s esophagus or the station coordinates used to generate the cardiac model. Using this information, the model may directly output station coordinates which correspond to medical views.

[0144] Optionally, the machine learning model may output information which is adjusted based on the patient. For example, the model may output information which is relative to the input cardiac model. In this example, the information may indicate apexes at which the TEE probe will extend imaging planes which are associated with cardiac views. As an example, the model may identify that a four-chamber view is associated with a specific orientation, position, and so on, relative to the cardiac model. That is, the four-chamber view is associated with a plane that extends from an apex at the specific orientation, position, and so on. The system may then use this output to determine the specific station coordinates for the patient. For example, the system may determine a length of translation by the TEE probe when obtaining sensor data to form the cardiac model. The system may also use information from the force or touch sensor to inform the contour of the esophagus. The system may also use CT data. Thus, the system may determine the station coordinates which correspond to output of the machine learning model.

[0145] Figure 6 illustrates an example cardiac model along with an imaging plane. In the illustrated example, a green and red line define a plane which is being imaged by the TEE probe. A user interface may present the cardiac model (e.g., a dynamic / beating model or a substantially fixed model). The user interface may additionally present a real-time view of a cardiac view. In some embodiments, Figure 6 may be presented to illustrate which plane corresponds to the cardiac view.

[0146] In some embodiments, the system may present a dynamic view of the patient’s heart. For example, the system may obtain ultrasound sensor data while at particular stationcoordinates and use that to inform updating of the cardiac model. In this example, the ultrasound sensor data may have limited visibility into the full heart. However, the movement of the heart as seen by the TEE probe may be monitored and used to update the cardiac model. For example, the movement may be used to cause or simulate motion of the cardiac model. In this example, the system may track specific landmarks or features which may be used to inform motion of the overall heart. In some embodiments, when the TEE probe is moving to navigate to station coordinates for a cardiac view, the TEE probe may traverse at a particular speed such that the TEE probe is substantially in phase with the cardiac phase of the heart.

[0147] To control the TEE probe, the system described herein may allow for fine-grain and course-grain controls. In some embodiments, the system may execute a machine learning model which maps input of simplified user input to complex controls of the TEE probe. As known by those skilled in the art, there may be a plethora of controls for an ultrasound sensor. Thus, a medical professional using a user interface to control the TEE probe may provide simplified input regarding where the TEE probe is to navigate or an intended goal (e.g., obtain a clearer picture, move to see a particular feature in more detail, and so on). This information may be analyzed, for example via a machine learning model, to output the specific fine-grain controls which would be required to implement the navigation or intended goal.

[0148] Figure 7 illustrates an example user interface. In this user interface, an indication of force being applied by the TEE probe is shown (e.g., 19 grams). As described above, this may be determined using a force sensor. In addition, the user interface includes a review option at the upper left. The review option may provide summary information associated with the cardiac views, cardiac model, and so on.

[0149] In some embodiments, the system may include functionality which may be organized into disparate layers. For example, the functionality may be organized similar to that of the open systems interconnection (OSI model). In this example, lower-level control information may represent a lower level layer. Upper level layers may include higher level data, such as a layer to extract a cardiac model, a layer to perform movement, commands which cause translation into fine-grain controls, and so on. Other Embodiments

[0150] All of the processes described herein may be embodied in, and fully automated, via software code modules executed by a computing system that includes one or more computersor processors. The code modules may be stored in any type of non-transitory computer- readable medium or other computer storage device. Some or all the methods may be embodied in specialized computer hardware.

[0151] Many other variations than those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain acts, events, or functions of any of the algorithms described herein can be performed in a different sequence or can be added, merged, or left out altogether (for example, not all described acts or events are necessary for the practice of the algorithms). Moreover, in certain embodiments, acts or events can be performed concurrently, for example, through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially. In addition, different tasks or processes can be performed by different machines and / or computing systems that can function together.

[0152] The various illustrative logical blocks, modules, and engines described in connection with the embodiments disclosed herein can be implemented or performed by a machine, such as a processing unit or processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor can be a microprocessor, but in the alternative, the processor can be a controller, microcontroller, or state machine, combinations of the same, or the like. A processor can include electrical circuitry configured to process computer-executable instructions. In another embodiment, a processor includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. A processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, a processor may also include primarily analog components. For example, some or all of the signal processing algorithms described herein may be implemented in analog circuitry or mixed analog and digital circuitry. A computing environment can include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computational engine within an appliance, to name a few.

[0153] Conditional language such as, among others, “can,” “could,” “might” or “may,” unless specifically stated otherwise, are understood within the context as used in general to convey that certain embodiments include, while other embodiments do not include, certain features, elements and / or steps. Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and / or steps are included or are to be performed in any particular embodiment.

[0154] Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is understood with the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (for example, X, Y, and / or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present.

[0155] Any process descriptions, elements or blocks in the flow diagrams described herein and / or depicted in the attached figures should be understood as potentially representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or elements in the process. Alternate implementations are included within the scope of the embodiments described herein in which elements or functions may be deleted, executed out of order from that shown, or discussed, including substantially concurrently or in reverse order, depending on the functionality involved as would be understood by those skilled in the art.

[0156] Unless otherwise explicitly stated, articles such as “a” or “an” should generally be interpreted to include one or more described items. Accordingly, phrases such as “a device configured to” are intended to include one or more recited devices. Such one or more recited devices can also be collectively configured to carry out the stated recitations. For example, “a processor configured to carry out recitations A, B and C” can include a first processor configured to carry out recitation A working in conjunction with a second processor configured to carry out recitations B and C.

[0157] It should be emphasized that many variations and modifications may be made to the above-described embodiments, the elements of which are to be understood as being amongother acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure.

Claims

WHAT IS CLAIMED IS:

1. A method implemented by a system of one or more processors, the system being in communication with a robotic transesophageal echocardiogram (TEE) tool configured for insertion into an esophagus of a patient, and the method comprising: initiating, using the robotic TEE tool, cardiac view registration to associate station coordinates with respective cardiac views, wherein individual station coordinates are indicative of, at least, distance and pose of the robotic TEE tool within the esophagus; determining individual station coordinates associated with individual cardiac views, wherein the robotic TEE tool is adjusted within the esophagus and images obtained from the robotic TEE tool are input into a classifier and / or regression model trained to assign confidence values indicative of the images depicting the cardiac views; and causing presentation, via a user device, of an interactive user interface, wherein the interactive user interface includes an individual image determined to depict an individual cardiac view.

2. The method of claim 1, wherein the robotic TEE tool includes a TEE probe at a terminal end of the robotic TEE tool, and wherein the station coordinates reflect a distance and pose of the TEE probe.

3. The method of claim 2, wherein the TEE probe includes an ultrasound sensor, wherein the ultrasound sensor is electronically adjustable in rotation, and wherein the station coordinates are further indicative of the rotation of the ultrasound sensor.

4. The method of claim 1, wherein determining individual station coordinates comprises: causing the robotic TEE tool to navigate to a plurality of station coordinates which include the individual station coordinates, for each of the plurality of station coordinates, obtaining images from the robotic TEE tool and computing a forward pass of the obtained images through the classifier and / or regression model, and determining, based on output from the classifier and / or regression model, the individual station coordinates associated with the individual cardiac views.

5. The method of claim 4, wherein the plurality of station coordinates are pre-stored and reflect discrete steps of distance and / or pose of the robotic TEE tool.

6. The method of claim 1, wherein the robotic TEE tool is adjusted according to a plurality of station coordinates, wherein the station coordinates are pre-stored and reflect discrete steps of distance and / or pose of the robotic TEE tool, and wherein the individual station coordinates are selected based on the classifier and / or regression model.

7. The method of claim 1, wherein the cardiac views include one or more of a two-chamber view, a four-chamber view, and a five-chamber view.

8. The method of claim 1, wherein the classifier and / or regression model is a convolutional neural network or a transformer network.

9. The method of claim 1, wherein the images are ultrasound images.

10. The method of claim 1, wherein the individual station coordinates are included in a data structure which associates station coordinates and cardiac views.

11. The method of claim 10, wherein the data structure is a look-up table.

12. The method of claim 1, wherein the cardiac view registration is initiated in response to user input received via the interactive user interface.

13. The method of claim 1, wherein the interactive user interface receives user input indicative of updating station coordinates associated with a particular cardiac view, and wherein the interactive user interface enables manual control of the robotic TEE tool to update the station coordinates associated with the particular cardiac view.

14. The method of claim 13, wherein the interactive user interface: responds to user input associated adjusting the robotic TEE tool through station coordinates, wherein individual degrees of freedom of the robotic TEE tool are separately adjusted in the interactive user interface, presents real-time images from the robotic TEE tool during adjustment, and causes storing, by the system, of the updated station coordinates.

15. The method of claim 1, wherein station coordinates reflect values assigned to a plurality of degrees of freedom associated with the robotic a TEE probe positioned at a terminal end of the TEE tool.

16. The method of claim 1, further comprising: triggering an updated cardiac view registration; and determining updated station coordinates associated with at least a subset of the cardiac views.

17. The method of claim 16, wherein triggering the updated cardiac view registration is based on determining movement of the patient and / or the TEE tool.

18. The method of claim 17, wherein determining movement is based on one or more force sensors or one or more accelerometers.

19. The method of claim 17, wherein determining movement is based on detecting movement in images from the TEE tool while the TEE tool is instructed to be at particular station coordinates.

20. The method of claim 19, wherein the detected movement reflects unexpected movement of image features within the images.

21. The method of claim 16, wherein determining updated station coordinates associated with a first cardiac view comprises: accessing prior images associated with station coordinates proximate to first station coordinates associated with the first cardiac view, the prior images being obtained during the cardiac view registration; comparing real-time images from the TEE tool with the prior images; and based on the comparisons, determining updated station coordinates associated with the first cardiac view.

22. The method of claim 21, wherein comparing comprises comparing real-time images associated with an end-diastolic phase with a subset of the prior images which are associated with the end-diastolic phase.

23. A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform the method of claims 1-22.

24. Non-transitory computer storage media storing instructions that when executed by one or more processors, cause the one or more processors to perform the method of claims 1-22.

25. A method implemented by a system of one or more processors, the system being in communication with a robotic transesophageal echocardiogram (TEE) tool configured for insertion into an esophagus of a patient, and the method comprising: controlling the TEE tool within the esophagus to be at a plurality of station coordinates, wherein the station coordinates reflect unique combinations of values of degrees of freedom of the TEE tool; analyzing, using a classifier and / or regression model, sensor information obtained from the TEE tool while at the station coordinates; and determining a subset of the station coordinates which are associated with respective cardiac views, wherein station coordinates associated with a cardiac view indicates that sensor information obtained from the TEE tool at the station coordinates depicts the cardiac view.

26. The method of claim 25, wherein the plurality of station coordinates are pre-stored and wherein the TEE tool is controlled to traverse to the station coordinates.

27. The method of claim 25, wherein the TEE tool includes a TEE probe positioned at a terminal end of the TEE tool, wherein individual station coordinates reflect values which correspond to a distance of the TEE probe within the esophagus, a rotation of the TEE probe, and an electronic rotation of an ultrasound sensor in the TEE probe.

28. The method of claim 27, wherein a value of the electronic rotation is set based on beamforming techniques.

29. The method of claim 25, wherein the classifier and / or regression model outputs confidence values indicative of the cardiac views.

30. The method of claim 25, wherein the sensor data includes ultrasound images.

31. The method of claim 30, wherein a plurality of ultrasound images are obtained at individual station coordinates, and wherein a subset of the ultrasound images which are end- diastolic are input into the classifier and / or regression model.

32. The method of claim 25, wherein the subset of station coordinates are included in a look-up table which associates individual station coordinates with individual cardiac views.

33. The method of claim 25, wherein images associated with the subset of station coordinates are configured for inclusion in an interactive user interface.

34. The method of claim 25, wherein the subset of the station coordinates are configured to be updated based on detection of movement of the patient or the TEE tool.

35. A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform the method of claims 25-34.

36. Non-transitory computer storage media storing instructions that when executed by one or more processors, cause the one or more processors to perform the method of claims37. A method implemented by a system of one or more processors, wherein the system is configured to present an interactive user interface, and wherein the interactive user interface: presents a plurality of images which depict respective cardiac views, wherein the cardiac views reflect views of the patient’s heart and the images are obtained from a robotic transesophageal echocardiogram (TEE) tool, and wherein confidence values associated with the images depicting the respective cardiac views are included; responds to user input associated with selection of a first cardiac view of the cardiac views, wherein the selection is indicative of a request (1) to cause navigation of the TEE tool to station coordinates associated with the first cardiac view or (2) to update the station coordinates associated with the first cardiac view, wherein station coordinates are indicative of, at least, distance and pose of the robotic TEE tool within an esophagus of the patient; and updates based on the selection.

38. The method of claim 37, wherein based on selection of the request to cause navigation of the TEE tool, the system controls the TEE tool to the station coordinates, and the interactive user input updates to present real-time images from the TEE tool.

39. The method of claim 38, wherein the TEE tool includes a TEE probe at a terminal end of the TEE tool, and wherein the TEE probe obtains the images.

40. The method of claim 37, wherein based on selection of the request to update the station coordinates, the interactive user interfaces: presents manual controls associated with adjusting the position and / or pose of the TEE tool; and presents real-time images during adjustment.

41. The method of claim 40, wherein the TEE tool includes a TEE probe at a terminal end of the TEE tool, and wherein the TEE probe obtains the images.

Images