Ultrasound imaging plane alignment guidance for neural networks and related devices, systems, and methods - Patents.com

Abstract

To provide a visual guidance or feedback which is generally related to ultrasound imaging, especially for aligning an imaging component with a desired imaging plane in such a situation that a current photographed image is given.SOLUTION: In one embodiment, a guidance system for obtaining a medical image includes a processor configured to obtain a motion control configuration for repositioning an imaging device from a first imaging position to a second imaging position with respect to a subject's body, the motion control configuration obtained based on a predictive network, an image of the subject's body captured while the imaging device is positioned at the first imaging position, and a target image view including a clinical property; and a display in communication with the processor and configured to display an instruction, based on the motion control configuration, for operating a control component in communication with the imaging device such that the imaging device is repositioned to the second imaging position.SELECTED DRAWING: None

Description

[Technical field]

[0001] Related Applications This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 641,540, filed March 12, 2018, which is incorporated by reference.

[0002]

[0001] The present disclosure relates generally to ultrasound imaging, and more particularly to providing visual guidance or feedback for aligning an imaging component with respect to a desired imaging plane given a currently captured image. [Background technology]

[0003]

[0002] Transesophageal echocardiography (TEE) uses ultrasound to capture images of the motion of a patient's heart and can be useful in clinical procedures. For example, a TEE probe is inserted into a patient's esophagus to capture images of the patient's heart. Some procedures that benefit from TEE include intraoperative procedures such as open heart and thoracic aortic surgery, guidance of transcatheter procedures such as transcatheter aortic valve implantation (TAVI), left atrial appendage occlusion (LAAC), transcatheter mitral valve repair (TMVR), and diagnostic procedures. Alternatively, transthoracic echocardiography (TTE) is an external imaging technique that can be used to capture various views of a patient's heart from outside the patient's body. For example, a TTE probe is placed in the patient's chest or abdomen to capture images of the patient's heart.

[0004]

[0003] In some cases, TEE is more effective than TTE in cardiac imaging. For example, the left atrial appendage (LAA) is not visible from TTE, but is visible from TEE. Recent studies have shown that LAA isolation is a treatment option for patients with atrial fibrillation. LAA isolation can reduce the risk of embolic events. LAA isolation is typically performed percutaneously using a LAAC device. Before performing an LAA isolation procedure on a patient, a clinician uses a TEE probe to image the patient's heart. Based on the image, the clinician determines the size and / or length of the LAA opening in the patient's heart and whether the size and / or length of the LAA opening is within the dimensions of the LAAC device. Given the complexity and anatomical variability of the LAA, such as the presence and number of lobes, imaging and depiction of the LAA from multiple views are typically required for a complete evaluation of the morphology. Summary of the Invention [Problem to be solved by the invention]

[0005]

[0004] However, it is difficult for clinicians to find the optimal view in the moving heart. For example, the TEE probe can be advanced or retracted in the patient's esophagus, rotated clockwise or counterclockwise, and bent in four different directions, and the imaging plane can be rotated from 0 degrees to 180 degrees. Due to the multiple degrees of freedom, manual adjustment of the TEE probe can be cumbersome and time-consuming. The TTE probe is moved along a Cartesian plane, so it is similarly difficult to locate the optimal imaging plane.

[0006]

[0005] Although existing procedures for using TEE or TTE probes have proven useful for clinical or therapeutic procedures, there remains a clinical need for improved systems and techniques to provide efficient, accurate, and automated procedures for aligning an imaging component to a desired imaging plane. One way to automate the alignment of an imaging component without relying on a similarity measure to a reference image is to use a statistical predictive network. The disclosed embodiments provide a mechanism for guiding a sonographer to align or reposition an imaging component to an optimal location to obtain a desired imaging view based on a currently captured image. The disclosed embodiments display guidance or motion on a display. The guidance provides instructions for the movement and / or control of the imaging component for alignment to a current position of the imaging component used to capture the current image. The guidance can be in the form of a graphical representation of the imaging probe or a control handle of the probe with corresponding controls (e.g., for bending and manipulating the probe). The graphical representation can indicate the direction and / or the degree or amount of motion to apply to the controls. The display of the graphical representation (e.g., different views of the control handle) can be controlled by the user (e.g., via a user input interface). The disclosed embodiments may provide guidance using deep convolutional networks to predict movement and / or control for alignment or repositioning. [Means for solving the problem]

[0007]

[0006] In one embodiment, a medical ultrasound imaging view guidance system is provided. The medical ultrasound imaging view guidance system includes a processor configured to obtain a motion control setting for repositioning an imaging device from a first imaging position to a second imaging position relative to a body of a subject, the motion control setting being obtained based on a predictive network, an image of the body of the subject is taken when the imaging device is disposed at the first imaging position, and a target image view includes a clinical characteristic; and a display in communication with the processor and configured to display instructions to operate a control component in communication with the imaging device based on the motion control setting so that the imaging device is repositioned to the second imaging position.

[0008]

[0007] In some embodiments, the display is further configured to display instructions by displaying a graphical view of the control component and a visual indicator indicating at least one of a direction of motion or an amount of the motion to operate the control component. In some embodiments, the graphical view includes a perspective view of the control component. In some embodiments, the medical ultrasound imaging view guidance system further comprises a communication device configured to communicate with the processor and receive a request for a first view of the plurality of views of the control component, the processor further configured to switch the graphical view from a current view of the plurality of views to the first view based on the request. In some embodiments, the plurality of views includes at least one of a forward view of the control component, a lateral view of the control component, or a rearward view of the control component. In some embodiments, the control component includes at least one of a first subcomponent for controlling a motion of the imaging device along a left-right plane of the subject's body, a second subcomponent for controlling a motion of the imaging device along an anterior-posterior plane of the subject's body, or a third subcomponent for controlling an orientation of an imaging plane of the imaging device. In some embodiments, the motion control settings include at least one of a first parameter for operating the first subcomponent, a second parameter for operating the second subcomponent, a third parameter for operating the third subcomponent, or a fourth parameter for rotating the imaging device relative to an axis of the imaging device. In some embodiments, the display is further configured to display instructions by displaying a visual indicator including at least one of an arrow or an on / off indicator based on at least one of the first parameter, the second parameter, the third parameter, or the fourth parameter. In some embodiments, the imaging device is a transesophageal echocardiography (TEE) probe. In some embodiments, the control component is located in a handle of the TEE probe.In some embodiments, the predictive network includes at least one convolutional neural network (CNN). In some embodiments, the medical ultrasound imaging view guidance system further comprises a communication device in communication with the processor and configured to receive the image, and the processor is further configured to obtain the motion control setting by applying the predictive network to the image and a plurality of parameters for operating the control component. In some embodiments, the communication device is further configured to receive the plurality of parameters.

[0009]

[0008] In one embodiment, a method for providing medical ultrasound imaging guidance is provided, comprising the steps of: obtaining a motion control setting for repositioning an imaging device from a first imaging position to a second imaging position relative to a body of a subject, the motion control setting being obtained based on a predictive network, an image of the body of the subject being captured when the imaging device is disposed at the first imaging position, and a target image view including a clinical characteristic; and displaying instructions on a display for operating a control component in communication with the imaging device to reposition the imaging device to the second imaging position based on the motion control setting.

[0010]

[0009] In some embodiments, the displaying step includes displaying a graphical view of the control component and a visual indicator indicating at least one of a direction of motion to operate the control component or an amount of the motion. In one embodiment, the method further includes receiving a request for a first view of the plurality of views of the control component; and switching the graphical view from a current view of the plurality of views to the first view based on the request. In some embodiments, the plurality of views includes at least one of a forward view of the control component, a side view of the control component, or a rearward view of the control component. In some embodiments, the control component includes at least one of a first subcomponent for controlling a motion of the imaging device along a left-right plane of the subject's body, a second subcomponent for controlling a motion of the imaging device along a front-back plane of the subject's body, or a third subcomponent for controlling an orientation of an imaging plane of the imaging device. In some embodiments, the motion control settings include at least one of a first parameter for operating the first subcomponent, a second parameter for operating the second subcomponent, a third parameter for operating the third subcomponent, or a fourth parameter for rotating the imaging device relative to an axis of the imaging device. In some embodiments, the displaying includes displaying a visual indicator including at least one of an arrow or an on / off indicator based on at least one of the first parameter, the second parameter, or the third parameter.

[0011]

[0010] Additional aspects, features, and advantages of the present disclosure will become apparent from the following detailed description.

[0012]

[0011] Exemplary embodiments of the present disclosure are described with reference to the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1]

[0012] FIG. 1 is a schematic diagram of a medical imaging system including a transesophageal echocardiography (TEE) probe according to an aspect of the present disclosure. [Figure 2A]

[0013] FIG. 1 is a schematic diagram showing a configuration for a TEE probe according to an aspect of the present disclosure. [Figure 2B]

[0014] FIG. 1 is a schematic diagram showing a configuration for a TEE probe, according to an embodiment of the present disclosure. [Figure 2C]

[0015] FIG. 1 is a schematic diagram showing a configuration for a TEE probe, according to an embodiment of the present disclosure. [Figure 2D]

[0016] FIG. 1 is a schematic diagram showing a configuration for a TEE probe, according to an embodiment of the present disclosure. [Diagram 3]

[0017] 1 is a schematic diagram of a medical imaging system including a transthoracic echocardiography (TTE) probe, according to an aspect of the present disclosure. [Figure 4]

[0018] 1 is a schematic diagram illustrating an approach for automating an imaging component alignment procedure, according to aspects of the present disclosure. [Diagram 5]

[0019] FIG. 1 is a schematic diagram illustrating an approach for acquiring a large image dataset, according to an aspect of the present disclosure. [Figure 6]

[0020] 1A-1C are schematic diagrams illustrating scenarios for repositioning a TEE probe, according to an embodiment of the present disclosure. [Figure 7]

[0021] 1A-1C are schematic diagrams illustrating scenarios for repositioning a TTE probe, according to an embodiment of the present disclosure. [Figure 8A]

[0022] 1 illustrates an example of a target image view of the LAA, according to an embodiment of the present disclosure. [Figure 8B]

[0023] 1 illustrates an example of an imaging plane for imaging the LAA, according to an embodiment of the present disclosure. [Figure 8C]

[0024] 1 illustrates an example image of the LAA taken by an imaging component, according to an embodiment of the present disclosure. [Figure 9]

[0025] FIG. 1 is a schematic diagram illustrating a configuration for a predictive convolutional neural network (CNN), according to an aspect of the present disclosure. [Figure 10]

[0026] FIG. 1 is a schematic diagram illustrating a configuration for fine-tuning CNN according to an embodiment of the present disclosure. [Figure 11]

[0027] FIG. 1 is a schematic diagram illustrating a configuration for a target CNN, according to an embodiment of the present disclosure. [Figure 12]

[0028] FIG. 1 is a schematic diagram illustrating an approach for training a predictive CNN according to an embodiment of the present disclosure. [Figure 13]

[0029] FIG. 1 is a schematic diagram illustrating an approach for training a fine-tuned CNN according to an embodiment of the present disclosure. [Figure 14]

[0030] FIG. 1 is a schematic diagram illustrating an approach for training a target CNN according to an embodiment of the present disclosure. [Figure 15]

[0031] 1 is a schematic diagram illustrating an approach for aligning an imaging component to a desired location, according to aspects of the present disclosure. [Figure 16]

[0032] 1A-1C are schematic diagrams illustrating an approach for providing guidance to align an imaging component with respect to a desired location, according to aspects of the present disclosure. [Figure 17A]

[0033] 1A-1C are schematic diagrams illustrating imaging component alignment guidance display views, according to aspects of the present disclosure. [Figure 17B]

[0034] 1A-1C are schematic diagrams illustrating imaging component alignment guidance display views, according to aspects of the present disclosure. [Figure 17C]

[0035] 1A-1C are schematic diagrams illustrating imaging component alignment guidance display views, according to aspects of the present disclosure. [Figure 18]

[0036] FIG. 1 is a flow diagram of a method for acquiring an imaging dataset for training a CNN, according to an embodiment of the present disclosure. [Figure 19]

[0037] FIG. 1 is a flow diagram of a method for training a CNN according to an embodiment of the present disclosure. [Figure 20]

[0038] FIG. 13 is a flow diagram of a method for applying a CNN to align an imaging component to a desired location, according to an aspect of the present disclosure. [Figure 21]

[0039] FIG. 1 is a flow diagram of a method for providing CNN-based imaging component alignment guidance according to an aspect of the present disclosure. [Figure 22]

[0040] FIG. 1 is a flow diagram of a method for automating medical testing according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014]

[0041] For the purpose of promoting understanding of the principles of the present disclosure, reference will be made to the embodiments shown in the drawings, and specific terms will be used to describe them. Nevertheless, it will be understood that no limitations on the scope of the present disclosure are intended. Any changes and further modifications to the described devices, systems and methods, and any further applications of the principles of the present disclosure, as would normally occur to one skilled in the art to which the present disclosure pertains, are fully contemplated and included in the present disclosure. In particular, it is fully contemplated that features, components and / or steps described with respect to one embodiment may be combined with features, components and / or steps described with respect to other embodiments of the present disclosure. However, for the sake of brevity, many iterations of these combinations will not be described separately.

[0015]

[0042] FIG. 1 is a schematic diagram of a medical imaging system 100 including a TEE probe 110 according to an embodiment of the present disclosure. The system 100 includes a host 130, a robotic system 120, and a TEE probe 110, which are in communication with each other. At a high level, the TEE probe 110 may be inserted through the mouth of the patient 102 into the esophagus to capture images of the patient's 102 heart, and the host 130 may instruct the robotic system 120 to reposition the TEE probe 110 to a desired location. For example, in a LAAC procedure, the host 130 instructs the robotic system 120 to position the TEE probe 110 such that images including the LAA of the heart are captured by the TEE probe 110. The system 100 may be an ultrasound imaging system, and the probe 110 may be an ultrasound probe.

[0016]

[0043] The TEE probe 110 includes a flexible elongate member 112, an imaging component 114, and a handle 116. The flexible elongate member 112 may be sized and / or shaped, structurally arranged, and / or otherwise configured to be placed within a body lumen of a patient, such as the esophagus. The member 112 is referred to as an endoscope. The imaging component 114 is attached to a distal end of the member 112. In some cases, the probe 110 may be a guidewire, catheter, or guide catheter. The imaging component 114 includes one or more ultrasonic sensor or transducer elements. The imaging component 114 is configured to emit ultrasonic energy toward an anatomical structure (e.g., the heart) of the patient 102. The ultrasonic energy is reflected by the vasculature and / or tissue structures of the patient. The ultrasonic transducer elements in the imaging component 114 receive the reflected ultrasonic echo signals. In some embodiments, the TEE probe 110 includes internal or integral processing components that can locally process the ultrasound echo signals to generate image signals representative of the anatomy of the patient 102 being imaged. The ultrasound transducer elements can be configured to provide two-dimensional (2D) or three-dimensional (3D) images of the anatomy of the patient 102. As described in more detail herein, the images acquired by the TEE probe 110 depend on the depth of insertion, as well as the rotation and / or tilt of the TEE probe 110.

[0017]

[0044] A handle 116 is coupled to a proximal end of the member 112. The handle 116 includes control elements for manipulating the imaging component 114 and / or the member 112. As shown, the handle 116 includes knobs 117 and 118 and a switch 119. The knob 117 bends the member 112 and the imaging component 114 along an anterior-posterior plane of the patient 102 (e.g., the heart). The knob 118 bends the member 112 and the imaging component 114 along a left-right plane of the patient 102. The switch 119 controls beam forming in the imaging component 114 (e.g., adjusting the angle of the imaging plane).

[0018]

[0045] The robotic system 120 includes electrical and / or mechanical components, such as motors, rollers, gears, etc., configured to control the handle 116 (e.g., to rotate the dials of the knobs 117 and 118 and / or to turn the switch 119 on and / or off). Alternatively, the robotic system 120 is configured to directly steer the TEE probe 110. The robotic system 120 receives motion control commands from the host 130 and controls the knobs 117 and 118 and / or the switch 119 on the handle 116 and / or directly drives the TEE probe 110 based on the motion control commands. The motion control commands are described in more detail herein.

[0019]

[0046] The host 130 includes a memory 132, a display 134, a processing component 136, and a communication interface 138. The processing component 136 is coupled to and in communication with the memory 132, the display 134, and the communication interface 138. The host 130 may be a computer workstation, a mobile phone, a tablet, or any suitable computing device.

[0020]

[0047] The memory 132 is any suitable data storage device, such as a cache memory (e.g., a cache memory of the processing component 136), a random access memory (RAM), a magnetoresistive RAM (MRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, a solid state memory device, a hard disk drive, a solid state drive, other forms of volatile or non-volatile memory, or a combination of different types of memory. The memory 132 is configured to store the database 140 and one or more CNNs 142.

[0021]

[0048] Processing component 136 includes a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a controller, a field programmable gate array (FPGA) device, another hardware device, a firmware device, or any combination thereof configured to perform the operations described herein. Processing component 136 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0022]

[0049] In an embodiment, the processing component 136 is configured to acquire images from a patient, such as the patient 102, or an anatomical model (e.g., a cardiac model), generate motion control parameters for controlling the TEE probe 110, determine labels or scores for qualifying or validating the acquired images and motion control parameters against target image views that include specific or selected clinical characteristics, and / or generate a database 140. The database 140 stores the acquired images in association with the motion control parameters and scores.

[0023]

[0050] In an embodiment, the processing component 136 is configured to train the CNN 142 to align the imaging component 114 with respect to a target imaging plane based on the database 140. In an embodiment, the processing component 136 is configured to apply the CNN 142 in a clinical environment to determine motion control commands for the robotic system 120 to align the imaging component 114 with respect to a patient, such as the patient 102, for a clinical procedure. For example, the imaging component 114 is aligned to acquire images of the LAA of the patient 102 for a LAAC procedure. Mechanisms for acquiring the datasets for the database 140, training the CNN 142, and applying the CNN 142 are described in more detail herein.

[0024]

[0051] In some embodiments, memory 132 includes a non-transitory computer-readable medium. Memory 132 stores instructions that, when executed by processing component 136, cause processing component 136 to perform operations described herein with reference to data acquisition, CNN training, and / or CNN application in connection with embodiments of the present disclosure. Instructions are also referred to as code. The terms "instructions" and "code" should be interpreted broadly to include any type of computer-readable instruction. For example, the terms "instructions" and "code" refer to one or more programs, routines, subroutines, functions, procedures, etc. "Instructions" and "code" include a single computer-readable instruction.

[0025]

[0052] The display 134 includes a computer screen or any suitable display for displaying a user interface (UI) 144. The UI 144 includes a graphical representation or view of the handle 116, including, for example, the knobs 117 and 118 and the switch 119. The UI 144 includes visual indicators indicating the direction and / or amount to turn the knobs 117 and 118, instructions to turn the switch 119 on or off, and / or the direction and / or degree to rotate the TEE probe 110, as described in more detail herein. Although the display 134 is illustrated as an integral component of the host 130, in some embodiments the display 134 may be external to the host 130 and communicate with the host 130 via the communication interface 138. For example, the display 134 may include a standalone display, augmented reality glasses, or a mobile phone.

[0026]

[0053] The communication interface 138 includes one or more transmitters, one or more receivers, one or more transceivers, and / or circuitry for transmitting and / or receiving communication signals. The communication interface 138 may include hardware and / or software components implementing a particular communication protocol suitable for carrying signals over the communication links 150 and 152. The communication interface 138 may be referred to as a communication device or a communication interface module. The communication interface 138 is configured to interface and communicate with the robotic system 120 and the imaging component 114 via the communication links 150 and 152, respectively. For example, the host 130 sends motion control commands to the robotic system 120 via the communication link 150 and receives acquired images from the imaging component 114 via the communication link 152. The communication links 150 and 152 include wireless and / or wired links. Examples of wireless links include a low power Bluetooth® wireless link, an Institute of Electrical and Electronics Engineers (IEEE) 802.11 (WiFi) link, or any suitable wireless link, etc. Examples of wired links include a Universal Serial Bus (USB) link, or any suitable wired link, etc.

[0027]

[0054] In some embodiments, the UI 144 displays instructions for manipulating the handle 116 to align the TEE probe 110 with respect to a desired location. For example, the processing component 136 outputs motion control commands in the form of instructions and / or visual indicators to the display 134 via the UI 144, and the clinician manually aligns the TEE probe 110 with respect to the patient 102 based on the instructions instead of using the robotic system 120. The communication interface 138 is further configured to receive user input, for example, via a keyboard, mouse, or touch screen. The UI 144 updates a particular display or view based on the user input. The UI 144 is described in more detail herein.

[0028]

[0055] Although the system 100 is shown with a TEE probe 110, the system 100 is configured to automatically align any suitable imaging component with respect to a patient for a clinical procedure. The imaging component provides an internal imaging modality, where the imaging component is inserted into the patient's body to obtain images of the patient's anatomy. Alternatively, the imaging component may provide an external imaging modality, where the imaging component is placed outside the patient's body to obtain images of the patient's anatomy. The imaging modalities include intravascular ultrasound (IVUS) imaging, forward-looking intravascular ultrasound (FL-IVUS) imaging, intravascular photoacoustic (IVPA) imaging, intracardiac echocardiography (ICE), optical imaging, optical coherence tomography (OCT), radiographic imaging, x-ray imaging, angiography, fluoroscopy, computed tomography (CT), magnetic resonance imaging (MRI), elastography, and the like.

[0029]

[0056] In some other embodiments, the system 100 includes any suitable sensing components, such as pressure sensors, flow sensors, temperature sensors, optical fibers, reflectors, mirrors, prisms, ablation elements, radio frequency (RF) electrodes, conductors, and / or combinations thereof to perform a clinical or therapeutic procedure, and images of the patient's anatomy undergoing the procedure are taken by the imaging component 114 before, during, and / or after the procedure.

[0030]

[0057] The TEE probe 110 is manipulated in various degrees of freedom. FIGS. 2A-2D show various mechanisms for manipulating the TEE probe 110. FIG. 2A is a schematic diagram showing a configuration 210 for the TEE probe 110, according to an embodiment of the present disclosure. The TEE probe 110 can be manually advanced into the patient's esophagus, as shown by arrow 212, or withdrawn from the patient's esophagus, as shown by arrow 214. The TEE probe 110 can be manually rotated leftward (e.g., counterclockwise) or rightward (e.g., clockwise) relative to the longitudinal axis 202 of the TEE probe 110, as shown by arrows 216 and 218, respectively. The rotation of the member 112 can be represented by a parameter shown as γ.

[0031]

[0058] 2B is a schematic diagram illustrating a configuration 220 for a TEE probe 110, according to an embodiment of the disclosure. The TEE probe 110 can be electronically rotated, for example, by controlling switch 119 on handle 116, from 0 degrees to 180 degrees (e.g., for beamforming) as indicated by arrows 222 and 224. The rotation of the imaging plane can be represented by a parameter shown as ω.

[0032]

[0059] 2C is a schematic diagram illustrating a configuration 230 for the TEE probe 110, according to an embodiment of the disclosure. The TEE probe 110 can be bent, for example, along an anterior-posterior plane relative to the patient's heart, by, for example, rotating a dial on the knob 118 of the handle 116, as indicated by arrows 232 and 234. Bending along the anterior-posterior plane can be represented by a parameter denoted as α.

[0033]

[0060] 2D is a schematic diagram illustrating a configuration 240 for a TEE probe 110, according to an embodiment of the disclosure. The TEE probe 110 can be bent, for example, along a left-right plane relative to the patient's heart, by, for example, rotating a dial on the knob 117 of the handle 116, as indicated by arrows 242 and 244. Bending along the left-right plane can be represented by a parameter denoted as β.

[0034]

[0061] 3 is a schematic diagram of a medical imaging system 300 including a TTE probe 310 according to an embodiment of the present disclosure. The system 300 is substantially similar to the system 100, except that the system 300 uses an external imaging component (e.g., the TTE probe 310) to align with a target imaging plane instead of an internal imaging component (e.g., the TEE probe 110). As shown, the system 300 includes a host 130, a robotic system 320, and a TTE probe 310, which are in communication with each other. At a high level, the TTE probe 310 is configured to capture images of the patient's 102 anatomy from outside the patient's 102 body, and the host 130 can instruct the robotic system 320 via the link 150 to reposition the TTE probe 310 to a desired location. The system 300 may be an ultrasound imaging system, and the probe 310 may be an ultrasound probe.

[0035]

[0062] The TTE probe 310 includes an imaging component that operates by a mechanism substantially similar to the imaging component 114. For example, the imaging component includes ultrasound transducer elements configured to capture 2D or 3D images of the anatomy (e.g., the heart) of the patient 102 by external imaging. Similarly, the TTE probe 310 transmits acquired image signals to the host 130 via the link 152.

[0036]

[0063] The robotic system 320 includes a number of links 322 coupled to a number of joints 324 configured to hold the TTE probe 310 and manipulate the TTE probe 310 over an external surface of the patient 102 (e.g., around the chest area for imaging the heart). The processing component 136 generates the database 140 and the CNN 142 using mechanisms substantially similar to those in the system 100, but as described in more detail herein, the motion control commands are in the format of velocities in Cartesian coordinates instead of rotations and bending of the TEE probe 110.

[0037]

[0064] In general, system 100, system 300, probe 110, probe 310, and / or other devices described herein may be utilized to inspect any suitable anatomical structure and / or lumen of a patient's body 102. In some cases, probe 110 may be positioned within an anatomical structure and / or lumen of a patient's body 102. In other cases, probe 110 may be positioned outside of body 102 to inspect an anatomical structure and / or lumen inside body 102. Anatomical structures and / or lumens refer to structures filled with or surrounded by fluid, both natural and artificial. For example, a probe of the present disclosure may be positioned within and / or used to inspect a patient's esophagus. In some embodiments, the probes of the present disclosure are used to inspect any number of anatomical locations and tissue types, such as, but not limited to, organs, such as the liver, heart, kidneys, gallbladder, pancreas, lungs, ducts, intestines, nervous system structures, such as the brain, dural sac, spinal cord, peripheral nerves, urinary tract, and valves in the blood, ventricles, or other parts of the heart, and / or other systems of the body. The anatomical structures and / or lumens in the body 102 may be blood vessels, such as arteries or veins, of the patient's vascular system, such as the cardiovascular system, peripheral vascular system, neurovascular system, renal vascular system, and / or any other suitable lumen in the body. In addition to natural structures, the probes of the present disclosure are used to inspect artificial structures, such as, but not limited to, heart valves, stents, shunts, filters, and other devices.

[0038]

[0065] 4 is a schematic diagram illustrating a method 400 for automating an imaging component alignment procedure according to an embodiment of the present disclosure. Method 400 is performed by host 130 in systems 100 and 300. Method 400 includes a data acquisition phase 402, a CNN training phase 404, and a clinical operation phase 406. Data acquisition phase 402 and CNN training phase 404 are performed offline, and clinical operation phase 406 is performed in a clinical environment.

[0039]

[0066] In the data acquisition phase 402, the data acquisition component 410 is configured to acquire image data using an imaging component (e.g., TEE probe 110 or TTE probe 310), generate motion control parameters for maneuvering the imaging component (e.g., using a robotic system 120 or 320) to various locations relative to a subject's body (e.g., patient 102 or anatomical model), and associate the locations of the imaging component and the corresponding images with target image views to generate the database 140. The data acquisition component 410 automatically annotates or labels the acquired images based on the relationship between the corresponding positions of the imaging component and the target views. The data acquisition component 410 repeatedly or periodically acquires data and updates the database 140. Mechanisms for acquiring data and generating the database 140 are described in more detail herein.

[0040]

[0067] In the CNN training phase 404, the CNN training component 420 is configured to train the CNN 142 based on the database 140. The CNN 142 includes a predictive CNN 142a, a fine-tuning CNN 142b, and a target CNN 142c. The predictive CNN 142a is trained to receive a currently acquired image from the imaging component and to infer or estimate a motion vector (e.g., including motion control parameters) that is most likely to cause the imaging component to reach a desired location to capture a target image view. For example, the predictive CNN 142a is trained for a target image view that includes a specific or predetermined clinical characteristic (e.g., LAA). The fine-tuning CNN 142b is trained to verify whether a pair of images have the same quality level or to select an image from a pair of images that has a higher quality level. The target CNN 142c is trained to determine whether a target image view (e.g., LAA) has been captured. In some embodiments, the CNN training component 420 trains the fine-tuning CNN 142b and / or the target CNN 142c using additional clinical data. The mechanisms for training the CNN 142 are described in more detail herein.

[0041]

[0068] In the clinical operation phase 406, the CNN application component 430 applies the trained CNN 142 to determine motion vectors for repositioning or aligning the imaging component to the patient's anatomical structure to obtain the target image view. The CNN application component 430 uses a closed-loop control algorithm for the alignment. For example, the CNN application component 430 applies the predictive CNN 142a to orient the imaging component and applies the target CNN 142c to check whether the imaging component is at the desired location. The CNN application component 430 repeatedly applies the predictive CNN 142a and the target CNN 142c until the target CNN 142c detects the target image view. Then, the CNN application component 430 applies the fine-tuning CNN 142b to orient the imaging component to the optimal location for capturing the target image view. The CNN application component 430 commands the robotic system to reposition or align the imaging component in the alignment process. Alternatively, the CNN application component 430 provides the motion vectors to the motion guidance component 432, for example, for display in the UI 144. The mechanism for applying CNN 142 is described in more detail herein.

[0042]

[0069] 5 is a schematic diagram illustrating a method 500 for acquiring a large image data set according to an aspect of the disclosure. Method 500 is performed by systems 100 and 300 and data acquisition component 410. Method 500 provides a more detailed view of the automated data acquisition process in data acquisition phase 402. Method 500 includes a motion generation component 510, a robotic system 520, a qualification component 530, a data association component 540, a database generation component 550, a database storage component 560, and an imaging device 570. In an embodiment, imaging device 570 corresponds to TEE probe 110 and robotic system 520 corresponds to robotic system 120. In another embodiment, imaging device 570 corresponds to TTE probe 310 and robotic system 520 corresponds to robotic system 330.

[0043]

[0070] The motion generation component 510 is configured to generate a set of imaging plane positions (e.g., for 2D imaging) and encode the imaging plane positions into motion control settings that are used to reposition the imaging device 570. The motion control settings include m t The motion control settings are represented by a motion vector shown as . The motion control settings include different parameters depending on the imaging component type in use. For example, when the imaging device 570 is a TEE probe 110, the motion control settings include parameters γ, ω, α, and β, as shown and described with respect to FIGS. 2A, 2B, 2C, and 2D, respectively. Alternatively, when the imaging device 570 is a TTE probe 310, the motion control settings are represented by linear velocity parameters and angular velocity parameters. In some embodiments, the motion generation component 510 uses a probability function to generate parameters for the motion control settings. The motion generation component 510 provides the motion control settings to the robotic system 520 and the data association component 540.

[0044]

[0071] In some embodiments, the motion control settings may include controls for operating the imaging device 570 to change the imaging plane within a volumetric ultrasound image in addition to changing the physical location of the imaging device 570. For example, the imaging plane may be tilted in elevation or at any suitable angle relative to the axis of the volumetric region during volumetric imaging.

[0045]

[0072] In general, a motion control setting may include any measurable data related to a specific position or specific movement of a device provided by a user or robot. Various motion control settings are described throughout this application, and the methods / systems herein may include any one or combination of these motion control settings. In some cases, a motion control setting includes one or more parameters. The parameters include a direction or velocity vector, and / or an imaging plane position.

[0046]

[0073] The robotic system 520 repositions the imaging device 570 based on the motion control settings. The imaging device 570 captures images of the subject's body 580 (e.g., a heart model) and t to the qualification component 530 and the data association component 540.

[0047]

[0074] The qualification component 530 determines whether the acquired image includes a particular clinical feature (e.g., LAA) that has meaning for a particular clinical procedure (e.g., LAAC procedure). By way of example, the method 500 may include a U target The qualification component 530 is configured to generate a database 140 for capturing target image views, which are shown as l t For example, when the selected motion control settings position the imaging device 570 in a location where the captured image successfully captures the target image view, the qualification component 530 outputs a score or label denoted as l tConversely, when a captured image fails to capture the target image view, the qualification component 530 outputs a score l t The eligibility verification component 530 outputs 0 as the value of score l. t to the data association component 540.

[0048]

[0075] In an embodiment, the data association component 540 t The image captured by the imaging device 570 at the current position shown as U t The robotic system 520 receives the current image, denoted as m t Based on this, the imaging device 570 is end This can be expressed as follows: m t =q end -q t (1) The qualification component 530 determines whether the imaging device 570 is at position q end and receiving a next image of the subject's body 580 captured by the imaging device 570 when the subject is repositioned at U t+1 The eligibility verification component 530 is shown as t+1 The target image view is target Image U based on whether it contains t+1 Generate a score for the movement command l t Alternatively, when the subject's body 580 is a phantom, the target positions for taking the target image views are known. Thus, the qualification component 530 determines the position q end score by comparing with the target position t Determine.

[0049]

[0076] The data association component 540 associates the current image U t Let m be the motion control vector t and score t In relation to d tThis can be expressed as follows: d t =(U t ,m t ,l t ) (2) The method 500 performs a sequence of K steps in each trial or attempt to reach a target image view, where K is a positive integer. n This can be represented as follows: S n ={d1,d2,...,d K} (3) where d t is a sequence S n The data association component 540 represents a data tuple at a particular step t in the sequence S n to the database generation component 550.

[0050]

[0077] Sequence S n can be terminated by various conditions. For example, the sequence S n is an image Ut+1 and the target image view U target The method terminates when the similarity measure between sequence S is large. Various mechanisms can be used to determine the similarity measure between two unimodal images. For example, normalized cross-correlation, sum of squared differences (SSD), template matching, and / or Siamese convolutional neural networks can be used to determine the similarity measure. Alternatively, the method can be used to determine the similarity measure between sequence S n Step d t Position for q end The sequence S nThe process also terminates when the number of trials or steps (e.g., K) exceeds a threshold without reaching the target image view, or after exhausting all motion control parameters, e.g., when a hardware limit for a particular motion is reached, or when a software limit is reached. An example of a hardware limit is a sweep α, β, or γ for bending and rotation of the TEE probe 110, respectively. An example of a software limit is a sweep ω for changing the imaging plane through all angles from 0 degrees to 180 degrees. In some cases, software limits are arbitrarily defined by the robotic system 120 for each degree of freedom α, β, γ, or ω.

[0051]

[0078] The database generation component 550 generates a sequence S obtained from the N trials. n , generate a database 140 by forming a relational data set, denoted as D, which can be expressed as follows: D = {S1, S2, ..., S N} (4) Here, N is a positive integer.

[0052]

[0079] The database storage component 560 stores the database 140 or the data set D in a memory, such as the memory 132. In some embodiments, the procedure 500 is performed using a phantom and then repeated using a clinical environment (e.g., taking images of the patient 102). In such embodiments, the repositioning of the imaging device 570 is performed manually by a user or automatically by the robotic system 520. When the repositioning of the imaging device 570 is performed manually, the motion is sensed by the motion sensing component, which records the motion control settings based on the sensed motion. In addition, image / position qualification is performed automatically by the qualification component 530 or manually by a user. In some cases, the UI 144 may provide a user with a function for qualifying the acquired images or imaging plane positions, or for the sequence S. nThis includes various settings to demonstrate the final steps in

[0053]

[0080] In an embodiment, ultrasound imaging parameters such as gain and depth are adjusted for acquisition. In an embodiment, a clinician or user selects a target image view U from a set of clinically relevant views recommended by the clinical community for a particular clinical procedure. target In an embodiment, the clinician selects a target image view U by an extensive search through the image volume. target In another embodiment, the clinician acquires several target image views having different imaging parameters or positions for acquisition.

[0054]

[0081] In an embodiment, after the method 500 generates a sufficient number of motion control settings to acquire an image data set for a portion of the imaging volume, the clinician repositions the imaging device 570 to a different location and repeats the method 500 to acquire a different portion of the imaging volume. In an embodiment, to increase the diversity of the acquired imaging data set, the method 500 is repeated using several other subject's bodies 580 (e.g., multiple heart models). In an embodiment, to increase the diversity of the imaging data set, the method 500 is repeated using several imaging devices 570 (e.g., multiple TEE probes 110 or multiple TTE probes 310). The number of imaging plane positions or motion control settings and the number of iterations are arbitrarily selected by the clinician. To support deep learning neural network training, the size of the imaging data set needs to be relatively large.

[0055]

[0082] Although the method 500 is shown for generating the database 140 for a particular target view, the method 500 may be repeated for different target image views, with the method 500 storing the data sets for the different target image views in the same database or in different databases.

[0056]

[0083] 6 is a schematic diagram illustrating a scenario 600 for repositioning a TEE probe, according to an aspect of the disclosure. Scenario 600 corresponds to repositioning imaging device 570 when imaging device 570 corresponds to TEE probe 110. As shown, TEE probe 110 includes an adjustable member 610 that bends or adjusts the positioning of member 112 and imaging component 114. Imaging component 114 can be adjusted along a series of positions, e.g., from a current position q t to the next position q t+1 and end position q end The movement of the TEE probe 110 ends at position q t+1 , the imaging component 114 captures an image 620 (e.g., t+1 ) is captured. The motion control vector m t can be expressed as follows: m t =q t+1 -q t =(α,β,γ,ω) T (5) Here, T is the transposition operator.

[0057]

[0084] 7 is a schematic diagram illustrating a scenario 700 for repositioning a TTE probe, according to an embodiment of the disclosure. The scenario 700 corresponds to repositioning an imaging device 570 when the imaging device 570 corresponds to a TTE probe 310. As shown, the TTE probe 310 is repositioned along a series of positions, e.g., t From the current position represented by p t+1 is moved to the next position represented by p end The movement ends at the end position represented by p t , p t+1 , and p end is a 4-by-4 transformation matrix (e.g., including translation and rotation). Hence, the motion control vector m tcan be represented by a 3D translation vector (e.g., linear velocity) and the change in orientation (e.g., angular velocity) about each axis (e.g., in xyz space) from the current position to the end position. t+1 When in the , the TTE probe 310 captures an image 710 (e.g., t+1 ) is captured. The motion control vector m t can be expressed as follows: m t =p t+1 -p t =(ν xt ,ν yt ,ν zt ,ω xt ,ω yt ,ω zt ) T (6) Here, v xt , v yt , and v zt represent the linear velocities along the x, y, and z axes, respectively, and ω xt , ω yt , and ω zt represent angular velocities about the x, y, and z axes, respectively. When applying the technique 500 using the TTE probe 310, the user optionally determines boundary or end conditions that limit where the TTE probe 310 is repositioned (e.g., the boundaries around the patient's chest area when imaging the patient's heart).

[0058]

[0085] 8A-8C show various example images or image views corresponding to images used in the method 500. FIG. 8A shows an example of a target image view 810 of an LAA 812, according to an embodiment of the present disclosure. For example, the target image view 810 is a U target8B illustrates an example of a position of an imaging plane 820 for imaging an LAA in a patient's heart 822, according to an embodiment of the present disclosure. For example, technique 500 may be used to reposition imaging device 570 to capture an image at imaging plane 820. FIG. 8C illustrates an example of a target image 830 of an LAA 832 captured by an imaging component, such as imaging device 570 or TEE probe 110, at imaging plane 820, according to an embodiment of the present disclosure. For example, target image 830 may be U target and LAA832 corresponds to the desired clinical properties.

[0059]

[0086] 9 to 11 show the internal configuration of the CNN 142. FIG. 9 is a schematic diagram showing a configuration 900 for a predictive CNN 142a according to an embodiment of the present disclosure. The predictive CNN 142a predicts that candidate motion control settings to be used for repositioning the imaging components (e.g., the TEE probe 110, the TTE probe 310, and the imaging device 570) when an input image 902 is given will be used to determine whether a particular target view (e.g., U target For example, the predictive CNN 142a may be used to predict whether the current position (e.g., q t A current image 902 (e.g., U t The predictive CNN 142a receives motion control setting candidates 904 (e.g., corresponding to m t904) can guide the imaging component to the target image view based on the current image. In some embodiments, the set of motion control setting candidates 904 can include controls for changing the imaging plane within the volumetric ultrasound image. The predictive CNN 142a calculates a likelihood 908 (e.g., output 908) of reaching the target image view for each of the motion control setting candidates, and selects the motion control setting candidate with the highest likelihood of reaching the target image view. The predictive CNN 142a includes a first set of convolution layers 910 and pooling layers 912, a spatial tiling layer 920, a summing component 930, and a second set of convolution layers 940 and pooling layers 942.

[0060]

[0087] The predictive CNN 142a applies a first set of convolutional layers 910 and pooling layers 912 to the current image 902. Each convolutional layer 910 includes a set of filters 914 configured to extract image features from the image 902. Each pooling layer 912 includes a set of filters 916 that reduce the dimensionality of the extracted image features.

[0061]

[0088] The predictive CNN 142a applies a spatial tiling layer 920 to each candidate motion control setting 904. The spatial tiling layer 920 includes a set of filters 922. The spatial tiling layer 920 converts or maps the candidate motion control setting 904 to an output spatial dimension of the last pooling layer 912 of the first set (e.g., depicted as poolN). A summing component 930 calculates a sum 906 (e.g., a point-wise sum) between the output of the last pooling layer 912 of the first set and the output of the spatial tiling layer 920.

[0062]

[0089] The predictive CNN 142a applies a second set of interleaved convolutional layers 940 and pooling layers 942 to the sum 906. The convolutional layers 940 and pooling layers 942 have a substantially similar structure to the convolutional layers 910 and pooling layers 912, respectively. For example, each convolutional layer 940 includes a set of filters 944, and each pooling layer 942 includes a set of filters 916. The predictive CNN 142a produces an output 908 representing the likelihood of reaching a target image view for each motion control setting candidate 904. Thus, the most likely motion control setting is selected from the motion control setting candidate 904. The filters 914, 916, 922, 944, and 946 may have any suitable order. The coefficients for the filters 914, 916, 922, 944, and 946 are learned or trained as described in more detail herein.

[0063]

[0090] 10 is a schematic diagram illustrating a configuration 1000 for fine-tuning CNN 142b according to an embodiment of the present disclosure. The fine-tuning CNN 142b is used to select an image having higher quality from a pair of input images 1002 and 1004. For example, as described in more detail herein, the fine-tuning CNN 142b is applied to improve the position of the imaging components (e.g., TEE probe 110, TTE probe 310, and imaging device 570) after the predictive CNN 142a orients the imaging components to a desired location. The fine-tuning CNN 142b includes a first set of convolutional layers 1010 and pooling layers 1012, a second set of convolutional layers 1020 and pooling layers 1022, and a selection component 1030.

[0064]

[0091] The fine-tuned CNN 142b applies a first set of interleaved convolutional layers 1010 and pooling layers 1012 to one input image 1002 and a second set of interleaved convolutional layers 1020 and pooling layers 1022 to the other input image 1004. Each convolutional layer 1010 includes a set of filters 1014, and each pooling layer 1012 includes a set of filters 1016. Similarly, each convolutional layer 1020 includes a set of filters 1024, and each pooling layer 1022 includes a set of filters 1026. The selection component 1030 is configured to select the image 1032 having a higher quality from the pair of images 1002 and 1004. The coefficients for the filters 1014, 1016, 1024, 1026 are learned or trained as described in more detail herein.

[0065]

[0092] 11 is a schematic diagram illustrating a configuration 1100 for a target CNN 142c, according to an embodiment of the present disclosure. The target CNN 142c is used to qualify an input image 1102 against a target image view. For example, the target CNN 142c may be configured to qualify an input image 1102 against a target image view (e.g., U target 1102。 In an embodiment, the target CNN 142c is trained to recognize a particular anatomical category, such as lung, liver, or heart. The target CNN 142c includes a set of convolutional layers 1110 and pooling layers 1112 and a region proposal network 1120. The target CNN 142c applies a set of interleaved convolutional layers 1110 and pooling layers 1112 to the input image 1102. Each convolutional layer 1110 includes a set of filters 1114, and each pooling layer 1112 includes a set of filters 1116. The set of convolutional layers 1110 and pooling layers 1112 outputs a feature map 1104.

[0066]

[0093] A region proposal network 1120 is applied to the feature map 1104. The region proposal network 1120 is configured as a CNN. For example, the region proposal network 1120 includes a set of convolutional and pooling layers, each of which includes a set of filters. The region proposal network 1120 scores the feature map 1104 based on a number of anatomical or imaging categories (e.g., heart, lung, liver, B-line imaging). Thus, the region proposal network 1120 classifies the input image 1102 into one of the categories based on the highest score. The target CNN 142c is trained to detect a particular category (e.g., image view of the heart) and produces an output 1106 indicating whether the target view or the particular category is detected. For example, when the input image 1102 includes a target image view, the output 1106 includes a value of 1. Conversely, when the input image 1102 does not include a target image view, the output 1106 includes a value of 0. The output 1106 is referred to as a label or score. The coefficients for filters 1114 and 1116 and the filters in region proposal network 1120 are learned or trained, as described in more detail herein.

[0067]

[0094] 12-14 show mechanisms for training the CNN 142. FIG. 12 is a schematic diagram illustrating a method 1200 for training the predictive CNN 142a according to an embodiment of the present disclosure. The method 1200 is performed by the system 100 or 300. The method 1200 provides a more detailed view of the operations performed by the CNN training component 420. The method 1200 is performed offline using simulated data (e.g., images taken from a phantom) and / or clinical data (e.g., images taken from a patient 102). The method 1200 is used to train the predictive CNN 142a to predict a motion for repositioning an imaging component (e.g., the TEE probe 110, the TTE probe 310, or the imaging device 570) to reach a specific target image view (e.g., the target image view 810). The target image view includes a specific clinical characteristic (e.g., the LAA 812) associated with a clinical procedure (e.g., a LAAC procedure). The method 1200 incrementally trains a predictive CNN 142a based on a relevant dataset (eg, database 140) that is periodically updated.

[0068]

[0095] In step 1210, an image of a subject's body (e.g., an anatomical model) is acquired using an imaging component. The placement of the imaging component is controlled by a robotic system (e.g., robotic system 120, 320, or 520) based on a set of motor commands, motion control settings, or motion vectors, for example, using a mechanism similar to that illustrated in technique 500 described above with respect to FIG.

[0069]

[0096] In step 1212, the target CNN 142c is applied to the acquired images. The target CNN 142c determines a score or label for each acquired image based on the target image view. For example, when the target CNN 142c detects a target image view (e.g., a desired clinical characteristic) in the acquired image, the target CNN 142c outputs a value of 1. Conversely, when the target CNN 142c fails to detect a target image view in the acquired image, the target CNN 142c outputs a value of 0. Step 1212 corresponds to the operation of the qualification component 530.

[0070]

[0097] In step 1214, the captured images are associated with corresponding motor commands and labels to generate associated data sets. For example, each associated data set may be a data tuple d t In step 1216, the data memory is updated with the relevant data set. For example, the data memory corresponds to database 140 stored in memory 132. Steps 1210, 1212, 1214, and 1216 are repeated periodically.

[0071]

[0098] In step 1218, the predictive CNN 142a receives a batch or subset of the relevant dataset. The subset is randomly selected from the data memory. In an embodiment, the coefficients for the filters 914, 916, 922, 944, and 946 in the predictive CNN 142a are initialized with arbitrary values. The predictive CNN 142a is applied to each relevant dataset in the subset, for example, using forward propagation. The coefficients for the filters 914, 916, 922, 944, and / or 946 are adjusted, for example, using backpropagation, to minimize the output error (e.g., the output error between the output 908 of the predictive CNN 142a and the labels for the corresponding relevant dataset).

[0072]

[0099] In step 1220, a decision is made to stop training or to continue training with additional relevant datasets. When a decision is made to continue training, the next subset or batch of the relevant dataset is retrieved (e.g., randomly selected) from the data memory, as indicated by arrow 1222, and training of predictive CNN 142a is repeated for the next subset of the relevant dataset.

[0073]

[0100] When it is determined to stop training, in step 1224, clinical data (e.g., images of patients taken in a clinical environment) is acquired to further train the predictive CNN 142a. In step 1226, a clinician annotates each clinical image with a label indicating whether the clinical image contains a target image view. As indicated by arrow 1228, the annotated clinical images are used to further train or fine-tune the predictive CNN 142a, e.g., to further adjust coefficients for filters 914, 916, 922, 944, and / or 946.

[0074]

[0101] 13 is a schematic diagram illustrating a method 1300 for training the fine-tuned CNN 142b according to an embodiment of the present disclosure. Method 1300 is performed by system 100 or 300. Method 1300 provides a more detailed view of the operations performed by CNN training component 420. Method 1300 is performed offline using images acquired from phantoms and / or clinical data. For example, in step 1310, a set of target images is acquired. The target images include a particular target image view (e.g., target image view 810).

[0075]

[0102] In step 1312, image pairs are randomly selected from the set of target images, and the image from each pair having the higher image quality is selected.

[0076]

[0103] In step 1314, the fine-tuning CNN 142b receives a subset of image pairs and corresponding selections (e.g., selections that indicate images with higher quality from the image pairs). In an embodiment, the coefficients for the filters 1014, 1016, 1024, and 1026 in the fine-tuning CNN 142b are initialized with arbitrary values. The fine-tuning CNN 142b is applied to each image pair, and the corresponding selections are made, for example, using forward propagation. The coefficients for the filters 1014, 1016, 1024, and / or 1026 are adjusted, for example, using back propagation, to minimize the output error (e.g., the output error between the output image 1032 of the fine-tuning CNN 142b and the selections). The training of the fine-tuning CNN 142b is repeated for the next subset of image pairs and corresponding selections.

[0077]

[0104] FIG. 14 is a schematic diagram illustrating a method 1400 for training a target CNN 142c according to an embodiment of the present disclosure. The method 1400 is performed by the system 100 or 300. The method 1400 provides a more detailed view of the operations performed by the CNN training component 420. The method 1400 is performed offline using images acquired from a phantom and / or clinical data. For example, in step 1410, a labeled dataset is acquired. The labeled dataset includes images classified into categories (e.g., lung, heart, liver, B-line, and LAA), and the images are labeled based on the classification category. In step 1412, acquired labeled images are acquired, for example, labeled images acquired using a phantom and a robotic system (e.g., robotic system 120 or 320) or acquired from a database 140.

[0078]

[0105] In step 1414, the target CNN 142c receives a subset of the labeled dataset and a subset of the obtained labeled images. In an embodiment, the coefficients for the filters 1114 and 1116 in the target CNN 142c are initialized with arbitrary values. The target CNN 142c is applied to the subset of the labeled dataset and the subset of the obtained labeled images, for example, using forward propagation. The coefficients for the filters 1114 and / or 1116 are adjusted, for example, using back propagation, to minimize the output error (e.g., the output error between the output 1106 of the target CNN 142c and the classification or label). The training of the target CNN 142c is repeated for the next subset of the labeled dataset and the subset of the obtained labeled images.

[0079]

[0106] 15 is a schematic diagram illustrating a method 1500 for aligning an imaging component with respect to a desired location, according to an embodiment of the disclosure. Method 1500 is performed by system 100 or 300. Method 1500 provides a more detailed view of the operations performed by CNN application component 430. Method 1500 is used in a clinical environment to acquire an image of a patient's anatomy (e.g., the heart of patient 102) prior to a clinical procedure (e.g., a LAAC procedure). A clinician obtains a target image view U target The clinician selects an initial position q (e.g., target image 830) near the patient's anatomical structure of interest. t Place in.

[0080]

[0107] In step 1510, an image U representing the patient's anatomy is captured by the imaging device 1504. t In step 1512, a set of motion vectors M is generated, for example by processing component 136, within a certain range of motion arbitrarily chosen by the clinician. The set of motion vectors M can be expressed as follows: M={m x_1 ,mx_2 ,...,m x_n} (7) where m x_1 From m x_n are the motion vectors, which are the m t and depends on the imaging device 1504. For example, when the imaging device 1504 is a TEE probe similar to the TEE probe 110, the motion vector m x_1 From m x_n includes parameters α, β, γ, and ω as shown in equation (5) above. Alternatively, when the imaging device 1504 is a TTE probe similar to the TTE probe 310, the motion vector m x_1 From m x_n is expressed by the parameter v xt , v yt , v zt , ω xt , ω yt , and ω zt In some other cases, the set of motion vectors includes parameters that change the imaging plane within the volumetric image.

[0081]

[0108] In step 1514, the prediction CNN 142a calculates the image U t and a set of motion vectors M. The prediction CNN 142a receives a target imaging view U target The most likely motion vector to reach x_k , which is selected or estimated from the set M. FIG. 15 shows how the prediction CNN 142a estimates one motion vector m x_k However, in some embodiments, the prediction CNN 142a selects the target imaging view U target The most likely combination of motion vectors to reach x_k1 ,m x_k2 ,m x_k3}) may be selected from set M.

[0082]

[0109] In step 1516, the robotic system 1502 (e.g., the robotic system 120, 320, or 520) calculates the determined motion vector m x_k The robotic system 1502 receives the motion vector m x_k The robotic system 1502 repositions the imaging device 1504 based on the motion vector m. The robotic system 1502 applies a control law, such as a proportional-integral-derivative (PID) control law, to control the positioning of the imaging device 1504. For example, the robotic system 1502 may apply a control law, such as a proportional-integral-derivative (PID) control law, to control the positioning of the imaging device 1504 based on the motion vector m x_k Based on this, the imaging device 1504 is moved to the next position q t+1 In step 1518, the imaging device 1504 is repositioned at position q t+1 When the patient is in the t+1 Photograph the following.

[0083]

[0110] In step 1520, the target CNN 142c computes the image U t+1 The target CNN 142c receives the image U t+1 In step 1522, the target CNN 142c applies the image U t+1 In the target image view U target It is determined whether the target CNN 142c detects the image U t+1 If the target image view is not found in step 1510, steps 1510 through 1522 are repeated.

[0084]

[0111] Target CNN142c is image U t+1 In the target image view U target If so, the method 1500 proceeds to step 1524. In step 1524, the motion vector m is calculated by modifying one or more of the motion parameters by a small amount. x_k has been corrected, m x_k’ The method 1500 produces a modified motion vector, denoted as m x_k’ For example, the robotic system 1502 repeats steps 1516 and 1518 for the motion vector m x_k’Based on this, the imaging device 1504 is moved to a position q t+n , and the imaging device 1504 is repositioned at position q t+n Images of the patient's anatomy when t+n Photograph the following.

[0085]

[0112] In step 1526, the fine-tuning CNN 142b calculates the image U t+1 and U t+1’ A pairwise comparison is performed on the motion vectors, and the image with the higher image quality is selected. In some embodiments, the refinement steps of 1524 and 1526 are repeated. For example, multiple revised motion vectors are generated, and the fine-tuning CNN 142b is applied to select the motion vector that leads to the image with the highest quality.

[0086]

[0113] 16 is a schematic diagram illustrating a method 1600 for providing guidance to align an imaging component with respect to a desired location, according to an embodiment of the present disclosure. Method 1600 is performed by system 100 or 300. Method 1600 uses mechanisms substantially similar to those in method 1500. For example, method 1600 may be used in a clinical environment to acquire images of a patient's anatomy (e.g., the heart of patient 102) prior to a clinical procedure (e.g., a LAAC procedure). A clinician may select a target image view U. target The clinician selects a target image view 810 (e.g., a target image view 810). The clinician positions the imaging device 1604 (e.g., the TEE probe 110, the TTE probe 310, or the imaging device 570 or 1504) at a position q t Method 1600 applies the predictive CNN 142a, the fine-tuning CNN 142b, and / or the target CNN 142c as described in method 1500. However, instead of automatically aligning the imaging device 1604 using the robotic system 120, method 1600 provides instructions to the clinician for manual alignment.

[0087]

[0114] In step 1610, an image U representing the patient's anatomy is captured by the imaging device 1604. t is obtained. In step 1612, a set of motion vectors M is generated, for example by processing component 136, within a certain range of motion arbitrarily chosen by the clinician. The set of motion vectors M may be expressed as shown in equation (7) above.

[0088]

[0115] In step 1614, the prediction CNN 142a calculates the image U t and a set of motion vectors M. The prediction CNN 142a receives a target imaging view U target The most likely motion vector to reach x_k Select or guess from the set M.

[0089]

[0116] In step 1616, the display 1602 (e.g., the display 134) displays the determined motion vector m x_k and displays instructions instructing the clinician to manipulate the imaging device 1604. As described in more detail herein below, the instructions may be in the format of a graphical representation of movement or control of the imaging device 1604 (e.g., in the UI 144).

[0090]

[0117] In step 1618, the clinician moves the imaging device 1604 to the next position q based on the instructions displayed on the display 1602. t+1 to be rearranged.

[0091]

[0118] In step 1620, the imaging device 1604 detects that the imaging device 1604 is at position q t+1 When the patient is in the t+1 Photograph the following.

[0092]

[0119] In step 1622, the target CNN 142c computes the image U t+1 applies to.

[0093]

[0120] In step 1624, the target CNN 142c computes the image U t+1 In the target image view U target It is determined whether the target CNN 142c detects the image U t+1 If the target image view is not found in step 1610, steps 1610 through 1624 are repeated.

[0094]

[0121] Target CNN142c is image U t+1 In the target image view U target If so, the method 1600 proceeds to step 1626. In step 1626, the clinician refines the position of the imaging device 1604 to obtain an optimal target view.

[0095]

[0122] 17A-17C show various display views for guiding a clinician to align imaging components (e.g., TEE probe 110, TTE probe 310, imaging devices 570, 1504, and 1604) relative to desired locations to obtain an optimal target view. FIG. 17A is a schematic diagram illustrating an imaging component alignment guidance display view 1710, according to an embodiment of the present disclosure. View 1710 corresponds to a display view on display 134 in system 200 or 300 or display 1602 in method 1600. FIG. 17B is a schematic diagram illustrating an imaging component alignment guidance display view 1720, according to an embodiment of the present disclosure. FIG. 17C is a schematic diagram illustrating an imaging component alignment guidance display view 1730, according to an embodiment of the present disclosure. Although FIGS. 17A-17C show the UI 144 displaying a movement view for operating the TEE probe 110, alternatively, the UI 144 may be configured to display a movement view for operating a TTE probe (e.g., the TTE probe 310) or another imaging device of a different imaging modality.

[0096]

[0123] In view 1710, the UI 144 displays a graphical representation of a forward view of the handle 116 of the TEE probe 110 and the movements needed to orient or align the TEE probe to a desired location. In view 1720, the UI 144 displays a graphical representation of a lateral view of the handle 116 and the movements needed to orient or align the TEE probe to a desired location. In view 1730, the UI 144 displays a graphical representation of a rearward view of the handle 116 and the movements needed to orient or align the TEE probe to a desired location. The UI 144 provides selection options 1712 that allow the user to select between the different views 1710, 1720, 1730.

[0097]

[0124] The UI 144 may indicate the direction to turn the dial of the knob 117 (as shown by arrow 1714) to bend the TEE probe 110, for example, along a left-right plane relative to the patient's heart as illustrated in FIG. 2D. In some embodiments, the UI 144 may also indicate the amount of movement required based on the parameter β of the corresponding motion vector shown in equation (5).

[0098]

[0125] The UI 144 may indicate the direction to turn the knob 118 (as shown by arrow 1716) to bend the TEE probe 110 in an anterior-posterior direction relative to the patient's heart as shown in FIG. 2C. In some embodiments, the UI 144 may also indicate the amount of movement required based on the parameter α of the corresponding motion vector shown in equation (5).

[0099]

[0126] The UI 144 may further indicate a direction in which to rotate the TEE probe 110 (as shown by arrow 1718), for example, relative to the longitudinal axis 202 of the TEE probe 110 as shown in FIG. 2A. In some embodiments, the UI 144 may also indicate an amount of movement required based on the parameter γ of the corresponding motion vector shown in equation (5).

[0100]

[0127] The UI 144 may also show a control (e.g., red button 1702 or green button 1704) for switch 119 to enable or disable rotation of the imaging plane of the TEE probe 110, for example, as shown in FIG. 2B. For example, when switch 119 is set to enable rotation, the imaging plane is swept from 0 to 180 degrees at a constant speed around an axis, the axis of rotation being configurable. In some embodiments, the UI 144 may also show the amount of movement required based on the parameter ω of the corresponding motion vector shown in equation (5).

[0101]

[0128] FIG. 18 is a flow diagram of a method 1800 of acquiring an imaging dataset for training a CNN according to aspects of the disclosure. The steps of the method 1800 may be performed by the systems 100 and 300. The method 1800 uses mechanisms similar to those in the techniques 400, 500, 1200, 1500, and 1600 described with respect to FIGS. 4, 5, 12, 15, and 16, respectively. As shown, the method 1800 has a number of enumerated steps, but embodiments of the method 1800 may have additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0102]

[0129] In step 1810, the method 1800 includes receiving a first image from an imaging component or imaging device, e.g., via a communication device similar to the communication interface 138, when the imaging component is disposed in a first imaging position. The imaging device may include an ultrasound transducer or an ultrasound transducer array. The imaging device may correspond to the TEE probe 110, the TTE probe 310, or the imaging device 570, 1504, 1604. The first image and the first imaging position may be received in the U-band mode as described in the method 500. t and q t and , respectively. In some cases, the first imaging location corresponds to a particular imaging plane within a volumetric ultrasound image.

[0103]

[0130] In step 1820, the method 1800 includes transmitting a motion control setting for repositioning the imaging component from the first imaging position to the second imaging position. The motion control setting and the second imaging position may be the same as those described in the method 500. t and q t+1 and , respectively. In some cases, the first imaging location corresponds to a particular imaging plane within a volumetric ultrasound image.

[0104]

[0131] In step 1830, the method 1800 may generate a score (e.g., the score described above) representing a relationship between the second imaging position and the first image for a target view (e.g., target image view 810) that includes the first image, the motion control settings, and the clinical characteristics (e.g., LAA 812). t , corresponding to the database 140).

[0105]

[0132] In an embodiment, the motion control setting includes one or more parameters for moving the imaging component to the second imaging position. For example, when the imaging component is a TEE probe 110, the motion control setting includes parameters α, β, γ, and ω, as shown in equation (5). Alternatively, when the imaging component is a TTE probe 310, the motion control setting includes parameters v xt , v yt , v zt , ω xt , ω yt , and ω zt The method 1800 may include determining one or more parameters.

[0106]

[0133] In an embodiment, the method 1800 includes acquiring a second image (e.g., the U image described above) representing the subject's body from the imaging component when the imaging component is disposed in a second imaging position. t+1and determining a score based on a comparison of the second image to the target image view. For example, when the second image matches the target image view, the score has a value of 1. Conversely, when the second image does not match the target image view, the score has a value of 0. The second image may be a U image as described in method 500. t+1 Corresponds to.

[0107]

[0134] In an embodiment, method 1800 may include determining a score based on a comparison of the second imaging position to a target imaging position of the imaging component for acquiring an image of the subject's body corresponding to the target image view. For example, when the subject's body is a phantom, a target imaging position for acquiring a particular clinical characteristic or view is registered, and thus the score is based on a comparison of the registered target imaging position to the second imaging position.

[0108]

[0135] In an embodiment, the method 1800 may include transmitting the motion control settings to a robotic system (eg, robotic systems 120, 320, 520, and 1502) that controls the imaging component.

[0109]

[0136] FIG. 19 is a flow diagram of a method 1900 for training a CNN according to an aspect of the disclosure. The steps of the method 1900 may be performed by the systems 100 and 300. The method 1900 uses mechanisms similar to those in the configurations 900, 1000, and 1100 and the techniques 1200, 1300, and 1400 described with respect to FIG. 9, FIG. 10, FIG. 11, FIG. 12, FIG. 13, and FIG. 14, respectively. As shown, the method 1900 has a number of enumerated steps, but embodiments of the method 1900 may have additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0110]

[0137] At step 1910, the method 1900 includes obtaining a dataset associating images with motion control settings and scores. The images represent a subject's body (e.g., patient 102). The images are taken by imaging components (e.g., TEE probe 110, TTE probe 310, and imaging devices 570, 1504, and 1604). The motion control settings are used to reposition the imaging components. The score represents a relationship between a plurality of motion control settings and a plurality of images for a target image view (e.g., target image view 810) that includes a clinical characteristic (e.g., LAA 812). In an embodiment, the dataset corresponds to D in equation (4) above. The images, motion control settings, and scores correspond to U in equation (2) above. t , m t , and l t correspond to the following:

[0111]

[0138] At step 1920, the method 1900 includes training a first CNN (eg, the predictive CNN 142a) based on the dataset.

[0112]

[0139] At step 1930, the method 1900 includes training a second CNN (eg, the fine-tuning CNN 142b) based on the dataset or clinical data.

[0113]

[0140] At step 1940, the method 1900 includes training a third CNN (eg, the target CNN 142c) based on the dataset or clinical data.

[0114]

[0141] In an embodiment, the first CNN has a configuration similar to configuration 900. For example, method 1900 may train the first CNN by applying at least a first convolutional layer (e.g., convolutional layer 910) and a first pooling layer (e.g., pooling layer 912) to a first image (e.g., image 902) of the multiple images. Method 1900 may include determining a sum based on an output of the first pooling layer and a first motion control setting of the multiple motion control settings associated with the first image. Method 1900 may include applying at least a second convolutional layer (e.g., convolutional layer 940) and a second pooling layer (e.g., pooling layer 942) to the sum. The method 1900 may include adjusting coefficients (e.g., coefficients of filters 914, 916, 922, 944, 946) in at least one of the first convolutional layer, the first pooling layer, the second convolutional layer, or the second pooling layer based on an output (e.g., output 908) of the second pooling layer and a first score of the plurality of scores associated with the first image. The trained first CNN may predict motion control settings for acquiring an image including clinical features based on the input image.

[0115]

[0142] In an embodiment, the second CNN has a configuration similar to the configuration 1000. The method 1900 may include obtaining an indication that a first image (e.g., image 1002) of the plurality of images has higher quality than a second image (e.g., image 1004) of the plurality of images for a target image view. The method 1900 may train the second CNN by applying at least a first convolutional layer (e.g., convolutional layer 1010) and a first pooling layer (e.g., pooling layer 1012) to the first image. The method 1900 may include applying at least a second convolutional layer (e.g., convolutional layer 1020) and a second pooling layer (e.g., pooling layer 1022) to the second image. The method 1900 may include adjusting coefficients (e.g., coefficients of filters 1014, 1016, 1024, and 1026) in at least one of the first convolutional layer, the first pooling layer, the second convolutional layer, or the second pooling layer based on an output of the first pooling layer, an output of the second pooling layer, and an index (e.g., image 1032). The trained second CNN may select a higher quality image from a pair of input images for a target image view.

[0116]

[0143] In an embodiment, the third CNN has a configuration similar to configuration 1100. Method 1900 may include obtaining a candidate image and a classification category that classifies the candidate image into a first category of a plurality of categories of anatomical parts. Method 1900 may train the third CNN based on the candidate image and the classification category of the first category. The trained third CNN may classify an input image into one of the plurality of categories.

[0117]

[0144] 20 is a flow diagram of a method 2000 of applying a CNN according to aspects of the disclosure. The steps of the method 2000 may be performed by the systems 100 and 300. The method 2000 uses mechanisms similar to those in the techniques 400, 1500, and 1600 described with respect to FIGS. 4, 15, and 16, respectively. As shown, the method 2000 has a number of enumerated steps, but embodiments of the method 2000 may have additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0118]

[0145] At step 2010, the method 2000 includes receiving a first image (e.g., image 902) from an imaging component. The image represents a subject's body (e.g., patient 102). The image is taken by the imaging component (e.g., TEE probe 110, TTE probe 310, and imaging devices 570, 1504, and 1604) when the imaging component is positioned at a first imaging position. The first imaging position may be a particular physical location of the ultrasound imaging component (e.g., q as described in method 500). t ) and / or refers to a specific imaging plane within a volumetric ultrasound image.

[0119]

[0146] At step 2020, the method 2000 includes obtaining a number of candidate motion control settings (eg, candidate motion control settings 904) for repositioning the imaging component.

[0120]

[0147] In an embodiment, the multiple motion control setting candidates may be obtained by randomly sampling a set of motion parameters for repositioning the imaging component. For example, when the imaging component is a TEE probe 110, the motion control setting may include parameters α, β, γ, and ω, as shown in equation (5). Alternatively, when the imaging component is a TTE probe 310, the motion control setting may include parameters vxt , v yt , v zt , ω xt , ω yt , and ω zt may include.

[0121]

[0148] At step 2030, the method 2000 includes applying a first CNN (e.g., predictive CNN 142a) to the first image and the candidate motion control settings to produce a first motion control setting (e.g., setting 904). The first CNN is trained based on a target image view (e.g., target image view 810) that includes clinical features (e.g., LAA 812).

[0122]

[0149] At step 2040, the method 2000 includes repositioning the imaging component based on the first motion control setting. For example, the imaging component is repositioned from the first imaging position to a second imaging position. The second imaging position may be a specific physical location of the ultrasound imaging component (e.g., the q position described in method 500). t+1 ) and / or refers to a specific imaging plane within a volumetric ultrasound image.

[0123]

[0150] In an embodiment, applying the first CNN includes applying at least a first convolution layer (e.g., convolution layer 910) and a first pooling layer (e.g., pooling layer 912) to the first image. Applying the first CNN may further include determining a sum based on an output of the first pooling layer and a first motion control setting candidate among the plurality of motion control setting candidates. Applying the first CNN may further include applying at least a second convolution layer (e.g., convolution layer 940) and a second pooling layer (e.g., pooling layer 942) to the sum. Applying the first CNN may further include determining a first motion control setting based on an output of the last prediction layer.

[0124]

[0151] In an embodiment, the method 2000 further includes applying a second CNN (e.g., the target CNN 142c) to the second image to qualify the second image against the target image view. When the second CNN indicates that the second image has not been qualified, the method 2000 may reapply the first CNN to the second image to produce a second motion control setting for repositioning the imaging component from the second imaging position to a third imaging position, and may reapply the second CNN to qualify the third image against the target image view. The method 2000 may repeat the steps of reapplying the first CNN and the second CNN until the imaging component is capable of capturing the target image view.

[0125]

[0152] On the other hand, when the second CNN indicates that the second image is qualified, the method 2000 may adjust the second imaging position based on a third CNN (e.g., fine-tuning CNN 142b). For example, the method 2000 may reposition the imaging component to a third imaging position and capture a third image of the subject's body when the imaging component is in the third imaging position. The method 2000 may apply the third CNN to the second image and the third image. The method 2000 may select the second image or the third image based on the output of the third CNN.

[0126]

[0153] In an embodiment, the method 2000 may send a command to a robotic system (e.g., robotic system 120, 320, or 1502) to instruct the robotic system to reposition the imaging component to a second imaging position based on the first motion control setting.

[0127]

[0154] FIG. 21 is a flow diagram of a method 2100 for providing CNN-based imaging component alignment guidance according to aspects of the disclosure. The steps of the method 2100 may be performed by the systems 100 and 300. The method 2100 uses mechanisms similar to those in the techniques 400 and 1600 and views 1710, 1720, and 1730 described with respect to FIGS. 4, 16, 17A, 17B, and 17C, respectively. As shown, the method 2100 has a number of enumerated steps, but embodiments of the method 2100 may have additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0128]

[0155] In step 2110, the method 2100 includes obtaining a motion control setting for repositioning an imaging component or an imaging device from a first imaging position to a second imaging position relative to the subject's body (e.g., patient 102). The imaging device may include an ultrasound transducer or an ultrasound transducer array. The imaging device corresponds to the TEE probe 110, the TTE probe 310, or the imaging device 570, 1504, 1604. The motion control setting may be obtained based on a predictive network (e.g., CNN 142), where an image of the subject's body is taken when the imaging component is positioned at the first imaging position, and a target image view (e.g., target image view 810) includes the clinical characteristic (e.g., LAA 812). The image may be obtained based on the image U in the method 1600. t The motion control setting corresponds to the motion vector m x_k The first and second imaging positions correspond to specific physical locations of the ultrasound imaging component (e.g., q t and q t+1 ) and / or refers to a specific imaging plane within a volumetric ultrasound image.

[0129]

[0156] In step 2120, the method 2100 includes displaying instructions based on the motion control setting to operate a control component (e.g., the handle 116) in communication with the imaging component to reposition the imaging component to a second imaging position.

[0130]

[0157] Method 2100 may display instructions using displays similar to those shown in views 1710, 1720, and 1730. For example, method 2100 may display a graphical view of the control component and a visual indicator showing at least one of a direction of movement or an amount of movement to operate the control component. The graphical view may include a perspective view of the control component.

[0131]

[0158] In an embodiment, the method 2100 may receive a request for a first view of the multiple views of the control component and, in response to the request, switch the graphical view from a current view of the multiple views to the first view. The multiple views may include at least one of a forward view of the control component, a lateral view of the control component, or a rearward view of the control component. The control component may include at least one of a first subcomponent (e.g., knob 117) for controlling motion of the imaging component along a left-right plane of the subject's body, a second subcomponent (e.g., knob 118) for controlling motion of the imaging component along an anterior-posterior plane of the subject's body, or a third subcomponent (e.g., switch 119) for controlling an orientation of an imaging plane of the imaging component. The motion control settings may include at least one of a first parameter for operating the first subcomponent, a second parameter for operating the second subcomponent, a third parameter for operating the third subcomponent, or a fourth parameter for rotating the imaging component relative to an axis of the imaging component. The first, second, third, and fourth parameters correspond to α, β, ω, and γ in equation (5) shown above. The displaying may include displaying a visual indicator including at least one of an arrow (e.g., arrows 1714, 1716, and 1718) or an on / off indicator (e.g., buttons 1702 and 1704). Although method 2100 is described in the context of a TEE probe, similar mechanisms may be applied to provide alignment guidance for a TTE probe or an imaging device of another imaging modality.

[0132]

[0159] 22 is a flow diagram of a method 2200 of automating medical testing according to aspects of the disclosure. Steps of method 2200 may be performed by systems 100 and 300. Method 2200 uses mechanisms similar to those in techniques 400, 1500, and 2000 described with respect to FIGS. 4, 15, and 20, respectively. As shown, method 2200 has a number of recited steps, but embodiments of method 2200 may have additional steps before, after, and between the recited steps. In some embodiments, one or more of the recited steps may be omitted or performed in a different order.

[0133]

[0160] In step 2210, the method 2200 includes receiving a first image from an imaging device when the imaging device is disposed in a first imaging position relative to a subject's body (e.g., patient 102). The first image is received via a communication device similar to the communication interface 138. The image represents the subject's body. The imaging device may include an ultrasound transducer or an ultrasound transducer array. The imaging device corresponds to the TEE probe 110, the TTE probe 310, or the imaging device 570, 1504, 1604.

[0134]

[0161] At step 2220, the method 2200 includes determining a first motion control setting for repositioning the imaging device from the first imaging position to the second imaging position based on the first prediction network (e.g., CNN 142a), the first image, and the target image view (e.g., target image view 810) including the clinical characteristic (e.g., LAA 812). The first and second imaging positions may correspond to specific physical locations of the ultrasound imaging component (e.g., q 100 as described in method 500). t and q t+1 ) and / or refers to a specific imaging plane within a volumetric ultrasound image.

[0135]

[0162] At step 2230, the method 2200 includes repositioning, by a robotic system coupled to the imaging device, the imaging device to a second imaging position based on the first motion control setting. The robotic system is similar to the robotic systems 120 and 320.

[0136]

[0163] In an embodiment, the method 2200 further includes receiving a second image representing the subject's body from the imaging device when the imaging device is positioned at a second imaging position relative to the subject's body, and determining whether the second image includes clinical characteristics of the target image view based on a second prediction network (e.g., CNN142b).

[0137]

[0164] The method 2200 may further include determining a second motion control setting when it is determined that the second image does not include the clinical characteristic of the target image view, repositioning the imaging device to a third imaging position based on the second motion control setting, and receiving a third image representative of the subject's body when the imaging device is disposed in the third imaging position relative to the subject's body. The method 2200 may further include repeating the steps of determining a motion control setting (e.g., the second, third, fourth, fifth, nth motion control setting), repositioning the imaging device to an imaging position (e.g., the third, fourth, fifth, nth imaging position), and receiving an image (e.g., the third, fourth, fifth, nth image) until an image including the clinical characteristic of the target image view is received from the imaging device.

[0138]

[0165] The method 2200 may further include determining an adjustment for the second imaging position when the second image is determined to include the clinical characteristic of the target image view. The method 2200 may further include repositioning, by the robotic system, the imaging device to a third imaging position based on the adjustment, receiving a third image representative of the subject's body from the imaging device when the imaging device is disposed in the third imaging position relative to the subject's body, and selecting a target imaging position from the second imaging position and the third imaging position based on the third predictive network. The method 2200 may further include receiving a fourth image representative of the subject's body from the imaging device when the imaging device is disposed in the selected target imaging position, and determining a medical test result associated with the clinical characteristic based on the fourth image.

[0139]

[0166] In an embodiment, the method 2200 may further include determining a plurality of candidate motion control settings by sampling a set of movements for repositioning the imaging device. Determining a first motion control setting may further include selecting a first motion control setting from the plurality of candidate motion control settings based on the first prediction network, the first image, and the target image view.

[0140]

[0167] In an embodiment, for example as described in method 1900, the first prediction network is trained by providing a plurality of images captured by an imaging device from at least two imaging positions to obtain a target image view, obtaining a plurality of motion control settings based on an orientation or movement of the imaging device associated with the at least two imaging positions, and assigning a score to a relationship between the plurality of motion control settings and the plurality of images relative to the target image view.

[0141]

[0168] Aspects of the present application may provide several benefits. For example, the use of anatomical statistical modeling in a predictive network may provide consistent guidance to the sonographer in aligning the imaging component in an optimal position to obtain the desired imaging view. Thus, the disclosed embodiments may reduce the amount of time required to search for the desired imaging view and may enable the sonographer to arrive at the desired imaging view consistently. The graphical representation provides ease of use and clarity to the guidance instructions. Although the disclosed embodiments are described in the context of repositioning a TEE or TTE probe, the disclosed embodiments are suitable for use in guiding any imaging component of any imaging modality.

[0142]

[0169] A further embodiment of the present disclosure includes a medical ultrasound imaging view guidance system including a processing component configured to obtain motion control settings for repositioning an ultrasound imaging component from a first imaging position to a second imaging position relative to a body of a subject, the motion control settings being obtained based on a predictive network, an image of the body of the subject being taken while the ultrasound imaging component is positioned at the first imaging position, and a target image view including a clinical characteristic; and a display in communication with the processing component and configured to display instructions to operate a control component in communication with the ultrasound imaging component such that the ultrasound imaging component is repositioned to the second imaging position based on the motion control settings.

[0143]

[0170] In some embodiments, the display is further configured to display instructions by displaying a graphical view of the control component and a visual indicator indicating at least one of a direction of motion or an amount of motion to operate the control component. In some embodiments, the graphical view includes a perspective view of the control component. In some embodiments, the system further comprises an interface configured to communicate with the processing component and receive a request for a first view of the plurality of views of the control component, the processing component further configured to switch the graphical view from a current view of the plurality of views to the first view based on the request. In some embodiments, the plurality of views includes at least one of a forward view of the control component, a lateral view of the control component, or a posterior view of the control component. In some embodiments, the control component includes at least one of a first subcomponent for controlling motion of the ultrasound imaging component along a left-right plane of the subject's body, a second subcomponent for controlling motion of the ultrasound imaging component along an anterior-posterior plane of the subject's body, or a third subcomponent for controlling an orientation of an imaging plane of the ultrasound imaging component. In some embodiments, the motion control settings include at least one of a first parameter for operating the first subcomponent, a second parameter for operating the second subcomponent, a third parameter for operating the third subcomponent, or a fourth parameter for rotating the ultrasound imaging component relative to an axis of the ultrasound imaging component. In some embodiments, the display is further configured to display instructions by displaying a visual indicator including at least one of an arrow or an on / off indicator based on at least one of the first parameter, the second parameter, the third parameter, or the fourth parameter. In some embodiments, the ultrasound imaging component is a transesophageal echocardiography (TEE) probe.In some embodiments, the control component is located on a handle of the TEE probe. In some embodiments, the predictive network includes at least one convolutional neural network (CNN). In some embodiments, the system further comprises an interface in communication with the processing component and configured to receive the image, the processing component further configured to obtain the motion control settings by applying the predictive network to the image and a plurality of parameters for operating the control component. In some embodiments, the interface is further configured to receive the plurality of parameters.

[0144]

[0171] A further embodiment of the present disclosure includes a method of providing medical ultrasound imaging guidance, the method comprising the steps of obtaining a motion control setting for repositioning an ultrasound imaging component from a first imaging position to a second imaging position relative to a body of a subject, the motion control setting being obtained based on a predictive network, an image of the body of the subject being taken while the ultrasound imaging component is positioned at the first imaging position, and a target image view including a clinical characteristic; and displaying instructions on a display to operate a control component in communication with the ultrasound imaging component to reposition the ultrasound imaging component to the second imaging position based on the motion control setting.

[0145]

[0172] In some embodiments, the displaying step includes displaying a graphical view of the control component and a visual indicator indicating at least one of a direction of motion to operate the control component or an amount of the motion. In one embodiment, the method further includes receiving a request for a first view of the plurality of views of the control component; and switching the graphical view from a current view of the plurality of views to the first view based on the request. In some embodiments, the plurality of views includes at least one of a forward view of the control component, a lateral view of the control component, or a posterior view of the control component. In some embodiments, the control component includes at least one of a first subcomponent for controlling motion of the ultrasound imaging component along a left-right plane of the subject's body, a second subcomponent for controlling motion of the ultrasound imaging component along an anterior-posterior plane of the subject's body, or a third subcomponent for controlling an orientation of an imaging plane of the ultrasound imaging component. In some embodiments, the motion control settings include at least one of a first parameter for operating the first subcomponent, a second parameter for operating the second subcomponent, a third parameter for operating the third subcomponent, or a fourth parameter for rotating the ultrasound imaging component relative to an axis of the ultrasound imaging component. In some embodiments, the displaying includes displaying a visual indicator including at least one of an arrow or an on / off indicator based on at least one of the first parameter, the second parameter, or the third parameter.

[0146]

[0173] Those skilled in the art will recognize that the above-described devices, systems, and methods may be modified in various ways. Thus, those skilled in the art will appreciate that the embodiments encompassed by the present disclosure are not limited to the specific exemplary embodiments described above. In this regard, while exemplary embodiments have been shown and described, a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure. It is understood that such variations may be made to the foregoing without departing from the scope of the present disclosure. It is therefore appropriate that the scope of the appended claims be broadly construed in a manner consistent with the present disclosure.

Claims

1. 1. A guidance system for acquiring medical images, comprising: a processor for obtaining motion control settings for repositioning an imaging device from a first imaging position to a second imaging position relative to a body of a subject, the motion control settings being obtained based on a predictive network, an image of the body of the subject being captured while the imaging device is positioned at the first imaging position, and a target image view includes a particular clinical characteristic; The prediction network comprises: providing a plurality of images acquired by the imaging device from at least two imaging positions to obtain the target image view; obtaining a plurality of motion control settings based on an orientation or a movement of the imaging device associated with the at least two imaging positions; assigning scores to relationships between the plurality of motion control settings for the target image view and the plurality of images; A guidance system trained by.

2. a display in communication with the processor and displaying instructions to operate a control component in communication with the imaging device based on the motion control setting such that the imaging device is repositioned to the second imaging position; The guidance system of claim 1 , wherein the display further displays the instructions by displaying a graphical view of the control component and a visual indicator showing at least one of a direction of movement or an amount of movement to operate the control component.

3. The guidance system of claim 2 , wherein the graphical view includes a perspective view of the control component.

4. 3. The guidance system of claim 2, further comprising a communications device in communication with the processor for receiving a request for a first view of a plurality of views of the control component, the processor further switching the graphical view from a current view of the plurality of views to the first view based on the request.

5. The guidance system of claim 4 , wherein the multiple views include at least one of a forward view of the control component, a side view of the control component, or a rearward view of the control component.

6. 3. The guidance system of claim 2, wherein the control component includes at least one of a first subcomponent that controls movement of the imaging device along a left-right plane of the subject's body, a second subcomponent that controls movement of the imaging device along an anterior-posterior plane of the subject's body, or a third subcomponent that controls an orientation of an imaging plane of the imaging device.

7. 7. The guidance system of claim 6, wherein the motion control settings include at least one of a first parameter for operating the first sub-component, a second parameter for operating the second sub-component, a third parameter for operating the third sub-component, or a fourth parameter for rotating the imaging device relative to an axis of the imaging device.

8. a display in communication with the processor and displaying instructions to operate a control component in communication with the imaging device based on the motion control setting such that the imaging device is repositioned to the second imaging position; 8. The guidance system of claim 7, wherein the display further displays the instructions by displaying a visual indicator including at least one of an arrow or an on / off indicator based on at least one of the first parameter, the second parameter, the third parameter, or the fourth parameter.

9. The guidance system of claim 1 , wherein the imaging device is a transesophageal echocardiography (TEE) probe.

10. The imaging device is a transesophageal echocardiography (TEE) probe; The guidance system of claim 2 , wherein the control component is mounted to a handle of the TEE probe.

11. The guidance system of claim 2 , further comprising a communication device in communication with the processor to receive the image, the processor further deriving the motion control settings by applying the predictive network to the image and a plurality of parameters for operating the control component.

12. The guidance system of claim 11 , wherein the communications device is further configured to receive the plurality of parameters.

13. A method of operating a guidance system for acquiring medical images, comprising: a processor of the guidance system obtaining motion control settings for repositioning an imaging device from a first imaging position to a second imaging position relative to a body of a subject, the motion control settings being obtained based on a predictive network, an image of the body of the subject being captured when the imaging device is positioned at the first imaging position, and a target image view including a specific clinical characteristic; the processor providing a plurality of images acquired by the imaging device from at least two imaging positions to obtain the target image view; obtaining a plurality of motion control settings based on an orientation or a movement of the imaging device associated with the at least two imaging positions; and training the prediction network by assigning scores to relationships between the plurality of motion control settings for the target image view and the plurality of images.

14. The method of claim 13, further comprising the step of: displaying instructions on a display, the processor causing a control component in communication with the imaging device to operate based on the motion control setting such that the imaging device is repositioned to the second imaging position; 14. The method of claim 13, wherein the displaying step comprises displaying a graphical view of the control component and a visual indicator showing at least one of a direction of movement or an amount of movement to operate the control component.

15. The method of claim 14, further comprising: receiving, by the processor, a request for a first view of a plurality of views of the control component; switching the graphical view from a current view of the plurality of views to the first view based on the request, The method of claim 14 further comprising:

16. The method of claim 15 , wherein the multiple views include at least one of a forward view of the control component, a side view of the control component, or a rearward view of the control component.

17. 15. The method of operating a guidance system of claim 14, wherein the control component includes at least one of a first subcomponent that controls movement of the imaging device along a left-right plane of the subject's body, a second subcomponent that controls movement of the imaging device along an anterior-posterior plane of the subject's body, or a third subcomponent that controls an orientation of an imaging plane of the imaging device.

18. 20. The method of claim 17, wherein the motion control settings include at least one of a first parameter for operating the first sub-component, a second parameter for operating the second sub-component, a third parameter for operating the third sub-component, or a fourth parameter for rotating the imaging device relative to an axis of the imaging device.

19. A method of operating a guidance system as described in claim 17, wherein the displaying step includes a step of displaying a visual indicator including at least one of an arrow or an on / off indicator based on at least one of the first parameter, the second parameter, or the third parameter.

Images