Systems, devices, and methods for intelligently controlled medical examinations

Abstract

Remotely controlled or supervised semi-autonomous and autonomous non-invasive medical imaging systems, devices and methods are disclosed in which a transducer mounted on a robotic unit arm assembly is used to produce imaging data from a patient. Pre-trained computational models are deployed to predict optimal transducer position and trajectories, and to adjust the trajectory using multi-modal inputs and real-time closed loop feedback. Remote control is also disclosed via a controller system that synchronizes with the transducer and provides haptic feedback to the operator. Remote operation may be remote from the patient generally, or entirely remote from the imaging environment.

Description

SYSTEMS, DEVICES, AND METHODS FOR INTELLIGENTLY CONTROLLEDMEDICAL EXAMINATIONSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent applications claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 638,063 filed on 24 April 2024 and entitled, “Systems and Methods for Intelligently Controlled Medical Examinations,” the disclosure of which is hereby incorporated by reference herein in its entirety and made part of the present patent application for all purposes.FEDERALLY SPONSORED RESEARCH

[0002] Research lending to various aspects of the present invention was made, at least in part, with Government support of Grant Nos. 2203203 and 2423628, awarded by the National Science Foundation. The United States Government may have certain rights in the invention.TECHNICAL FIELD

[0003] The disclosed inventive subject matter relates in general to human assistive technologies in healthcare, and more specifically to systems, devices, and methods that use collaborative robotics and artificial intelligence to assist skilled individuals in performing non-invasive ultrasound diagnostics.BACKGROUND OF THE INVENTION

[0004] The following description of the background of the invention is provided simply as an aid in understanding the invention and is not admitted to describe or constitute prior art to the invention.

[0005] Non-invasive ultrasound diagnostics, such as transthoracic echocardiograms (TTEs), are currently constrained by a variety of factors that inhibit individuals from accessing essential healthcare services. TTE is one type of an ultrasound test that uses high-frequency sound waves to produce detailed images of the heart’s structure and function for preventative care and / or for diagnosing various cardiovascular issues such as, for example, myocardial infarction, vulvar heartdisease, heart failure, cardiomyopathies, congenital structural heart defects, stenosis, cardia tumors, and the like. The orientation and placement of an ultrasound transducer on an individual’s chest produces different views of the individual’s heart. A full standard TTE includes several key views of the heart: parasternal long axis (PLAX), parasternal short axis (PSAX), apical views, subcostal view, and suprasternal notch view. Other views may be included without departing from the scope of the present disclosure, such as but not limited to the right sternal border, for instance. Currently, obtaining a comprehensive set of TTE views requires a skilled technician (e.g. sonographer, ultrasound technician, etc.) to be physically present in an examination room with the individual, and manually manipulate an ultrasound transducer along the individual to obtain numerous, optimal images of the heart. These requirements have resulted in: (i) extended outpatient wait times for TTEs; (ii) limited availability of sonographers in rurally located and critical access hospitals; (Hi) rushed TTE examinations requiring reimaging due to poor image quality, a missed image in a protocol, and / or inadvertent mistakes; (iv) increased number of sonographers suffering from work-related musculoskeletal disorders; (v) increased workload and sonographer burnout; (vi) sonographer shortages; (vii) strains on emergency services; (viii) barriers to expanding services in healthcare facilities; and / or (ix) lost revenues in healthcare facilities. All of these constraints create a substantial barrier to diagnosing and initiating treatment for cardiovascular diseases, where time-sensitive intervention could be crucial and / or where conditions could be managed or mitigated with early detection.

[0006] Various video conference ultrasound machines have been developed in an attempt to address sonographer shortages and extended TTE wait times. In such machines, an untrained operator or minimally trained operator conducts a full TTE exam under the guidance of a remote sonographer. The remote sonographer instructs the untrained or minimally trained operator how to move the ultrasound transducer along the individual to obtain the desired views. However, these video conference ultrasound machines require two individuals to conduct a TTE exam, while also making the untrained or minimally trained operator responsible for capturing images and properly positioning the transducer.

[0007] Additionally, point of care ultrasound (POCUS) transducers have attempted to eliminate or greatly reduce sonographer involvement by utilizing deep learning algorithms to instruct untrained users on how to acquire quick views of the heart. While this technology has made asignificant impact in emergency room settings where the emergency department physicians can quickly check ejection fraction or pericardial effusion, the capabilities of these POCUS transducers are greatly limited in scope beyond triage examinations. In fact, the increased use of these transducers by untrained operators has actually increased the demand for full TTEs as the healthcare facilities do not want to be liable if issues were missed by the untrained operator.

[0008] Accordingly, there is an ongoing need for non-invasive systems, devices, and methods that leverage collaborative robotics and artificial intelligence to assist a sonographer with acquiring high-quality diagnostic ultrasound images in TTEs and other ultrasound applications. The disclosed technology, when paired with the clinical expertise of a sonographer, amplifies the impact of its sonographer, while solving the obstacles described above that exist in practice today.BRIEF SUMMARY OF THE INVENTION

[0009] The following provides a summary of certain example implementations of the disclosed technology. This summary is not an extensive overview and is not intended to identify key or critical aspects or elements of the disclosed technology or to delineate its scope. However, it is to be understood that the use of indefinite articles in the language used to describe and claim the disclosed technology is not intended in any way to limit the described technology. Rather the use of “a” or “an” should be interpreted to mean “at least one” or “one or more”.

[0010] As an example solution to the above-described problems in the art, a system for conducting an imaging examination on a living being in an imaging environment is described including a transducer configured to capture image data coupled to a robotic unit arm assembly, and a pretrained computational model executed by one or more processors to initiate the imaging examination in response to receiving a command from an operator, and by way of the pre-trained computational model, predict a target location on the living being to position the transducer for a first viewpoint of the imaging examination, and provide a set of movement commands to the robotic unit to place the transducer in an initial position at the target location.

[0011] In some embodiments, the system includes a spatial registration system. The spatial registration system has a capture device configured to gather real-time spatial data from the imaging environment, and the one or more processors are further configured to receive the real-time spatial data, and in response, resize and normalize the visual data, apply the resized and normalized visual data to the pre-trained computational model, and in response, determine an X,Y coordinate location that corresponds to the target location and determine a depth coordinate with respect to the X,Y coordinate location such that an X,Y,Z coordinate location that corresponds to the target location is defined.

[0012] Further exemplary embodiments of the system may be configured to receive a set of image data from the transducer at the target location, generate a quality score derived from the set of image data, and based on the quality score, generate a set of updated movement commands for the robotic unit that are predicted to increase the quality score.

[0013] In some aspects, a quality score may be computed by using the pre-trained computational model to identify a segmentation mask for one or more anatomical structures depicted in the set of image data, calculate an area within the segmentation mask, and compute the quality score based on the calculated area and a ground truth segmentation mask area.

[0014] In one embodiment, the computational model is trained based on one or more training datasets comprising images previously captured during an imagining examination performed by an expert demonstrator.

[0015] In embodiments provided with admittance control, the system may include one or more sensors configured to generate real-time force data arising from the movement of the transducer along the living being, wherein the one or more processors are further configured to apply admittance control over the movement of the transducer based on the real-time force data to ensure constant contact between the transducer and the living being during the imaging examination. Further, some embodiments may be provided wherein the admittance control models the movement of the transducer as a spring using a damping factor and a stiffness factor in combination with the real-time force data.

[0016] The disclosed system may be adapted for autonomously transitioning between imaging viewpoints, and in some embodiments the one or more processors are further configured to, by way of the pre-trained computational model identify a second target location on the living being to position the transducer for a second viewpoint of the imaging examination, and in response,provide a set of movement commands to the robotic unit to move the transducer at the second target location.

[0017] One advantage of the invention disclosed herein is provided in a system that includes a controller system in communication with the robotic unit, wherein the controller system includes a controller that is movable by the operator to correspondingly move the arm assembly with the transducer, wherein the controller is movable along a curved scanning surface. In further embodiments, the one or more processors are further configured to autonomously position the controller at a location on the curved scanning surface that corresponds to the initial position of the transducer. In some cases, the controller includes one or more controls that, when actuated, allow the operator to move the controller along the curved scanning surface, and an index marker that facilitates orientation and provides the operator with the position of the transducer relative to that of the controller.

[0018] In an exemplary embodiment, the system is particularly suited for circumstances wherein the imaging examination is a transthoracic echocardiogram (TTE) procedure comprising a plurality of viewpoints, wherein the plurality of viewpoints include a parasternal long axis (PLAX) view, a parasternal short axis (PSAX) view, an apical view, a subcostal view, and a suprasternal notch view.

[0019] An exemplary advantage of the invented system is found in applications enabling the operator to be remotely located from both the robotic unit and the living being undergoing the imaging examination. In some embodiments, the operator is not present in the examination environment and is remote therefrom.

[0020] Exemplary embodiments are described utilizing a spatial registration system having a capture device configured to gather real-time visual data of the living being, the transducer, and the arm assembly of the robotic unit, wherein the one or more processors are further configured to predict an optimal transducer position based on the real-time visual data, and calculate the quality score as a function of a difference in position between the transducer at the target location from the optimal transducer position.

[0021] As yet another example, disclosed herein is a system for conducting an imaging examination on a living being in an imaging environment having a transducer configured tocapture image data, wherein the transducer is coupled to an arm assembly of a robotic unit, a plurality of sensors configured to collect real-time data including force, pose, and transducer velocity, and a pre-trained computational model executed by one or more processors, wherein the one or more processors are configured to, receive a sequence of image data, sensor data and transducer pose data, segment the sequence into discrete action blocks corresponding to clinically relevant transducer movements, determine, based on the sequence and a task objective, a predicted action block to execute, and provide a set of control commands to the robotic unit to execute the predicted action block.

[0022] In some embodiments, the system is provided such that the computational model is a transformer-based neural network trained using expert demonstrations. In a further embodiment, the system is provided such that the computational model is trained using reinforcement learning to maximize an image quality score or anatomical completeness score. In an embodiment, the system is provided such that the computational model is configured to generate a reward function via inverse reinforcement learning. In another embodiment, the system is provided such that the computational model fuses input data from ultrasound image features, force-torque readings, and transducer pose encodings. In a further embodiment, the system is provided such that the computational model is pre-trained using self-supervised or few-shot learning to improve generalization to new patient anatomies.

[0023] An exemplary embodiment of the disclosed invention is provided as a controller system for use in a non-invasive imaging examination of an individual. In some embodiments, the controller system may include a controller that receives motion input from an operator to correspondingly move an arm of a robotic unit, wherein a transducer is coupled to the arm of the robotic unit and is positioned in a predetermined orientation at a predetermined location on the individual undergoing the non-invasive ultrasound examination, and at least one processor configured to receive location data of the transducer at the predetermined location, and in response, generate commands to autonomously position the controller at a location on a scanning surface that corresponds to the predetermined location of the transducer and in an orientation that corresponds to the predetermined orientation of the transducer.

[0024] In one embodiment, an exemplary controller may include a handle, and a ball-type joint coupled to the handle, wherein the ball-type joint maintains contact with the scanning surfaceduring the motion input. In some applications the controller system may include a scanning surface having a curved surface representing a torso. Further embodiments may include a controller having a control that, when actuated, allows the operator to move the controller with respect to the scanning surface. Yet other embodiments are provided in which the controller includes an index marker that facilitates orientation and provides the operator with the position of the transducer on the individual relative to that of the controller on the scanning surface.

[0025] As yet another example, exemplary controller systems are disclosed herein in which the non-invasive imaging examination is a transthoracic echocardiogram (TTE) procedure. In some TTE embodiments, the one or more predetermined locations on the individual correspond to a parasternal long axis (PLAX) view, a parasternal short axis (PSAX) view, an apical view, a subcostal view, a suprasternal notch view, or combinations thereof.

[0026] A further exemplary advantage of the disclosed controller system is found in applications enabling the operator and the controller to be remote from both the robotic unit and the living being undergoing the imaging examination. In some embodiments, the operator and the controller are not present in the examination environment and is remote therefrom.

[0027] Moreover, also disclosed herein are controller systems that have in some embodiments, an arm assembly having at least six degrees of freedom, wherein the controller is coupled to the arm assembly.

[0028] In some embodiments, the controller system includes one or more sensors configured to collect real-time force data during the motion input on the controller from the operator, and one or more feedback devices in communication with the at least one processor, wherein the one or more feedback devices are configured to provide a haptic response to the operator during movement of the controller. In some of these exemplary embodiments are disclosed further embodiments wherein the at least one processor is further configured to define a pre-set working area on the scanning surface for the controller, monitor the pre-set working area, and by way of the one or more feedback devices, provide the haptic response to the operator when the controller is moved outside of the working area.

[0029] Additionally, the present disclosure includes an embodiments of a system for controlling a robotic unit for use in non-invasive diagnostic imaging of an individual having a scanning surface,an arm assembly, a controller coupled to the arm assembly, at least one processor configured to capture position and orientation encoding data from the system arising from movements of the controller by an operator, and transmit the position and orientation encoding data to the robotic unit wherein the robotic unit causes a transducer to be moved in a corresponding orientation to a corresponding location on the individual undergoing the non-invasive ultrasound examination.

[0030] In some exemplary embodiments, the arm assembly is configured with three translational degrees of freedom with respect to the scanning surface, the controller is configured with three degrees of rotational freedom with respect to the arm assembly, or both.

[0031] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the technology disclosed herein and may be implemented to achieve the benefits as described herein. Additional features and aspects of the disclosed system, devices, and methods will become apparent to those of ordinary skill in the art upon reading and understanding the following detailed description of the example implementations. As will be appreciated by the skilled artisan, further implementations are possible without departing from the scope and spirit of what is disclosed herein. Accordingly, the descriptions provided herein are to be regarded as illustrative and not restrictive in nature.BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated into and form a part of the specification, schematically illustrate one or more example implementations of the disclosed technology and, together with the general description given above and detailed description given below, serve to explain the principles of the disclosed subject matter, and wherein:

[0033] FIGURE 1 is an example implementation of the disclosed ultrasound system showing the basic components of the system, wherein the system comprises one or more communication networks; a computer system; a robotic unit; a controller system; a live display; an ultrasound machine; and a user interface;

[0034] FIGURE 2 is a perspective view of the non-limiting, exemplary robotic unit of FIGURE 1, wherein an ultrasound transducer is coupled to an arm assembly of the robotic unit;

[0035] FIGURE 3 depicts the robotic unit of FIGURE 2 co-located with a patient on an examination platform;

[0036] FIGURE 4 is a perspective view of the exemplary controller system of FIGURE 1 comprising a controller coupled to an arm assembly;

[0037] FIGURE 5 is another perspective view of the controller system of FIGURE 4;

[0038] FIGURE 6 is a perspective view of the exemplary controller of FIGURE 4;

[0039] FIGURE 7 is another perspective view of the controller of FIGURE 6;

[0040] FIGURE 8 depicts a non-limiting, exemplary scanning surface configured for use with the controller system of FIGURE 4;

[0041] FIGURE 9 depicts the non-limiting, exemplary ultrasound machine of FIGURE 1;

[0042] FIGURE 10 is a detailed view of the ultrasound machine of FIGURE 9, showing an exemplary pressure knob;

[0043] FIGURE 11 depicts the non-limiting, exemplary user interface of FIGURE 1, showing a plurality of interactive elements;

[0044] FIGURES 12A-12B illustrate the position and orientation of the robotic unit of FIGURE 2 and the controller system of FIGURE 4 when obtaining ultrasound images of the patient’s heart along a parasternal long axis (PLAX), wherein FIGURE 12A illustrates the position and orientation of the robotic unit, and wherein FIGURE 12B illustrates the synchronized position and orientation of the controller system on the scanning surface;

[0045] FIGURES 13A-13B illustrate the position and orientation of the robotic unit of FIGURE 2 and the controller system of FIGURE 4 when obtaining ultrasound images of the patient’s heart along a parasternal short axis (PSAX), wherein FIGURE 13A illustrates the position and orientation of the robotic unit, and wherein FIGURE 13B illustrates the synchronized position and orientation of the controller system on the scanning surface;

[0046] FIGURES 14A-14B illustrate the position and orientation of the robotic unit of FIGURE 2 and the controller system of FIGURE 4 when obtaining ultrasound images of the patient’s heart in an apical viewpoint, wherein FIGURE 14A illustrates the position and orientation of the roboticunit, and wherein FIGURE 14B illustrates the synchronized position and orientation of the controller system on the scanning surface;

[0047] FIGURES 15A-15B illustrate the position and orientation of the robotic unit of FIGURE 2 and the controller system of FIGURE 4 when obtaining ultrasound images of the patient’s heart in a subcostal viewpoint, wherein FIGURE 15A illustrates the position and orientation of the robotic unit, and wherein FIGURE 15B illustrates the synchronized position and orientation of the controller system on the scanning surface;

[0048] FIGURES 16A-16B illustrate the position and orientation of the robotic unit of FIGURE 2 and the controller system of FIGURE 4 when obtaining ultrasound images of the patient’s heart in a suprasternal viewpoint, wherein FIGURE 16A illustrates the position and orientation of the robotic unit, and wherein FIGURE 16B illustrates the synchronized position and orientation of the controller system on the scanning surface;

[0049] FIGURE 17 is a flowchart of one non-limiting, exemplary method or process for using the ultrasound system of FIGURE 1 to perform a transthoracic echocardiogram (TTE) procedure, with a sonographer co-located with the patient and / or at a location remote from the patient;

[0050] FIGURE 18 is a flowchart of an exemplary set of steps that may be performed with the flowchart of FIGURE 17 to autonomously position and orient the ultrasound transducer on or near the patient in a selected TTE acoustic window;

[0051] FIGURES 19A-19B depict another flowchart of an exemplary set of steps that may be performed using the ultrasound system of FIGURE 1 to perform the TTE procedure, with the sonographer co-located with the patient or at a location remote from the patient;

[0052] FIGURES 20A-20D depict an exemplary process for training one or more computational models used in the ultrasound system of FIGURE 1, wherein FIGURE 20A provides the exemplary set of steps for training the one or more computational models, wherein FIGURE 20B depicts an exemplary convolutional neural network (“CNN”) that may be used in the process, wherein FIGURE 20C depicts an exemplary time-dependent neural network that may be used in the process, and wherein FIGURE 20D depicts a long short term memory (LSTM) network that may be used as the time-dependent neural network in the process;

[0053] FIGURE 21 is a flowchart of an exemplary set of steps that may be performed with the system of FIGURE 1 to provide admittance control to the robotic unit of FIGURE 2;

[0054] FIGURES 22A-22C provide exemplary, non-limiting operational flow diagrams for the ultrasound system of FIGURE 1, wherein FIGURE 22A provides an input stage, a data preprocessing stage, a feature extraction stage, a core processing stage, a decision making stage, and an action handling and output stage, wherein FIGURE 22B is a detailed view of the input stage, the data preprocessing stage, the feature extraction stage, and the core processing stage, and wherein FIGURE 22C is a detailed view of the decision making stage and the action handling and output stage;

[0055] FIGURE 23 is a schematic diagram of an exemplary transformer-based action chunking architecture used to generate control commands for an autonomous imaging transducer based on multi-modal input; and

[0056] FIGURE 24 is another example implementation of the disclosed ultrasound system showing the basic components of the system, wherein the system comprises a first bedside system having a patient and a robotic unit; a second bedside system having a patient and a robotic unit; one or more communication networks; a computer system; a controller system; a live display; an ultrasound machine; and a user interface.DETAILED DESCRIPTION

[0057] Example implementations are now described with reference to the Figures. Reference numerals are used throughout the detailed description to refer to the various elements and structures. Like reference numerals are used to refer to like elements throughout. Although the following detailed description contains many specifics for the purposes of illustration, a person of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the disclosed technology. Accordingly, the following implementations are set forth without any loss of generality to, and without imposing limitations upon, the claimed subject matter.

[0058] The examples discussed herein are examples only and are provided to assist in the explanation of the apparatuses, devices, systems, and methods described herein. None of thefeatures or components shown in the drawings or discussed below should be taken as required for any specific implementation of any of these the apparatuses, devices, systems or methods unless specifically designated as such. For ease of reading and clarity, certain components, modules, or methods may be described solely in connection with a specific Figure. Any failure to specifically describe a combination or sub-combination of components should not be understood as an indication that any combination or sub-combination is not possible. Also, for any methods described, regardless of whether the method is described in conjunction with a flow diagram, it should be understood that unless otherwise specified or required by context, any explicit or implicit ordering of steps performed in the execution of a method does not imply that those steps must be performed in the order presented but instead may be performed in a different order or in parallel.

[0059] The invention is described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the drawings, the size and relative sizes of layers and regions may be exaggerated for clarity.

[0060] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the use of the word “or” is intended to be non-exclusive unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof.

[0061] Embodiments of the invention are described herein with reference to illustrations that are schematic illustrations of idealized embodiments (and intermediate structures) of the invention. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques or tolerances, are to be expected. Thus, embodiments of the invention should not beconstrued as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing.

[0062] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0063] The disclosed technology includes non-invasive systems, devices, and methods that leverage collaborative robotics and artificial intelligence to assist a sonographer or other skilled medical professional or operator in acquiring high-quality diagnostic ultrasound images of patients (defined herein as all living beings, including humans and animals, receiving an ultrasound, but not necessarily for the purpose of treatment) during full transthoracic echocardiograms (TTEs) or other ultrasound examinations (ex. peripheral vessel or vascular, abdominal, thoracic, fetal, pelvic, breast, scrotal, carotid, musculoskeletal, etc.). The disclosed systems, devices, and methods identify clinically relevant anatomical landmarks or keypoints on the patient’s body that correspond to key image windows, and optimize image quality by safely repositioning or angling an ultrasound transducer on a robotic unit using ultrasound image feedback. The disclosed systems, devices, and methods permit sonographers / operators to conduct and / or monitor ultrasound diagnostics on a patient without having to manually manipulate the ultrasound transducer along various positions on the patient.

[0064] Certain implementations of the disclosed technology provides systems, devices, and methods in which the sonographer is co-located (i.e. in the same room) with the patient being examined. In this implementation, the sonographer can direct an in-room, robotic unit having an ultrasound transducer to perform ultrasound diagnostics on the patient through operation of the disclosed controller system, thereby reducing sonographer arm strain during examinations, reducing sonographer burnout, and maintaining staffing levels. Additionally, the disclosed systems, devices, and methods can autonomously optimize image quality and conduct full TTE examinations or other ultrasound diagnostics on the patient with sonographer supervision.

[0065] Other implementations of the disclosed technology permits the sonographer to direct ultrasound diagnostics on a patient from a remote location (ex. different areas of the same facility, different locations in the same state, different states, different countries / globally, etc.). In this implementation, the sonographer is physically distant from both the patient being examined and the robotic unit performing the ultrasound examination. The sonographer, who is remote from the patient and robotic unit, is in control of the robotic unit by operation of the disclosed controller system, thereby providing sonographers the ability to assess patients / individuals without spending time traveling. Additionally, the disclosed systems, devices, and methods can autonomously optimize image quality and conduct full TTE examinations or other ultrasound diagnostics on the patient with sonographer supervision.

[0066] Still other implementations of the disclosed technology permit the sonographer to simultaneously direct ultrasound diagnostics on a plurality of patients from the remote location. In this implementation, the plurality of individuals may be physically distant from each other (ex. different areas of the same facility, different locations in the same state, different states, different countries / globally, etc.) with their own in-room robotic unit, and the sonographer may be physically distant from one or more of the plurality of patients being examined. The sonographer who is remote from one or more of the patients can selectively switch between the patients to control the respective robotic unit by operation of the disclosed controller system, thereby allowing one sonographer to access multiple patients / individuals at once. Additionally, the disclosed systems, devices, and methods can autonomously optimize image quality and conduct full TTE examinations or other ultrasound diagnostics on the plurality of patients with sonographer supervision.

[0067] FIGURE 1 provides a block diagram of an example implementation of the disclosed robotic imaging system 10 showing the basic components of the system. The imaging system 10 shown in the exemplary embodiment depicted in FIGURE l is a robotic ultrasound system, and includes the following components that when used together assist a sonographer or similar ultrasound technician in acquiring high-quality, non-invasive images of a patient's anatomy, including but not limited to ultrasound images of a patient’s heart along the parasternal long axis (PLAX), the parasternal short axis (PSAX), apical views, subcostal view, and suprasternal notch view during a full, standard TTE protocol. The ultrasound system 10 is used on a patient 50 andmay include various combination of an examination platform 75; one or more communication networks 100; a computer system 200; a robotic unit 300; a controller system 400; a live display 500; an ultrasound machine 600; and a user interface 700. Both the patient 50 and the robotic unit 300 can be co-located with, or at a location remote from, the computer system 200, the controller system 400, the live display 500, the ultrasound machine 600, the user interface 700, or a combination of those components.

[0068] The patient 50 can be any living being, including humans and animals, receiving a non- invasive imaging procedure such as an ultrasound, but not necessarily for the purpose of a medical treatment. By way of a non-limiting example, the patient 50 may be an individual testing or recalibrating the ultrasound system 10. The examination platform 75 can be any standard examination table or hospital bed that permits the patient 50 to lie down in a supine position, a left lateral decubitus position, or a right lateral decubitus position.

[0069] The communication network 100 can be used to interconnect the computer system 200, the robotic unit 300, the controller system 400, the live display 500, the ultrasound machine 600, and the user interface 700. The communication network 100 can be embodied as any number of various wired or wireless communication networks and protocols capable of communicating data, instructions, and general digital or analog signals as needed among the components of the imaging system 10. For example, the communication network 100 may include one or more local area networks (LANs), one or more wide area networks (WANs), one or more cellular networks, one or more publicly-accessible, global networks such as the Internet, or combinations thereof. Additionally, the communication network 100 can include any number of additional devices (e.g., wireless access points, bridges, switches, routers, etc.) to facilitate communication between the components of the ultrasound system 10. As discussed further herein, some components may be directly connected to one another in some embodiments, while others may communicate via a general purpose network.

[0070] The computer system 200 includes one or more processors 210, one or more memories 220, and one or more data stores 230 that receive, transmit, analyze, modify, and store information. The one or more memories 220 may store code 222 for execution by the one or more processors 210 to provide intelligence for controlling the robotic unit 300 based on input from the controller system 400, the live display 500, the ultrasound machine 600, or the user interface 700. The code222 may comprise processor-executable instructions that are resident on a computer-readable storage medium, such as memory 220, and may take the form of software, firmware, or the like. The one or more data stores 230 may comprise one or more relational databases, one or more file systems, or other suitable data storage devices support the decision-making processes of the computer system 200.

[0071] While FIGURE 1 illustrates the computer system 200 communicating with the components of the ultrasound system 10 across the communication network 100, it is to be appreciated that that the computer system 200 need not be implemented remotely from the robotic unit 300, the controller system 400, the live display 500, the ultrasound machine 600, or the user interface 700, all or in part. By way of a non-limiting example, some or all of the components of the computer system 200 may be deployed in the same room at the patient 50 as part of, or connected to, the robotic unit 300. By co-locating some or all of the components of the computer system 200 with the robotic unit 300, communication latency over the network 100 can be effectively eliminated or greatly reduced, thereby improving the speed at which the computer system 200 controls operation the robotic unit 300. By way of another non-limiting example, some or all of the components of the computer system 200 may be deployed as part of, or connected to, the controller system 400. Co-location of some or all of the components of the computer system 200 with the controller system 400 can improve the speed at which the computer system 200 coordinates operations with the computer system 400.

[0072] With reference to FIGURES 1-3, the robotic unit 300 includes an arm assembly 305 having a mount or similar attachment device 310 that is configured to receive and hold an imaging transducer, such as an ultrasound transducer 315. The mount 310 is a versatile attachment device that holds various types of ultrasound transducers, while enabling fast interchangeability. The arm assembly 305, by way of a plurality of arm segments and joints, is controllably articulable with respect to a distal end 317 to within at least six degrees of freedom (e.g. six DOF, seven DOF, or eight DOF) to move, position, or maintain the ultrasound transducer 315 in one or more desired locations relative to the patient 50. For example, the arm assembly 305 can move up / down (heave), forward / back (surge), left / right (sway) in three perpendicular axes (x, y, z) and pitch, roll, and yaw (p, r, y). The arm assembly 305 is preferably mounted to a mobile base assembly 307 that permits the robotic unit 300 to be easily maneuvered within or between examination rooms or to bepositioned adjacent to the patient 50 located on the examination platform such that the ultrasound transducer 315 can reach the patient’ s 50 chest area, although those skilled in the art will appreciate that other mounting options are available without departing from the scope of the claimed invention, such as the examination platform 75.

[0073] In a preferred embodiment, the robot unit 300 includes 8 total DOF to mimic torso and shoulder movements available to technicians performing imaging manually. This particular configuration has been found to be advantageous in positioning the transducer 315 to obtain optimal images, improving accuracy and speed overall. In the embodiment shown in connection with FIGURES 1-3, for example, the robotic arm assembly 305 is fixed to a swing arm 318 that is in turn mounted to the mobile base assembly 307 such that it is rotatable about the assembly 307 via swing arm motor 319. The swing arm 318 and swing arm motor 319 may be combined with a six- or seven-DOF arm assembly 305, for instance, to yield a seven- or eight-DOF robot unit 300.

[0074] The robotic unit 300 may include an on-board computer system 320 having a processor 322, a memory 324, and a communication device 326. The on-board computer system 320 can translate commands from the computer system 200 into position encodings for the arm assembly 305 to move, position, or maintain the ultrasound transducer 315 in one or more desired locations relative to the patient 50. The communication device 326 may be any wired or wireless communication device capable of exchanging information with remote or nearby components of the ultrasound system 10 (e g., cables or other direct data connections, Bluetooth transceiver, WiFi transceiver, cellular data transceiver, etc.). In some embodiments, the transducer 315 is directly connected to the imaging machine 600 via communication tether 316, separately from the general communication network 100. This configuration permits use of third-party imaging systems with the invented system 10 more easily.

[0075] The robotic unit 300 may further include one or more sensors 330 that sense data indicative of one or more characteristics of the movement of the ultrasound transducer 315. The one or more sensors 330 can collect real-time force and torque data arising from movements of the robotic unit 300 or movements of the ultrasound transducer 315 along the patient 50 during the TTE examination. The real-time force and torque data can represent force and torque values in multiple degrees and dimensions of the robotic unit 300 (e.g., in at least six degrees of freedom as discussed above) to enable nuanced control over the articulated movement of the arm assembly 305 and theultrasound transducer 315. The sensor 330 can include one or more force-torque sensors that are located on the mount 310, on the arm assembly 305 near the ultrasound transducer 315 or the mount 310, or on the ultrasound transducer 315 itself. The sensor 330 can also include one or more current sensors located within the robotic unit 300 that detects and measures the electric current used in articulating the arm assembly 305 of the robotic unit 300.

[0076] Further still, the robotic unit may include an on-board spatial registration system 340 having one or more capture devices 342 that spatially identifies the real-time position of the arm assembly 305 and the ultrasound transducer 315 relative to the patient 50 via a common frame of reference. The capture device 342 can rely on radio frequency sensing, such as through radio frequency identification (RFID), take the form or one or more cameras that obtain image or audio data of the arm assembly 305, the ultrasound transducer 315 and nearby areas, or a combination thereof. As such, the capture device 342 may include one or more: 2D cameras, 3D cameras (e.g. LIDAR), RGB cameras, or RGB-D cameras. The RGB-D cameras may include a plurality of stereoscopic cameras using time-of-flight (TOF) for depth sensing. The data generated by the capture device 342 can be used to provide the sonographer (either remotely or co-located) with views of the ultrasound transducer 315 relative to the patient 50 (via live display 500), as well as to generate data indicative of the positioning and movement / trajectory of the ultrasound transducer 315 while TTE examination is being performed. The spatial registration system 340 allows ultrasound images that are generated by the ultrasound transducer 315 to be tagged with position data that can be translated to a coordinate system of the robotic unit 300.

[0077] The robotic unit 300 may optionally include a communication interface 350 or an emergency stop 355. The communication interface 350 may be a display screen or the like that provides the patient 50 or an assistant near the patient 50 with a visual connection to the sonographer who is remote from, or co-located with, the patient 50. The emergency stop 355 may include a button, switch, knob, or similar control device that allows the patient 50, the assistant, or the sonographer to stop movement of the robotic unit 300 in emergency situations (e.g. medical emergencies, improper articulation of the robotic unit, etc.).

[0078] Now referring to FIGURES 1 and 4-8, the controller system 400 includes a controller 410 coupled to an arm assembly 405. The controller system 400 is configured to include at least six degrees of freedom (e.g. six DOF, seven DOF, or eight DOF) that allow the sonographer tomanually manipulate the controller 410 to move, position, or maintain the ultrasound transducer 315 in one or more desired locations relative to the patient 50, thereby emulating manual holding and movement of an ultrasound transducer. In a preferred embodiment, the arm assembly 405 includes three translational degrees of freedom and the controller includes three rotational degrees of freedom. The controller system 400 is co-located with the sonographer and can either be colocated with, or remote from, the patient 50 and the robotic unit 300.

[0079] The controller 410 includes a ball-type joint 412 and a handle 414. The ball-type joint 412 is coupled to both a distal end 406 of the arm assembly 405 and the handle 414, and generally provides a curved surface 407 for contact with a scanning surface 800. The ball-type joint 412 provides full freedom of movement of the controller 410 and the handle 414 (e.g. up / down, forward / back, left / right, pitch, roll, and yaw) with respect to the arm assembly 405. In the present non-limiting embodiment, the handle 414 is of an ergonomic design having a grip portion 416 configured to receive the sonographer’ s hand. The handle 414 includes at least one control, such as button 418 that, when actuated (e.g. compressed), allows the sonographer to move the controller 410 along a scanning surface 800 (see FIGURE 8), rotate the handle 414 about the ball-type joint 412, or re-center or re-position the controller 410 on the scanning surface 800. Deactivating the control 418 (e.g. released) causes the current position of the controller 410 to be fixed or held in place. The control 418 may take the form of a variety of equivalent mechanisms, including but not limited to one or more buttons, touch sensors, optical occlusion sensors, force and torque sensors, and the like.

[0080] The handle 414 further includes an index marker 420 that corresponds to the orientation of an index marker present on the ultrasound transducer 315. The index marker 420 facilitates orientation and provides the sonographer with an additional reference point to indicate location, position, and orientation information of the ultrasound transducer 315 relative to the current location, position, and orientation of the controller 410. The index marker 420 may be a notch, LED, or any other indicator suitable for the described purpose.

[0081] Still referring to FIGURES 1 and 4-8, the scanning surface 800, which is co-located with the sonographer, is a stationary object having a rounded or curved surface 802 that in one embodiment substantially resembles that of a human chest or torso. The rounded surface 802 allows the sonographer grip and move the controller 410 in the same way and in the same anglesas the sonographer would with manual scanning of a patient. As the controller 410 is manipulated / moved along the rounded surface 802, it is preferred in some embodiments that the ball-type joint 412 must maintain contact with the rounded surface 802 for the robotic unit 300 to move in accordance with the sonographer’ s movements.

[0082] It is to be appreciated that controller 410 could include ajoystick, elongated handle, or any other electronic or computer-compatible motion control device providing at least six degrees of freedom in combination with an arm assembly 405.

[0083] The controller system 400 further includes one or more sensors 430 that sense movement characteristics of the controller 410 in response to forces exerted thereon by the sonographer. The one or more sensors 430 can collect real-time force and torque data during manipulations of the controller 410 within the at least six degrees of freedom. The sensor 430 can include one or more force-torque sensors that are located on the arm assembly 405, on the controller 410, or on or near the handle 414, for example. The sensor 430 can also include one or more current sensors located within the controller system 400 that detects and measures the electric current used in articulating the arm assembly 405 or the controller 410.

[0084] The controller system can further include an on-board processor 440, memory 450, and communication device 460. The processor 440 can execute instructions stored in the memory 450 to capture positional encoding data from the arm assembly 405 arising from movements of the controller 410 or capture the sensed movement characteristics from the one or more sensors 430 so that feedback can be provided to the sonographer during operation of the controller 410 (see feedback device 470 discussed below). The positional encoding data, or measured, sensed or calculated movement characteristics can be provided across communication network 100 to the computer system 200 or directly to the other components of the ultrasound system 10. The communication device 460 may be any wired or wireless communication device capable of exchanging information with remote or nearby components of the ultrasound system 10 (ex. cables or other direct data connections, Bluetooth transceiver, Wi-Fi transceiver, cellular data transceiver, etc.).

[0085] The controller system 400 further includes one or more feedback devices 470 that provides a haptic, tactile, or other feedback response to the sonographer during operation of the controller410 to guide control inputs and signal key control events during operation. The processor 440, in communication with the feedback device 470, can change servo motor power and characteristics within the controller system 400 to provide feedback response to the sonographer. For example, when electrical current through the servo motor is low, the motors are free to move and the controller 410 can be moved with little to no resistance; however, when electrical current is high, the motors resist movement and the controller 410 is more difficult to move.

[0086] Additionally, the processor 440 or the one or more feedback devices 470 can define a preset working area on the scanning surface 800 in which the sonographer is able to move or manipulate the controller 410. This working area can be pre-set based on, for example, the geometry of the controller 410 or the handle 414. The pre-set working area can be monitored by the processor 440 to ensure the controller 410 remains therein during sonographer manipulations. If the sonographer attempts to move the controller 410 or the handle 414 outside of the defined working area, the processor 440 can apply a current to the servo motors and prevent movement thereof, and the one or more feedback devices 470 can alert the sonographer. The current can be applied according to a graduation function (e.g. a linearly increasing current) as movement of the controller 410 or the handle 414 approaches the boundary of the working area, thereby gradually increasing the difficulty in movement of the controller 410 or the handle 414 the closer they get to the working area’s boundary.

[0087] In a preferred embodiment, one or more techniques are employed to bound the movement of the robotic unit or provide feedback to the operator. During an examination, pre-set regions or computer vision can be used to identify areas in which the ultrasound transducer 315 should not be placed. By way of one non-limiting example, when imaging at the suprasternal notch, pre-set regions or computer vision can be used to prevent the ultrasound transducer 315 from inadvertently contacting the patient’s 50 face or head. The spatial registration system 340 may be used to detect regions / areas in which the robotic unit 300 should avoid, and the controller system 400 can halt movement of the robotic unit 300 if the sonographer moves the controller 410 or the handle 414 in a manner that would cause the robotic unit 300 to trespass one of the detected regions / areas, before the trespass actually occurs. This type of movement bounding and feedback can also be applied in some embodiments using force and torque sensors 330 on the robotic unit 300. These data can be used to detect collision force to provide collision awareness and feedback to theoperator (preferably haptic), increasing patient safety and reducing the risk of damage to the robot unit 300 and transducer 315. In some embodiments, robotic arm 305 collision detection may also result in the feedback device 470 and the processor 440 restricting the operator's further motion in the direction of the collision, for instance, and collisions are detected by comparing the forces on the end of the robotic arm sensors 330 and the torques on each of the joints of the robotic arm assembly 305 and swing arm motor 319. In this manner, the sensor 330 output is used to calculate expected individual joint torques, which are compared to actual sensed torques to determine whether any measurement falls outside of the expected range.

[0088] Now referring to FIGURE 1, as images, videos, and audio are captured in some embodiments by the one or more capture devices 342 on the robotic unit 300, the live display 500 presents, to the sonographer, real-time data of the positions and movements of the arm assembly 305 and the ultrasound transducer 315 relative to the patient 50. The live display 500 is co-located with the sonographer, and may be an independent component of the ultrasound system or may be part of the controller system 400, the ultrasound machine 600, or the user interface 700. The live display 500 can include a liquid crystal display (LCD) or light-emitting diode display (LED) display, ranging from a simple multi-segment alpha-numeric display, to a full color high resolution display, that displays each position of the ultrasound transducer 315 in real-time to the sonographer. The live display 500 may include audio interface capabilities that permit verbal communication between the sonographer and the patient 50 or the assistant near the patient 50.

[0089] With reference to FIGURES 1 and 9-10, the ultrasound machine 600 includes the following components that when used together allow the sonographer to view in real-time the ultrasound images being generated by the ultrasound transducer 315. The ultrasound machine 600 comprises a processor 605, a memory 610, a communication device 615, and a user interface 620 having one or more display screens 622 and a plurality of controls 624. The ultrasound machine 600 may be co-located with the sonographer, or may be co-located with the patient depending on operational needs in a given application. When remote scanning is preferred, the imaging machine 600 may preferably remain co-located with the patient and robotic unit 300. In some cases, the imaging machine and the robotic unit may be combined together as a single unit (not shown). The communication device 615 can be any wired or wireless communication device capable of exchanging imaging information, diagnostic information, software configurations, softwareapplications, and other information with the robotic unit 300 or the computer system 200, either remotely or nearby (e.g. cables or other direct data connections, Bluetooth transceiver, Wi-Fi transceiver, cellular data transceiver, etc.). The one or more display screens 622 can display the real-time ultrasound images to the sonographer, and the plurality of controls 624 provide the sonographer with various options for evaluating and / or completing measurement of the generated ultrasound images, or confirming the accuracy and correct positioning of the ultrasound transducer 315 on the patient 50 for obtaining the desired ultrasound images.

[0090] In embodiments deployed for co-located imaging procedures, the ultrasound machine 600 may further include a mount for a pressure knob, button, slide, or similar control device 630 that, when activated, increases or decreases pressure of the ultrasound transducer 315 against the patient 50. The amount of pressure of the ultrasound transducer 315 against the patient 50 may be controlled through an admittance control architecture, as will be discussed in greater detail below. By way of one non-limiting example, rotation of the pressure knob 630 in a first, clockwise direction increases the pressure of the ultrasound transducer 315 against the patient 50, and rotation of the pressure knob 630 in a second, counter-clockwise direction reduces the pressure of the ultrasound transducer 315 against the patient 50. The amount or level of pressure between the ultrasound transducer 315 and the patient can be adjusted by way of the pressure knob 630 at any time during scanning. It is to be appreciated that pressure may need to be adjusted depending on the patient’s 50 body fat content such as subcutaneous fat. The amount of pressure currently applied between the ultrasound transducer 315 and the patient 50 during scanning is presented on the user interface 700 for sonographer reference. While the figures illustrate the pressure knob 630 mounted on or incorporated with the ultrasound machine 600, it is to be appreciated that the pressure knob could be located on or near the controller system 400, incorporated into the user interface 700 as a touch-screen initiated feature, or an independent component of the ultrasound system 10 co-located with the sonographer.

[0091] With reference to FIGURES 1 and 11, the user interface 700 is co-located with the operator such as a sonographer, allowing the sonographer to start and stop the imaging procedure (e.g., a TTE procedure) or allows for autonomous initial positioning of the robotic unit 300 or ultrasound transducer 315 relative to the patient 50. The user interface 700 may be a graphical user interface that is displayed on a touchscreen display, as in the present non-limiting embodiment, or may bea non-touchscreen display. The user interface 700 may include a number of interactive elements that, when activated, position the robotic unit 300 or the ultrasound sound transducer 315 at a desired position, including location and orientation, relative to the patient 50. “HOME” element 702 can autonomously position or return the robotic unit 300 to an initial starting position or other reference position. The initial starting position of the robotic unit 300 can be any position where the ultrasound transducer 315 is not in contact with the patient 50; for example, the ultrasound transducer 315 is pointing downward towards the ground, perpendicular to the examination platform 75, at a distance away from the patient 50. “START” element 704 moves the ultrasound transducer 315, by way of the arm assembly 305, into contact with the patient 50 at a position associated with a sonographer-selected TTE acoustic window: “PARASTERNAL LONG AXIS” element 706; “PARASTERNAL SHORT AXIS” element 708; “APICAL” element 710; “SUBCOSTAL” element 712; and “SUPRASTERNAL” element 714. Each of the elements 706, 708, 710, 712, 714, and their respective views and transducer placement, will be discussed in greater detail below. “STOP” element 716 will pause movement of the robotic unit 300. The “START” element 704 can also be used to restart / continue the currently selected command. “CANCEL” element 718 will cancel or undo the currently initiated command. When the TTE procedure is finished, first select the “STOP” element 716 followed by the “HOME” element 702. “RETRACT” element 720 will withdraw or pull-back the ultrasound transducer 315 if the sonographer-desired position relative to the patient 50 is improper.

[0092] The user interface 700 may further include various indicators 730 that provide positional information of the ultrasound transducer 315 relative to the patient 50 during the TTE examination, such as, for example, the amount of force or pressure between the transducer 315 and the patient 50, the angle of the transducer 315, or whether the controller 410 is synched with the robotic unit 300

[0093] Each of the interactive elements described with reference to the user interface 700 may be buttons, interfaces, or other similar controls that allow the sonographer to manually select the desired command or may illuminate when selected to indicate to the sonographer the currently selected settings. It is to be appreciated that the user interface 700 includes components such as a processor, memory, and communication device capable of exchanging information with the components of ultrasound system 10 wirelessly or via a wired connection. The user interface 700may be an independent component of the ultrasound system 10 or may be part of the controller system 400, the live display 500, or the ultrasound machine 600.

[0094] The orientation and placement of a transducer 315 on a patient 50 produces unique imaging views of the patient’s 50 internal anatomical structures. In some embodiments, an operator manually manipulates the robotic unit 300 to position the transducer 315 in a desirable starting orientation at a location corresponding to a desired imaging window. In a preferred embodiment, the user interface 700 is utilized to direct the transducer 315 mounted to the arm assembly 305 of the robotic unit 300 to a predetermined position associated with an imaging window preselected by an operator. With references to FIGURES 1, 11-16B, use of the system 10 to autonomously position the transducer 315 at exemplary starting positions for particular TTE acoustic windows is described. In addition to these manual and autonomous initial positioning options, other semi- autonomous methods are disclosed in further detail below.

[0095] With reference to FIGURES 1, 11, and 12A-12B, in one non-limiting example, an operator may choose to select the “PARASTERNAL LONG AXIS” element 706 to proceed with this view during a TTE ultrasound procedure, followed by the “START” element 704. The system 10 will autonomously move the ultrasound transducer 315 into contact with the patient 50 at the position (location / orientation) needed to obtain high-quality ultrasound images in the PLAX acoustic window. In the PLAX view, the ultrasound transducer 315 is generally placed on the upper torso of the patient 50, on the left side of their sternum, at or near the 4th rib, with the index marker on the ultrasound transducer 315 pointing towards the patient’ s 50 right shoulder (see FIGURE 12A). The PLAX view assesses the general condition of the heart and the left and right ventricular functions. Once the ultrasound transducer 315 is correctly positioned and oriented on the patient 50 in the PLAX window, the controller 410 will optionally autonomously reposition and synchronize (i.e. sync) with the current angle, orientation, or position of the ultrasound transducer 315 on the scanning surface 800 (see FIGURE 12B). The index marker 420 on the handle 414 will mimic the position of the index marker on the ultrasound transducer 315 and point towards upper right comer of the scanning surface 800.

[0096] With reference to FIGURES 1, 11, and 13A-13B, in another non-limiting example, selecting the “PARASTERNAL SHORT AXIS” element 708, followed by the “START” element 704, will autonomously move the ultrasound transducer 315 into contact with the patient 50 at theposition (location / orientation) needed to obtain high-quality ultrasound images in the PSAX acoustic window. In the PSAX view, the ultrasound transducer 315 is generally placed on the upper torso of the patient 50, on the left side of their sternum, at or near the 4th rib, with the index marker on the ultrasound transducer 315 pointing towards the patient’s 50 left shoulder (see FIGURE 13A) The PSAX view provides cross-sectional images of the heart, allowing for assessment of the left ventricular function, wall motion, and morphology of the aortic and mitral valves. Once the ultrasound transducer 315 is correctly positioned and oriented on the patient 50 in the PSAX window, the controller 410 will optionally autonomously reposition and sync with the current angle, orientation, or position of the ultrasound transducer 315 on the scanning surface 800 (see FIGURE 13B). The index marker 420 on the handle 414 will mimic the position of the index marker on the ultrasound transducer 315 and point towards upper left comer of the scanning surface 800.

[0097] With reference to FIGURES 1, 11, and 14A-14B, in yet another non-limiting example, selecting the “APICAL” element 710, followed by the “START” element 704, will autonomously move the ultrasound transducer 315 into contact with the patient 50 at the position (location / orientation) needed to obtain high-quality ultrasound images in an apical acoustic window. In the apical view, the ultrasound transducer 315 is positioned at the point of maximal impulse, typically at or around the 5th intercostal space at a midclavicular line, with the index marker on the ultrasound transducer 315 pointing downward towards the patient’s left armpit (see FIGURE 14A). This acoustic view provides assessment of the functionality of all four heart chambers, the atrioventricular valves, the septa, ejection fraction, cardiac output, and detection of regional wall motion abnormalities. Once the ultrasound transducer 315 is correctly positioned and oriented on the patient 50 in the acoustic window, the controller 410 will optionally autonomously reposition and sync with the current angle, orientation, or position of the ultrasound transducer 315 on the scanning surface 800 (see FIGURE 14B). The index marker 420 on the handle 414 will point downward towards the left side of the scanning surface 800 to mimic the position of the index marker on the ultrasound transducer 315.

[0098] With reference to FIGURES 1, 11, and 15A-15B, in still another non-limiting example, selecting the “SUBCOSTAL” element 712, followed by the “START” element 704, will autonomously move the ultrasound transducer 315 into contact with the patient 50 at the position(location / orientation) needed to obtain high-quality ultrasound images in the subcostal window. In the subcostal view, the ultrasound transducer 315 is positioned at or near the patient’s 50 xiphoid process, angled toward the patient’s chest, with the index marker on the ultrasound transducer 315 pointing towards the patient’s 50 left side (see FIGURE 15A). The subcostal view provides an alternative imaging window, especially beneficial for patients with pulmonary conditions or when other views are obstructed, focusing on the atria, atrial septum, and the inferior vena cava. Once the ultrasound transducer 315 is correctly positioned and oriented on the patient 50 in the subcostal window, the controller 410 will optionally autonomously reposition and sync with the current angle, orientation, or position of the ultrasound transducer 315 on the scanning surface 800 (see FIGURE 15B) The index marker 420 on the handle 414 will point downward towards the left side of the scanning surface 800 to mimic the position of the index marker on the ultrasound transducer 315.

[0099] With reference to FIGURES 1, 11, and 16A-16B, in still another non-limiting example, selecting the “SUPRASTERNAL” element 714, followed by the “START” element 704, will autonomously move the ultrasound transducer 315 into contact with the patient 50 at the position (location / orientation) needed to obtain high-quality ultrasound images in the suprasternal window. In the suprasternal view, the ultrasound transducer 315 is positioned at or near the patient’s 50 suprasternal notch, angled downward, with the index marker on the ultrasound transducer 315 pointing towards the patient’s 50 left shoulder (see FIGURE 16A). The suprasternal view facilitates examination of the aorta, which is useful for identifying aortic dissection, coarctation, and evaluating the aortic arch’s size and shape. Once the ultrasound transducer 315 is correctly positioned and oriented on the patient 50 in the suprasternal window, the controller 410 will optionally autonomously reposition and sync with the current angle, orientation, or position of the ultrasound transducer 315 on the scanning surface 800 (see FIGURE 16B). The index marker 420 on the handle 414 will point downward towards the patient’s 50 left shoulder to mimic the position of the index marker on the ultrasound transducer 315.

[0100] It is to be appreciated that each of the TTE acoustic windows (PLAX, PSAX, apical, subcostal, suprasternal) can be selected and performed sequentially without needing the robotic unit 300 to return to its initial starting position upon completion of each window scan. For example, once the ultrasound transducer 315 obtains sufficient images in the PLAX view, the ultrasoundtransducer 315 can move or be moved to the position needed to obtain images in the PSAX view. It is to be appreciated that the TTE acoustic windows can be selected in any order.

[0101] FIGURE 17 is a flowchart of one non-limiting, exemplary method or process for using the system 10 to perform an imaging procedure, such as a full TTE ultrasound procedure, with a sonographer co-located with the patient or at a location remote from the patient. The method or process detailed in FIGURE 17 can be performed by one or more of the disclosed components of the ultrasound system 10. With reference to FIGURE 17, at step 900, the patient 50 is positioned on the examination platform 75 in a supine position, a left lateral decubitus position, or a right lateral decubitus position. The sonographer or sonographer assistant applies a gel or similar substance to the patient’s 50 chest where scanning will occur at step 905. At step 910, the sonographer selects, by way of the user interface 700, a desired TTE acoustic window for scanning (e.g., PLAX, PSAX, apical, subcostal, suprasternal). Once the desired acoustic window is chosen in step 910, the ultrasound transducer 315, which is mounted to the arm assembly 305 of the robotic unit 300, is positioned and oriented at or near the patient 50 in the selected acoustic window in step 915.

[0102] Step 915 can be performed manually, autonomously, or semi-autonomously. For manual positioning, the sonographer or sonographer assistant manually positions the robotic unit 300 with the ultrasound transducer 315 at the desired TTE acoustic window. This manual positioning can be performed by using one or more handles or flanges extending from the robotic unit 300 to move the robotic unit 300 and ultrasound transducer 315 into the correct position and orientation. In some embodiments, step 915 can be performed semi-autonomously using a pre-programmed approach position that is at a distance from the patient. The robotic arm 305 is moved autonomously using a pre-planned trajectory into the approach position, after which the operator switches the system 10 to admittance mode. In this mode, the operator may manipulate the controller to guide the transducer 315 movement as it is moved toward the patient in order to improve the starting location, while the robotic unit 300 moves toward the patient until experiencing a predetermined pressure level. During this mode in some embodiments, the controller 410 is continuously synched with the position of the transducer 315 during its approach to the target location while the operator is engaged with the controller 410, providing them withhaptic feedback during initial positioning and the ability to stop controller 410 or transducer 315 motion. Autonomous positioning is described in greater detail below with regard to FIGURE 18.

[0103] Still referring to FIGURE 17, once the ultrasound transducer 315 is positioned (located / oriented) on or near the patient 50 at the desired acoustic window, the controller 410 may optionally sync to the corresponding position (location / orientation) of the ultrasound transducer 315 at step 920. Syncing of the controller 410 with the ultrasound transducer 315 is described above with regard to FIGURES 12A-12B, 13A-13B, 14A-14B, 15A-15B, and 16A-16B. At step 925, the positioned ultrasound transducer 315 generates ultrasound image data of the patient 50 that are provided to ultrasound machine 600 for presentation to the sonographer via the one or more display screens 622, to the computer system 200, or a combination thereof. The sonographer can manipulate and move the handle 414 of the controller 410 along the scanning surface 800 at step 930 if, for example, additional or various views are needed within the selected TTE window. Any movement or manipulation of the controller 410 correspondingly re-positions (e.g., relocates, re-orients both) the ultrasound transducer 315 on or near the patient 50, in step 935. At step 940, the re-positioned ultrasound transducer 315 generates updated ultrasound image data of the patient 50 that then are provided to the computer system, the ultrasound machine 600 for presentation to the sonographer via the one or more display screens 622, or a combination thereof.

[0104] At decision step 945, the sonographer determines whether examination in the selected TTE acoustic window is complete. Examination can be complete, for example, when all necessary views in the selected acoustic window have been obtained or the generated images are of sufficient quality determined by the sonographer. If examination is incomplete, the system 10 loops back to step 930 for sonographer manipulation of the controller 410; if examination is complete, the method or process advances to decision step 950. At decision step 950, the sonographer determines whether additional TTE acoustic windows are to be evaluated. If no additional windows are needed, the process ends at step 955. However, if additional windows are to be evaluated, the process loops back to step 910, and the sonographer selects the additional TTE acoustic window for examination.

[0105] FIGURE 18 is a flowchart of an exemplary set of steps that may be performed with the steps of FIGURE 17 to autonomously position and orient the ultrasound transducer 315 on or near the patient 50 in a selected TTE acoustic window. At step 915a, the spatial registration system 340gathers data of the patient 50 at the TTE acoustic window via the one or more capture devices 342. The spatial data gathered by the spatial registration system 340 will be referred to as images; however, it is to be appreciated that the data could include any visual data, audio data, or frequency data. The images gathered in step 915a can optionally be resized and normalized for efficient operation of the computation model, such as, for example, taking all of the pixel values (e.g. 0- 255) and converting the pixels to a decimal point scale (e.g. 0-1).

[0106] At step 915b, the gathered images are applied to a machine learning model, expert system, or artificial intelligence, such as a convoluted neural network (CNN) (collectively referred to as the “pre-trained computation model”), and processed to identify defined anatomical landmarks for locating a desired initial position (i.e. a target location) for placing the ultrasound transducer 315 on or near the patient 50. The target location can be defined in part as a function of the type of ultrasound examination to be performed or the desired acoustic view. A training dataset used to train the computational model may include images of a patient or an inanimate patient model annotated or marked to identify keypoints or anatomical landmarks to be used for locating the desired target location on the patient 50. For example, ultraviolet (UV) markers and UV light may be used to mark the patient or inanimate patient model with the key anatomical landmarks. The images or training masks may be fed into the computation model for training. In one non-limiting embodiment, the CNN is an EfficientNet CNN. In another non-limiting embodiment, the CNN may be employed as a You Only Look Once (YOLO) CNN, such as YOLOv8. In another nonlimiting embodiment, the spatial registration system 340 is configured to track the motion of a patient using keypoints, optical flow techniques or the like, and the final optimal position of the transducer 315. Those data are used to back calculate from the optimal position of the transducer 315 one or more keypoints before the transducer was moved into the optimal position, taking into account patient motion data. In both non-limiting examples, the pre-trained computational model is trained on training data to predict a target location for use initial positioning of the transducer.

[0107] At step 915c, the target location on the patient 50 is identified with current (real-time) session data using the pre-trained computational model. The output of step 915c can comprise a pixel value that correlates with X,Y coordinates (i.e. 2-dimensional) of the location on the patient’ s body where the ultrasound transducer 315 is to be placed. The X,Y coordinates can be referenced to a coordinate system of the spatial registration system 340.

[0108] In one non-limiting example, the patient 50 can have their own coordinate system with a predefined origin location and coordinate system orientation, such as the suprasternal notch acting as the origin location (0,0) and where the length of the patient 50 is the positive Y-axis. The sonographer sees the patient 50, for example, through the live display 500 and defines the origin location on the patient 50 on an image obtained from the live display 500. It is to be appreciated that referencing the patient’s 50 coordinate system can also be done using computer vision algorithms that automatically detect desired keypoints on the patient 50.

[0109] At step 915d, a planned trajectory is determined for moving the ultrasound transducer 315 to the identified target location. This trajectory determination can be based on translation and mapping of the coordinate space of the images to a three-dimensional (3D) coordinate space of the area around the patient 50, the robotic unit 300, and the ultrasound transducer 315. For example, the depth (i.e. Z-coordinate) of the X,Y location identified at step 915c can be calculated, and a vector normal to the surface of the patient’s 50 body at the X,Y,Z location can be determined. The Z-coordinate can be derived from a depth image captured by the one or more capture devices 342 of the spatial registration system 340, and the calculation of the Z-coordinate can be determined by using averaged 3D depth data from a neighborhood of points in the image around the X,Y location. Depth determinations can be determined, for example, using data collected with 3D stereo, time of flight cameras, and the like. Optionally, a smoothing neural network may be applied. The plane used in determining the normal may be the plane defined by the Z-coordinate and the averaged 3D depth data. The X,Y,Z location is then converted into a 3D pose that is referenced to the coordinate frame of the arm assembly 305 of the robotic unit 300. The position / orientation of the spatial registration system 340 relative to the coordinate system of the of the arm assembly 305 is known, having an X,Y,Z point in space with a vector corresponding to the surface normal along the Z-axis. Thus, the X,Y location and Z vector values in the coordinate frame of the arm assembly 305 can be calculated using coordinate transformation mathematics. In a preferred embodiment, the planned trajectory is determined for moving the transducer 315 to an initial location that is offset from the target location or otherwise corresponds to the target location.

[0110] At step 915e, commands are generated for controllably articulating the robotic unit 300 to move the ultrasound transducer 315 to the identified target location based on the determined trajectory. The commands can include moving the arm assembly 305 and the ultrasound transducer315 to a location that is aligned along the normal, positioned just above the skin of the patient 50 at the X,Y,Z location without contacting the patient 50. The commands can further include articulating the arm assembly 305 to move the ultrasound transducer 315 along the normal toward the patient 50, while reading output data from the one or more sensors 330 to determine when the ultrasound transducer 315 has contacted the patient 50 at the target location.

[0111] The commands generated at step 915e are provided to the robotic unit 300 at step 915f. The commands can be provided to the robotic unit 300 via the communication network 100 or direct data connections. At step 915g, the robotic unit moves in response to the received commands to position the ultrasound transducer 315 at the identified target location on the patient 50.

[0112] For purposes of tracking the position of the ultrasound transducer 315 during examination, position data for the transducer 315 can be captured using a fiducial tracking system (e.g. optitrack fiducial tracking) where a set of markers are placed on the patient 50 and the transducer 315. As images are captured from the transducer 315, the location of the transducer 315 and the patient 50 are recorded and stored in association with each image, relative to one or more keypoints. Tracking can be performed, for example, through the spatial registration system 340 or the computer system 200. In some embodiments, tracking may be performed by collecting robotic unit 300 positional data by recording, for instance, positions of the robotic arm 305 joints during a scanning session. In further embodiments, the spatial registration system 340 data are combined with the positional data to calculate the transducer 315 position over time.

[0113] The processing and computation operations described in steps 915a-915g may be performed by the computer system 200 via execution of the code 222 by processor 210. However, it is to be appreciated that one or more of the operations may be performed by one or more of the onboard components of the robotic unit 300, or other components of the ultrasound system 10 suitable for this purpose without departing from the scope of the invention.

[0114] FIGURES 19A-19B provide a flow chart of another exemplary method or process for using the ultrasound system 10, with the sonographer co-located with the patient 50 or at a location remote from the patient 50. The method or process detailed in FIGURES 19A-19B can be performed by one or more of the disclosed components of the ultrasound system 10. The process detailed in FIGURES 19A-19B is a semi-autonomous process; however, it is to be appreciatedthat such process or steps can be completed autonomously, as explained in greater detail below. Steps 1000, 1005, 1010, 1015, 1020, 1025 have similar operations as steps 900, 905, 910, 915, 920, 925, respectively, as described above in FIGURE 17. At step 1000, the patient 50 is positioned on the examination platform 75 in a supine position, a left lateral decubitus position, or a right lateral decubitus position. The sonographer or sonographer assistant applies a gel or similar substance to the patient’s 50 chest where scanning will occur at step 1005. At step 1010, the sonographer selects, by way of the user interface 700, the desired TTE acoustic window for scanning. Once the desired acoustic window is selected in step 1010, the ultrasound transducer 315 is positioned and oriented at or near the patient 50 in the selected acoustic window in step 1015. Step 1015 can be performed manually, autonomously (as described above in steps 915a- 915g of FIGURE 18), or semi-autonomously. The controller 410 optionally but preferably syncs to the corresponding position of the ultrasound transducer 315 at step 1020. The synchronization process may include the use of a virtual offset between the position of the transducer 315 and the position of the controller 410 to improve ergonomic orientation and usability, absorb system variances, and improve operational characteristics. For example, during subcostal view imaging acquisition it has been found desirable to offset the controller 410 orientation to be further away from the scanning surface 800 than the transducer 315 orientation to the patient 50. Furthermore, small differences in orientation due to lower positional resolutions of the controller 410 with respect to the robotic unit 300 can be optionally but preferably compensated for via offset instead of alternatively synchronizing upon re-engagement with the controller 410 by the operator. Once the ultrasound transducer 315 contacts the patient 50 at the target location within the selected TTE window, the robotic unit 300 can begin following a predefined coordinate path based on the initial target location of the ultrasound transducer 315 in the TTE acoustic window.

[0115] At step 1025, the ultrasound transducer 315 generates ultrasound image data of the patient 50 that is provided to ultrasound machine 600 for presentation to the sonographer via the one or more display screens 622, the computer system 200, or a combination thereof, for analysis. At step 1030, the computer system 200 performs an automated analysis of the ultrasound image data using machine learning, expert system, or artificial intelligence, such as a CNN (collectively referred to as the “pre-trained computational model”), to identify one or more anatomical structures depicted in the ultrasound image data. The pre-trained computational model processes the ultrasound imagedata to segment anatomical regions depicted therein, and the segmented ultrasound images are scored for quality. The anatomical regions that the computation model is trained to detect and segment (via pictures, expert input, etc.) can vary as a function of the type of examination that is being performed, the specific imaging view being acquired, (e g., for a selected TTE acoustic window), or combinations thereof. For example, if the ultrasound transducer 315 is generating images in the PLAX view, the computational model can be trained to detect and segment the left ventricle, the aorta, the right ventricular outflow tract, and the left atrium. In another non-limiting example, for a subcostal view, the computational model can be trained to detect and segment the right ventricle, the right atrium, left ventricle, and left atrium. In a preferred embodiment, models may be trained separately corresponding to particular imaging views.

[0116] In one non-limiting embodiment, the computational model can be trained to receive ultrasound images, from the ultrasound transducer 315, as inputs and produce segmentation masks for regions of the ultrasound images with known anatomical segments. The ultrasound images are saved and stored during the full TTE examination. Masks corresponding to different mask labels can then be created on the ultrasound images to distinguish which anatomical regions of the images correspond to each mask label. The ultrasound images can optionally be augmented through processes of adjusting brightness and definition before being fed to the computational model.

[0117] Still referring to step 1030, the segmented anatomical regions can be totaled, or otherwise combined or aggregated, to score a quality of the generated ultrasound images and whether the anatomical regions identified correspond to the desired anatomical structure views. This scoring can be used to govern decision-making regarding positioning and orientation of the ultrasound transducer 315. The scoring can employ any scoring mechanism suitable for quantifying image quality, now known or later developed. In one non-limiting example, when using segmentation masks, the computational model can calculate the area of the masks and use the area versus a ground truth to determine a score. For each view to be evaluated, a ground truth can be established using an ideal image or set of ideal images. The computational model can be trained to compare an acquired image with the ground truth for each view. The greater or higher the degree of match between the segmented acquired image and the ground truth, the closer the score is to a maximum score. By way of a non-limiting example, a per-pixel accuracy can be used for training against the training data set, and the same can be applied to the ground truth for the segmentation technique.For the image classification technique, a dataset of images can be created, each with an assigned quality score indicating how close an image is to ideal. The computational model is trained on these images and are validated when the output score matches the manually applied score.

[0118] In some embodiments, the quality score may be optionally provided by predicting a distance from an optimal transducer position predicted for the current position of the transducer 315. Motion capture and computer vision inputs may be used in these embodiments to label ideal or optimal images and corresponding ideal or optimal transducer position characteristics (e.g., X,Y,Z,r,p,y). Those optimal image characteristics are used by a model to calculate a score that is based on the distance from the final motion frame at the optimal image location. In a preferred embodiment this score is a negative distance value and lower scores correspond to lower quality predictions. In these embodiments, the quality score is a function of a difference in position between the transducer at the time of measurement and the optimal transducer position. These quality scores may be calculated during flight with real-time image capture data received from the spatial registration system, for instance.

[0119] In one non-limiting embodiment, the pre-trained computational model can use a U-Net or Dense U-Net architecture to perform image segmentation for identifying the various anatomical regions. The starting U-Net can have no weights, but it is to be appreciated that a model for biomedical image segmentation that has good transfer learning properties could be used. In one or more non-limiting embodiments, the pre-trained computational model can be a deep CNN such as the VGG-16 CNN architecture model. The CNN model can be trained on training image datasets having labeled anatomical regions to recognize such anatomical regions in non-invasive images generated by a transducer, such as ultrasound images generated by the ultrasound transducer 315.

[0120] At step 1035, a determination is made whether the generated, analyzed ultrasound images obtained a maximum score. If a maximum score has been obtained, the robotic unit 300 and ultrasound transducer 315 maintain their current position at step 1040 and continue to generate additional ultrasound images. At step 1045, a determination is made whether additional TTE acoustic windows are to be evaluated. If yes, the process loops back to step 1010 and the sonographer selects the additional TTE views; if no, the process ends at step 1050.

[0121] Still referring to step 1030, if the ultrasound images have not obtained the maximum score, the process progress to decision step 1055, which determines if the sonographer is to intervene with manual manipulation of the controller 400. If the sonographer does wish to intervene, the sonographer can manipulate the controller 400 along the scanning surface 800 at step 1060 to obtain the desired position or orientation of the ultrasound transducer 315. The process then loops back to step 1025 and new ultrasound images are generated and presented to the sonographer or the computer system 200. If the sonographer does not wish to intervene at step 1055, the ultrasound images are fed to a computational model, such as a time-dependent network, temporal neural network, or Long Short-Term Memory (LSTM) network, at step 1065 which analyzes the image data sets to determine an adjusted trajectory, position, location, or orientation of the ultrasound transducer 315 that are predicted to increase the image score. The image data sets may be a timeseries of a recent window of the ultrasound images and corresponding transducer 315 pose data for the image embeddings. The corresponding transducer 315 pose data can identify the position of the arm assembly 305 or transducer 315 when the corresponding ultrasound image within the time-series was generated. The current ultrasound image and its corresponding transducer 315 post data can be fed to the computational model that has been trained to output a direction of travel for the ultrasound transducer 315 that is estimated or predicted to improve (i.e. optimally improve) the image score.

[0122] Once the adjusted trajectory is determined at step 1065, commands are generated and sent to the robotic unit 300 to move the arm assembly 305 and ultrasound transducer 315 along the adjusted trajectory at step 1070. Thereafter, the process loops back to step 1055 to generate new ultrasound images that can be subsequently analyzed for score improvements.

[0123] In one non-limiting embodiment, the process or method illustrated in FIGURES 19A-19B can optionally include a time-determination step for analyzing the amount of time that has passed since the ultrasound image score has increased or improved. The defined amount of time may be represented by a threshold number of units of time (e.g. X seconds), where the threshold is selected to be any value sufficient to define a tolerance to be given to the system for attempting to find an incremental adjustment of the trajectory that is expected to improve the image score. If a determination is made that too much has passed since score improvement with respect to the threshold period, the computer system 200 can compute a new trajectory for positioning theultrasound transducer 315. If the score has improved within the defined amount of time, then the ultrasound images can be fed to the computational model such as in step 1065 to produce the adjusted trajectory.

[0124] It is to be appreciated that the process or methodology described above in FIGURES 19A- 19B may be autonomously performed with sonographer, or another trained individual, supervision. For example, the sonographer may select the TTE examination (or another ultrasound procedure) to begin at step 1010, and the system 10 may autonomously conduct a full TTE examination on the patient 50. The system 10 may autonomously: (i) position the ultrasound transducer 315 on or near the patient 50 in a first TTE acoustic window; (ii) generate ultrasound images; (Hi) analyze the images; (iv) adjust the position of the ultrasound transducer 315 to obtain better quality images; and (v) repeat scanning, analysis, and corrections for the additional TTE acoustic windows. The sonographer or trained individual, co-located with the patient or remote, may monitor the autonomous process via the live display 500 or the ultrasound machine 600, for instance, and may intervene by way of the controller 400 if needed.

[0125] The processing and computation operations described in steps 1000-1070 may be performed by the computer system 200 via execution of the code 222 by processor 210; however, it is to be appreciated that one or more of the operations may be performed by one or more of the onboard components of the robotic unit 300, or other components of the ultrasound system 10.

[0126] FIGURES 20A-20D depict an exemplary process for training the computational model discussed in step 1065 of FIGURE 19B for producing the adjusted trajectory of the robotic unit 300 to improve image scores. Training of the computational model can include various phases of operation that include: (i) expert demonstration collection, where the ultrasound images and ultrasound transducer pose settings for the trajectory of an exemplary ultrasound examination are acquired; (ii) data pre-processing, where the paired ultrasound images and transducer pose data are processed in preparation of anatomical feature extraction by a computational model such as a CNN; (Hi) anatomical feature extraction using the computational model, where the computational model procedures a feature vector from one of the last layers of the model to serve as a dense representation of the ultrasound image that captures relevant anatomical features for automated image analysis; (iv) data sequencing, where a sliding window sequence of ultrasound image frames is combined with normal pose data for the ultrasound transducer and / or arm of the robotic unit,along with velocities and accelerations at each timestamp associated with the image frames; and / or (v) a time-dependent network, where the sliding window of paired image data and pose data is processed to produce an estimate or prediction of the pose of the ultrasound transducer and / or arm of the robotic unit to improve image quality.

[0127] With reference to FIGURES 20A-20D, training may be premised on the acquisition of one or more sets of training data that serve as a ground truth, where one or more users who serve as expert demonstrators (e.g. sonographer, sonography experts, etc.) perform one or more ultrasound examinations on one or more patients (step 1100). In a preferred embodiment, this step may be implemented by the performance of repeated discrete steps or actions that would be performed during a complete imaging examination, such as moving the transducer from random points around a patient's chest to the final location repeatedly, with each attempt recorded as a separate trajectory. To obtain the one or more sets of training data, a motion capture system may be used to track an ultrasound transducer being moved during an ultrasound examination, resulting in 3D location data of the ultrasound transducer and the patient (step 1105). For example, motion capture may be performed at a location of the patient and the transducer pose in 3D space relative to the center location of the patient (0,0), positional data may be collected from the robotic unit, or a combination thereof. At step 1110, the images acquired by the ultrasound transducer are saved and assigned a timestamp that corresponds to a timestamp of the motion capture data. The timestamps for the ultrasound images are synchronized with the transducer and patient pose data during the full ultrasound examination, which may serve as a ground truth for training the computational model. Steps 1100, 1105, 1110 may be repeated using a diverse patient population and varied expertise.

[0128] At step 1115, each ultrasound image from the examination is downsized and normalized to improve system speed. Image compression can be used to reduce the size of the images (for example, a reduction to a size of 512 x 512 pixels). Computational models generally expect a standard input size and will perform better on images that are smaller and contained less information. Larger images typically require larger training models, more training data, and take longer to process. At step 1120, the downsized / normalized images are transformed, such as, for example, by image cropping. The ultrasound images may include regions or spaces with irrelevantinformation. Cropping such regions or spaces from the ultrasound images improves the overall speed and performance of the computational model.

[0129] The transformed images are then fed to the pre-trained model, such as a CNN (see FIGURE 20B), which is used to identify anatomical regions depicted in the images (step 1125). The CNN used in step 1125 may be the same CNN used in step 915b of FIGURE 18 or in step 1030 of FIGURE 19B. Embeddings from the last pooling layer in the CNN of step 1125 are extracted from the convolve layer (step 1130), and the extracted embeddings are combined with the pose data in a frame at the time when the ultrasound image was acquired (step 1135). The frame is then windowed with a set of previously collected frames to form a data set with a timeseries component (step 1140). In one non-limiting example, the time-series data set formed at step 1140 may cover a time window that includes at least one cardiac cycle for a patient.

[0130] At step 1145, a computational model, such as a time-dependent neural network (see FIGURE 20C), can be trained on the time-series data set formed in step 1140. In the present nonlimiting embodiment, the time-dependent neural network is an LSTM network having an input layer, a plurality of LSTM layers, and an output layer (see FIGURE 20D). The output layer may provide regression using at least 6 neurons operating on the at least 6 degrees of freedom coordinates (X,Y,Z, r, p, y), with tanh or sigmoid activation functions for 0-1. It is to be appreciated that admittance control (described in greater detail below) may be used instead of calculating the depth / height value (e.g. Z-axis), thereby reducing the size of the LSTM network for improved speed and efficiency. Once in operation, the trained LSTM network can compare an actual timeseries trajectory of a window of synchronized pose data with the trajectory for the training data set to generate a regression estimation of an appropriate pose for the ultrasound transducer or robotic arm to acquire improved or optimal ultrasound views (step 1150). Onboard computer system 320 of the robotic unit 300 operates on the output from the time-dependent network (where the output may be expressed as Cartesian commands) to translate this output into real-time commands for controlled articulation of the arm assembly 305 or ultrasound transducer 315.

[0131] FIGURE 21 is a flowchart of an exemplary set of steps that may be performed with the system 10 to provide admittance control to the robotic unit 300. With reference to FIGURE 21, the ultrasound system 10 can include admittance control that utilizes a motion control strategy to provide constant, safe, and correct pressure between the ultrasound transducer 315 and the surfaceof the patient’s 50 body during an imaging procedure. The force / pressure needed to maintain constant contact between the transducer 315 and the patient 50 may change during examination based on one or more of: (i) the area of the patient 50 being imaged; (ii) body shape, size, or other physical characteristics of the patient 50; (Hi) patient 50 breathing; or (iv) perturbations in patient position or other such movements. The disclosed admittance control model guides the arm assembly 305 so that ultrasound transducer 315 behaves like a spring that can be damped and stiffened accordingly to achieve a desired vector force output. As can be seen in Equation 1 below, the admittance control model includes the difference between desired and current positions of the ultrasound transducer 315 along a relevant dimension (x-dimension, in this non-limiting example) and models acceleration along the relevant dimension as a contact force (F) for the transducer 315 relative to the patient 50 minus a damping factor multiplied by a velocity along the relevant dimension minus a stiffness factor multiplied by the difference between the desired and current positions along the relevant dimension, all divided by a mass value.Equation 1:

[0132] The mass value is derived from a spring equation that evaluates the effects of the spring on a specific mass. In the present non-limiting embodiment, the admittance control is tuned to mimic the behavior of a system in response to a mass, such a predetermined mass of the ultrasound transducer 315 connected to the transducer mount 310 or a scaling factor for reactivity. The mass value is a tuned parameter that establishes desired performance. The present admittance control model is shown for the x-dimension, but it to be appreciated that the same model can be applied to the other movement dimensions. The damping and stiffness factors may be adjusted to adapt to the movement of the ultrasound transducer 315 in response to patient 50 breathing, shifting, or other movement.

[0133] As the ultrasound transducer 315 moves along the surface of the patient 50 during TTE examination (either manually, autonomously, or semi-autonomously), the one or more sensors 330 of the robotic unit 300 gather force and torque data arising from articulation of the arm assembly 305 and movement of the transducer 315. The force and torque data can be fed to the computer system 200 or the onboard computer system 320 to provide admittance control over the motion ofthe arm assembly 305 and transducer 315. It is to be appreciated that any processor of the system 10 suitable for such purpose may operate on the force and torque data to provide admittance control.

[0134] Still referring to FIGURE 21, admittance controller 1228 may be fed with input data from motion planning sources (1202 or 1204) and feedback input data the one or more sensors 330 (step 1214). For example, inputs from the autonomously controlled robotic unit 300 via a neural network as discussed herein (step 1204) or from the controller system 400 (step 1202) may provide movement guidance (e.g. twist commands 1206, 1210) to Cartesian planning step 1208. Modelbased feedback is preferred but optional, as the disclosed invention may be implemented with admittance control based purely on operator controller input and error correction as described further herein below.

[0135] At step 1208, the movement guidance can be translated via real-time Cartesian planning into a desired velocity 1212 for the ultrasound transducer 315 being moved by arm assembly 305. For example, step 1208 may evaluate the current position of the arm assembly 305 in Cartesian space using forward kinematics (e.g. examining joint angles and link lengths) and apply the desired twist to the current position to obtain translated Cartesian velocities (X,Y,Z). As such, the operations at step 1208 may translate the twist commands (1206, 1210) into the desired velocity 1212 using inverse kinematic calculations. The admittance controller 1228 then outputs to low- level joint position and velocity controllers to impart articulation of the arm assembly 305.

[0136] At step 1214, the input data generated from the one or more sensors 330 along feedback path 1248 can be configured as sensed wrench data 1216. The wrench data 1216 can comprise liner force or angular torque along all axes that are applied to the robotic unit 300. At step 1218, filtering and gravity compensation can be applied to the wrench data 1216 to generate refined wrench signals 1220.

[0137] In one non-limiting example in which the transducer 315 is not uniform or integrated in the robotic unit 300, at the beginning of the imaging procedure, it can be positioned for calibration for example by pointing downward towards the ground, perpendicular to the examination platform 75, at a distance away from the patient 50. The system 10 can capture raw force / torque output from the one or more sensors 330 at this position over a period of time, for example one second. Thiscaptured output over the period of time can be averaged to become the baseline for the zero force input, with only the measured weight of the transducer 315. Additionally, the calculated value of the mass and location of the center of mass (COM) of the transducer 315 can be predetermined. Subtracting the mass / location value from the calibration value of the sensors 330 at each time step provides the external forces measured by the sensors 330. It is to be appreciated that the transducer 315 pose can be taken into account by subtracting the values from the calibrated values in all directions. In this manner, the system 10 is configured to easily accommodate a variety of transducers 315 from various manufacturers.

[0138] Still referring to FIGURE 21, a control loop mechanism such as a proportional integral derivative (PID) controller 1222 may compute a force error value 1226 in the linear (X,Y,Z) and angular (p,r,y) dimensions from the refined wrench value 1220. The PID controller 1222 may also compute the force error value 1226 from a force set point 1224 that has been preset or defined by the operator during the procedure. The force error value 1226 may represent a compensation that is needed for the sensed force data to between approximate the desired force set point 1224. As such, if the set point 1224 for the force is 2 Newtons (N) and the PID controller reads a sensed force 1220 of IN, the error value 1226 would be IN.

[0139] Once the admittance controller 1228 receives the desired velocity 1212 and error value 1226, the inputs are modeled as outlined above with respect to Equation 1, and produces twist commands (e.g. velocities) 1230 for the arm assembly 305 in the at least six degrees of freedom, such as (X,Y,Z,p,r,y). The admittance controller 1228 may operate to prevent the transducer 315 from pressing too hard into patient 50 or inadvertently separating from the patient 50 during an imaging procedure. In one non-limiting example, if the neural network 1204 produced twist commands 1210 that cause the transducer 315 to press too hard into the patient 50, the admittance controller 1228 would intercept the commands 1210, read the actual forces, and determine the actual twist output in order safely provide contact between transducer 315 and patient 50. The admittance controller 1228 may be implemented as a script that runs on the computer system 320 of the robotic unit 300. Such co-location of the admittance controller 1228 with the robotic unit 300 can improve performance and responsiveness of the arm assembly 305 and consequently the ultrasound transducer 315 to admittance control. The admittance controller 1228 may run every loop so that close control can be exercised over movement of the robotic unit 300 components. Itis to be appreciated that the admittance controller 1228 may be incorporated into the computer system 200 or other components of the system 10.

[0140] At step 1232, the output from the admittance controller 1228 is converted into a set of joint angle data 1234 that governs movement of the arm assembly 305 to a desired pose. This conversion from a Cartesian input to joint angles may be accomplished by way of an inverse kinematic calculation of a desired position for the transducer 315 to the joint angles that are required by the arm assembly 305 to established said position of the transducer 315. At step 1236, the computer system 200 or computer system 320 of the robotic unit 300 apply jerk limits, acceleration limits, or velocity limits to articulation of the arm assembly 305 via commands to the robotic unit hardware at step 1238. Meanwhile, the ultrasound transducer 315 will move correspondingly with the arm assembly 305 and will generate ultrasound images 1240 that are processed using a feedback loop through the neural network 1204 to continue controlling movement of the arm assembly 305. The feedback processing may include: (i) image analysis at step 1242 that produces an image score 1246, as previously discussed; or (ii) auto-encoding at step 1244 that produces encoded images features 1248 for use by the neural network at step 1204, as previously discussed. The neural network 1204 can then output twist commands 1210 in (X,Y,Z,p,r,y) dimensions based on the image score 1246 or encoded image features 1248, as previously discussed.

[0141] FIGURE 22A is an exemplary, non-limiting operational flow diagram for the ultrasound system 10. FIGURES 22B-22C depict detailed views of the operational stages of FIGURE 22A, wherein FIGURE 22B is a detailed view of the input stage, the data preprocessing stage, the feature extraction stage, and the core processing stage, and wherein FIGURE 22C is a detailed view of the decision making and action handling and output states. With reference to FIGURES 22A-22C, the exemplary operational flow of the ultrasound system 10 can include: (i) an input stage 1300; (ii) a data preprocessing stage 1310 performed subsequent to the input stage 1300; (Hi) a feature extraction stage 1320 performed subsequent to the data preprocessing stage 1310; (iv) a core processing stage 1330 performed subsequent to the feature extraction stage 1320; (v) a decision making stage 1340 performed subsequent to the core processing stage 1330; and (vi) an action handling and output stage 1350 performed subsequent to the decision making stage 1340. The operational flow may further comprise a feedback loop 1360, a visualization and monitoring component 1370, or a maintenance and learning component 1380. Any of the disclosedcomponents and techniques with regard to the ultrasound system 10 may be used in any of these operational stages.

[0142] At the input stage 1300, the following data may be input to the system 10: ultrasound image data 1300a; spatial registration system 340 data 1300b; sensor 330 data 1300c; arm assembly 305 joint states 1300d; or controller system 400 data 1300e. During the data preprocessing stage 1310, signal / image conditioning of data 1300a, 1300b, 1300c is performed at step 1312 for noise reduction, normalization, or data synchronization. Data 1300d and the output of the signal / image conditioning of step 1312 are transferred to sensor fusion 1314. The controller system 400 data 1300e is smoothed and corrected for errors at step 1316. Data preprocessing may utilize wavelet transforms or other advanced signal processing techniques to enhance the feature extraction stage before feeding data to the core processing / decision making stages.

[0143] During the feature extraction stage 1320, a pre-trained computation model (such as CNN) is used to segment anatomical features (step 1322) from the output of the signal / image conditioning of step 1312. The output of the sensor fusion step 1314 and the segmented anatomical features 1322 are provided to step 1324 for windowing and feature evaluation. During the core processing stage 1330, data from 1324 is fed to: a deep learning neutral network 1332 (e.g. LSTM, GRU, etc.); a transformer module 1334 (e.g. self-attention mechanism, pre-trained transducer model for time series, etc.) or a classical machine learning module 1336 (e g. SVM, Random Forest, or the like for pattern recognition). Dynamic time wrapping (DTW) or similar algorithms may be used to align temporal sequences of varying lengths to enhance the ability of the temporal sequence analysis to learn from the data.

[0144] At the decision making stage 1340, a computation model with pre-trained decision algorithms 1342 (e.g. Q-learning, DQN, policy gradient methods, variational encoders, sparse encoders, etc.) receives output from the deep learning neutral network 1332, the transformer module 1334, the classical machine learning module 1336, or the smoothed and corrected data of step 1316. Output from the computation model 1342 is fed to an action planning step 1352 (e.g. inverse kinematics solvers and trajectory planning) of the action handling and output stage 1350. The smoothed and corrected data of step 1316 is also fed to the action planning step 1352. The system 10 then: (i) filters outputs, performs safety checks, and complies with operational constraints at step 1354; (ii) generates actuation commands for post processing operations at step1356; and (Hi) provides the actuation commands to the robotic unit 300 at step 1358 for movement and positioning of the arm assembly 305.

[0145] The feedback loop 1360 may provide real-time tracking of the performance of system 10 and provide any adjustments based on feedback. The visualization and monitoring component 1370 may provide the sonographer with system 10 performance indicators such as system alerts for anomalies or errors. The visualization and monitoring component 1370 may include the user interface 700, the live display 500, or other components of the system 10. The maintenance and learning component 1380 provides the system 10 with updates, learning, and retraining from new data to improve performance.

[0146] In certain implementations, the system 10 may employ advanced artificial intelligence models for autonomous control of the transducer 315 during diagnostic procedures. These technique embodiments utilize models trained to receive, interpret, and act upon multi-modal data sources such as image data streams, joint positions of the robotic arm assembly 305 or controller system 400, pose encodings, real-time force-torque measurements, and other variables upon which non-invasive medical imaging quality may be dependent. In one exemplary embodiment, for instance, the system 10 may utilize action chunking, a control strategy in which longer robotic tasks are segmented into discrete, interpretable units of behavior referred to as "action chunks." Each action chunk corresponds to a clinically meaningful phase of a scanning routine, such as initiating a parasternal sweep, adjusting tilt near the apex, or applying pressure in a subcostal view.

[0147] An exemplary implementation of action chunking techniques according to the present invention is depicted in connection with a schematic diagram of chunking architecture shown in FIGURE 23. In this embodiment, at least one input variable 1402 upon which non-invasive medical imaging quality is dependent is provided, preferably as time-series datasets, such as ultrasound image streams 1404, pose data 1406, or force-torque data 1408. Where a plurality of input variables or data types are used the inputs may first be processed by a multi-modal encoder 1410 to fuse image features with kinematic and tactile sensor data.

[0148] In some embodiments, the fused state representation from the encoder 1410 is be passed to a transformer-based neural network 1420 configured to model temporal sequences of input data and identify transitions between chunks. These models leverage self-attention mechanisms toevaluate and prioritize prior time steps in relation to the current scanning context, enabling more efficient and interpretable planning of robotic motion. For example, a preferred embodiment implements a time-series-based network (e.g., LSTM) or transformer model to guide action prediction over time. In some cases, meta-learning or self-supervised learning strategies may be employed to pre-train feature encoders on large unlabeled datasets or enable rapid adaptation to new patients, probe types, or anatomical configurations.

[0149] In some embodiments, action chunking is provided as an optional solution wherein the system incorporates a policy selector 1430 such as a policy network trained via reinforcement learning to select among available control options (e.g., action chunks, velocity vectors, pressure adjustments) to maximize a given reward function. Reward signals may include, without limitation: image quality scores, keypoint visibility, transducer stability, or real-time user feedback. Alternatively, or in combination with reinforcement learning, the system may use inverse reinforcement learning (IRL) methods to infer an implicit reward function from expert demonstrations. These demonstrations may include paired data comprising transducer trajectory, contact force, and ultrasound image quality during human-in-the-loop procedures.

[0150] The output of an action chunking transformer may be a decoder 1425 that produces a discrete latent code, control token, or action label that informs downstream decision layers responsible for robotic motion planning or admittance control via a command generator 1440, wherein generated commands are transmitted to the robotic unit 1450 to articulate the robotic arm assembly thereby imparting transducer motion at 1460. By organizing scanning behaviors into structured units, the system can better generalize to new patient anatomies, recover from deviations, and improve robustness over time.

[0151] In some configurations, the system may also include or interface with a generative model to simulate diverse ultrasound imaging conditions or augment training data when patient samples are limited. Such models may include, for instance, a diffusion model, variational autoencoder (VAE), generative adversarial network (GAN), or other suitable alternatives now known or later developed as will be appreciated by those skilled in the art.

[0152] FIGURE 24 provides a block diagram of another example implementation of the disclosed ultrasound system showing the basic components of the system. The ultrasound system 2000includes the following components that when used together assist a sonographer or similar ultrasound technician in acquiring high-quality ultrasound images of a plurality of patients undergoing a full, standard TTE examination. The ultrasound system 2000 includes a first bedside system 2010 and a second bedside system 2020. The first bedside system 2010 includes a patient 2012, an examination platform 2014, and a robotic unit 2016. The second bedside system 2020 includes a patient 2022, an examination platform 2024, and a robotic unit 2026. The ultrasound system 2000 further includes one or more communication networks 2030; a computer system 2040; a controller system 2050; a live display 2060; an ultrasound machine 2070; and a user interface 2080. The first bedside system 2010 is at a remote location from the second bedside system 2020. The first bedside system 2010, the second bedside system 2020, or both may be at a location remote from the computer system 2040. The controller system 2050, the live display 2060, the ultrasound machine 2070, and the user interface 2080.

[0153] The ultrasound system 2000 includes similar components that function similarly to those features and operational techniques previously described with reference to the ultrasound system 10, the difference being that ultrasound system 2000 allows a sonographer or trained individual to perform or monitor a full TTE examination, or other ultrasound procedure, on a plurality of patients at once while being at a remote location from one or more of the patients. The ultrasound examination can be performed on the plurality of patients in an autonomous or semi-autonomous manner. It is to be appreciated that all or some of the operations, techniques, and procedures described previously with regard to system 10 can be applied to like components of the ultrasound system 2000.

[0154] While the present disclosure describes the use of trained CNN for anatomical feature extraction and segmentation in ultrasound images, it is to be appreciated that other forms of artificial intelligence and machine learning models could be used. For example, capsule networks may be utilized to preserve spatial hierarches between features which can offer robust image interpretation in scenarios with varying orientations and positions of the ultrasound transducer. Alternatively, other generative models suitable augmenting image data with synthetic but realistic ultrasound images may be deployed (e.g., diffusion models, deep generative models, VAE models, GANs and the like), which may provide robustness of the feature extraction phase against uncommon or noisy data.

[0155] While the present disclosure describes the use of LSTM networks for temporal sequence analysis, it is to be appreciated that other forms of time dependent networks for temporal sequence analysis may be used. For example, gated recurrent units (GRUs) may be used as an efficient alternative to LSTM networks as GRUs reduce computational requirements while maintaining similar performance characteristics. Additionally, transformer networks may also be used for processing temporal sequences as such networks provide robust performance in terms of parallelizability and handling of long-range dependencies.

[0156] As a general alternative to an Al framework, the disclosed systems may employ reinforcement learning (RL) and / or hybrid models. An RL-based system may be designed to learn control policies directly from interaction with the environment, aiming to optimize a reward function that encapsulates the quality of ultrasound imaging. A hybrid approach can combine supervised learning for feature extraction with RL for sequence generation and control, which may yield improved results by leveraging the strengths of both methodologies.

[0157] Having shown and described a preferred embodiment of the invention, those skilled in the art will realize that many variations and modifications may be made to affect the described invention and still be within the scope of the claimed invention. Additionally, many of the elements indicated above may be altered or replaced by different elements which will provide the same result and fall within the spirit of the claimed invention. It is the intention, therefore, to limit the invention only as indicated by the scope of the claim.

[0158] All literature and similar material cited in this application, including, but not limited to, patents, patent applications, articles, books, treatises, and web pages, regardless of the format of such literature and similar materials, are expressly incorporated by reference in their entirety. Should one or more of the incorporated references and similar materials differs from or contradicts this application, including but not limited to defined terms, term usage, described techniques, or the like, this application controls.

[0159] As previously stated and as used herein, the singular forms “a”, “an”, and “the” refer to both the singular as well as plural, unless the context clearly indicates otherwise. The term “comprising” as used herein is synonymous with “including”, “containing” or “characterized by” and is inclusive or open-ended and does not exclude additional, unrecited elements or methodsteps. Although many methods and materials similar or equivalent to those described herein can be used, particular suitable methods and materials are described herein. Unless context indicates otherwise, the recitations of numerical ranges by endpoints include all numbers subsumed within that range. Furthermore, references to “one implementation” are not intended to be interpreted as excluding the existence of additional implementations that also incorporate the recited features. Moreover, unless explicitly stated to the contrary, implementations “comprising” or “having” an element or a plurality of elements having a particular property may include additional elements whether or not they have that property.

[0160] The terms “substantially” and “about” describe and account for small fluctuations, such as due to variations in processing or operational ranges that are evident from the disclosure to those skilled in the art, for instance. For example, these terms can refer to less than or equal to ±5%, such as less than or equal to ±2%, such as less than or equal to ±1%, such as less than or equal to ±0.5%, such as less than or equal to ±0.2%, such as less than or equal to ±0.1%, such as less than or equal to ±0.05%, or 0%.

[0161] Underlined or italicized headings and subheadings are used for convenience only, do not limit the disclosed subject matter, and are not referred to in connection with the interpretation of the description of the disclosed subject matter. All structural and functional equivalents to the elements of the various implementations described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and intended to be encompassed by the disclosed subject matter. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the above description.

[0162] There may be many alternate ways to implement the disclosed technology. Various functions and elements described herein may be partitioned differently from those shown without departing from the scope of the disclosed technology. Generic principles defined herein may be applied to other implementations. Different numbers of a given module or unit may be employed, a different type or types of a given module or unit may be employed, a given module or unit may be added, or a given module or unit may be omitted.

[0163] Regarding this disclosure, the term “a plurality of’ refers to two or more than two. Unless otherwise clearly defined, orientation or positional relations indicated by terms such as “upper” and “lower” are based on the orientation or positional relations as shown in the Figures, only for facilitating description of the disclosed technology and simplifying the description, rather than indicating or implying that the referred devices or elements must be in a particular orientation or constructed or operated in the particular orientation, and therefore they should not be construed as limiting the disclosed technology. The terms “connected”, “mounted”, “fixed”, etc. should be understood in a broad sense. For example, “connected” may be a fixed connection, a detachable connection, or an integral connection, a direct connection, or an indirect connection through an intermediate medium. For an ordinary skilled in the art, the specific meaning of the above terms in the disclosed technology may be understood according to specific circumstances.

[0164] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail herein (provided such concepts are not mutually inconsistent) are contemplated as being part of the disclosed technology. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the technology disclosed herein. While the disclosed technology has been illustrated by the description of example implementations, and while the example implementations have been described in certain detail, there is no intention to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and modifications will readily appear to those skilled in the art. Therefore, the disclosed technology in its broader aspects is not limited to any of the specific details, representative devices and methods, and / or illustrative examples shown and described. Accordingly, departures may be made from such details without departing from the spirit or scope of the general inventive concept.

Claims

CLAIMSWhat is claimed:

1. A system for conducting an imaging examination on a living being in an imaging environment, comprising: a transducer configured to capture image data, wherein the transducer is coupled to an arm assembly of a robotic unit; and a pre-trained computational model executed by one or more processors, wherein the one or more processors are configured to: initiate the imaging examination in response to receiving a command from an operator; and by way of the pre-trained computational model, predicting a target location on the living being to position the transducer for a first viewpoint of the imaging examination; and provide a set of movement commands to the robotic unit to place the transducer in an initial position at the target location.

2. The system of claim 1, further comprising: a spatial registration system having a capture device configured to gather real-time spatial data from the imaging environment, wherein the one or more processors are further configured to: receive the real-time spatial data, and in response, resize and normalize the visual data; apply the resized and normalized visual data to the pre-trained computational model, and in response, determine an X,Y coordinate location that corresponds to the target location; and determine a depth coordinate with respect to the X,Y coordinate location such that an X,Y,Z coordinate location corresponding to the target location is defined.

3. The system of claim 1, wherein the one or more processors are further configured to, by way of the pre-trained computational model: receive a set of image data from the transducer at the target location; generate a quality score derived from the set of image data; and based on the quality score, generate a set of updated movement commands for the robotic unit that are predicted to increase the quality score.

4. The system of claim 3, wherein the one or more processors are further configured to, by way of the pre-trained computational model: identify a segmentation mask for one or more anatomical structures depicted in the set of image data; calculate an area within the segmentation mask; and compute the quality score based on the calculated area and a ground truth segmentation mask area.

5. The system of claim 1, wherein the computational model is trained based on one or more training datasets comprising images previously captured during an imagining examination performed by an expert demonstrator.

6. The system of claim 1 , further comprising one or more sensors configured to generate realtime force data arising from the movement of the transducer along the living being, wherein the one or more processors are further configured to apply admittance control over the movement of the transducer based on the real-time force data to ensure constant contact between the transducer and the living being during the imaging examination.

7. The system of claim 6, wherein the admittance control models the movement of the transducer as a spring using a damping factor and a stiffness factor in combination with the real-time force data.

8. The system of claim 1, wherein the one or more processors are further configured to, by way of the pre-trained computational model identify a second target location on the livingbeing to position the transducer for a second viewpoint of the imaging examination, and in response, provide a set of movement commands to the robotic unit to move the transducer at the second target location.

9. The system of claim 1, further comprising a controller system in communication with the robotic unit, wherein the controller system includes a controller that is movable by the operator to correspondingly move the arm assembly with the transducer, wherein the controller is movable along a curved scanning surface.

10. The system of claim 9, wherein the one or more processors are further configured to autonomously position the controller at a location on the curved scanning surface that corresponds to the initial position of the transducer.

11. The system of claim 10, wherein the controller comprises: one or more controls that, when actuated, allow the operator to move the controller along the curved scanning surface; and an index marker that facilitates orientation and provides the operator with the position of the transducer relative to that of the controller.

12. The system of claim 1 , wherein the imaging examination is a transthoracic echocardiogram (TTE) procedure comprising a plurality of viewpoints, wherein the plurality of viewpoints include a parasternal long axis (PLAX) view, a parasternal short axis (PSAX) view, an apical view, a subcostal view, and a suprasternal notch view.

13. The system of claim 1, wherein the operator is remote from both the robotic unit and the living being undergoing the imaging examination.

14. The system of claim 1, further comprising: a spatial registration system having a capture device configured to gather real-time visual data of the living being, the transducer, and the arm assembly of the robotic unit, wherein the one or more processors are further configured to: predict an optimal transducer position based on the real-time visual data; and calculate the quality score as a function of a difference in position between the transducer at the target location from the optimal transducer position.

15. A system for conducting an imaging examination on a living being in an imaging environment, comprising: a transducer configured to capture image data, wherein the transducer is coupled to an arm assembly of a robotic unit; a plurality of sensors configured to collect real-time data including force, pose, and transducer velocity; and a pre-trained computational model executed by one or more processors, wherein the one or more processors are configured to: receive a sequence of image data, sensor data and transducer pose data; segment the sequence into discrete action blocks corresponding to clinically relevant transducer movements; determine, based on the sequence and a task objective, a predicted action block to execute; and provide a set of control commands to the robotic unit to execute the predicted action block.

16. The system of claim 15, wherein the computational model comprises a transformer-based neural network trained using expert demonstrations.

17. The system of claim 15, wherein the computational model is trained using reinforcement learning to maximize an image quality score or anatomical completeness score.

18. The system of claim 15, wherein the computational model is configured to generate a reward function via inverse reinforcement learning.

19. The system of claim 15, wherein the computational model fuses input data from ultrasound image features, force-torque readings, and transducer pose encodings.

20. The system of claim 15, wherein the computational model is pre-trained using selfsupervised or few-shot learning to improve generalization to new patient anatomies.

21. A controller system for use in a non-invasive imaging examination of an individual, comprising: a controller that receives motion input from an operator to correspondingly move an arm of a robotic unit, wherein a transducer is coupled to the arm of the robotic unit and is positioned in a predetermined orientation at a predetermined location on the individual undergoing the non-invasive ultrasound examination; and at least one processor configured to receive location data of the transducer at the predetermined location, and in response, generate commands to autonomously position the controller at a location on a scanning surface that corresponds to the predetermined location of the transducer and in an orientation that corresponds to the predetermined orientation of the transducer.

22. The controller system of claim 21, wherein the controller comprises: a handle; and a ball-type joint coupled to the handle, wherein the ball-type joint maintains contact with the scanning surface during the motion input.

23. The controller system of claim 21, wherein the scanning surface comprises a curved surface representing a torso.

24. The controller system of claim 21, wherein the controller comprises a control that, when actuated, allows the operator to move the controller with respect to the scanning surface.

25. The controller system of claim 21, wherein the controller comprises an index marker that facilitates orientation and provides the operator with the position of the transducer on the individual relative to that of the controller on the scanning surface.

26. The controller system of claim 21, wherein the non-invasive imaging examination is a transthoracic echocardiogram (TTE) procedure.

27. The controller system of claim 26, wherein the one or more predetermined locations on the individual correspond to a parasternal long axis (PLAX) view, a parasternal short axis (PSAX) view, an apical view, a subcostal view, a suprasternal notch view, or combinations thereof.

28. The controller system of claim 21, wherein both the controller and operator are at a location remote from both the individual and the robotic unit.

29. The controller system of claim 21, further comprising an arm assembly having at least six degrees of freedom, wherein the controller is coupled to the arm assembly.

30. The controller system of claim 21, further comprising: one or more sensors configured to collect real-time force data during the motion input on the controller from the operator; and one or more feedback devices in communication with the at least one processor, wherein the one or more feedback devices are configured to provide a haptic response to the operator during movement of the controller.

31. The controller system of claim 30, wherein the at least one processor is further configured to: define a pre-set working area on the scanning surface for the controller; monitor the pre-set working area; and by way of the one or more feedback devices, provide the haptic response to the operator when the controller is moved outside of the working area.

32. A system for controlling a robotic unit for use in non-invasive diagnostic imaging of an individual, comprising: a scanning surface; an arm assembly; a controller coupled to the arm assembly; at least one processor configured to: capture position and orientation encoding data from the system arising from movements of the controller by an operator; and transmit the position and orientation encoding data to the robotic unit wherein the robotic unit causes a transducer to be moved in a corresponding orientation to a corresponding location on the individual undergoing the non-invasive ultrasound examination.

33. The system of claim 32, wherein the arm assembly is configured with three translational degrees of freedom with respect to the scanning surface.

34. The system of claim 32, wherein the controller is configured with three degrees of rotational freedom with respect to the arm assembly.

35. The system of claim 32, wherein the controller comprises: a handle; and a ball-type joint coupled to the handle, wherein the ball-type joint maintains contact with the scanning surface during the motion input.

36. The system of claim 32, wherein the scanning surface comprises a curved surface that resembles a torso.

37. The system of claim 32, wherein the controller comprises: a control that, when activated, allows the operator to move the controller along the scanning surface; andan index marker that facilitates orientation and provides the operator with the position of the ultrasound transducer on the individual relative to that of the controller on the scanning surface.

38. The system of claim 32, wherein both the controller and operator are at a location remote from both the individual and the robotic unit.

39. The controller system of claim 32, further comprising: one or more sensors configured to collect real-time force data resulting from the movements of the controller by the operator; and one or more feedback devices in communication with the at least one processor, wherein the one or more feedback devices are configured to provide a haptic response to the operator during movement of the controller, wherein the at least one processor is further configured to: define a pre-set working area on the scanning surface for the controller; monitor the pre-set working area; and by way of the one or more feedback devices, provide the haptic response to the operator when the controller is moved outside of the working area.

Images