Systems, methods, and apparatus for remote ultrasound examination

Abstract

The present disclosure generally relates to systems, methods, and devices for remote ultrasound examinations. The system may include a local control system including an input device and at least one local processor coupled to the input device; a remote robotic ultrasound system comprising a motion subsystem and at least one local processor coupled to the motion subsystem wherein the at least one local processor of the local control system is operable to: detect a new position state of the input device; obtaining position data corresponding to the new position state; generating input side motion demand data based on the position data; and transmitting the input-side motion demand data to the telerobotic system, wherein at least one remote processor of the telerobotic system is operable to: receive the input-side motion demand data; and adjusting a position state of the motion subsystem based on the input side motion demand data.

Description

Technical Field

[0001] The present disclosure relates generally to ultrasound examinations, and in particular, to systems, methods, and apparatus for remote ultrasound examinations. Background Art

[0002] Medical imaging tools are commonly used to assess, estimate, and diagnose medical conditions. It is estimated that approximately two-thirds of medical conditions can be assessed using medical imaging techniques. Among the various imaging options available, ultrasound imaging is widely considered to be a safe, cost-effective, and practical option.

[0003] However, ultrasound imaging generally requires well-trained and highly skilled technicians (e.g., sonographers) to operate complex ultrasound equipment. This presents a challenge in providing ultrasound examination services to many remote and underserved communities. In particular, patients in these communities are often forced to travel long distances to reach the nearest urban center where ultrasound scanning services are readily available. Summary of the invention

[0004] The following introduction is provided to introduce the reader to the more detailed discussion that follows. This introduction is not intended to limit or define any claimed or unclaimed invention. One or more inventions may reside in any combination or subcombination of elements or process steps disclosed in any part of this document, including its claims and drawings.

[0005] In a broad aspect, a system for remote ultrasound inspection is provided, comprising: a local control system including an input device and at least one local processor coupled to the input device; a remote robotic ultrasound system including a motion subsystem and at least one local processor coupled to the motion subsystem, wherein the at least one local processor of the local control system is operable to: detect a new position state of the input device; obtain position data corresponding to the new position state; generate input-side motion requirement data based on the position data; and transmit the input-side motion requirement data to the remote robotic system, wherein the at least one remote processor of the remote robotic system is operable to: receive the input-side motion requirement data; and adjust the position state of the motion subsystem based on the input-side motion requirement data.

[0006] At least one remote processor of the remote robotic ultrasound system may be further operable to: monitor a change in the position state of the motion subsystem; and in response to detecting a change in the position state, generate robot-side motion demand data corresponding to a new position state of the motion subsystem; and transmit the robot-side motion demand data to the local control system.

[0007] The at least one local processor of the local control system may be further operable to: receive robot-side motion demand data; and adjust a position state of the input device based on the robot-side motion demand data to generate a tactile feedback effect at the input device.

[0008] The input device may include a sensor subsystem, and detecting a new position state of the input device may include receiving new sensor data from the sensor subsystem.

[0009] The sensor subsystem may include a plurality of rotary encoders, and the position data may include encoder data generated by the plurality of rotary encoders.

[0010] The sensor subsystem may include an inertial measurement unit (IMU), and the position data may include IMU data generated by the IMU.

[0011] Generating the input-side motion requirement data may include determining an input-side transformation matrix based on the position data, wherein determining the input-side transformation matrix includes solving a forward kinematics model.

[0012] Adjusting the position state of the motion subsystem may include analyzing an input-side transformation matrix by solving an inverse kinematics model.

[0013] Generating the robot-side motion requirement data may include determining a robot-side transformation matrix based on the new position state, wherein determining the robot-side includes solving a forward kinematics model.

[0014] The input device may include a movable arm system.

[0015] The motion subsystem may include a robotic arm.

[0016] The motion subsystem may include a tool holding end effector for holding a tool.

[0017] The end effector may include a first connector that engages a second connector associated with the tool.

[0018] The tool may include one or more of an ultrasound transducer, a gel dispenser, and a transvaginal probe.

[0019] The tool may be disposed inside a housing including the flexible sleeve, and the second connector may be attached to the housing.

[0020] The input side motion demand data can control one of the spatial position and the orientation position of the tool holding end effector.

[0021] The local control system may include an input interface, and the at least one local processor of the local control system may be operable to: receive one or more user inputs from the input interface; and send the one or more user inputs to the telerobotic ultrasound system.

[0022] The one or more user inputs may include one of an ultrasound transducer configuration and a command to swap an actively selected tool.

[0023] The input interface may include an ultrasound input interface.

[0024] The telerobotic ultrasound system may include a pain threshold monitor coupled to at least one teleprocessor of the telerobotic ultrasound system, and wherein the at least one teleprocessor of the telerobotic ultrasound system is further operable to: monitor for an activation signal from the pain threshold monitor; and in response to detecting the activation signal, one of disabling the motion subsystem and reducing the force applied by the motion subsystem on the patient.

[0025] In another broad aspect, a method for remote ultrasonic inspection is provided, comprising: detecting a new position state of an input device of a local control system; obtaining position data corresponding to the new position state; generating input-side motion requirement data based on the position data; and sending the input-side motion requirement data from the local control system to a remote robotic system, receiving the input-side motion requirement data at the remote robotic system; and adjusting the position state of a motion subsystem of the remote robotic system based on the input-side motion requirement data.

[0026] The method may include: monitoring changes in the position state of a motion subsystem at a remote robot system; and in response to detecting the changes in the position state, generating robot-side motion requirement data corresponding to a new position state of the motion subsystem; and transmitting the robot-side motion requirement data to the local control system.

[0027] The method may include: receiving robot-side motion demand data at a local control system; and adjusting a position state of an input device at the local control system based on the robot-side motion demand data so as to generate a tactile feedback effect at the input device.

[0028] The input device may include a sensor subsystem, and detecting a new position state of the input device may include receiving new sensor data from the sensor subsystem.

[0029] The sensor subsystem may include a plurality of rotary encoders, and the position data may include encoder data generated by the plurality of rotary encoders.

[0030] The sensor subsystem may include an inertial measurement unit (IMU), and the position data may include IMU data generated by the IMU.

[0031] Generating the input-side motion requirement data may include determining an input-side transformation matrix based on the position data, wherein determining the input-side transformation matrix includes solving a forward kinematics model.

[0032] Adjusting the position state of the motion subsystem may include analyzing an input-side transformation matrix by solving an inverse kinematics model.

[0033] Generating the robot-side motion requirement data may include determining a robot-side transformation matrix based on the new position state, wherein determining the robot-side includes solving a forward kinematics model.

[0034] The input device may include a movable arm system.

[0035] The motion subsystem may include a robotic arm.

[0036] The motion subsystem may include a tool holding end effector for holding a tool.

[0037] The end effector may include a first connector that engages a second connector associated with the tool.

[0038] The tool may include one or more of an ultrasound transducer, a gel dispenser, and a transvaginal probe.

[0039] The tool may be disposed inside a housing including the flexible sleeve, and the second connector may be attached to the housing.

[0040] The input side motion demand data can control one of the spatial position and the orientation position of the tool holding end effector.

[0041] The local control system may include an input interface, and the method may include: receiving one or more user inputs from the input interface at the local control system; and transmitting the one or more user inputs from the local control system to the remote robotic ultrasound system.

[0042] The one or more user inputs may include one of an ultrasound transducer configuration and a command to swap an actively selected tool.

[0043] The input interface may include an ultrasound input interface.

[0044] The telerobotic ultrasound system may include a pain threshold monitor coupled to at least one teleprocessor of the telerobotic ultrasound system, and the method may include: monitoring for an activation signal from the pain threshold monitor; and in response to detecting the activation signal, one of disabling the motion subsystem and reducing the force applied by the motion subsystem on the patient.

[0045] According to another broad aspect, an input device is provided, comprising: a mounting structure; a user-controllable arm system, comprising: (i) a robotic arm extending between a first arm end and a distal second arm end, and (ii) a rotatable member rotatably coupled to the mounting structure, wherein the robotic arm is fixed to the rotatable member at the second arm end; a motor subsystem, the motor subsystem comprising a plurality of motors for controlling a position state of the arm system; a sensor subsystem for monitoring the position state of the arm system; and at least one processor, the at least one processor being coupled to each of the motor subsystem and the sensor subsystem.

[0046] The robotic arm may include a plurality of arm segments that are rotatably connected.

[0047] A user engageable joystick may be mounted to the first arm end, the joystick providing an engagement interface for moving the arm system.

[0048] The joystick may include one or more contact sensors coupled to at least one processor.

[0049] The joystick may include an activation toggle coupled to the at least one processor.

[0050] The input device may include a power supply coupled to at least one processor, wherein the at least one processor is operable to control power supplied from the power supply to the motor subsystem.

[0051] When the contact sensor is not engaged, the at least one processor may supply current from the power supply to the motor subsystem to maintain the arm system in the static position.

[0052] The at least one processor is operable to: receive motion demand data from a remote system; and apply increased current to one or more motors of the motor subsystem to increase motor torque and generate a haptic feedback effect.

[0053] The rotatable connections between the arm segments and between the rotatable member and the mounting structure may include a rotatable shaft.

[0054] The sensor subsystem may include one or more rotary encoders coupled to the rotatable shaft, each rotary encoder generating encoder data corresponding to a rotational position of the respective shaft.

[0055] One or more of the rotatable shafts may be coupled to a corresponding motor of the motor subsystem.

[0056] The coupling may be one of a direct coupling and an indirect coupling, and the indirect coupling may include a belt and pulley mechanism.

[0057] At least one processor is operable to control the motor subsystem to adjust the rotational position of the rotational axis to adjust the positional state of the arm system.

[0058] The sensor subsystem may include an inertial measurement unit mounted to the joystick.

[0059] The user-controllable arm system may include: (i) an directional positioning control portion; and (ii) a spatial position control portion, wherein the directional position control portion is configured to control the directional position of an end effector associated with a motion subsystem in a remote environment, and the spatial position control portion is configured to change the spatial position of the end effector.

[0060] The orientation control portion may include at least one arm segment of a robotic arm.

[0061] The at least one arm segment may include a forward wrist arm segment, a middle arm rotation segment, and a rear arm rotation segment.

[0062] The forward wrist arm section may be rotatably coupled to the intermediate arm rotation section and rotate about a first yaw axis to control a yaw angle of the end effector in a remote environment.

[0063] The intermediate arm rotating section may be rotatably connected to the rear arm rotating section and rotated about a second pitch axis to control a pitch angle of the end effector in a remote environment, wherein the second axis is orthogonal to the first axis.

[0064] The spatial position control portion may include one or more arm segments and a rotatable member of the robotic arm.

[0065] The one or more arm segments may include a front translating arm segment and a rear translating arm segment.

[0066] The rear arm rotation section may be rotatably connected to the forward translation rotation section and rotate about a third roll axis to control a roll angle of the end effector in a remote environment, wherein the third axis is orthogonal to both the first axis and the second axis.

[0067] The front translation arm segment may be rotatably connected to the rear translation arm segment and rotate about a fourth axis to control vertical spatial movement of the end effector in the remote environment.

[0068] The rear translation arm segment may be rotatably connected to the rotatable member and rotated about a fifth axis to control forward and rearward spatial movement of the end effector in the remote environment.

[0069] The rotatable member may be rotatably connected to the mounting structure and rotated about a sixth axis to control lateral horizontal spatial movement of the end effector in the remote environment.

[0070] According to the following detailed description, other features and advantages of the present application will become apparent. However, it should be understood that the detailed description and specific examples, although indicating embodiments of the present application, are only given in an illustrative manner, and the scope of the claims should not be limited by these embodiments, but should be given the broadest interpretation consistent with the entire description. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The drawings included herein are used to illustrate various examples of the systems, methods, and devices of the present specification and are not intended to limit the scope of what is taught in any way. In the drawings:

[0072] Figure 1A is a simplified block diagram of an example system for remote ultrasound examination;

[0073] Figure 1B is a schematic diagram of an example local control system in communication with a remote robotic ultrasound system;

[0074] Figure 2 is a simplified electrical hardware block diagram of an example local control system that communicates with a remote robotic ultrasound system;

[0075] Figure 3 is a simplified block diagram showing the data flow between a local control system and a remote robotic ultrasound system;

[0076] Figure 4A is a side front perspective view of an example input device associated with a local control system;

[0077] Figure 4B is a side rear perspective view of an input device;

[0078] Figure 4C is a side view of the input device;

[0079] Figure 4D is along Figure 4A A side cross-sectional elevation view of the input device taken along section line 4D-4D';

[0080] Figure 4E is along Figure 4A A side perspective cross-sectional view of the input device taken along section line 4D-4D';

[0081] Figure 4F is a front perspective view of an input device;

[0082] Figure 4G is along Figure 4DA perspective cross-sectional view of the input device taken along section line 4F-4F';

[0083] Figure 4H is along Figure 4F A perspective cross-sectional view of the input device taken along section line 4H-4H';

[0084] Fig. 4I is a simplified electrical hardware block diagram of an example input device;

[0085] Figure 5A is a schematic diagram of a tool rotating in clockwise and counterclockwise directions in a remote environment;

[0086] Figure 5B is a schematic diagram of a tool for scrolling forward and backward in a remote environment;

[0087] Figure 5C is a schematic diagram of a tool for scrolling left and right in a remote environment;

[0088] Figure 5D It is a schematic diagram of tools in a remote environment being translated to different spatial locations;

[0089] Fig. 6A is a schematic diagram showing a side view of an example telerobotic ultrasound system;

[0090] Figure 6B is a schematic diagram showing another side view of another example of a telerobotic ultrasound system;

[0091] Figure 6C It is shown Figure 6B A schematic diagram of a front view of a telerobotic ultrasound system;

[0092] Fig. 7A is a schematic diagram showing a side view of an example robotic arm for use in a telerobotic ultrasound system;

[0093] Figure 7B is a schematic diagram showing a side view of another example robotic arm for use in a telerobotic ultrasound system;

[0094] Fig. 8A is a simplified electrical hardware block diagram of an example end effector of a robotic arm;

[0095] Figure 8B is an illustration of an example of a tool holding mechanism for use with an end effector;

[0096] Figure 8C is an illustration of another exemplary tool holding mechanism for use with an end effector;

[0097] Fig.8Dis a diagrammatic representation of a plurality of tools disposed in a tool holder area;

[0098] Fig. 8E is an illustration of an exemplary housing body for holding an ultrasonic transducer;

[0099] Figure 8F is an illustration of an exemplary housing body for holding a transvaginal probe;

[0100] Figure 8G is an illustration of another exemplary tool holding mechanism for use with an end effector;

[0101] Figure 8H is an illustration of another exemplary tool holding mechanism for use with an end effector;

[0102] Fig. 9 is an illustration of an example setup configuration for multiple object tracking sensors and cameras within a remote environment;

[0103] Fig.10 is a process flow of an example method for position control and feedback between an input device and a telepresence robotic system; and

[0104] Fig.11 is a process flow of an exemplary method for remote ultrasound examination.

[0105] Other aspects and features of the examples described herein will become apparent from the following description taken in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0106] Various embodiments according to the teachings of this article will be described below to provide an example of at least one embodiment of the subject matter claimed for protection. The embodiments described herein do not limit any of the subject matter claimed for protection. The subject matter claimed for protection is not limited to a device, system or method having all the features of any of the devices, systems or methods described below, or is not limited to multiple or all common features in the devices, systems or methods described herein. It is possible that there may be a device, system or method described herein that is not an embodiment of any of the subject matter claimed for protection. Any subject matter described herein that is not claimed for protection in this document may be the subject of another protective document, for example, a continuing patent application, and the applicant, inventor or owner does not intend to abandon, declare or disclose such subject matter to the public in the disclosure herein.

[0107] For simplicity and clarity of explanation, reference numerals may be repeated in the accompanying drawings to indicate corresponding or similar elements. In addition, many specific details are set forth in order to provide a thorough understanding of the subject matter described herein. However, it will be appreciated by those of ordinary skill in the art that the subject matter described herein may be practiced without these specific details. In other cases, well-known methods, processes, and components are not described in detail to avoid obscuring the subject matter described herein. This description should not be construed as limiting the scope of the subject matter described herein.

[0108] It should also be noted that the terms "coupled" or "coupling" as used herein may have several different meanings depending on the context in which these terms are used. For example, the terms coupled or coupling may have mechanical, fluid, or electrical meanings. For example, as used herein, the terms coupled or coupling may indicate that two elements or devices may be connected to each other directly or via electrical or magnetic signals, electrical connections, electrical elements, or mechanical elements through one or more intermediate elements or devices, depending on the specific context. In addition, coupled electrical elements may send and / or receive data.

[0109] Unless the context requires otherwise, throughout the specification and following claims, the word "comprise" and variations such as "comprises" and "comprising" should be interpreted in an open, inclusive sense, ie, to mean "including, but not limited to."

[0110] It should also be noted that, as used herein, the term "and / or" is intended to mean an inclusive or. That is, "X and / or Y" is intended to mean, for example, X or Y or both. As a further example, "X, Y and / or Z" is intended to mean X or Y or Z or any combination thereof.

[0111] It should be noted that as used herein, terms of degree such as "substantially", "about" and "approximately" mean a reasonable amount of deviation of the modified term such that the end result is not significantly changed. These terms of degree may also be interpreted as including deviations of the modified term, such as 1%, 2%, 5% or 10%, for example, if the deviation does not negate the meaning of the modified term.

[0112] In addition, numerical ranges recited herein by endpoints include all numbers and fractions contained within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, and 5). It should also be understood that all numbers and fractions thereof are assumed to be modified by the term "about," meaning that variations of up to a certain amount, e.g., 1%, 2%, 5%, or 10%, of the referenced number may be allowed without significantly changing the end result.

[0113] References throughout this specification to "one embodiment," "an embodiment," "at least one embodiment," or "some embodiments" mean that one or more particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, unless otherwise specified as incombinable or as an alternative.

[0114] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. It should also be noted that the term "or" is generally used in its broadest sense, i.e., as meaning "and / or," unless the content clearly dictates otherwise.

[0115] Similarly, throughout this specification and the appended claims, the term "communication," as in "communication path," "communicative coupling," and in variations such as "communicatively coupled," is generally used to refer to any engineered arrangement for transmitting and / or exchanging information. Exemplary communication paths include, but are not limited to, conductive paths (e.g., conductive wires, conductive traces), magnetic paths (e.g., magnetic media), optical paths (e.g., optical fibers), electromagnetic radiation paths (e.g., radio waves), or any combination thereof. Exemplary communicative couplings include, but are not limited to, electrical couplings, magnetic couplings, optical couplings, radio couplings, or any combination thereof.

[0116] Throughout this specification and the appended claims, infinitive verb forms are frequently used. Examples include, but are not limited to: "detect," "provide," "send," "communicate," "process," "route," etc. Unless the specific context requires otherwise, such infinitive verb forms are used in an open, inclusive sense, i.e., "at least detect," "at least provide," "at least transmit," etc.

[0117] The example systems and methods described herein can be implemented as a combination of hardware or software. In some cases, the examples described herein can be implemented at least in part by using one or more computer programs executed on one or more programmable devices including at least one processing element and a data storage element (including volatile memory, non-volatile memory, storage element, or any combination thereof). Depending on the nature of the device, these devices may also have at least one input device (e.g., keyboard, mouse, touch screen, etc.) and at least one output device (e.g., display screen, printer, radio, etc.).

[0118] Some elements for realizing at least a part of the system, method and device described herein can be realized via software written in a high-level process language (such as object-oriented programming).Program code can be written in C++, C#, JavaScript, Python or any other suitable programming language, and can include modules or classes, as known to those skilled in the art in the field of object-oriented programming.Alternately or in addition, some of these elements realized via software can be written in assembly language, machine language or firmware as required.In either case, language can be a compiled or interpreted language.

[0119] At least some of these software programs may be stored on a computer-readable medium, such as but not limited to a ROM, a disk, an optical disk, a USB key, etc., which can be read by a device having at least one processor, an operating system, and associated hardware and software for implementing the functionality of at least one of the methods described herein. The software program code, when read by the device, configures the device to operate in a new, specific, and predefined manner (e.g., as a special-purpose computer) in order to perform at least one of the methods described herein.

[0120] In addition, at least some of the programs associated with the systems and methods described herein may be capable of being distributed in a computer program product that includes a computer-readable medium carrying computer-usable instructions for one or more processors. The medium may be provided in various forms, including non-transitory forms, such as, but not limited to, one or more disks, optical disks, tapes, chips, and magnetic and electronic memories. Alternatively, the medium may be transient in nature, such as, but not limited to, wire transmission, satellite transmission, Internet transmission (e.g., download), media, digital and analog signals, etc. The computer-usable instructions may also be in various formats, including compiled and non-compiled code.

[0121] Ultrasound imaging is an important tool for diagnosing a wide range of medical conditions. As non-limiting examples, examples of medical conditions that can be diagnosed using ultrasound imaging include retinal detachment, pericardial effusion, acute appendicitis, kidney stones, urinary retention, testicular torsion, deep vein thrombosis, and muscle tears and ruptures. Ultrasound imaging is also more widely used in obstetrics (e.g., for fetal anatomy scanning and automatic placental positioning), oncology (e.g., for assessing endometrial thickness and determining the need for endometrial biopsy), emergency medicine (e.g., for accurate and early diagnosis of acute cholecystitis), and for monitoring various chronic conditions (e.g., heart failure and diabetic complications, such as peripheral arterial insufficiency).

[0122] However, ultrasound scanning typically requires the presence of a highly skilled technician, known as a sonographer, to operate the complex ultrasound equipment. Many rural and remote communities often suffer from a lack of available sonographers. For patients in these communities, accessing ultrasound services requires a difficult and often inconvenient trip to larger urban centers where these services are readily available. For this reason, patients living in these communities often do not have immediate access to ultrasound services, which often results in delays in medical diagnosis, and subsequently, delays in follow-up medical treatment.

[0123] The requirement for an on-site ultrasound technician also limits access to ultrasound services in other settings. For example, during many long-distance trips (i.e., travel via ship, airplane, etc.), access to ultrasound services is limited if not completely unavailable. Likewise, as deep space exploration becomes a reality - the requirement for an on-site sonographer becomes a barrier to accessing important diagnostic ultrasound services for those in space.

[0124] While mobile diagnostic imaging can increase access to ultrasound services, this approach fails to fully address the issue of access to diagnostic quality ultrasound. More specifically, mobile diagnostic imaging services involve transporting mobile ultrasound imaging equipment and skilled sonographers to remote communities. However, these services are hampered by the difficulty of recruiting expert staff who are able and willing to work in these remote communities. Furthermore, these services suffer from high operating costs, as well as complex logistics in coordinating staff to work in remote communities.

[0125] The present disclosure provides systems, methods, and apparatus for remote ultrasound examinations. The disclosed systems and methods can alleviate at least some of the disadvantages inherent in existing systems and thereby provide improved access to diagnostic quality ultrasound.

[0126] As described in further detail below, a remote ultrasound inspection system may include a local control system and a remote ultrasound system. The local control system may be located, for example, in an area accessible to a skilled ultrasound technician (e.g., a city center). In contrast, the remote system may be located in a peripheral site that is geographically remote from the local system. This may include, for example, a remote community, or otherwise include a transport ship, an aircraft, a space station, or any other desired environment.

[0127] The remote system may include one or more robotic devices operable to perform ultrasound scanning. The robotic device may be installed, for example, in a dedicated scanning room in a remote site. The robotic device may include one or more robotic arms. The robotic arms may hold the tools required for ultrasound scanning (e.g., gel dispensers, ultrasound transducers, etc.) and translate these tools across the patient's body as needed. The remote ultrasound system may be used to capture sonograms of the patient, and these sonograms may be sent back to the local control system. The captured sonograms may then be viewed locally by a sonographer and further analyzed by a medical specialist (e.g., a radiologist).

[0128] The telerobotic system may be configured in one of three modes of operation: (i) a non-autonomous or guided mode; (ii) a fully autonomous or non-guided mode; and / or (iii) a semi-autonomous or semi-guided mode.

[0129] In a "non-autonomous" mode, the robotic system can be completely directed by the local control system. For example, the input device of the local control system can direct the movement of the remote robotic arm. In this way, the local system operator (e.g., a skilled sonographer) can control the remote robotic arm to pick up and replace various tools, apply gel to the desired location on the patient, move and re-orient the ultrasound transducer to capture sonograms, etc. The local operator can also control image capture using the ultrasound transducer, and configure ultrasound parameters to obtain enhanced ultrasound images.

[0130] The remote system may enable tactile feedback. Through tactile feedback, the local user operator may receive feedback about the forces and pressures experienced by the remote robotic arm. In turn, this may help the local user operator avoid applying excessive or excessive pressure to the patient. Tactile feedback may also assist in some types of medical diagnosis.

[0131] Conversely, in “fully autonomous” mode, all functions of the remote system are fully automated, including gel application, ultrasound scanning, image capture, and preliminary medical predictions for diagnosis.

[0132] To this end, in order to achieve fully automated scanning, the remote system can host one or more trained machine learning models. The machine learning model can be trained to identify abnormalities in the target organ based on the ultrasound data generated by the ultrasound transducer. The remote system can then control and adjust itself to capture more optimized images of the abnormalities and otherwise modify the ultrasound transducer position and configuration parameters to obtain enhanced images. The captured images can then be sent back to the local system for preliminary review by the ultrasound technician and further analysis by the radiologist.

[0133] In a fully automated mode, the remote system may also perform an automated preliminary diagnosis of the patient. For example, the remote system may host a machine learning model that is trained to analyze abnormalities in the sonogram and provide a preliminary diagnosis of the associated medical condition (e.g., in real time or near real time). The preliminary diagnosis may then be subsequently verified and / or confirmed by a medical expert (e.g., a radiologist).

[0134] In a "semi-autonomous guidance" mode, the remote system may perform at least some functions under the control (or guidance) of the local control system, and may perform other functions autonomously and independently. For example, the movement of the robotic arm may still be controlled by the local control system, however, the remote system may automatically control the robotic arm to avoid obstacles and prevent collisions. In other cases, the local operator may control the robotic arm to capture a sonogram, but the robot may otherwise automatically analyze the captured sonogram to provide a preliminary diagnosis.

[0135] The remote system may be configured to dynamically change between different operating modes. For example, this may occur in response to receiving a user input to change the operating mode. The user input may be received at the local control system, or directly at the remote system.

[0136] In other cases, the remote system may change its operating mode based on a triggering event. For example, the system may initially operate in a guided, non-autonomous mode. However, in response to detecting a connection delay, the remote system may automatically revert to a semi-guided mode or a fully autonomous mode.

[0137] In both guided and semi-guided modes, the remote system can be deployed at a distance from a skilled technician. This can extend ultrasound services to areas far from skilled sonographers, i.e., remote communities. This, in turn, minimizes the "hidden costs" for patients living in rural and / or underserved areas and improves access to diagnostic quality ultrasound in these communities.

[0138] The non-autonomous or guided mode requires only a single sonographer to perform the ultrasound. This avoids the need for a healthcare professional to serve as an on-site attendant at the remote system. This simplifies deployment logistics and reduces costs associated with hiring additional medical professionals.

[0139] In autonomous mode, the need for skilled practitioners can be completely eliminated on both the control system side and the remote system side. The use of autonomous mode can solve important problems associated with communication delays. In particular, operation in autonomous mode does not rely on a stable network connection between the remote system and the sonographer at the local system. Therefore, in areas with poor connectivity (e.g., remote communities, deep space exploration, etc.), autonomous mode can enable full operation of the ultrasound machine, including automatic capture of sonograms and / or analysis of sonograms to provide a preliminary diagnosis.

[0140] Remote Ultrasound System

[0141] The following is a discussion of an example remote ultrasound system, which may be used alone or in any combination or sub-combination with any other one or more of the features described herein (e.g., input devices, local user control systems, remote robotic control systems, methods for operating a remote ultrasound system, etc.).

[0142] Reference now Figure 1A , which shows a simplified block diagram of an example system 100a for remote ultrasound scanning.

[0143] As shown, the system 100a may include a local control system 102 (also referred to as a local user control system) connected to one or more telerobotic ultrasound systems 104a , 104b via a network 110 .

[0144] The local system 102 may control or otherwise interact with the remote system 104. The local system 102 may be operated by a skilled technician or sonographer. The local system 102 may be located, for example, in a city center where there is a large number of skilled sonographers.

[0145] The remote ultrasound system 104 can be located away from the local system 102. The remote system 104 can be installed in various remote rural communities. In other example applications, the remote system 102 is installed in any other remote, mobile or non-mobile environment (e.g., an aircraft, a ship, a space station, etc.). To this end, the remote system can be installed in a scanning room or a dedicated room located in a remote area. Although only two remote systems 104 are illustrated, it should be understood that the system 100a can include any number of remote systems 104.

[0146] It should also be understood that the remote ultrasound systems 104 need not be located in a remote or rural location, but rather they are not located in the same location as the local control system 102. For example, the remote ultrasound system 104 may be located within the same facility (e.g., a hospital) as the local control system 102, but in a different room or building than the local control system 102.

[0147] As explained herein, each remote system 104 can be operated to capture sonograms of patients located in the respective remote regions. The sonograms are then sent back to the local system 102 and / or server 106 and can be subject to further review by local experts (e.g., sonographers, radiologists, etc.). Thus, ultrasound services can be provided on demand to remote patients, thereby expediting medical diagnostic evaluations for these patients.

[0148] Optionally, remote system 104 may operate in one of three operating modes including: (i) a non-autonomous mode, (ii) a semi-autonomous mode, and / or (iii) a fully autonomous mode.

[0149] In particular, as described above, in the non-autonomous mode, the operation of the remote system 104 is completely directed by the local system 102. In the semi-autonomous mode, the remote system 104 may be partially controlled by the local system 102 and may be partially controlled automatically. In the fully autonomous mode, the remote system 104 may operate completely independently of the local system 102. In particular, in the fully autonomous mode, the remote system 104 does not have to be connected to the network 110.

[0150] As provided herein, the system 100a enables a single sonographer operating a local system 102 to remotely control multiple remote systems 104. Thus, a single sonographer can provide ultrasound examination services to multiple remote systems located in the same or different geographic regions. In turn, this avoids the need to provide a separate sonographer in each remote region. Thus, diagnostic quality ultrasound can be provided to many remote locations in a scalable and economically sustainable manner.

[0151] In other example cases, the system 100a may also include more than one local system 102. For example, a separate local system 102 may be provided for each remote system 104.

[0152] The network 110 may be connected to the Internet. Typically, the connection between the network 110 and the Internet may be made via a firewall server (not shown). In some cases, there may be multiple links or firewalls or both between the network 110 and the Internet. Some organizations may operate multiple networks 110 or virtual networks 110, which may be networked or isolated. For ease of illustration, these are omitted; however, it should be understood that the teachings herein may be applied to such systems. The network 110 may be comprised of one or more computer network technologies, such as IEEE 802.3 (Ethernet), IEEE 802.11, and similar technologies.

[0153] The system 100a may also include one or more servers 106 (although only one is shown for simplicity) connected to the network 110. The servers 106 may include, for example, virtual or "cloud" servers.

[0154] The server 106 may perform various functions in the system 100a, such as providing a storage database for images (i.e., sonograms) captured by the remote system 104. The database may be accessed by the computer of the local system 102 (or any other computer). Sonograms captured by the remote system 104 in a semi-autonomous or fully autonomous mode may be uploaded to the server 106. At a later point in time, a sonographer may review the captured images to provide image quality control. The images may also be accessed and reviewed by a medical specialist (e.g., a radiologist) to provide a diagnostic assessment of the imaged abnormalities. The server 106 may also store various preliminary diagnostic assessments automatically generated by the remote system 104 in association with the ultrasound images.

[0155] The server 106 may also store an action log of various commands and data sent and received between the local and remote systems 102, 104. The action log may be used to monitor and / or audit system performance.

[0156] Reference now Figure 1B , which shows a schematic diagram 100b of an example portion of system 100a. More specifically, Figure 1B An example of a local system 102 and a remote system 104 is shown.

[0157] As shown, the local control system 102 includes various features that allow a local operator (eg, a skilled sonographer) to interact with the remote ultrasound system 104. For example, the local control system 102 may include an input device 180 and a computer workstation 190.

[0158] The input device 180 enables a user to control a motion subsystem in a remote environment. To this end, the remote system 104 may include a motion subsystem that controls the position of a tool (e.g., a gel dispenser, an ultrasound transducer, etc.) relative to the patient. The remote motion subsystem may include, for example, one or more robotic arms 150 having an integrated end effector 152 that can be used to hold (and manipulate) the tool. Alternatively, the motion subsystem may include, for example, a gantry-type system.

[0159] In operation, the input device 180 can allow a local operator (e.g., a sonographer) to control the remote motion subsystem and thereby change the position of the tool-carrying end effector 152. For example, the local operator 180 can use the input device 180 to translate an ultrasound transducer on a patient in a remote environment.

[0160] As described herein, position control by input device 180 may include spatial position control (e.g., X, Y, Z position control) and directional position control (e.g., roll, yaw, and pitch control) of remote end effector 152. In general, input device 180 may be used to control the pose of end effector 152 and a tool held by end effector 152. For example, input device 180 may be used when remote system 104 is operating in a guided mode or a semi-guided mode.

[0161] Any suitable input interface 180 may be provided in the local system 180. As shown in the illustrated example, the input device 180 may include a movable joystick arm 180a supported by a frame 180b. Thus, the local sonographer may manipulate the spatial and directional position of the joystick 180a. In turn, the local system may track the changing position of the joystick 180a and may generate control instructions (also referred to herein as "motion demands") to achieve mutual control of the remote motion subsystems.

[0162] Optionally, the input device 180 may also receive return data from the remote system 104. For example, the input device 180 may be configured to mirror the forces and pressures experienced by the remote system 104 in order to provide tactile feedback effects. For example, these may include the forces experienced by the remote motion subsystem when colliding with obstacles in the remote environment (e.g., the patient's body or external objects). Tactile feedback may help the local operator ensure that they are not applying excessive or excessive forces to the patient during an ultrasound scan (i.e., when operating the remote motion subsystem).

[0163] Alternatively or additionally, input device 180 may include any other suitable interface for receiving input. For example, input device 180 may be any other type of Cartesian-style input device, and may include any suitable parallel manipulator configuration design. In still other example cases, input device 180 may include a virtualized computer interface (e.g., a touch screen display) that allows an operator to manipulate the remote motion subsystem via inputting computer commands.

[0164] The local system 102 may also include a computer workstation 190. The computer workstation 190 may enable a user operator to input additional control data for controlling the remote system 104. For example, the workstation 190 may include an input interface 190a, such as a keyboard and / or a mouse. The keyboard may be a general-purpose keyboard and / or a dedicated ultrasound keyboard. The input interface (e.g., keyboard 190a) may be used to adjust various ultrasound parameters of the ultrasound transducer in the remote environment. The input interface 190a may also be used, for example, to instruct the remote system to capture an image, interchange or swap tools that are actively used (e.g., swap between different ultrasound transducers), adjust camera switching of various cameras located in the remote system 104, pan, tilt, or zoom cameras located in the remote system 104, and the like.

[0165] In some example cases, the input interface 190a may be an at least partially virtual computerized interface. For example, the input interface 190a may include a touch screen display or a virtual keyboard that can receive various inputs.

[0166] The workstation 190 can also include a display interface 190b. The display interface 190a allows a local operator to view ultrasound images received from the remote system 104. For example, this can include viewing a real-time or near real-time feed from an actively deployed ultrasound transducer. In other cases, the display 190 can be used to view previously acquired sonograms (e.g., stored on the server 106).

[0167] The display interface 190a may also display video or image feeds from one or more cameras 164 located in the remote environment. This in turn may enable a local operator to better monitor the remote environment, as well as observe a remote patient.

[0168] Optionally, the workstation 190 itself can include a camera and / or microphone. The camera and / or microphone can allow a local operator to interact with a remote patient. For example, a local operator can use a camera and microphone to instruct the patient to change their posture in order to facilitate ultrasound scanning.

[0169] The computer workstation 190 may be coupled to the input device 180 and may act as a communication intermediary between the input device 190 and the robotic system 104. For example, the computer 190 may receive control data generated by the input device 180 and may send the data to the robotic system 104. In the reverse case, the computer 190 may receive current position data from the robotic system 104 and may send the data to the input device 180 (e.g., to implement tactile feedback). Alternatively, the input device 180 may be a stand-alone device and may include its own communication interface for independent communication with the remote system 104.

[0170] Turning now to the remote system 104—As previously mentioned, the remote system 104 may include a motion subsystem for holding various ultrasound tools and manipulating the position of these tools relative to the patient. The motion subsystem may be directed, in whole or in part, by control instructions received from the input device 180. Alternatively, the motion subsystem may operate autonomously, for example, to independently capture ultrasound images of the patient.

[0171] The motion subsystem may include a controllable robotic arm 150. The robotic arm 150 may include a distal end effector 152 that holds and replaces various tools 154 (e.g., gel dispenser, ultrasound transducer probe, etc.). The robotic arm 150 may be positioned above a patient lying on a bed frame 160, for example, in a supine position.

[0172] Although only a single robotic arm 150 is illustrated, it should be understood that the system may include any other number of robotic arms 150. Alternatively or additionally, the motion system may include any other suitable motion mechanism, such as an XY stage system.

[0173] As explained herein, the remote system may also include one or more object tracking sensors (e.g., LiDAR or depth sensing cameras), as well as a camera 164 and a display interface 162. The camera 164 may capture a live video and / or audio feed of the patient area. The feed may be sent back to the local user system 102. In this way, a local system operator (e.g., a sonographer) may monitor the patient, as well as observe and track the movement of the motion subsystem (e.g., the robotic arm 150).

[0174] Reference now Figure 2 , which shows a simplified electrical hardware block diagram 200 of an example of each of the local system 102 and the remote system 104 .

[0175] As shown, local user system 102 may include a processor 202a coupled via a data bus to a user input interface 190a, a display device 190b, an input device 180, and one or more of an audio / video (A / V) subsystem 208a and a communication interface 212a.

[0176] The processor 202 a and components 204 a - 210 a may be housed within the computer workstation 190 .

[0177] The processor 202a may be a computer processor, such as a general-purpose microprocessor. Alternatively, the processor 202a may be a field programmable gate array, an application-specific integrated circuit, a microcontroller, or other suitable computer processor.

[0178] The processor 202a may be coupled to the memory 204a via a computer data bus. The memory 204a may include both volatile and non-volatile memory. The non-volatile memory stores a computer program consisting of computer executable instructions, which may be loaded into the volatile memory as needed for execution by the processor 202a. Those skilled in the art will understand that the reference herein to the local system 102 performing a program or action in a particular manner means that the processor 202a is executing instructions (e.g., software programs) stored in the memory 204a and may send or receive input and output via one or more interfaces. The memory 204a may also store data input to or output from the processor 202a during the execution of the computer executable instructions.

[0179] The memory 204a may store various programs and algorithms that enable the conversion of input data generated by the input device 180 into control instructions to be sent to the remote system 104. In other cases, the memory 204a may store programs and algorithms that convert position data received from the remote system 104 into control data for the input device 180.

[0180] Communication interface 206a is one or more data network interfaces, such as IEEE 802.3 or IEEE 802.11 interfaces, for communicating over a network (eg, network 110).

[0181] The audio / visual (A / V) subsystem 208a may include one or more cameras, microphones, and speakers. The A / V system 208a may allow a local user operator to interact and communicate with a remote patient, for example, to provide instructions during a scanning operation.

[0182] Input device 180 may include any interface for generating motion commands for a remote motion subsystem. Figure 2 204a, and communication interface 206a.

[0183] The user input interface 190a may include any interface for receiving user input, including a keyboard and a mouse. As described above, the input interface 190a may include, for example, an ultrasonic keyboard.

[0184] The display device 190b may be any suitable display (e.g., a desktop monitor) for outputting information and data required by various computer programs. In particular, the display 190b may display a graphical user interface (GUI). In some cases, the display interface 190b may include a touch screen and thus may integrate the user input interface 190a.

[0185] Continue to refer Figure 2 The remote control system 104 may also include a processor 202b coupled via a data bus to one or more of a memory 204b, an audio / visual (A / V) subsystem 206b, a communication interface 208b, a motion subsystem 210b, an object detection subsystem 212b, an ultrasonic tool subsystem 214b, and / or a patient feedback device 216b.

[0186] Processor 202b, memory 204b, and communication interface 208b may be similar to processor 202a, memory 204a, and communication interface 206a, respectively.

[0187] To this end, those skilled in the art will understand that reference herein to the remote system 104 performing a function or action in a particular manner means that the processor 202b is executing instructions (e.g., a software program) stored in the memory 204b and may be sending or receiving input and output via one or more interfaces.

[0188] The memory 204b may store various programs for controlling the motion system 210b. The memory 204b may also store one or more trained machine learning models. The trained machine learning model may be operable to control the motion subsystem 210b to automatically move various ultrasound tools to automatically capture ultrasound images of abnormalities detected in the patient's body. The trained machine learning model may also be used to automatically analyze abnormalities in the captured sonogram to provide a preliminary diagnosis of the patient's medical condition. Therefore, the trained machine learning model may enable the remote system 104 to operate in a fully autonomous or semi-autonomous mode.

[0189] The audio / visual (A / V) subsystem 206b may include, for example, one or more monitors (e.g., Figure 1B The A / V subsystem 206b allows the patient to receive a video feed from a remotely located sonographer to allow the sonographer to interact with the patient.

[0190] The motion subsystem 210b may include any suitable system for holding and translating an ultrasonic tool. For example, the motion subsystem 210 may include one or more movable robotic arms 150 ( Figure 1B ). To this end, the motion subsystem 210b may include various controllable motors 210b1 for motion control. The motion subsystem 210b may also include sensors 210b2. The sensors 210b2 may include various position tracking sensors (e.g., encoders) and force sensors. The force sensors may be used to monitor the pressure and force experienced by the motion subsystem, such as to implement tactile feedback at the input device 180.

[0191] The object detection subsystem 212b may include various depth perception cameras and sensors (eg, time of flight sensors such as LiDAR). As explained herein, the object detection subsystem 212b may be used to monitor for obstacles that obstruct the path of the motion subsystem 210b.

[0192] The ultrasonic tool subsystem 214b may include various electronic hardware required to operate the tools required for ultrasonic scanning. For example, this may include a driver for an ultrasonic transducer, a pump for operating a gel dispenser, etc.

[0193] Optionally, the remote system 104 may also include a patient feedback device 216b. The patient feedback device 216b may include a pain threshold monitor that the patient may use to indicate whether the remote system 104 is applying excessive or excessive pressure. Input from the input device 216b may cause the system to disengage or otherwise reduce the applied pressure.

[0194] Reference now Figure 3 , Figure 3 is a simplified block diagram 300 illustrating the data flow between the local user control system 102 and the remote robotic ultrasound system 104 .

[0195] As shown, the local control system 102 may generate and transmit various user-generated data 302 to the remote system 104. The user-generated data 302 may include motion demand controls generated by the input device 180. The motion demand data may be used to control and manipulate the remote motion subsystem 210b (e.g., the robotic arm 150).

[0196] The user-generated data 302 may also include various data generated by the user input interface 190a, including (i) ultrasound control parameters for controlling ultrasound transducers in the remote environment (e.g., generated using an ultrasound keyboard); (ii) camera switching of a camera in the remote system 104; (iii) camera adjustments (e.g., pan, tilt zoom) of a camera in the remote system 104; and (iv) other user input commands (e.g., controlling tool change switching, operating tools, changing the operating mode of the remote system 104, etc. via the motion subsystem 210b). The user-generated data 302 may also include various audio and video data generated by the A / V system 208a. For example, this may include images and speech generated when a local operator interacts with a remote patient.

[0197] The remote system 104 may receive the user generated data 302 and, in response, may control the motion subsystem 210b and various ultrasonic tools 214b. The remote system 104 may in turn generate and send its own remote system data 304. For example, this may include current position data regarding the current position of the motion subsystem 210b. This may be based, for example, on the motion subsystem 210b ( Figure 2 ) is determined by sensor data generated by sensor 210b2 of a remote system. The remote system data 304 may also include ultrasound images generated by ultrasound tool 214b (e.g., ultrasound transducer). In various cases, video and audio data 304 may also be transmitted, including video and audio of the patient interacting with the remote sonographer.

[0198] In at least some cases, remote system data 304 may also be sent to server 106. For example, server 106 may store a log of actions and a database of various captured images.

[0199] General description of input device

[0200] The following is a discussion of example input devices that may be used alone or in any combination or sub-combination with any other one or more features described herein (e.g., a remote ultrasound system, a local user control system, a remote robotic control system, a remote ultrasound system, etc.).

[0201] Input devices are described herein and may be used alone, as part of a local control system 102, or in other ways in conjunction with the local control system 102. Figure 1A One or more components of the system 100a in are used in combination.

[0202] As described above, input device 180 enables control of motion subsystem 210b in a remote environment. Motion subsystem 210b may include, for example, remote robotic arm 150, or any other suitable system (eg, a gantry-type system) for holding and moving tools in a remote environment for ultrasound scanning purposes.

[0203] As described herein, the disclosed input device 180 provides one or more of spatial and directional control of an end effector of the motion subsystem 210b. The end effector is the portion of the motion subsystem that holds a tool for ultrasound scanning.

[0204] More specifically, spatial control enables the input device 180 to control the translation of the end effector to any spatial direction (e.g., along the X, Y, Z axes defined in the remote reference frame). In turn, spatial control allows the local operator to move, for example, a gel dispenser or an ultrasound transducer to different locations on the patient's body. Orientational control enables the input device 180 to control the remote tool to maintain the pitch, yaw, and roll of the end effector. This can, for example, allow the local operator to roll, shake, or rotate an ultrasound transducer attached to the end effector (e.g., to obtain enhanced ultrasound images).

[0205] In the example shown, the input device 180 includes a user movable arm system. The arm system can be moved to different spatial and directional positions by a local operator (e.g., a sonographer). In turn, the movement of the arm system enables the motion subsystem 210b to be controlled accordingly in a remote environment. The arm system can include a joystick (also referred to herein as a "pseudo-probe") that provides a simplified engagement interface for the user to engage the arm system.

[0206] To this end, the arm system includes one or more arm segments that are rotatably connected. The arm segments can provide up to six degrees of freedom of motion in spatial and orientation dimensions. In the example shown, the rotational connection between the arm segments is provided by various shafts and belt and pulley mechanisms.

[0207] As illustrated herein, the input device 180 may also include a sensor subsystem. The sensor subsystem may be used to sense and track the positional movement of the arm system. The data generated by the sensor subsystem may be used to generate a motion demand control for controlling the remote motion subsystem 210b. The sensor subsystem 212a may include, for example, one or more encoders (e.g., rotary and / or linear encoders). The encoder may be coupled to a rotatable shaft or a belt and pulley mechanism to track the rotation and movement of these mechanisms.

[0208] The input device 180 can additionally provide a tactile feedback function. Tactile feedback enables the input device 180 to mirror the forces and pressures experienced by the motion subsystem 210b in the remote environment. As provided herein, tactile feedback can be implemented by one or more high-torque DC motors of the input device 180. The DC motor can be coupled to a rotatable shaft and can apply a counter-rotational force to simulate the effects of drag. The DC motor can also stabilize the arm system in a static position against gravity.

[0209] To this end, the tactile feedback function has a number of understandable advantages: (i) first, the tactile feedback ensures that the user is prevented from moving the input device 180 to a position state that is otherwise inaccessible to the motion subsystem 210b in the remote environment (e.g., due to obstacles in the path of the motion subsystem 210b). In turn, the input device 180 can more closely track the motion in the remote system 104; (ii) second, certain diagnoses can be facilitated by monitoring different levels of pressure applied to the patient's body, which is facilitated by tactile feedback; and (iii) third, the local operator is prevented from applying excessive pressure to the patient, which may otherwise injure the patient.

[0210] Optionally, the input device 180 may also include an activation switch. The activation switch may operate as an input device clutch to connect or disconnect the motion of the input device 180 with the motion subsystem 210b. The activation switch may be mounted directly to the input device 180 (e.g., a joystick 450) to allow an operator to easily adjust the activation switch while manipulating the input device 180. Alternatively or additionally, the activation switch may be provided by the user input interface 190a.

[0211] The activation switch can be adjusted between an activated position and an inactivated position. The activation switch can be a contact sensor, a soft switch (e.g., a touch-sensitive area of ​​a touch screen on the input device 180), or a physically movable switch such as a slider, a pivot switch, a depressible button (such as a trigger), etc.

[0212] When the switch is in the activated position, the movement of the input device 180 can be connected to the motion subsystem 210b, so that the position movement of the arm system is used to generate the motion demand control for controlling the remote motion subsystem 210b. When the switch is in the inactive position, the movement of the input device 180 is disconnected from the motion subsystem 210b, so that the position movement of the arm system does generate the motion demand control for controlling the remote motion subsystem 210b. This can allow the operator to reposition or reset the input device 180 to rest or readjust to a more comfortable control position.

[0213] Reference now 4A to 4F , which illustrates an example of the input device 180.

[0214] As shown, the input device 180 may include a user-controllable arm system 402 supported to a mounting structure 404 .

[0215] As shown, the arm system 402 includes a robotic arm 410 extending between a first arm end 410a and a distal second arm end 410b. Figure 4A As shown in the figure, the robot arm 410 can be arranged along the horizontal longitudinal axis 420b ( Figure 4C) extends between the first end 410a and the second end 410b.

[0216] The robotic arm 410 itself may be mounted to a rotatable member 412. In the example shown, the rotatable member 412 comprises a rotatable half disk ( Figure 4G ). Alternatively, the rotatable member 412 may have any other suitable shape or design. The rotatable member 412 may in turn be secured to the mounting structure 404 (ie, rotatably secured).

[0217] As described herein, the combination of the robotic arm 410 and the rotatable member 412 enables a user to control the spatial and directional position of a tool holding end effector in the remote motion subsystem 210b. In the example shown, the robotic arm 410 - in combination with the rotatable element 412 - provides up to six degrees of freedom of motion control (i.e., spatial and rotational control).

[0218] The first end 410a of the robotic arm 410 may be coupled to a joystick-type device 412 (also referred to herein as a "virtual probe"). The joystick 412 may be engaged by a user to enable the user to move the arm system 402. The joystick 450 thus serves as a "stand-in" for a tool (e.g., a gel dispenser or an ultrasound transducer) located in a remote environment. The joystick 450 may include one or more contact sensors 442 ( Figure 4A ). The contact sensor 442 may be used to monitor user engagement of the joystick 412 .

[0219] As further shown, the second arm end 410b of the robotic arm 410 is indirectly fixed to the rotatable half-disc member 412. As explained, rotation of the half-disc member 412 allows for a corresponding rotation of the entire arm structure 410.

[0220] As further illustrated, the robotic arm 410 includes one or more arm segments 414 that are rotatably coupled. In the example shown, the robotic arm 410 includes five arm segments 414a-414e ( Figure 4A ). In combination, the arm segments provide up to five degrees of freedom of motion. The rotatable member 412 provides a sixth degree of freedom of motion.

[0221] Alternatively, the robot arm 410 may be provided with a different number of arm segments, such as three arm segments, which may reduce the degree of freedom of movement of the robot arm 410 while simplifying the control process.

[0222] For more details, refer to Figure 4C , the user-controllable arm system 402 can be segmented into two overall portions: (i) a first portion 418a for directional position control (ie, directional position control portion 418a), and (ii) a second portion 418b for spatial position control (ie, spatial position control portion 418b).

[0223] The directional control portion 418a enables a user to adjust the directional position of the joystick 450. In turn, the user can adjust the orientation of a tool located in a remote environment. In the example shown, the directional control portion 418a provides up to three degrees of freedom of motion. These three degrees of motion correspond to the yaw, pitch, and roll of the joystick 450. Thus, a user can interact with the input device 180 to, for example, roll, shake, or rotate the ultrasound transducer in the remote environment (e.g., to obtain an enhanced ultrasound image). In other cases, portion 418a may provide three or fewer degrees of freedom of motion.

[0224] In contrast, the spatial control portion 418b enables the user to adjust the spatial position of the joystick 450. The spatial control portion 418b may provide up to three degrees of freedom of motion (i.e., X, Y, and Z spatial movement). In turn, this may allow the operator to control the spatial position of a remote tool in the remote system 104. For example, the operator may move a gel dispenser or an ultrasound transducer to different locations on the patient's body.

[0225] Each of the system portions 418a, 418b is now discussed in greater detail herein, beginning with the directional control portion 418a.

[0226] refer to Figure 4A and 4C , the directional control portion 414a may include multiple arm segments of the robotic arm 410. These segments include: (i) a forward "wrist" arm segment 414a for yaw or rotation control; (ii) an intermediate arm rotation segment 414b for pitch or roll control; and (iii) a rear arm rotation segment 414c for roll control.

[0227] The forward arm segment 414a enables control of the rotational orientation of the joystick 450. For example, Figure 4F As shown, this includes moving the joystick 450 in both a clockwise direction 424a and a counterclockwise direction 424b about a first rotation axis 422a. The first axis 422a may also be referred to herein as a "yaw" axis 422a, such that the arm segment 414a enables control of the "yaw" angle of the joystick. In the initial position state, the first axis 422b may be substantially aligned with the vertical axis 422a in the local environment (i.e., the z-axis defined in the local reference frame 490).

[0228] like Figure 5AAs shown, in a remote environment, rotation of the arm segment 414a of the input device can control a corresponding rotation of the end effector 502 of the remote motion subsystem 210b. For example, the arm segment 414a of the input device can control a clockwise 508a (500a1) or counterclockwise 508b (500a2) rotation of the end effector 502. The end effector 502 can be used to hold various tools, such as an ultrasonic transducer. In the case of an ultrasonic transducer, the rotation can enable the user to switch between viewing the long axis and the short axis of a body structure or organ. The rotation can also be applied to any other tool connected to the end effector 502.

[0229] like Figure 4C As shown, the "wrist" arm segment 414a of the input device extends between a first end 414a1 and a second end 414a2 along a rotation axis 422a. The first end 414a1 can be rigidly connected to the joystick 450. More generally, the first end 414a1 also defines the first end 410a of the robotic arm 410. The second end 414a2 can be rotatably connected to the intermediate arm rotation segment 414b and rotate about the first axis 422a.

[0230] Now turning to the middle arm section 414b - the middle arm section 414b enables the joystick 450 to roll forward and backward. Figure 4E As shown, the middle arm segment 414b can rotate about a second rotation axis 422b that is orthogonal to the first axis 422a. The middle arm segment 414b can rotate in a counterclockwise direction 426a to provide forward rolling, and can rotate in a clockwise direction 426b to provide backward rolling. To this end, forward rolling 426a can refer to rolling away from the robot arm 410, while backward rolling 426b can refer to rolling toward the robot arm 410.

[0231] The second rotation axis 422b may also be referred to herein as a "pitch" axis 422b to enable control of the "pitch" angle of the joystick 450. When the arm system 402 is in the initial position state, the second axis 422b may be substantially aligned with the lateral horizontal axis 422a in the local environment (i.e., the x-axis defined in the local reference frame 490) ( Figure 4F ).

[0232] Brief reference Figure 5B , the rotation of the middle arm segment 414b enables the end effector 502 to roll accordingly in the remote environment. In this way, the ultrasound transducer 504 coupled to the end effector 502 can roll forward (500b1) and backward (500b2). In turn, this enables the operator to control the "deployment" of the transducer on the surface 506. As is known in the art, the deployment of the transducer facilitates the visualization of multiple cross-sectional images of the organ structure of interest.

[0233] like Figure 4C As best shown in FIG. 1 , the intermediate arm segment 414b also extends between respective first and second ends 414b1, 414b2 and along an axis that is co-linear with the first axis 422a. The first end 414b1 may be rotatably connected to the forward wrist segment 414a to enable rotation about the first axis 422a. The second end 414b2 may be rotatably connected to the rear arm segment 414c to enable rotation about the second axis 422b.

[0234] Finally, rear arm segment 414c controls the "roll" of joystick 450, or alternatively, the left-right rolling of joystick 450. Figure 4A and 4F , the rear arm segment 414b can rotate about a third axis of rotation 422c. The third axis 422c can be orthogonal to both the first axis 422a and the second axis 422b. The third axis of rotation 422c can also be referred to herein as a "roll" axis 422b so as to be able to control the "roll" angle of the joystick 450. When the arm system 402 is in the initial position state, the third axis 422c can be substantially aligned with, i.e., parallel to, the longitudinal horizontal axis 420c in the local reference system (i.e., the y-axis defined in the local reference system 490).

[0235] For this reason, Figure 4F As shown, rear arm segment 414 c may rotate about third axis 422 c in a clockwise rotation 428 a to facilitate leftward rolling of joystick 450 , and in a counterclockwise rotation 428 b to facilitate rightward tilting 450 of joystick 450 .

[0236] like Figure 5C As shown, the rotation of the rear arm segment 414c provides a corresponding roll of the end effector 502 in the remote environment. For example, the end effector 502 can roll, or roll in the left (500c1) or right (500b2) direction. With the end effector 502 holding the ultrasonic transducer, this movement enables the operator to "shake" the ultrasonic transducer. As is known in the art, shaking the ultrasonic transducer helps to visually "center" the desired area of ​​interest. It will be noted that Figure 5C Based on the axis in which the rolling occurs and Figure 5B Distinguish.

[0237] Return to reference Figure 4C , the rear arm segment 414c also extends along the third axis 422c between the respective first end 414c1 and the second end 414c2. The first end 414b1 can be rotatably connected to the intermediate arm segment 414b to enable rotation around the second axis 422a. The second end 414b2 can be rotatably connected to the translation arm segment 414d to enable rotation around the third axis 422c.

[0238] Thus, and in view of the foregoing, the three arm segments 414a-414c forming the orientation control portion 414a provide complete orientation control in any of the yaw, pitch, and / or roll directions.

[0239] Reference is now made to the spatial position control portion 418b of the arm system 402. The spatial control portion 418b includes (i) a forward translation arm segment 414d; (ii) a rearward translation arm segment 414e; and (iii) a rotating half disk 412 ( Figure 4C ).

[0240] The combination of these components enables the user to spatially move the joystick 450 in any X, Y, Z position within the local Cartesian reference frame 490. In turn, the user can control the tool to maintain the end effector 502 in a corresponding spatial position ( Figure 5D ).

[0241] In more detail, the forward translation arm segment 414d allows the joystick 450 to be translated upward and downward ( Figure 4C ), i.e., along the vertical axis 420a (i.e., the z-axis defined in the local reference frame). To this end, the arm segment 414d can rotate about a fourth axis 422d ( Figure 4F and 4E ) rotates, and the fourth rotation axis 422d is parallel to the x-axis in the local reference system. Figure 4E As shown, the arm segment 414d can rotate 430a counterclockwise about the fourth axis 422d to move vertically upward, and rotate 430b clockwise to move vertically downward.

[0242] like Figure 5D As shown, the operator can use arm segment 414d to control the corresponding vertical position of end effector 502 in the remote environment. That is, the operator can move the robotic arm end effector 502 upward (500d1) and downward (500d2) along the z-axis defined in the remote reference frame 510.

[0243] In the example case where the end effector 502 holds the ultrasound transducer 504, the transducer 504 can be moved downward to bring the transducer closer to the contact surface 506 (ie, the patient's body in the supine position) to generate an ultrasound image. In contrast, the tool 502 can be moved upward to disengage from the contact surface 506.

[0244] Optionally, the transducer 504 can be moved downward to apply pressure to the contact surface 506. For example, this can increase the visualization area and thereby improve the quality of the image captured. In other cases, downward pressure can be applied on one side of the transducer to tilt the probe. In turn, this can allow the operator to direct or tilt the ultrasound beam in a desired direction. Further, the downward pressure can allow compression of a vein, or push an anatomical structure away from the path of the intended needle.

[0245] like Figure 4C As shown, the forward translation arm segment 414d extends between a first end 414d1 and a second end 414d2 along an axis that is colinear with the third rotation axis 422c. The first end 414d1 can be rotatably connected to the rear arm segment 414c and rotate around the third rotation axis 422c. The second end 414d2 can be rotatably connected to the rear translation arm segment 414e to allow rotation around the fourth axis 422d.

[0246] Turning now to the rear translation arm segment 414e, this arm segment may allow translation of the joystick 450 forward 432a and rearward 432b ( Figure 4E ). Forward and backward motion can be defined along the longitudinal horizontal axis 420c (i.e., defined along the y-axis in the local reference frame) ( Figure 4C ). As used herein, the forward direction 432a may refer to a direction approaching a user operator, and the rearward direction 432b may refer to a direction away from the user operator.

[0247] like Figure 5D As shown, the operator can use the rear arm segment 414e to control the corresponding lateral position of the end effector 502 in the remote environment. That is, the operator can translate the end effector 502 forward (500d3) or backward (500d4) along the y-axis defined in the remote reference system 510. In turn, the tool 504 can be repositioned at different locations along the contact surface 506. This enables the ultrasonic transducer or gel dispenser to be spatially repositioned.

[0248] Return to reference Figure 4C , the rear translation arm segment 414e extends between a first end 414e1 and a distal second end 414e2. The first end 414e1 can be rotatably coupled to the forward translation arm segment 414d so as to be rotatable about a fourth rotation axis 422d ( Figure 4E The second end 414e2 may be rotatably mounted to enable rotation about a fifth axis of rotation 422e, for example to allow clockwise 432a and counterclockwise 432b rotation ( Figure 4E In the example shown, the second end 414e2 is rotatably mounted to the mounting bracket 436 ( Figure 4C), the mounting bracket 436 is connected to the rotating half disk 412.

[0249] Reference now Figure 4G As shown, the rotating half disk 412 is rotatably mounted to the mounting structure 404 and rotates about a sixth rotation axis 422e ( Figure 4E and 4G In the example shown, the rotation axis 422e is aligned with the local reference frame 490 ( Figure 4E ) and the longitudinal horizontal axis 420c ( Figure 4C ) are collinear.

[0250] Rotation of the rotatable half disk 412 causes the entire robotic arm 410 attached to the half disk 412 via the mounting bracket 436 to move in the right direction 434a and the left direction 434b. This, in turn, allows the joystick 450 to move along the x-axis defined in the local reference frame 490.

[0251] like Figure 5D As shown, rotation of half disk 412 enables corresponding movement of end effector 502 on the x-axis in remote reference frame 510. In this manner, an operator can remotely translate tool 504 in the right (500d5) and left (500d6) directions.

[0252] In view of the foregoing, the combination of front and rear translation arm segments 410d, 410e and rotating half-disc 412 provides complete spatial control along any Cartesian axis in both the local and remote reference frames.

[0253] In the example shown, a rotating shaft member is used to achieve the rotatable connection between the various robot arm segments 410 (and the rotating half-disc 412). The shaft member is disposed, for example, within the hollow interior of each arm segment.

[0254] As an example, Figure 4D As shown, the first shaft 430a rotatably connects the middle arm segment 414b to the wrist arm segment 414a. The first shaft 430a extends along the first rotation axis 422a. The second shaft 430b rotatably connects the middle arm segment 414b to the rear arm segment 414c and extends along the second rotation axis 422b. The third shaft 430c extending along the third axis 422c rotatably connects the rear arm segment 414c to the front translation arm segment 414d.

[0255] Similarly, shaft 430d rotatably connects front translation arm segment 414d and rear translation arm segment 414e together and extends along fourth axis 422d. Another shaft 430e extends along fifth rotation axis 422e ( Figure 4H) and rotatably connects the rear translation arm segment 414e to the mounting bracket 436. Further, the shaft 430f can extend along the sixth rotation axis 422e and can rotatably connect the rotation element 412 to the mounting structure 404. The diameter of the shaft 430f can be larger because it can bear the weight of the entire movable arm 410. Optionally, the shaft 430f is mounted inside the shaft receiving housing 438.

[0256] Alternatively, any other mechanism may be used to rotatably couple the various structural elements together.

[0257] As previously mentioned, the input device 180 ( Figure 2 ) can monitor and track the movement of the various arm segments 414 and rotating half disks 412. In the example shown, the sensor subsystem includes a plurality of encoders 440 (e.g., rotary encoders). Each encoder can be attached to the end of a rotating shaft 430 and can track the rotation of the shaft in clockwise and counterclockwise directions. For example, each shaft 430A-430F can have a corresponding rotary encoder 440A-440F coupled to one end of the shaft (see, e.g., Figure 4D as well as Figure 4B and 4C ).

[0258] Alternatively, the sensor subsystem may include an inertial measurement unit (IMU) coupled to the joystick 450. The IMU may monitor the pose (position and orientation) of the joystick 450.

[0259] The input device 180 may also include a plurality of motors 444 (eg, DC motors). The motors 444 may be coupled to one or more of the shafts 430a-430f and may be used to drive the shaft within each rotatable connection.

[0260] In more detail, the motors 444 can operate in one of two modes: (i) in a default mode, each motor 444 can run at a low current and apply only enough torque to keep each rotatable connection in a static position. That is, each motor 444 can control the corresponding axis 430 to keep the arm segment 414 and the rotating member 412 in a static state. In turn, this prevents the movable arm 410 from collapsing under the action of gravity when the user disengages the joystick 450; and (ii) in a tactile feedback mode, a higher current can be applied to one or more of the motors 444. A higher current can be applied so that the motor 444 produces a higher torque for a given axis. In turn, when trying to move the rotatable connection in a given direction, the user faces increased resistance.

[0261] The default mode may be activated based on the engagement or disengagement of the contact sensor 442 in the joystick 450. For example, once the user disengages the contact sensor 442, the motor 444 may operate in the default mode to hold the arm system in position. In contrast, once the user engages the contact sensor 442, the motor 444 may be ready to operate in the tactile feedback mode. Alternatively or additionally, when the contact sensor 442 is engaged, the default mode may be activated to help the user support the arm system against the downward force of gravity.

[0262] As shown in the illustrated example, at least six motors 444a-444e may be provided. Figure 4D As shown, at least three motors 444A-444C are respectively coupled to the rotatable shafts 440A-440C and otherwise drive the rotatable shafts 440A-440C. The motors 444a-444c can, for example, be respectively disposed within the hollow interior of the corresponding arm segments 414a-414d.

[0263] like Figure 4F As best shown in FIG. 4 , two additional motors 444d, 444e are provided for controlling the rotation of the front translation arm segment 414d and the rear translation arm segment 414e, respectively. Figure 4H As shown, motor 444e is connected to and controls shaft 430e associated with rear arm segment 414e.

[0264] Motor 444d may not be directly connected to shaft 414d, which is associated with front translation arm segment 414d. Figure 4E As best shown in FIG. 4 , motor 444d can be connected to shaft 444d via a belt and pulley mechanism 446. Optionally, belt and pulley mechanism 446 extends through the hollow interior of rear arm segment 414e. A larger motor may be required to control the heavy forward translation arm segment 414d. Due to its size, this larger motor may not be received within the arm segment (i.e., as shown in motors 444a-444c), and therefore may be positioned distally of arm segment 414d. Belt and pulley mechanism 446 thus connects distal motor 444d to drive the forward translation arm segment 414d.

[0265] In more detail, motor 444d can be connected to and drive motor shaft 448. Motor shaft 448 itself is connected to a first pulley or gear 446a. In addition, shaft 444d that rotatably connects front arm segment 414d and rear arm segment 414e can be connected to a second pulley (or gear) 446b. Belt 446c is wrapped around each of gears 446a, 446b and can be fully tensioned by one or more tensioning rollers 446d. In this way, motor 444d can first rotate pulley 446a (via motor shaft 448), which in turn can cause rotation of second pulley 446b, thereby causing rotation of front translation arm segment 446b (via shaft 444d).

[0266] like Figure 4E and 4G As best shown in FIG. 4 , a similar belt and pulley mechanism 452 may be used to drive the rotatable half disk 412. For example, the rotatable half disk 412 may be driven by a motor 444e, which in turn drives a corresponding motor shaft 454. The motor shaft 454 may be rotatably coupled at one end to a pulley (or gear) 452a ( Figure 4E ). A second pulley 452b may be coupled to the shaft 444e, which controls the rotation of the rotatable half disk 412. A belt 452c may be wound around each of the pulleys 452a, 452b and may be passed through one or more tensioning rollers 452d ( Figure 4G ) is sufficiently tensioned. Thus, motor 444e can rotate pulley 452a (via motor shaft 454), which in turn can cause rotation of second pulley 452b, thereby causing rotation of half disk 412 (via shaft 444e).

[0267] Return to reference Figure 4C , the input device 180 may also include a control box 456. The control box 456 may house, for example, a processor connected to each of the motors 444A-444E, various encoders 440, and a joystick contact sensor 442. The control box 456 may also house a power supply for powering the motors 444.

[0268] As previously described, in order to accommodate the weight of the movable arm system 402, the input device 180 may include a mounting structure 404. Figure 4C As best illustrated in FIG. 4 , the mounting structure 404 may include a support frame 406. In an upright position, the support frame 406 may include a base portion 406a and a vertical portion 406b coupled to the base. Figure 4A and 4B As shown, each of the base and vertical portions 406a, 406b may include one or more respective support beams 406a1-406a3 (ie, base portions) and 406b1-406b2 (ie, vertical portions).

[0269] In the illustrated example, the mounting plate 408 is supported between the vertical members 406a, 406b ( Figure 4B ). The mounting plate 408 can be used to mount various components of the input device 180 (eg, the movable arm 410 , etc.). In other examples, the mounting plate 408 can be integrally formed with the frame 406 .

[0270] Reference now Fig. 4I , which shows a simplified electrical hardware block diagram 400i of the input device 180.

[0271] As shown, the input device 180 may include an input device processor 402i, which is coupled to one or more of the following: a memory 404i, a motor subsystem 406i, a power supply 408i, a sensor subsystem 410i, an input / output (I / O) interface 412i, and one or more contact sensors 442 in a joystick 450.

[0272] The processor 402i and the memory 404i may be similar in architecture to Figure 2 The processor 202a and the memory 204a in.

[0273] The motor subsystem 406 i may include one or more motors 440 a - 440 f for adjusting the position state of the arm system 402 .

[0274] The power supply 408i may include, for example, one or more batteries for powering the motor 444. In other cases, the power supply 406i may include an electrical outlet. To this end, the processor 402i may control the power supplied from the power supply 406i to the motor 444. For example, as previously described, in the default operating mode, the processor 402i may supply a low current to each of the motors 444. Otherwise, in the tactile feedback mode, the processor 402i may supply a higher current (e.g., 0 to 60 amps) to each of the motors 444 as needed.

[0275] The sensor subsystem 410i may include one or more sensors for monitoring the positional state of the arm system 402. For example, these may include encoders 440a-440f.

[0276] The I / O interface 412i may couple the input device 180 to the workstation 190 ( Figure 2). The I / O interface 412i can enable the input device 180 to send encoder data from the encoder 440 to the workstation 190. In turn, the workstation 190 can generate and send control instructions to control the motion subsystem 210b in the remote system 104. In other cases, the input device memory 404i itself can store a program that can directly generate control instructions from the encoder data. The I / O interface 412i can also receive control instructions from the workstation 190. For example, the workstation 190 can receive position data from the remote motion system. The control instructions are generated by the workstation 190 to achieve control of the motor 444 to provide a tactile feedback effect to the user. In other cases, the input device memory 404i itself can store a program that can directly generate control instructions for the motor 444 based on the position data from the remote motion system.

[0277] Optionally, the input device 180 may also include a communication interface that may allow for direct communication between the input device 180 and the remote system 104 (eg, via the network 110 ).

[0278] Remote robot control system

[0279] The following is a discussion of an example telerobotic control system, which may be used alone or in any combination or sub-combination with any other one or more features described herein (e.g., teleultrasound systems, input devices, local user control systems, methods for operating teleultrasound systems, etc.).

[0280] As described in more detail herein, the remote control system may include a robotic arm. The distal end of the robotic arm may include an end effector that may include a tool holding mechanism for holding various tools required to perform ultrasound scanning (e.g., a gel dispenser, an ultrasound transducer, etc.). The robotic arm may be configured for up to six degrees of freedom of motion. The remote control system may also include one or more object detection cameras that may be used to monitor the environment for any obstacles that may collide with the robotic arm.

[0281] Reference now Figures 6A-6C , which shows an example of a remote robotic control system 104.

[0282] As shown, the remote system 104 may include a motion subsystem 210b for performing ultrasound scanning. In the example shown, the motion subsystem 210b includes a robotic arm 150.

[0283] As further illustrated, the robotic arm 150 extends between a first arm end 150a and a distal second arm end 150b. The first arm end 150a can be coupled to an end effector 152 for holding and retaining various tools for performing ultrasound scanning (e.g., ultrasound transducer, gel dispenser, etc.). The second arm end 150b can be mounted to a mounting surface to support the weight of the robotic arm 150. The mounting surface can include, for example, a wall 602 ( Fig. 6A ), such as a side wall or ceiling in a scanning room. The mounting surface may also include a movable robot body 158 ( Figure 6B , 6C ). To this end, the robot body 158 can allow the robot arm 150 to be more easily transported between different environments.

[0284] As shown, the robotic arm 150 can be arranged so that at least the end effector 152 engages the patient 604. In the case where the patient is lying in a supine position (e.g., on a bed frame 160), the robotic arm 150 can be positioned vertically above the patient. In other cases, the patient can be in a standing position, in which case the robotic arm 150 can be positioned, for example, in front of the patient.

[0285] As described herein, the robotic arm 150 can be adapted for several degrees of freedom of motion. The robotic arm 150 (similar to the input device 180) can be configured for up to six degrees of freedom of motion. This in turn enables the robotic arm 150 to change the spatial position (e.g., X, Y, Z Cartesian position) and orientation position of the end effector 152.

[0286] In particular, changing the spatial position of the end effector allows the end effector 152 to reach different locations on the patient's body, and more generally, to reach different spatial locations in a remote environment (e.g., to access a tool holder for tool change). In addition, the ability to configure the end effector 152 in different directional positions can further assist in the use of a particular tool. For example, as discussed, the ability to rotate, shake, or roll an ultrasound transducer attached to the end effector 152 can enable the capture of more enhanced ultrasound images.

[0287] Now briefly refer to Fig. 7A , which shows a schematic diagram of an example robotic arm 150.

[0288] As shown, similar to the input device 180, the robotic arm 150 may include a plurality of arm segments 194a-194b rotatably connected via rotatable joints 706a-706f.

[0289] To achieve six degrees of freedom of motion, the robot arm 150 may include: (i) a first part 702a for directional position control of the end effector 152 (i.e., directional control part 702a), and (ii) a second part 702b for spatial position control of the end effector 152 (i.e., spatial position control part 702b).

[0290] Similar to input device 180, orientation control portion 702a may include: (i) a forward "wrist" arm segment 194a for yaw or rotation control; (ii) an intermediate arm rotation segment 194b for pitch or roll control; and (iii) a rear arm rotation segment 194c for roll control. End effector 152 may be attached to forward wrist arm segment 194a.

[0291] As shown, the rotatable connection 706a rotatably connects the wrist arm segment 194a to the middle arm segment 194b. In addition, the rotatable connection 706b rotatably connects the middle arm segment 194b to the rear arm segment 194c.

[0292] It should be appreciated that the arm segments 194A-194C may have a similar mode of operation as the corresponding directional control arm segments 414A-414C in the input device 180 so as to also provide up to three degrees of freedom of movement.

[0293] The spatial position control portion 702 b of the robotic arm 150 may also include: (i) a forward translation arm segment 194 d ; (ii) a rear translation arm segment 194 e ; and (iii) a rotation support member 712 .

[0294] The arm segments 194D-194E and the rotating member 712 may also have similar modes of operation as the corresponding arm segments 414D-414E and the rotating member 412 in the input device 180, respectively. For example, the forward arm segment 194d enables the end effector 152 to translate forward or backward (i.e., move the end effector 152 along the y-axis defined in the remote Cartesian reference frame 714). The rear translation arm segment 194e enables the end effector 152 to translate vertically upward and downward (i.e., z-axis translation in the remote reference frame 714). The rotating member 714 can allow the end effector 152 to "swing" in the left and right directions (i.e., x-axis translation as defined in the reference frame 714).

[0295] Although not shown, multiple internal motors and shaft assemblies can control the rotation of each of the rotary joints 706a-706f. The motors can be controlled by a controller or processor of the remote system 104 (e.g., Figure 2 The processor 202 b in the embodiment is controlled to achieve the desired position configuration of the end effector 152 .

[0296] Encoders (e.g., rotary encoders) 718a-718e can be connected to the end of each shaft in each motor and shaft assembly. The encoders can generate encoder data that can be used to determine the angular position of each joint 706a-706f. In turn, this data can be used to monitor and track the spatial and orientation position of the end effector 152. In other cases, any other position tracking sensor can be used to track the position movement of the robot arm 150. For example, an inertial measurement unit (IMU) can be mounted to the end effector 152. The IMU can directly measure the posture (position and orientation) of the end effector 152.

[0297] As described herein, the robotic arm 150 may operate in various modes including: (i) a non-autonomous guided mode; (ii) a semi-autonomous, semi-guided mode; and / or (iii) a fully autonomous non-guided mode.

[0298] As previously described, in the guided mode, the robotic arm 150 can be fully controlled by the input device 180. That is, the robotic arm 150 can adjust the position (e.g., spatial and directional position) of the end effector 152 based on the control instructions received from the input device 180. In the semi-autonomous mode, the robotic arm 150 can be partially controlled by the input device 180. For example, the robotic arm 150 can be controlled by the input device 180 in some cases, but can operate autonomously in other cases. For example, the path movement of the robotic arm 150 can be automatically controlled to avoid detected obstacles. In the fully autonomous mode, the robotic arm 150 is also fully autonomously controlled by the controller of the remote system 104.

[0299] Figure 7B Another example design configuration of a robotic arm 150 is shown.

[0300] Reference now Fig. 8A , which shows a simplified hardware block diagram of an example end effector 152 of a robotic arm 150 .

[0301] As shown, the end effector 152 may include multiple subcomponents, including a tool holding mechanism 802 , and one or more cameras 804 , pressure sensors 806 , and / or object detection sensors 808 .

[0302] The tool holding mechanism 802 can be used to hold and replace various tools used for ultrasound scanning (eg, gel dispenser, ultrasound transducer, etc.) It should be understood that the tool holding mechanism 802 can have one of a variety of possible configurations. FIG. 8B to FIG. 8F An example configuration of a tool holding mechanism is shown. In particular, Figure 8B An example configuration using a mechanical gripper is shown, Figure 8C An example configuration using permanent magnets is shown, and Figures 8D-8F An example configuration using a mechanical coupling mechanism is shown.

[0303] First reference Figure 8B , the end effector 512 may include a main body portion 850 extending between opposite first and second ends 850a, 850b. The first end 850a may be coupled to the robotic arm 150. The second end 850b may include mechanical grippers 852a, 852b for holding various tools 154. As shown, the grippers 852a, 852b may expand outward to disengage or release the tool (see 800b1) and may retract inward to engage or hold the tool (see 800b2). Once the tool is held, the tool may be moved around and redirected by the robotic arm 150. In some example cases, the main body portion 850 may house one or more servo motors 854, which may be used to control the gripper 852.

[0304] Although in Figure 8B Only two mechanical grippers 852 are shown in FIG. 8 , but it should be understood that the end effector 152 may include any number of grippers.

[0305] exist Figure 8B (as well as Figure 8C and 8D ), the end effector 152 is configured to hold and manipulate a tool 154 in an orientation that is co-linear with the direction in which the end effector 152 extends from the robotic arm 150.

[0306] Alternatively, the end effector 152 can be configured to hold the tool at a different position or angle. For example, the end effector 152 can be configured to hold the tool 154 at an offset position parallel to but not co-linear with the direction in which the end effector 152 extends from the robot arm 150.

[0307] Alternatively or in addition, the end effector 152 may be configured to hold the tool 154 at approximately 90 degrees or a right angle, such as Figure 8G When the end effector 152 holds the tool 154 in a co-linear orientation, this can introduce singularities into the determination of the motion path of the end effector 152. Adjusting the angle at which the tool 154 is held can reduce these singularities.

[0308] As another example, the end effector 152 can be configured to hold the tool 154 at an angle between 0 degrees and 90 degrees, such as about 45 degrees. Figure 8H This can reduce singularities in motion path determination while providing a more natural tool holding and manipulation position that can reduce unintentional collisions with patients and other objects.

[0309] Figure 8C Another example configuration of the end effector 152 is shown. In the example shown, a permanent magnet 860 is disposed within the hollow interior 856 of the body 850 of the end effector. The permanent magnet 860 can translate between an unengaged position (800c1) and an engaged position (800c2). In the engaged position (800c2), the magnet 860 can translate toward the second body end 850b. In this position, the permanent magnet 860 can attract a corresponding magnet 864 located inside or on top of the tool 154. The magnetic attraction between the magnets 860, 864 allows the end effector 152 to hold the tool 154.

[0310] In the unengaged position (800c1), the permanent magnet 860 can be translated away from the second housing end 850b and toward the first housing end 850a. When translated toward the first end 850a, the magnetic attraction between the magnets 860, 864 is weakened. This in turn allows the end effector 152 to disengage and release the tool 154 (e.g., switch to a new tool).

[0311] In the example shown, the permanent magnet 862 can translate between the engaged position and the disengaged position along the axial track 862. One or more servo motors 854 can be used to translate the permanent magnet 862 along the axial track 862. In other examples, other translation mechanisms can be used to translate the magnet 862 between the engaged position and the disengaged position.

[0312] Alternatively, an electromagnet may be used instead of a permanent magnet. The electromagnet may be disposed proximal to the second end 850b of the end effector body 850. The electromagnet may be switched between an activated state and an inactivated state. In the activated state, an electric current is applied to the magnet to activate the magnet and allow the magnet to engage a magnet 864 associated with the tool 154. In the inactivated state, the applied current is reduced or otherwise removed, thereby releasing the tool 154 from engagement with the end effector 152. The current may be applied using, for example, a power source located in the remote system 104.

[0313] Fig.8D Yet another example configuration of an end effector 152 is shown. In this example configuration, the end effector 152 includes a mechanical connector. The mechanical connector is used to mate with a reciprocating connector associated with a tool 154.

[0314] In more detail, the end effector 152 can include a female connector that engages with a male connector 870 associated with the tool 154. Any suitable mating engagement between the two connectors can be used. For example, the male connector 870 of the tool can include a protruding member 880. The protruding member 880 can be slidably received in a slot located in the female connector of the end effector. In this way, the end effector 152 can quickly engage and disengage various tools 154 located in the tool holder area 156.

[0315] In other cases, a female connector may be associated with tool 154 and a male connector may be associated with end effector 152 .

[0316] like Fig. 8E As shown, each tool 154 can be retained within a housing shell 872. The housing shell 872 can have any suitable design. In the example shown, the housing shell 872 is formed by two housing portions 872a, 872b that are connected to clamp around at least a portion of the tool 154. The housing shell 872 can further extend between a first end 872a and a second end 872b. The first end 870a can be coupled to the connector 870, while the second end 870b can include an opening through which the tool 154 protrudes. In the case of the transvaginal probe 154c, the housing 872 can have a right angle design to facilitate insertion of the probe into the patient's vagina ( Figure 8F ).

[0317] In some example cases, the housing shell 872 can also hold a flexible sleeve 874. The flexible sleeve 874 can include a hole 876 that can receive at least a portion of the tool 154 within the shell 872. The flexible sleeve 874 can provide freedom of movement between the tool 154 and the shell 872. For example, this can allow the transvaginal probe 154c to have some flexibility of movement when inserted and navigated within the patient's vagina. In turn, this prevents injury to the patient that may occur if the probe 154c is rigidly fixed to the shell 872.

[0318] Return to reference Fig. 8A , the end effector 152 may also include one or more of a camera 805 , a pressure sensor 806 , and / or an object tracking sensor 808 .

[0319] The camera 804 may be positioned on the end effector 152 to assist in capturing various images and video feeds (eg, a live video feed) (see, e.g., Figure 7B These images and video feeds may be transmitted back to the local control system 102 and may be displayed to a local user operator (e.g., at Figure 2This can allow the operator to more closely track the position of the end effector 152 relative to the patient. To this end, the camera 804 can form the A / V subsystem 206b of the remote system ( Figure 2 ) part.

[0320] The pressure sensor 806 (or multiple pressure sensors 806) can be used to monitor the pressure applied to the patient by the end effector 152. For example, the pressure sensor 806 can monitor the applied pressure to determine whether the pressure is within an acceptable range. In turn, this can ensure that the robotic arm 150 does not otherwise apply excessive force to the patient, which could otherwise injure the patient.

[0321] In some example cases, if the recorded pressure data is above a predetermined threshold, the remote system 102 may disable the robotic arm 150, or otherwise automatically adjust the robotic arm 150 to minimize the applied pressure. In some cases, "disabling" the robotic arm 150 may involve automatically moving the end effector 152 to a predetermined spatial position away from the patient.

[0322] In some example cases, pressure data generated by pressure sensor 806 may be transmitted to local system 102 to allow a local operator to monitor applied pressure during a scanning procedure.

[0323] Example pressure sensors 806 (or sensors 806) that may be incorporated into end effector 152 include various types of load cells, including pneumatic load cells, capacitive load cells, strain gauge load cells, and / or hydraulic load cells. In some cases, multiple pressure sensors 806 may be provided on end effector 152 to monitor applied forces or pressures from multiple directions.

[0324] The object tracking sensor 808 may include, for example, various object tracking cameras, including time-of-flight (ToF) sensors. The object tracking sensor 808 (or multiple object tracking sensors 808) may be used to automatically detect obstacles in the path of the end effector. In the semi-guided mode and the autonomous mode, the motion data generated by the sensor 808 (or multiple sensors 808) may be used to automatically control the robotic arm 150 to avoid the detected obstacles. More generally, the sensor 808 may also be used to detect the position and movement of the patient to achieve precise tool positioning relative to the patient, as well as to adapt the movement of the robotic arm 150 relative to the movement of the patient.

[0325] Return to reference Figure 6B and 6C, various tools required to perform ultrasound scanning can be held in the tool holder area 156. For example, the tool holder 156 can hold various types of ultrasound transducers and gel dispensers. The tool holder 156 can be mounted on the lateral side of the robot body 158, for example.

[0326] To this end, when it is desired to exchange tools, the remote system 104 can translate the robotic arm 150 and the end effector 152 to the tool holder area 156. The end effector 152 can then disengage (or drop) the currently held tool and engage (or pick up) the new tool.

[0327] Alternatively, the remote control system 104 may have advanced knowledge of the exact location (eg, X, Y, Z location) of the tool holder area 156 within the remote reference environment. This information may be stored, for example, in the memory 204b of the remote control system 104.

[0328] Thus, each time a tool change is desired, the remote system 104 can automatically control the robotic arm 150 to translate and align the end effector 152 with a known position of the tool holder area 152. In a guided or semi-guided mode, this can occur in response to receiving a user input command to swap tools from the local system 102. In a fully autonomous mode, this can occur automatically in response to the system determining that a new tool is needed.

[0329] In some other cases, the remote control system 104 may also have knowledge of the exact location of each tool within the tool holder area 156. For example, each tool may be associated with a predefined coordinate position (e.g., X, Y, Z position) in the remote reference system, corresponding to its designated position in the tool holder area 156. Thus, when it is necessary to drop a tool or otherwise pick up a tool, the remote system 104 may reference that coordinate position. In some cases, the control system memory 204b may store a table or database of tools and their associated coordinate positions in the remote reference system.

[0330] It should be appreciated that in various circumstances, the tools 154 may not necessarily be gathered into a single tool holder area 156. For example, the tools may be dispersed across different locations within a remote environment.

[0331] like Figure 6B and 6C As further shown in FIG. 1 , the electronic tool 154 can be connected via a data cable 690 to send or receive data. For example, the data cable 690 can connect the ultrasonic transducer 154a to the controller of the remote system 104. The controller can include not only the remote system 104 ( Figure 2) processor 202b, and may also include various drivers required to operate the ultrasonic transducer. In the example shown, the controller can be housed within the robot body 158, and the data cable 690 can be connected to the input port 606 provided on the robot body 158. In other cases, the controller can be provided in an external housing 650 ( Fig. 6A )middle.

[0332] To this end, the data cable 690 can be used to transmit commands to change the configuration parameters of the ultrasound transducer 154a. For example, these can include user commands received from the local system 102. These can also include commands automatically generated by the remote system 104. The transducer 504a can also transmit ultrasound images to the controller via the data cable 604.

[0333] In other cases, some of the electronic tools 154 may also be adapted for wireless communication.

[0334] In the case of gel dispenser 154b, gel dispenser 154b may be connected to gel tank 610 via conduit 608. Gel tank 610 may supply gel to gel dispenser 154b via a controllable pump mechanism 612 (ie, controlled by a remote system controller). In other cases, gel may be gravity fed from gel tank 610.

[0335] In some cases, remote system 104 can monitor the amount of gel consumed in gel tank 610. For example, this can be performed via a level sensor located inside gel tank 610. In other cases, monitoring can be based on a known maximum fill volume of gel tank 610, combined with a known amount of gel previously dispensed. The amount of gel dispensed can be determined, for example, based on a time period that pump 612 is activated, combined with a known flow rate of pump 612. When it is determined that the amount of remaining gel is below a predetermined threshold, an alarm can be generated to refill gel tank 610.

[0336] Alternatively, the gel dispenser may be provided in the form of a disposable gel container (e.g., a plate or bottle). The gel may be provided by the gel container, and the end effector 152 may manipulate the ultrasound transducer probe to extract the gel from the container (e.g., by dipping the probe into the container). The probe may then be manipulated to dispense the gel onto the patient. This may avoid challenges caused by the gel becoming stuck or drying up within the dispensing tube.

[0337] like Fig. 6A As shown, the remote system 104 may also include one or more object tracking sensors 164 mounted around the environment. The object tracking sensors 164 may include, for example, object tracking cameras (eg, LiDAR or other time-of-flight sensors) to monitor the environment around the robotic arm 150.

[0338] Similar to the object tracking sensor 808 in the end effector 152, the object tracking sensor 164 can be used to detect obstacles in the remote environment, for example to prevent collisions between the obstacle and the robotic arm 150. For example, in response to detecting an obstacle, the remote system 104 can stop the movement of the robotic arm 150 and wait for further instructions from the local system 102. Otherwise, in semi-guided and fully autonomous modes, the remote system 104 can automatically plan the movement of the robot's arm around the obstacle. The object tracking sensor 614 can also be used more generally to detect the position and movement of the patient, again to achieve precise tool positioning relative to the patient, and to adapt the movement of the robotic arm 150 relative to the patient's movement.

[0339] like Fig. 9 As shown, object tracking sensors 614 can be positioned around remote environment 900 in different configurations to enable motion capture from different angles. For example, object tracking sensors 614a-614e can be positioned along different lateral sidewalls 902-908 of scanning room 900 and ceiling 910.

[0340] More generally, the remote system 104 may also include one or more cameras 164 for generating a video feed (e.g., a real-time or near real-time feed). The feed generated by the camera 164 may be sent back to the local system 102. In particular, the camera feed may enable a local user operator to monitor the patient environment, as well as observe and track robotic arm movements from multiple perspectives.

[0341] like Fig. 9 As shown, the cameras 164a-164d may also be positioned at different locations around the bed frame 160. Thus, each camera 164 may provide a different perspective angle of the patient and the robotic arm 150. The cameras 164 may also be mounted in other locations, such as along one or more of the lateral side walls 902-908 or the ceiling 910 of the room. The cameras 164 may be configured to be remotely controlled (e.g., pan, tilt, zoom) by the local control system 102 in response to operator input. This may allow the operator to adjust the view of the patient environment in response to changes in the environment (e.g., a patient in a different position or with a different size).

[0342] To this end, the cameras 164A-164D may form the A / V subsystem 206b of the remote system ( Figure 2 ). In some example cases, such as Fig. 6AAs best shown in , the A / V subsystem 206b may also include a display device 616 that can display a video feed of a local operator to the patient. In this way, a remote local operator can interact with the patient to provide guidance instructions while also allowing the patient to interact with the local operator. The A / V subsystem 206b may also include a speaker and a microphone to send and receive audio.

[0343] like Fig. 6A As best shown in FIG. 1 , the remote system 104 may also include a pain threshold monitor 618, which is similar to the patient feedback device 216b ( Figure 2 ). The patient may activate the pain threshold monitor 618 when the robotic system applies excessive or uncomfortable force to the patient. Once activated, the threshold monitor 618 may send an activation signal to the system processor 202b. In response, the remote system may reduce the force applied to the patient by the robotic arm 150, or may otherwise disengage the arm 150 entirely. Upon disengaging the arm 150, the system may move the arm 150 to a predetermined default position away from the patient, or may otherwise stop any further movement of the arm 150. In some cases, the system may also cut off power to the arm 150 when the pain threshold monitor 618 is activated.

[0344] The pain threshold monitor 618 can have any suitable design. For example, the monitor 618 can include a clickable button. In some example cases, the button can be pressed to generate an activation signal. In other cases, the user can initially hold the button and then release the button to generate an activation signal.

[0345] Alternatively, the pain threshold monitor 618 may include a hand clamp type mechanism that can be squeezed or released to generate an activation signal. In some cases, the clamp may incorporate a pressure sensor that can monitor the pressure applied by the user.

[0346] The pain threshold monitor 618 may not necessarily generate a binary output (i.e., activated or deactivated). Instead, the monitor 618 may generate a level, gradient, or intensity of an activation signal. For example, where the monitor 618 comprises a hand clamp, the user may vary the degree of squeeze or release of the clamp to indicate different levels of pain experienced. The system may then adjust and adapt the intensity of the received activation signal by reducing the force applied by the robotic arm 150 in a linear or non-linear ratio.

[0347] Method for operating a remote ultrasound system

[0348] The following is a discussion of example methods for operating a remote ultrasound system, which may be used alone or in any combination or sub-combination with any other feature or features described herein (e.g., a remote ultrasound system, an input device, a local user control system, a remote robotic control system, etc.).

[0349] Reference now Fig.10 , which illustrates a process flow of an example method 1000 for controlling a telepresence robotic system 104 using an input device 180 of a local control system 102. The method 1000 may be performed, for example, when the telepresence robotic system 104 is operating in a non-autonomous, guided mode or otherwise in a semi-autonomous, semi-guided mode.

[0350] As shown, at 1002, the local control system 102 can detect a new position state of the input device 180. For example, when the input device 180 includes the user-controllable arm system 402 (ie, FIG. 4A to FIG. 4I ), action 1002 may involve detecting a spatial or rotational position change in arm system 402. The position change may be caused, for example, by a local operator engaging joystick 450 and applying force to joystick 450. To this end, detecting the position change at action 1002 may involve detecting a position change caused by sensor subsystem 410i ( Fig. 4I In particular, this may include detecting new encoder data generated by encoders 440A-440F that are coupled to a plurality of rotating axes in input device 180, as previously described.

[0351] If a change in position is detected, the local control system 102 may obtain position data associated with the new position state of the input device at 1004. For example, this may involve obtaining encoder data generated by each encoder 440a-440f associated with each rotatable connection in the input device 180 (e.g., the rotatable connections between the respective arm segments 414a-414e and the rotatable member 412). Optionally, at act 1004, the local control system 102 may perform initial filtering of the obtained encoder data to produce filtered encoder data (or more generally, filtered sensor data).

[0352] At 1006, the local control system 102 may process the new position data (eg, encoder data) to solve the forward kinematics model. Figures 4A-4I The forward kinematics model can express the new position (eg, new spatial and directional position) of the joystick 450 for the input device 180. An input-side transformation matrix can then be generated that includes various parameters of the solved forward kinematics model.

[0353] At 1008, the local control system 102 may transmit the input-side motion demand data to the remote system 104. The input-side demand data may include the determined input-side transformation matrix. As explained herein, the input-side motion demand data constitutes control instructions for controlling the motion subsystem 210b (e.g., the robotic arm 150) in the remote environment.

[0354] At 1010 , the telepresence robotic system 104 may receive input-side motion demand data from the local control system 102 , such as via the network 110 .

[0355] At 1012, the telepresence robotic system 104 may analyze the input-side transformation matrix to determine a new position state of the motion subsystem 210b. To this end, action 1012 may involve solving an inverse kinematics model based on the parameters included in the input-side transformation matrix. In the example case where the motion subsystem 210b includes a robotic arm 150, solving the inverse kinematics model may allow for determination of new joint angles for each of the joints 706a-706f of the robotic arm.

[0356] At 1014, the telerobotic system 104 may adjust the motion subsystem 210b to the new position state. That is, the telerobotic system 104 may manipulate the motion subsystem 210b to generate a new position for the end effector 152. In the example case where the motion subsystem 210b comprises a robotic arm 150, the action 1014 may involve controlling the arm's internal motors, which are connected to each of the robotic arm's joints 706a-706f, to achieve the desired target joint angles. A proportional-integral-derivative (PID) loop may be used to achieve the new position of the end effector in the motion subsystem.

[0357] At 1016, the telepresence robotic system 104 may monitor for a change in the position state (e.g., a change in the spatial or directional position state) of the motion subsystem 210b. That is, the telepresence robotic system 104 may monitor whether the position state of the system's end effector 152 has changed from the last position generated at act 1014. In various example situations, the change in position state may result in a situation where an external force is applied to the motion subsystem, such as a situation where the motion subsystem 210b collides with an obstacle (e.g., a patient's body or an external object).

[0358] In at least some example cases, where the motion subsystem 210b includes a robotic arm 150, the action 1016 can be performed by monitoring data generated by sensors in the robotic arm 150. For example, this can involve monitoring encoder data generated by each of the robotic arm's encoders 718A-718E, which are coupled to the various arm joints 706A-706F ( FIG. 7 ).

[0359] At 1018, the telepresence robot system 104 may generate a robot-side transformation matrix. The robot-side transformation matrix may be generated by solving a forward kinematics model based on encoder data generated, for example, by encoders 718A-718E of the robot arm. In at least some cases, the sensor data (e.g., encoder data) may initially be pre-filtered to produce filtered sensor data. The filtered sensor data may then be used to solve the forward kinematics model to generate the robot-side transformation matrix.

[0360] At 1020 , the telepresence robotic system 104 may transmit robot-side motion demand data including a robot-side transformation matrix to the local control system 102 .

[0361] At 1022 , the local control system 102 may receive robot-side motion demand data, such as via the network 110 .

[0362] At 1024, the local control system 102 may analyze the robot-side transformation matrix included in the received robot-side motion demand data. Based on the analysis, the local control system 102 may determine a new position state (e.g., a new spatial and orientation state) of the input device 180. For example, where the input device 180 includes a user-controllable arm system 402, action 1024 may involve determining a new position state of the joystick 450. This may be determined, for example, by solving an inverse kinematics model based on the received robot-side transformation matrix. Solving the inverse kinematics model may allow determination of a new angle for each rotatable connection in the input device 180.

[0363] At 1026, the input device 180 may be controlled to adjust its position state to the determined new position state. For example, this may involve controlling the motors 444a-444e in the motor subsystem 406i of the input device to adjust the angle of each rotatable connection in the input device 180. In turn, this may adjust the spatial and / or directional position of the joystick 450 in the input device 180. For example, a PID loop may be used to implement the new position state.

[0364] Here, it should be appreciated that actions 1016 - 1026 provide a haptic feedback effect in input device 180 because input device 180 is controlled to mirror the external force that may be applied to the remote motion subsystem at action 1016 .

[0365] After act 1026 , method 1000 may return to act 1002 to again detect a new position state of input device 180 , ie, resulting from interaction of the local user operator with input device 180 .

[0366] Optionally, before performing action 1002 - the local control system 102 and the telerobotic system 104 may perform an initialization protocol. The initialization protocol may involve the telerobotic system 104 generating an initial robot-side transformation matrix and transmitting it to the local control system 102. The initial robot-side transformation matrix may reflect an initial position state of the motion subsystem. The local control system 102 may also generate its own input-side transformation matrix, which corresponds to an initial position state of the input device.

[0367] The local control system 102 can then analyze the robot-side transformation matrix to determine the offset of its origin frame from the robot origin frame based on the difference between the two transformation matrices. This can be performed to account for the significantly smaller working volume of the input device 180 compared to the workspace of the remote robot system. The system 102, 104 can then use this dynamically offset representation of the working volume of the input device in the robot's workspace to obtain a one-to-one (or scaled value) position match of the input device position with the position of the end effector in the remote robot workspace (also referred to as traversing the robot workspace).

[0368] Reference now Fig.11 , which illustrates an example control data flow 1100 between the local control system 102 and the telepresence robotic system 104 for controlling various configuration parameters of the telepresence robotic system 104 .

[0369] As shown, at 1102, a local user operator may input a user command 1102 into the local control system 102. For example, this may include a command to change or modify a tool currently being used by the remote system 104. The input command may also include a command to adjust various ultrasound configuration parameters of an ultrasound transducer being used by the remote system 104. The user input may be received, for example, at an input interface 190a of the local control system 102.

[0370] At 1104 , the user input command is transmitted to the telepresence robotic system 104 , such as via the network 110 .

[0371] At 1106-1122, the telepresence robotic system 104 may receive and execute the received command. For example, at actions 1106-1110, the telepresence robotic system 104 may execute the command to adjust the ultrasound parameters of the currently deployed ultrasound transducer. This may involve, for example, receiving an ultrasound machine command (1106), and in response, modifying a probe signal associated with the ultrasound transducer (1108) and / or controlling an ultrasound probe signal multiplexer (1110) based on the received command.

[0372] In other cases, at actions 1112-1116, the telepresence robotic system 104 may receive and execute a command to pick up a new tool, or otherwise change the currently held tool to a new tool. In particular, the telepresence robotic system 104 may receive a tool change command (1112), and in response, the telepresence robotic system 104 may control the motion subsystem 210b to drop the currently held tool and pick up a new target tool (1114), or otherwise simply pick up a new target tool (1116). The target tool may be, for example, an ultrasonic transducer or a gel dispenser.

[0373] At act 1118, the remote system 104 may use the currently held (or currently selected) tool. For example, if the tool is a gel dispenser, the remote robotic system 104 may control the gel dispenser to deposit gel on the desired area of ​​the patient's body, i.e., the area where the scan is to be performed. In some example cases, act 1118 may involve controlling the gel dispenser pump 612 to deposit the gel ( Figure 6C ).

[0374] In other cases, at 1122, if the tool is an ultrasonic transducer, the remote system 104 may use the ultrasonic transducer to perform an ultrasonic scan. To this end, the transducer may be configured according to the configuration parameters selected at actions 1108-1110. Once the scan is complete, the remote system 104 may then generate an ultrasound image and send it to the local system, i.e., via the network 110 (1124). The local system 102 may receive the ultrasound image (1126), which may then be read and interpreted by the local system operator (e.g., a sonographer). The data flow 1100 may then return to action 1102 to iterate until the local user operator has obtained the desired ultrasound image.

[0375] Although the above description provides examples of one or more methods or apparatus or compositions, it should be understood that other methods or apparatus or compositions may be within the scope of the appended claims.

[0376] To the extent that any modification, characterization, or other assertion previously made regarding any art, prior or otherwise (in this patent application or patent or any related patent application or patent, including any parent, sibling, or child) could be construed as a disclaimer of any subject matter supported by the present disclosure of this application, Applicant hereby revokes and withdraws such disclaimer. Applicant also respectfully submits that any prior art previously considered in any related patent application or patent (including any parent, sibling, or child) may need to be revisited.

Claims

1. A system for remote ultrasound examination, comprising: - a local control system comprising an input device and at least one local processor coupled to the input device; - a telerobotic ultrasound system comprising a motion subsystem and at least one local processor coupled to the motion subsystem, wherein the at least one local processor of the local control system is operable to: - detecting a new position state of the input device; - obtaining position data corresponding to the new position state; - generating input-side motion demand data based on the position data; and - transmitting the input side motion requirement data to the remote robot system, wherein the at least one teleprocessor of the telerobot system is operable to: - receiving the input-side motion demand data; and -Adjusting the position state of the motion subsystem based on the input-side motion demand data.

2. The system according to claim 1, wherein: The at least one remote processor of the telerobotic ultrasound system is further operable to: - monitoring changes in the position state of the motion subsystem; and - in response to detecting a change in the position state, generating robot-side motion demand data corresponding to a new position state of the motion subsystem; and -Transmitting the robot-side motion requirement data to the local control system.

3. The system according to claim 2, wherein: The at least one local processor of the local control system is further operable to: - receiving the robot-side motion requirement data; and - adjusting the position state of the input device based on the robot-side motion requirement data so as to generate a tactile feedback effect at the input device.

4. The system of any one of claims 1 to 3, wherein the input device further comprises a sensor subsystem, and detecting a new position state of the input device comprises receiving new sensor data from the sensor subsystem.

5. The system according to claim 4, wherein: The sensor subsystem includes a plurality of rotary encoders, and the position data includes encoder data generated by the plurality of rotary encoders.

6. The system according to claim 4, wherein: The sensor subsystem includes an inertial measurement unit (IMU), and the position data includes IMU data generated by the IMU.

7. The system according to any one of claims 1 to 6, wherein generating the input-side motion requirement data comprises determining an input-side transformation matrix based on the position data, wherein determining the input-side transformation matrix comprises solving a forward kinematics model.

8. The system according to any one of claims 1 to 7, wherein: Adjusting the position state of the motion subsystem includes analyzing the input-side transformation matrix by solving an inverse kinematics model.

9. A system according to claim 2 and any one of claims 3 to 8 when dependent on claim 2, wherein: Generating robot-side motion requirement data includes determining a robot-side transformation matrix based on the new position state, wherein determining the robot-side includes solving a forward kinematics model.

10. The system according to any one of claims 1 to 9, wherein: The input device includes a movable arm system.

11. The system according to any one of claims 1 to 10, wherein: The motion subsystem includes a robotic arm.

12. The system according to any one of claims 1 to 11, wherein: The motion subsystem includes a tool holding end effector for holding a tool.

13. The system according to claim 12, wherein: The end effector includes a first connector that engages a second connector associated with the tool.

14. The system according to claim 12, wherein: The tool includes one or more of an ultrasound transducer, a gel dispenser, and a transvaginal probe.

15. The system of claim 12, wherein: The tool is disposed inside a housing including a flexible sheath, and the second connector is attached to the housing.

16. The system of any one of claims 12 to 15, wherein the input-side motion demand data controls one of a spatial position and an orientation position of the tool holding end effector.

17. A system according to any one of claims 1 to 16, wherein: The local control system further comprises an input interface, and the at least one local processor of the local control system is further operable to: - receiving one or more user inputs from said input interface; and - sending the one or more user inputs to the telerobot ultrasound system.

18. The system of claim 17, wherein: The one or more user inputs include one of an ultrasound transducer configuration and a command to swap an actively selected tool.

19. A system according to any one of claims 17 or 18, wherein: The input interface includes an ultrasound input interface.

20. The system according to any one of claims 1 to 19, wherein: The telerobotic ultrasound system further comprises a pain threshold monitor coupled to the at least one teleprocessor of the telerobotic ultrasound system, and wherein the at least one teleprocessor of the telerobotic ultrasound system is further operable to: - monitoring an activation signal from said pain threshold monitor; and - In response to detecting the activation signal, one of disabling the motion subsystem and reducing the force applied by the motion subsystem to the patient.

21. A method for remote ultrasound examination, comprising: - Detecting a new position state of an input device of a local control system; - obtaining position data corresponding to the new position state; - generating input-side motion demand data based on the position data; and - transmitting the input-side motion demand data from the local control system to the telerobotic system, - receiving the input-side motion demand data at the telepresence robot system; and -Adjusting the position state of the motion subsystem of the telepresence robot system based on the input-side motion demand data.

22. The method according to claim 21, further comprising: - monitoring changes in the position state of the motion subsystem at the telepresence robotic system; and - in response to detecting a change in the position state, generating robot-side motion demand data corresponding to a new position state of the motion subsystem; and -Transmitting the robot-side motion requirement data to the local control system.

23. The method according to claim 22, further comprising: - receiving the robot-side motion requirement data at the local control system; and - Based on the robot-side motion demand data, adjusting the position state of the input device at the local control system so as to generate a tactile feedback effect at the input device.

24. The method of any one of claims 21 to 23, wherein the input device comprises a sensor subsystem, and detecting a new position state of the input device comprises receiving new sensor data from the sensor subsystem.

25. The method according to claim 24, wherein: The sensor subsystem includes a plurality of rotary encoders, and the position data includes encoder data generated by the plurality of rotary encoders.

26. The method according to claim 24, wherein: The sensor subsystem includes an inertial measurement unit (IMU), and the position data includes IMU data generated by the IMU.

27. The method according to any one of claims 21 to 26, wherein generating the input-side motion requirement data comprises determining an input-side transformation matrix based on the position data, wherein determining the input-side transformation matrix comprises solving a forward kinematics model.

28. The method according to any one of claims 21 to 27, wherein: Adjusting the position state of the motion subsystem includes analyzing the input-side transformation matrix by solving an inverse kinematics model.

29. The method according to claim 28, wherein: Generating robot-side motion requirement data includes determining a robot-side transformation matrix based on the new position state, wherein determining the robot-side includes solving a forward kinematics model.

30. The method according to any one of claims 21 to 29, wherein: The input device includes a movable arm system.

31. The method according to any one of claims 21 to 30, wherein: The motion subsystem includes a robotic arm.

32. The method according to any one of claims 21 to 31, wherein: The motion subsystem includes a tool holding end effector for holding a tool.

33. The method of claim 32, wherein: The end effector includes a first connector that engages a second connector associated with the tool.

34. The method of claim 32, wherein: The tool includes one or more of an ultrasound transducer, a gel dispenser, and a transvaginal probe.

35. The method of claim 32, wherein: The tool is disposed inside a housing including a flexible sheath, and the second connector is attached to the housing.

36. The method of any one of claims 32 to 35, wherein the input-side motion requirement data controls one of a spatial position and an orientation position of the tool holding end effector.

37. The method according to any one of claims 21 to 36, wherein: The local control system further comprises an input interface, and the method further comprises: - receiving, at the local control system, one or more user inputs from the input interface; and - transmitting the one or more user inputs from the local control system to the telerobotic ultrasound system.

38. The method of claim 37, wherein: The one or more user inputs include one of an ultrasound transducer configuration and a command to change an actively selected tool.

39. The method of any one of claims 37 and 38, wherein the input interface comprises an ultrasound input interface.

40. The method according to any one of claims 21 to 39, wherein: The telerobotic ultrasound system further includes a pain threshold monitor coupled to the at least one teleprocessor of the telerobotic ultrasound system, and the method further includes: - monitoring activation signals from a pain threshold monitor; and - In response to detecting the activation signal, one of disabling the motion subsystem and reducing the force applied by the motion subsystem to the patient.

41. An input device, comprising: - Mounting structure; - A user-controllable arm system comprising: (i) a robotic arm extending between a first arm end and a distal second arm end, and (ii) a rotatable member rotatably coupled to the mounting structure, wherein the robotic arm is fixed to the rotatable member at the second arm end; - a motor subsystem comprising a plurality of motors for controlling the position state of the arm system; - a sensor subsystem for monitoring the position status of the arm system; - At least one processor coupled to each of the motor subsystem and the sensor subsystem.

42. The input device of claim 41, wherein the mechanical arm comprises a plurality of arm segments rotatably connected.

43. An input device as claimed in any one of claims 41 or 42, wherein a user engageable joystick is mounted to the first arm end, the joystick providing an engagement interface for moving the arm system.

44. The input device of claim 43, wherein the joystick comprises one or more contact sensors coupled to the at least one processor.

45. The input device of claim 43, wherein the joystick comprises an activation switch coupled to the at least one processor.

46. ​​An input device according to any one of claims 39 to 45, further comprising a power supply coupled to the at least one processor, wherein the at least one processor is operable to control power supplied from the power supply to the motor subsystem.

47. An input device according to claim 46 when dependent on claim 44, wherein when the contact sensor is not engaged, the at least one processor supplies current from the power supply to the motor subsystem to maintain the arm system in a static position.

48. An input device according to any one of claims 46 or 47, wherein the at least one processor is operable to: - receiving motion demand data from a remote system; and -Applying increased current to one or more motors of the motor subsystem to increase motor torque and generate a haptic feedback effect.

49. The input device of claim 42, wherein the rotatable connections between the arm segments and between the rotatable member and the mounting structure comprise rotatable shafts.

50. The input device of claim 49, wherein the sensor subsystem comprises one or more rotary encoders coupled to the rotatable shaft, each rotary encoder generating encoder data corresponding to a rotational position of a corresponding shaft.

51. The input device of claim 50, wherein one or more of the rotatable shafts are coupled to a corresponding motor of the motor subsystem.

52. The input device of claim 51, wherein the coupling is one of a direct coupling and an indirect coupling, and the indirect coupling comprises a belt and pulley mechanism.

53. An input device according to any one of claims 51 or 52, wherein the at least one processor is operable to control the motor subsystem to adjust the rotational position of the rotation axis so as to adjust the position state of the arm system.

54. An input device according to claim 43 and any one of claims 44 to 53 when dependent on claim 43, wherein: The sensor subsystem includes an inertial measurement unit mounted to the joystick.

55. An input device according to any one of claims 41 to 54, wherein the user-controllable arm system comprises: (i) Directional positioning control part; and (ii) spatial position control part, The directional position control part is configured to control the directional position of an end effector associated with a motion subsystem in a remote environment, and the spatial position control part is configured to change the spatial position of the end effector.

56. An input device according to claim 55 when dependent on claim 42, wherein the directional control portion comprises at least one arm segment of the robotic arm.

57. The input device of claim 56, wherein the at least one arm segment comprises a forward wrist arm segment, a middle arm rotation segment, and a rear arm rotation segment.

58. The input device of claim 57, wherein the forward wrist arm segment is rotatably coupled to the intermediate arm rotation segment and rotates about a first yaw axis to control a yaw angle of the end effector in the remote environment.

59. An input device according to any one of claims 57 or 58, wherein the intermediate arm rotation section is rotatably connected to the rear arm rotation section and rotates around a second pitch axis to control the pitch angle of the end effector in the remote environment, wherein the second axis is orthogonal to the first axis.

60. The input device according to any one of claims 57 to 59, wherein the spatial position control portion comprises one or more arm segments of the robotic arm and the rotatable member.

61. The input device of claim 60, wherein the one or more arm segments include a front translation arm segment and a rear translation arm segment.

62. An input device according to claim 61 when dependent on claim 59, wherein the rear arm rotation section is rotatably connected to the forward translation rotation section and rotates about a third roll axis to control the roll angle of the end effector in the remote environment, wherein the third axis is orthogonal to both the first axis and the second axis.

63. The input device of any one of claims 61 and 62, wherein the front translation arm segment is rotatably connected to the rear translation arm segment and rotates about a fourth axis to control vertical spatial movement of the end effector in the remote environment.

64. An input device according to any one of claims 61 to 63, wherein the rear translation arm segment is rotatably connected to the rotatable member and rotates about a fifth axis to control forward and rearward spatial movement of the end effector in the remote environment.

65. An input device according to any one of claims 61 to 64, wherein the rotatable member is rotatably connected to the mounting structure and rotates about a sixth axis to control lateral horizontal spatial movement of the end effector in the remote environment.

Images