Image guided robotic system with remote guidance, image steering, and multi-planar imaging

Abstract

Described herein are interventional robotic devices paired with imaging devices for which a remote user selects targets or "waypoints" on a remote interface and the robot subsequently maneuvers to satisfy the commands implied by those targets or waypoints. By coupling the robot targeting device to an ultrasound imager, the imaging plane can be adjusted to match the measured position and orientation of the instrument tip, and / or the plane and the robotic device can be adjusted to be optimally positioned and oriented with respect to the anatomical target. By coupling the robot targeting device to an ultrasound imager capable of multiple planes, the expected trajectory and / or intersection of the instrument on / in the imaging plane can be displayed on top of the ultrasound image. By coupling the robot targeting device to an ultrasound system with needle visualization enhancement features, the measured position and orientation of the instrument can be used as input to the visualization features to most effectively enhance the instrument.

Description

Technical Field

[0001] This application relates to the field of medical imaging systems.

[0002] Cross-references to related applications

[0003] This application claims priority to U.S. Provisional Application No. 63 / 565,416, filed March 14, 2024, which is incorporated herein by reference in its entirety. This application also relies on U.S. Patent No. 12,011,239, which is incorporated herein by reference in its entirety. Background Technology

[0004] High-quality medical interventions often require reliable clinical judgment, technical expertise, and timely and effective care delivery. However, delivering high-quality interventions on a large scale is challenging due to current healthcare allocation methods. Healthcare providers with sufficient skills to perform interventions are often located far from the best locations for the intervention to occur, resulting in complex coordination between departments and surgical sites within an institution, practitioners having to serve multiple geographically dispersed facilities, and / or patients having limited or no access to the intervention at all.

[0005] Specifically, for minimally invasive image-guided surgery, users must possess spatial reasoning abilities and know how to effectively utilize imaging and minimally invasive equipment. While affordable, radiation-free, real-time imaging such as ultrasound exists, its use is limited by the challenging interpretation required. Due to the complexity of the procedure, few healthcare providers perform image-guided interventions in high volumes. This complexity can be associated with four fundamental challenges.

[0006] Image acquisition Image acquisition can be challenging for healthcare providers due to a variety of factors, including but not limited to poor motor skills, poor acoustic coupling, and a poor understanding of how to hold, apply, rotate, and tilt the probe relative to the body to establish a relevant imaging window.

[0007] Image Explanation Interpreting images (e.g., interpreting ultrasound images) can be challenging for providers due to various factors, including differing levels of understanding of how ultrasound images are formed, how anatomical structures and pathologies appear on ultrasound, and how cognitive three-dimensional (3D) information is created from a series of 2D planar images.

[0008] Intervention PlanFor providers, defining the optimal trajectory for an instrument (e.g., a needle) to consistently hit a predetermined target while avoiding anatomical constraints (e.g., avoiding contact with unplanned portions of anatomical structures) is challenging due to various factors, including the inconsistency in acquiring and interpreting 2D images and deriving 3D volume from that information. Another challenge providers may face is the inability to contextualize 2D images with respect to surrounding anatomical structures, whereas contextualization using 3D information is much more intuitive.

[0009] Intervention and execution Even with an optimal trajectory defined for the instrument, it is challenging for the provider to consistently initiate and execute their planned objectives due to various factors, including the use of the non-dominant hand, the ability to introduce the instrument with appropriate force and control it with one hand, the challenges associated with obtaining an optimal ultrasound view of the instrument tip while maintaining a view of the relevant anatomy, poor motor skills, steep needle insertion angles producing undesirable acoustic reflections, and the planned needle path compared to the actual needle path due to deflection. Furthermore, manual techniques require the provider to create an intervention plan in their mind based on a mental map generated from scanned anatomy. Intervention execution requires the provider to then use a single image as a reference to manipulate the needle through the skin to the target.

[0010] Overcoming these challenges requires extensive training and practice to estimate the optimal entry point, trajectory, and depth for interventional instruments to reach the target imaged by the US probe. This expertise is difficult to acquire and prevents many clinicians from successfully performing ultrasound-guided percutaneous interventions.

[0011] In certain limited application areas, there are solutions for more efficiently allocating clinical judgment. For example, telepsychiatry, teleprimary care, and teleradiology allow remote providers to connect with patients or examinations and make clinical decisions. Other existing applications include remote guidance in the operating room, where physicians can connect via two-way video and audio to supervise and / or support cases using verbal or gestural instructions.

[0012] However, these existing virtual communication and collaboration solutions are limited to situations without physical interaction. The remote deployment of physical capabilities or manual skills has not yet been proven on a large scale in healthcare. This is largely due to the significant technical requirements (hardware and software) of the systems that would enable such remote intervention capabilities. One area where this has been attempted is remote surgery, such as remote laparoscopic surgery using the da Vinci Surgical System, where the user remotely operates a robotic system located in a different location from the control unit. However, this type of system, relying on continuous, very low-latency round-trip communication using video and telemetry data, cannot be deployed with modern internet infrastructure. This limitation will exist for any system where users expect to feel as if they are controlling the equipment in real time.

[0013] In scenarios where the provider intending to perform minimally invasive image-guided interventions has expertise in interpreting images, it is often useful to be able to visualize instruments and anatomical structures in an imaging plane that is not directly aligned with the long or short axis of the probe during the procedure.

[0014] A first example of this is a scenario where the provider begins inserting the needle and it deviates from the plane of the ultrasound image. The provider will be unable to maintain a visual field of the instrument and its initially identified anatomical target because the instrument is veering away from that target. When this happens, the provider needs to determine how the instrument is now aimed so they can decide whether and how to make adjustments. Adjustments typically involve retracting the instrument and restarting the procedure, which is clinically undesirable. Continuing without adjustments usually requires the provider to move the imager back and forth between the target and the instrument and attempt to track their relative 3D positions in the mind, which is both challenging maneuvering and generally diminishes the value of image guidance.

[0015] The second example is a surgery in which the provider cannot obtain an optimal view of the target via ultrasound beam at the start of the procedure, making it difficult or impossible to perform the surgery or plan the optimal instrument trajectory with a view of the instruments and anatomical structures.

[0016] In ultrasound, very few products allow for extensive adjustments to the ultrasound beam plane. For those ultrasound systems that do have this capability, configuring the ultrasound beam is often so complex that it is not presented to the user as a controllable parameter. Even if it were controllable, users struggling to overcome the aforementioned challenges would not understand in what direction to manipulate the beam, as they would not know (in both cases) where the instrument has gone or what the optimal imaging plane is. These problems, combined with other issues of ultrasound-guided surgery, include the difficulty in mastering (1) selecting the optimal trajectory, (2) keeping the instrument within that trajectory during insertion, and (3) knowing where the needle tip is.

[0017] Visualizing instruments and anatomical structures in more than one imaging plane can also be useful. A first example is during ultrasound-guided vascular access, where procedures are sometimes performed in an "in-plane" manner, where both the long axis of the needle and the long axis of the vessel are in the ultrasound imaging plane. This allows the needle to be fully visualized as it approaches the vessel, and the needle tip to be visualized as it penetrates through the top wall of the vessel. However, this method cannot show how the needle is oriented relative to the vessel's cross-section, and the needle may be close to the sidewall without the operator's awareness (i.e., the needle is not centered in the cross-section). Some healthcare providers use an "out-of-plane" technique, where the cross-section of both the needle and the vessel is in the ultrasound imaging plane. This technique has the opposite consequence; the needle's centering relative to the vessel is easily determined, but the needle's trajectory and the depth of the needle tip are relatively difficult to ascertain.

[0018] The second example is a biopsy, where the provider will typically use a single technique (in-plane or out-of-plane) to place the needle in the anatomical target of interest. It is beneficial to know which area of ​​the target the provider will sample from, which will require seeing the needle tip's location in both in-plane and out-of-plane views.

[0019] Very few products exist that can deliver ultrasound images concurrently across multiple imaging planes. Using these types of ultrasound products can be helpful in the clinical scenarios described above, but the operator must also contend with selecting and maintaining the trajectory and knowing where the needle tip is during insertion. Errors in any of these tasks, when multiple imaging planes are displayed, can lead to even more confusion than when the operator is using only one imaging plane.

[0020] Related to the third issue (knowing where the needle tip is during insertion), some commercially available ultrasound devices are designed to enhance the echogenicity of the needle as it is inserted through tissue. Poor needle echogenicity can be due to a variety of factors, including sound waves reflecting off the needle surface and away from the ultrasound transducer. Needle visualization enhancement features typically utilize a combination of (1) adjusting the reception and transmission of ultrasound waves so that they are oriented toward the intended position and angle of the needle and (2) using post-processing image adjustments to enhance and highlight linear structures (e.g., the needle) in the resulting ultrasound image. Both approaches require the ultrasound system to assume information about the position and orientation of the needle relative to the transducer. In commercially available systems, this is typically addressed by requiring the user to input the intended needle angle on the user interface. However, due to the aforementioned issues, it is difficult for the user to identify and maintain the optimal approach angle when using freehand techniques, making it difficult to know precisely what approach angle to input into the ultrasound system.

[0021] Devices exist to help providers overcome some of the challenges mentioned above, such as robotic systems that help users acquire images, identify anatomical structures, plan approach methods, and perform surgery. These devices can have different levels of automation and robotic control, ranging from vision-guided only to full automation of the surgery. An intermediate operating mode is reverse-drive, where the user manually and collaboratively adjusts the needle trajectory or interacts with the robot in a haptic manner. Direct reverse-drive can be used to move the robot to a set configuration or quickly establish an initial position, after which the user can switch to a graphical user interface for interaction with various aiming modes, such as clicking and dragging on the interface, using automatically segmented pixels for automatic aiming, or utilizing guidance from a remote provider, all of which enable fine, precise aiming of the robotic device.

[0022] In many cases, due to a more natural workflow / usability and / or enhanced sensory interaction (such as force-haptic / tactile feedback), users may still want to manually interact with the arm for fine aiming. Unconstrained backdrive may not adequately serve the user's need for the ability to finely aim at targets as expected with automated motion while maintaining the direct user interaction associated with backdrive. Summary of the Invention

[0023] This article describes an interventional robotic device paired with an imaging device, for which a remote user can select a target or "waypoint" on a remote interface, and the robot will then maneuver to meet commands implied by those targets or waypoints.

[0024] This article describes the combination of adjustable ultrasound imaging planes with robotic guidance techniques. Robotic and non-robotic ultrasound guidance systems primarily focus on holding the instrument along the long or short imaging axis and perpendicular to the transducer, rather than using non-principal or non-perpendicular planes or adjusting the imaging plane during surgery. By coupling a robotic aiming device to an ultrasound imager capable of adjusting the imaging plane, the imaging plane can be adjusted to match the position and orientation of the measured instrument tip, and / or the plane and robotic device can be adjusted to be optimally positioned and oriented relative to the anatomical target.

[0025] This article describes the combination of multiplanar ultrasound imaging and robotic guidance technology. Robotic and non-robot ultrasound guidance systems primarily focus on holding the instrument within a single ultrasound imaging plane, rather than using multiple concurrent imaging planes to visualize the intended trajectory and / or establish that trajectory. By coupling a robotic aiming device to an ultrasound imager capable of achieving multiplanarity, the intended trajectory and / or intersections of the instrument on / within the imaging plane can be displayed on top of the ultrasound image.

[0026] This paper describes the combination of needle visualization enhancement features and robot-guided techniques. Existing needle visualization enhancement features rely on the ultrasound system's assumptions about the needle's position and orientation upon entry into the body. By coupling a robot aiming device to an ultrasound system with needle visualization enhancement features, the measured position and orientation of the instrument can be used as input to the visualization features to most effectively enhance the instrument.

[0027] This paper describes constraint-backed actuation based on inverse kinematics (IK), which allows users to focus only on the parts they need to control, while kinematically assisting the rest of the motion in a human-in-the-loop manner. Given the same framework, it can be very flexible and easy to implement a range of constraint-backed actuation patterns to meet specific needs and / or migrate to different robot designs. The user experience depends primarily on the given robot's motion constraints and the choice of user touchpoints.

[0028] This paper describes a robotic system with software-controlled angle adjustment. A user interface can be used to initiate software commands to move the joints of the robotic device to change the robot's angle, while maintaining one or more fixed constraints, such as target position or end effector height. Attached Figure Description

[0029] Figure 1 This is an example of a user selecting a target on a remote interface; Figure 2 This is an example of a conduit-based robotic system; Figure 3 This is an example of a remote center of motion (RCM) constraint; Figure 4 This is an example of how the system translates guidance or commands shown on the external view into similar guidance or commands in the imager's coordinate system; Figure 5 This is an example block diagram of a communication architecture that facilitates remote support for robotic systems; Figure 6 It is an example ultrasound system with an adjustable imaging plane; Figure 7 Here is an example of a robot mounted on an ultrasound probe; Figure 8 This is an example of the instrument being further inserted into the patient; Figure 9 This is an example of an ultrasound imager directional beam to obtain multiple views of an anatomical structure; Figure 10 This is an example of a multi-plane robot using ultrasound to aim at a target; Figure 11 This is an example of enhanced needle visualization with and without robots; Figure 12 This is an example flowchart of a constraint-driven reverse framework; Figure 13 This is an example of a 5-DoF robot design; Figure 14 This is an example of a 3-DoF robot design; Figures 15a and 15b illustrate examples of RCM reverse drive; Figures 16a and 16b illustrate examples of PnP reverse drive; and Figure 17 This is an example of a patient-side robotic system used for percutaneous instrumental intervention.

[0030] Detailed Description of the Embodiments (see attached figures)

[0031] Image-guided robotic interventional systems (“IGRIS”) may include a robotic instrument guide, imaging equipment, a sterile barrier, a patient-side computer with a user interface, and system software including the user interface. Healthcare professionals (i.e., users) can utilize each of these when performing one or more medical procedures on a patient. IGRIS relies on imaging to provide the user with a real-time view or image of the patient's anatomy and pathology, as well as the intended target or multiple targets for the procedure, and on software that allows the user to plan approach methods or trajectory paths using real-time imaging or robotic equipment. This software may allow the user to convert a series of 2D images into a 3D volume, and positioning sensors determine the orientation of the device relative to the patient, the patient's anatomy, and the pathology.

[0032] IGRIS may include the following hardware components: (1) an imaging device capable of imaging objects such as patient anatomy and pathology; (2) a computer for performing calculations and algorithms based on those images; (3) a robotic arm (e.g., a robotic manipulator and instrument guide) that influences the use of instruments that interact with patient anatomy and pathology; and (4) external sensors (e.g., an IMU and a camera). Here, the healthcare provider (“HCP”) or user interacts with the system and potentially with the instruments themselves to perform the procedure.

[0033] Imaging can be real-time ultrasound; the instrument is designed to be introduced percutaneously and guided in a generally straight line toward the anatomical target. The imaging device, computer, and robotic arm can be handheld. In some embodiments, computer processor operations (including, but not limited to, machine learning and / or artificial intelligence) can be offloaded to the cloud for processing.

[0034] Alternatively, the component may be mounted on at least one positioning device, where it is manually manipulated or manipulated via robot actuation.

[0035] In this way, HCP interacts with IGRIS to physically stabilize the system and acoustically couple the ultrasound transducer to the patient; acquire and interpret ultrasound images; configure, plan, and execute interventions using typical computer input interfaces (e.g., touchscreen, mouse, keyboard, joystick, trackball, etc.); and can collaboratively manipulate instruments to optimize interventions.

[0036] IGRIS may not include an imaging device and can be operated to couple to an existing imaging device. For example, IGRIS can be attached to a standard ultrasound device and operate in the same way as an IGRIS with an integrated imaging device. This can be represented as a "modular robotic arm".

[0037] Modular robotic arms can be used on objects (e.g., human anatomy) in a variety of percutaneous surgeries. In some embodiments, a user can hold an imaging device (e.g., an ultrasound probe) with the modular robotic arm mounted on one hand, the modular robotic arm pointing an instrument guide to the target, and the user manually inserting a needle through the instrument guide. In some embodiments, the instrument guide can be removable and replaceable. Preferably, the modular robotic arm can operate in a sufficiently large workspace to accommodate needle insertion near and away from the ultrasound probe to enable a variety of percutaneous procedures. Furthermore, the robotic joints and instrument guide of the modular robotic arm are slim in profile to avoid collisions with the imaging device, the robot itself (self-collision), and surrounding external anatomy. The modular robotic arm may also include gripping and insertion mechanisms to allow the modular robotic arm to measure and / or control the degrees of freedom for instrument insertion. Modular robotic arms (1) may be universally mountable to multiple imaging devices, (2) can be mounted in multiple mounting locations around existing wiring (which may be non-removable), (3) may include homogeneous kinematic architectures, and (4) may include low-profile wrists via remote actuators (e.g., cable-driven).

[0038] Modular robotic arms can support reverse drive, allowing users to manually adjust the instrument guide. This reverse drive motion can be constrained or unconstrained to help users more easily adjust the needle trajectory. For example, the instrument guide (e.g., a needle guide) can point towards a target while allowing the user to adjust the entry point—or vice versa. To make the reverse drive mode intuitive, the robot can incorporate homogeneous kinematics throughout the workspace, thus keeping it away from singularities.

[0039] In some embodiments, the modular robotic arm can be symmetrically mounted on the imaging device to ensure flexibility in probe / robot orientation, thereby facilitating left- and right-handed use, as well as the arrangement of the robot in front and behind. This can also facilitate in-plane and out-of-plane clinical surgery.

[0040] Modular robotic arms can be coupled to a computer. The computer is calibrated such that defining anatomical structures in image space (pixels) allows the robot to point its end effector to target the same anatomical structure in physical space.

[0041] The modular robotic arm may include a first joint that can be mounted around existing imaging device wiring (e.g., fixed ultrasound probe cables) but still be free to rotate. In other embodiments, the mount may not rotate freely but instead allows the robot to be rigidly attached to the probe at multiple fixed rotation angles. The most common angles are used to facilitate base point positions: in-plane right-hand, in-plane left-hand, out-of-plane behind-the-image, and out-of-the-image in front of the probe. The modular robotic arm can be attached to the imaging device via a semi-permanently mounted imaging device-specific socket. The socket / robot interface can be standardized, allowing the robot to be attached to any imaging device with a corresponding socket.

[0042] Modular robot assemblies can be mounted to any imaging device (e.g., any ultrasound probe) using an interface. Specific interfaces may exist for each compatible imaging device. One end of the interface may include a unique design coupled to the outer housing of the imaging device. The other end of the interface may include a coupling geometry attached to the modular robot arm. The coupling geometry may include a number of features: (1) keying geometry to control the orientation of the robot's mechanical mounting to the imaging device; (2) locking features such that the modular robot assembly can be locked onto the top of the imaging device; and (3) electrical contacts to provide power and communication to accessories embedded in the interface, including buttons, cameras, and IMUs.

[0043] In some embodiments, the modular robot assembly includes complementary coupled geometries for the interface attached to the first rotary joint. In some embodiments, the robot arm includes relief notches that allow the joint to slide over wiring that drives the imaging device. For example, these relief notches can produce a unique C-shaped geometry compared to the conventional O-shaped geometry of conventional robot joints. In some embodiments, components of the joint (e.g., bearings, bushings, etc.) must “cross the clearance” to provide stiffness over a large range of rotation. In some embodiments, recirculating bearings may be used. In some embodiments, the joint can still rotate 360 ​​degrees around the imaging device when paired with multiple drive gears. In some embodiments, the joint can allow nearly 360 degrees of rotation when paired with a single drive gear.

[0044] Various aspects of the present invention can be operated using a number of kinematic geometry configurations as understood by those skilled in the art, including but not limited to parallel manipulator systems. In some embodiments, the first rotary joint can be mounted such that the axis of rotation is along the axis of the probe, operating similarly to a turret. In some embodiments, the turret can position the remainder of the modular robotic arm symmetrically around the imaging device. This allows the modular robotic arm to be arbitrarily and symmetrically positioned around the imaging device, both rotationally and in a mirror manner.

[0045] The most common kinematic architecture is 6 degrees of freedom (DoF), allowing the instrument to be positioned arbitrarily relative to the imager, including arbitrary position and orientation of the instrument. A particularly useful kinematic architecture is 5 DoF, where the sixth degree of freedom, which controls rotation around the instrument, is uncontrolled. This is especially useful for robots designed to guide percutaneous instruments where rotation around the instrument—or needle—is constrained or left to the user. In this configuration, the trajectory taken by the needle along its axis of symmetry is entirely controlled by the robot, and needle rotation is uncontrolled.

[0046] A simpler version is a 3 DoF robot, where the robot is capable of orienting instruments or needles in a plane. This includes two DoFs for position and one angle for orientation. Such a configuration allows the user to set the skin entry position and the entry angle for the instrument to hit the correct target. This is particularly useful when the imager is ultrasound (where the image is a cross-section of the anatomical structure). Aligning the robot so that the control plane is consistent with the ultrasound image is clinically most relevant. Similarly, for out-of-plane surgery, orthogonalizing the plane to the imaging plane is also clinically relevant.

[0047] The extreme case of a robot having only a single degree of freedom is also relevant in clinical practice. In this embodiment, the entry point relative to the ultrasound probe can be fixed, and the robot only controls the angle of instrument insertion.

[0048] Modular robotic arms include corresponding motors or actuators for rotating joints. In some embodiments, the motor may be housed in a joint link adjacent to the joint of interest. The motor or actuator may be coupled to the joint rotation or translation using a mechanical drive system such as gears or cables and winches.

[0049] The system may include the ability to remotely control a robotic system. Figure 1 An example is shown where a user selects a target 101 on a remote user interface 102 (such as a laptop computer 100) and the robotic system 103 physically aims at the corresponding location in an ultrasound image 104. In this embodiment, the remote user interface is a copy of the patient-side robotic system user interface (not shown).

[0050] Selecting a pixel location in an ultrasound image to command a robot to point an instrument at a target is called tap-to-target (T2T). Dragging a finger on the user interface to adjust the instrument angle is called drag-to-rotate (D2R), and is further explained under the general category of "Software-Controlled Angle Adjustment" below. These are the basic user interactions in a graphical user interface for commanding a robot to guide an instrument to a target.

[0051] The proposed system includes an interventional robotic device paired with an imaging device, for which a remote user can select a target or "waypoint" on a remote interface, and the robot will then maneuver to meet commands implied by those targets or waypoints.

[0052] The use of "pathpoint" commands in interventional systems differs from teleoperation for traditional surgical systems described in the background, partly due to less stringent latency requirements and the elimination of the need for rapid round-trip video and telemetry data. Robotic motion is quasi-static (rather than continuous), thus eliminating the need for real-time closed-loop control or a remote user in the loop.

[0053] In addition to the benefits of solving the problems mentioned above, such a system can enable training, with remote trainers or supervisors overseeing and guiding on-site users, or remote assistance when on-site assistance is not available (e.g., if someone can remotely control those functions that interact with the system, change settings, associate images with patient records, etc., then there is no need for an expensive second person in the room to perform those functions).

[0054] This could include a remote user tapping an ultrasound image on a mirrored user interface, causing the robot to automatically aim at a target, similar to the tap-to-target (“T2T”) functionality used by field users for robotic systems. The remote user can select multiple waypoints, and the system can progressively meet them. For example, a remote user can select several biopsy targets (waypoints), and the robotic device can move to those targets in a step-by-step manner. Remote setting of waypoints can be used with various types of robotic devices, such as orthopedic robots for spinal surgery (where a remote user can select waypoints and approach angles associated with pedicle screw placement on top of one or more fluoroscopic images), or laparoscopic surgical robots (where a remote user can select waypoints the robotic arm will achieve during setup on top of a live camera image, ensuring they are correctly positioned relative to the patient and, consequently, the cannula). This waypoint control approach can be applied to different types of robotic systems. Figure 2 This is an example of a catheter-based robotic system in which a remote user can use a "bird's-eye view" to tap several waypoints 201 along the airway 202 (or other cavities, such as GI channels or blood vessels), and the system can then automatically advance the catheter 203 by successively advancing to each waypoint 201.

[0055] In one embodiment, a remote user controls the entire mirrored user interface, interacting with the UI as if they were a field user. This may include allowing them to: (i) activate other robotic functions, such as software-controlled angle adjustments; (ii) control ultrasound functions, such as gain, depth, freeze, and presets; (iii) save image or video media; (iv) annotate or measure on top of ultrasound images; and (v) configure system settings.

[0056] Software-controlled angle adjustment involves using software and robot joints to adjust the robot's angles to meet user requests while maintaining a target position. In this case, remote motion center (RCM) constraints are used to control the robot. Figure 3 This is an example of a remote center of motion (RCM) constraint. The user inputs the desired angle change 301 on the user interface and defines the target position on the medical image 302. This is called drag-and-rotate (D2R). The system sends commands to the robot 303 (in this case, a three-DOF planar robot with joints A1, A2, and A3) to change the instrument angle 304 while maintaining aim at the target. In this robot setup 303, there are three degrees of freedom 305 for controlled joint movements (e.g., ...). Figure 3 Points A1, A2, and A3 in the diagram are used to change only the angle and maintain the target position. Figure 3 This illustrates how degree of freedom 305 is used to comply with additional kinematic constraints that maintain the height of the end effector 306.

[0057] The system may optionally include external views of the on-site operator, surgical area, patient, and physical systems, such as via an overhead camera, as well as video displayed to a remote operator. The system may also include one-way or two-way audio communication with the remote operator. Remote users may also provide guidance to the on-site operator using remote annotations (remote drawing overlays on the user interface or on top of the external views). Remote users may optionally control the external views by directly controlling the camera or by controlling the resulting image (e.g., digital zoom).

[0058] Remote operators can interact with a variety of interfaces, including web applications, smartphones, tablets, or their own physical user interface instances, such as their own carts and touchscreens. The remote interface can include a digital visualization of the robotic system, providing external users with a visual context of the robot's posture at any given time.

[0059] Remote interfaces can be used for device-related support or service operations. For example, during surgery, support or service personnel can use the sharing of audio, video, user interface elements, or medical images to troubleshoot or triage problems. They can also use the ability to communicate using these same mechanisms to request on-site users to test or attempt various operations performed with the system. In embodiments where remote users have the ability to remotely modify the system software, they can use this ability to run system diagnostics, troubleshoot problems, and otherwise guide users. Modifying the software can include changing configuration parameters, updating software modules, or remotely installing a new disk image including the operating system and robotic software applications.

[0060] The remote interface can have a portal for connecting to historical or real-time system data, such as robot telemetry data, system health status, faults and problems, and usage. This system data can be used simultaneously with real-time audio and video capabilities to more effectively provide remote assistance or troubleshooting for novice users. The portal can automatically associate real-time audio or video calls with historical or real-time data associated with the current system using identifiers such as device IDs, or associate real-time audio or video calls with the current user using identifiers such as provider names or IDs.

[0061] For robotic devices where the advance, retraction, and / or activation of the attached instrument can be remotely initiated, a remote user can initiate these functions. This could include activating instrument insertion by moving the robotic arm toward the patient. In an alternative embodiment, a remote user can trigger an actuator to physically drive the needle forward or backward until it reaches a pre-selected target.

[0062] In another alternative embodiment, a remote user can actuate the device functions when the device reaches the target. This includes, but is not limited to, thermal, radio frequency (RF) energy, injection, aspiration, ablation, biopsy, mechanical tissue interaction (grabbing, shearing), high-intensity ultrasound, etc.

[0063] In other embodiments, a remote user can control the imager itself. This could include remotely steering the ultrasonic beam by requesting different transducer parameters. Alternatively, if the ultrasonic probe is attached to a robotic arm, the remote user can trigger movement of the probe in any of the six degrees of freedom (three positional and three rotational). Another example could include a field user holding the robot's end effector, and then the software controlling the imager's position by "inversely" running a kinematics solver (i.e., treating the probe as an end effector).

[0064] The system may include an approval workflow where either a remote or field user can propose a target / waypoint that must be approved by the other party. This can be an optional feature that enhances the security of remote commands (by requiring the field user to approve them before the system moves), reduces the burden on the remote user (by allowing the field user to propose targets), or both. The approval workflow can be designed to be unidirectional or bidirectional; targets can be proposed on either side, and targets can be approved on either side. For systems with onboard AI to assist in target / waypoint / path identification, the approval pipeline can allow either the field user or the remote user to approve AI-suggested waypoints.

[0065] Approval can be a one-time selection or a series of selections (e.g., a dead-man's switch). Two example approval workflows are: a. Workflow A i. Remote users select a target on top of the ultrasound image in their user interface; ii. A virtual target and a confirmation button appear on the on-site user interface; iii. The user on site taps the confirmation button; iv. The robot moves to achieve the selected position; b. Workflow B i. Remote users select a target on top of the ultrasound image in their user interface; ii. A virtual target and a confirmation button appear on the on-site user interface; iii. The on-site user presses and holds the confirmation button (disabling switch); iv. While the button is pressed, the robot moves to achieve the selected position; v. If the user releases the confirmation button, the robot stops moving.

[0066] Because this system can be used collaboratively by both on-site and remote users, it may have mechanisms to handle inconsistencies between on-site and remote commands. For example, target or waypoint rejection could be implemented. a. Based on latency / time: If the latency is too high (e.g., slow internet speed); b. Image-based: If the image is sufficiently different from the image shown when the target was selected (e.g., the on-site user moved the imager); and c. Location-based: If the robot / end effector is detected to be in a different location (e.g., a field user reverse-drives the robot, or a needle is inserted / retracted).

[0067] Conflicts can be handled through the approval system described above.

[0068] This system allows on-site users to request remote user assistance during surgery. This is done by queuing on-site requests, assigning the first available remote user to the on-site user, and connecting to the on-site user. The system can also automatically suggest when it might be useful to introduce a remote user based on the ongoing surgery, measurements from sensors on the robot or probes, or user interactions on the user interface.

[0069] Remote users can trigger media exports to PACS or other media repositories, similar to on-site users, but remote users can also initiate media exports sent to them (remote users). An example could be a remote user triggering a DICOM image to be sent to them for subsequent billing purposes.

[0070] In a more advanced embodiment, the system can translate guidance or commands shown on the external view into similar guidance or commands in the coordinate system of the imager or the image itself. For example, a remote user can draw an "up" arrow in the external view, which could correspond to a "right" movement of the imager from the field user's perspective; as long as the system understands the imager's orientation relative to the external view camera, the software can be able to display guidance telling the user to move the imager "right". This embodiment utilizes information about the medical imager's attitude relative to the external view, such as using optical tracking methods or information from accelerometers. Using one or more cameras, which may or may not include the camera generating the external view, the orientation of the medical imager relative to the external view can be determined.

[0071] Figure 4 This is an example of how the system translates guidance or commands shown on the external view into similar guidance or commands in the imager's coordinate system. In the third-person view, the external camera and its coordinate system 401 (denoted by subscript 'a'), the medical imager and its coordinate system 402 (denoted by subscript 'b'), and the medical image and its coordinate system 403 (denoted by subscript 'c') are shown. The dashed line 404 indicates the view from the external camera. Once the optical tracking and accelerometer data from 401 and 402 are used to identify the transformation from coordinate system 'a' to coordinate system 'c', the drawing on the screen in the external view 405 (such as +x) can be translated. a The dotted line in the direction is converted to a similar instruction relative to medical image 406, which is in +x c and -y c Direction.

[0072] Additionally or alternatively, a remote user can select targets in external view 405, which are then converted into robot targets or waypoints. For example, a user can select a desired skin entry point in external view 405; the robot can move to meet that target, provided the system understands the robot's position and orientation relative to the external view camera. This embodiment will likely require the use of heuristics to determine the remote user's intent, as it is difficult to select precise 3D targets using 2D views (such as external cameras), yet this 3D information is needed to perform transformations similar to those described above, which include both orientation and position information. These heuristics may include, for example, template matching based on relative size calculations; if the dimensions of the medical imager or robotic device are known, the expected dimensions at various distances from the camera can be calculated a priori, and the estimated distance from the camera can be calculated based on its relative size in the camera image. More advanced optical tracking (with optical tracking sensors and reference points) can also be used for this purpose.

[0073] The term “remote” is broad and implies a range of distances between the remote user and the on-site user. This could include across the Internet (e.g., the on-site user is in California and the remote user is in New York), on a local area network (e.g., both users are in the same hospital, but the on-site user is in a different area than the remote user), or via another wireless mode (e.g., the user is in an adjacent control room connected via Bluetooth).

[0074] Figure 5 This is an example block diagram of a communication architecture that facilitates remote support for a robotic system. Robotic system 501 connects to a backend service 502, which coordinates connections with both robotic system 501 and remote users 503. This includes establishing secure connections between robotic system 501, backend service 502, and remote users 503. Multiple remote users 503 can connect simultaneously, requiring backend service 502 to authorize and coordinate user capabilities. For example, it may be necessary to ensure that only one remote user 503 controls robotic system 501 at a time. This avoids race conditions from conflicting commands from multiple remote users 503. Simultaneous annotation can be enabled to allow multiple clinicians to collaborate on a case. Real-time ultrasound imaging must also be provided to ensure that remote users 503 can perform tasks requiring clinical judgment based on the current ultrasound scan. Backend service 502 can provide one-way or two-way video and audio communication to allow a seamless conferencing experience between the healthcare provider on the robotic system 501 side and the remote users 503. Finally, backend service 502 can generate and maintain updated software configuration parameters, software module updates, application updates, driver updates, robot firmware, or disk images, including the operating system and all other software / firmware, which can be remotely deployed to robot system 501.

[0075] This article describes a combination of adjustable ultrasound imaging planes and robot-guided techniques. By coupling a robot aiming device to an ultrasound imager capable of adjusting the imaging plane, the imaging plane can be optimized based on the instrument's posture, and / or the plane and the robot device can be adjusted for optimal positioning and orientation relative to the anatomical target.

[0076] Figure 6 This is an example ultrasound system with an adjustable imaging plane. A typical imaging plane associated with imaging centered on probe 601 (such as a typical ultrasound probe) is denoted by the letter 'o'. However, certain probe technologies allow the imaging plane to be turned to the left (e.g., 'a', 'b') and to the right (e.g., 'c', 'd'). These technologies include phased arrays and matrix transducer arrays. The attached figure shows this turning as a change in the beam angle, but it can also be performed as a lateral offset. For ultrasound imagers with multiple imaging beams, this turning can be accomplished using one or more of those beams.

[0077] Figure 7 This is an example of a robot mounted on an ultrasound probe. When combined with a robot aiming device 701 that controls or measures the position of the instrument 702, the ultrasound transducer 703 can be actuated in such a way that the imaging plane follows the instrument tip 704 (or other location of interest) as the instrument passes through the tissue. When the instrument is initially inserted, the cross-section of the instrument tip 704 is imaged by the imaging plane 'd' 705.

[0078] Figure 8 This is an example of the instrument being further inserted into the patient. As the instrument 801 is further inserted, the cross-section of the instrument tip 802 is imaged by the imaging plane 'o' 803. In this way, the ultrasound image can follow the needle to the target, thus preserving the cross-section of the instrument tip 802 and the surrounding anatomical structures.

[0079] This closed-loop tracking of the instrument tip 802 via the ultrasonic imager 804 is uniquely achieved because the robot 805 is able to accurately control and measure the orientation of the instrument 801 relative to the probe. The robot 805 can then transmit commands to the ultrasonic probe 804 to image the correct plane, thus showing the view of the instrument as desired by the user.

[0080] like Figures 6-8As shown, although the imaging plane is fixed except for left-right movement, this concept can be extended to the general case where a robot holds a needle in any posture (position and rotation) relative to the probe. If the ultrasound imager is complex enough to provide images at any angle relative to the transducer, closed-loop control can be used to image in a plane that always shows the needle contained in the imaging plane, as well as the cross-section of the needle tip (or other important locations on the instrument simultaneously). In other words, steerable ultrasound imaging can be extended to six-DOF instrument posture cases, where the imager has enough degrees of freedom to steer the imaging plane toward the instrument in any posture.

[0081] For example, users can place the system on the patient, such as Figure 7 As shown, an initial scan is performed using the imaging plane 'o'. As the needle is inserted, the robot guide can detect that the needle has been inserted into the imaging plane represented by 'd'. The system can automatically or in response to user input update the ultrasound plane to 'd'. As the user continues to insert the needle (thus intersecting with 'c' and 'o'), the system can update the ultrasound imaging plane accordingly, either continuously or discretely. As described above, for sufficiently complex imagers with multiple degrees of freedom, the imaging plane can be rotated to match any six-degree-of-freedom position of the needle tip.

[0082] An imager with a steerable ultrasound beam can be configured to acquire multiple views of anatomy and determine which view is the optimal view for intervention. Figure 9 An ultrasound imager 901 is shown scanning an anatomical structure 902 to obtain several views 903a, 903b, and 903c. Once the optimal view is identified, whether automatically, via a user interface, or by a remote user, an attached robot with sufficient degrees of freedom can be configured to align within the imaging beam because the position and orientation of the ultrasound beam relative to the probe are known. If it does not have sufficient degrees of freedom, guidance can be provided to the user via the user interface to encourage them to move the imager so that the optimal view aligns with a pose that the robot can satisfy.

[0083] This article describes a combination of multiplanar ultrasound imaging and robotic guidance technology. Multiplanar ultrasound imaging is very uncommon in the ultrasound market. Robotic and non-robot ultrasound guidance systems primarily focus on keeping the instrument within a single ultrasound imaging plane, rather than using multiple concurrent imaging planes to visualize and / or establish the desired trajectory. By coupling a robotic aiming device with an ultrasound imager possessing multiplanar capabilities, the desired trajectory and / or intersections of the instrument on / within the imaging plane can be displayed on top of the ultrasound image. Furthermore, the user can select the desired trajectory and / or intersections on one or more ultrasound images, and the robotic aiming device can move to accommodate this input. Multiple views can be made available simultaneously, as can all trajectory selection modes associated with those images.

[0084] Figure 10 This is an example of using ultrasound for multiplanar robotic aiming. The robotic aiming device 1001 can be coupled to a multiplanar ultrasound probe 1002 having one or more adjustable imaging planes. This results in a combination of the aforementioned adjustable planes and multiplanar capabilities. Multiplanar ultrasound imaging can assist less experienced users in performing ultrasound-guided surgery because they can simultaneously visualize needles and anatomical structures in multiple imaging planes and leverage the advantages of robotics to overcome the aforementioned planning and hand-eye coordination challenges.

[0085] exist Figure 10 In the image, the user is accessing a blood vessel, which is imaged in cross-section on the left (1003) and longitudinally on the right (1004). To fully define the desired position of the needle tip in 3D space, the user will need to define the target in both views 1003 and 1004. The longitudinal view (right) 1004 is needed to define the desired position of the needle tip along the length of the blood vessel. The cross-section (left) 1003 is needed to define the centering of the needle within the blood vessel. Either image can be used to define the depth to which the needle is inserted. The user can define these parameters sequentially by starting with either the longitudinal or cross-sectional view.

[0086] This article describes a combination of needle visualization enhancement features and robot-guided techniques. By coupling a robot aiming device to an ultrasound system with needle visualization enhancement features, the measured position and orientation of the instrument can be used as input to the visualization features to most effectively enhance the instrument.

[0087] Figure 11This is an example of how combining robot guidance with an imager capable of needle visualization enhancement can improve enhancement performance in ways that would be impossible without robotics. View 'a' 1101 shows how poor needle echo reflectivity can occur during ultrasound imaging because the emitted sound waves reflect off the instrument and do not return to the transducer, resulting in poor needle echo reflectivity. View 'b' 1102 shows a version of needle visualization enhancement where the beam is steered toward the instrument, resulting in improved signal return (shown as the returned echo is closer to the transducer in 'b' than in 'a'), but the beam alignment is not optimal. This is because, in views 'a' and 'b', the angle between the needle and the transducer (defined by theta-a(Q)) differs. a ) and theta-b (Q b () indicates that the angle was not measured. Existing needle visualization enhancements assume this value via default parameters, or require the user to estimate the needle angle and input the value via the user interface. View 'c'1103 shows the angle theta-c (Q) of the needle. c Knowing the robot's kinematics, one can determine how the beam can be optimally steered. Needle visualization enhancement, performed via image post-processing (i.e., without steer the ultrasonic beam), relies on similar information about the needle's orientation. The enhanced needle visualization can then be presented on the user interface in several ways. Examples include enhancing image contrast in known areas of the instrument, or by adding a color overlay.

[0088] An alternative embodiment of needle enhancement involves performing machine learning segmentation on the image region where the needle is to be visualized. This can be used in conjunction with the beam steering embodiment described above, but can also be implemented independently. By focusing the segmentation algorithm on a smaller region of the image, the performance and speed of needle recognition can be improved.

[0089] This paper describes constraint-backward actuation based on inverse kinematics (IK). This involves the user controlling all degrees of freedom of a given robot's motion. Doff Unlike conventional unconstrained inverse kinematics (IK) inverse kinematics (IK) based constrained inverse kinematics (IK) allows users to focus only on the parts they need to control, while kinematically assisting the rest of the motion in a human-like manner within the loop. Given the same framework, a range of constrained inverse kinematics (IK) patterns can be implemented very flexibly and easily to meet specific needs and / or migrated to different robot designs. The user experience depends primarily on the given robot's motion constraints and the choice of user touchpoints.

[0090] This paper describes the common framework of IK-based constrained back-drive and the design of three specific constrained back-drive modes: a) RCM (Remote Center of Motion) back-drive; b) PnP (Placement and Pointing) back-drive; and c) PnT (Placement and Tracking) back-drive. Each of the three constrained back-drive modes has a corresponding automated motion mode for similar aiming requirements: a) RCM back-drive corresponds to T2T; b) PnP back-drive corresponds to D2R; and c) PnT back-drive corresponds to dynamic aiming (such as automatic tracking of anatomical structures identified by segmenting pixels in ultrasound images). User touchpoints for different back-drive modes may or may not overlap. Furthermore, the back-drive modes are adapted for both 3-DoF and 5-DoF robots.

[0091] Figure 12 This is an example flowchart of a constraint-driven reverse framework. Figure 12 Not only is the common process of constrained back drive illustrated, but the switching between all robot operating modes is also shown, where unconstrained back drive can be regarded as a special case of zero-constraint back drive. Figure 12 Step 1201, which occurs when the reverse drive contact is activated and when the reverse drive contact is not activated, is described in detail.

[0092] When the reverse drive contact is activated 1201a, the system can begin to indicate the constraint reverse drive mode 1202. Control of selective robot joints can then be enabled, and other joints can be allowed to coast 1203. The reverse drive joint positions can then be read from the coasted robot joints 1204. The task-space target pose 1205 can then be synthesized from the constraints of this mode and the reverse drive joint positions. The joint position target 1210 can then be generated using an inverse kinematics solver.

[0093] When the reverse drive contact is not activated 1201b, control of all robot joints can be activated 1206. The system can then wait for the final task-space target pose to be achieved 1207. Then the automated motion mode can begin 1208. Then the task-space target pose can be generated 1209. Then the inverse kinematics solver can be used to generate the joint position target 1210.

[0094] In both the reverse-drive frames 1201a and 1201b, the trajectory planner can then generate smooth joint position commands 1211. The movement of the activated robot joints can then be controlled 1212.

[0095] For different constraint-driven modes Figure 12Only two prominent areas require conceptualization based on aiming requirements and robot design: the controlled selective robot joint 1203 and the synthetic task-space target pose 1205. This paper describes the details of these three modes.

[0096] The first mode could be RCM reverse drive. RCM reverse drive is a situation where the user needs to maintain a constant position at the skin entry point but wants to adjust the needle angle to point towards a new target. Therefore, control can be enabled on the proximal joint to maintain (or converge to) a constant task-space end effector position, while allowing free movement of (one or more) distal rotational joints to allow the user to adjust the orientation. Specifically, for Figure 13 In the exemplary 5-DoF robot (described in more detail below), A0 to A2 1301 are under control for 3-DoF task-space end effector positions, and A3 to A4 1302 are in free movement; the reverse-driven joint angles of A3 / A4 are transformed to constrained task-space positions to synchronize the complete 5-DoF task-space target pose, and are thus fed into its IK solver to calculate the target pose of A0 to A2.

[0097] for Figure 14 The exemplary 3-DoF robot (described in more detail below) has A1–A2 1401 under control for 2-DoF task-space positioning, and A3 1402 in a free-moving position; the synthesized 3-DoF task-space target pose is provided to its IK solver to calculate the pose of A1–A2 1401. Generally, the task-space end effector position that is kept constant refers to the RCM. This point may coincide with the tip of the needle guide, the position along the insertion of the instrument into the skin, or the position in space associated with the anatomy of interest.

[0098] Figure 15 illustrates an example of RCM reverse drive. Here, RCM is a point defined in the task space that is fixed below the ultrasound probe. As shown in Figure 15b, the user reverse drives joint A3 1503, and joints A1 1501 and A2 1502 are controlled to extend the arms so that the instrument still points at the RCM at a newly defined angle.

[0099] The second mode can be PnP reverse drive. PnP reverse drive can complement RCM reverse drive because the user reverse drives the proximal joints that define the position of the end effector, while the distal joints are controlled to ensure the instrument consistently points toward the target of interest. Therefore, control can be enabled on (one or more) distal rotational joints for pointing, while the proximal joints are left free to allow the user to reverse drive the task-space position. Specifically, for an exemplary 5-DoF robot, A0–A2 are free to move, and A3–A4 are under control; a 2-DoF orientation is derived from the target and robot end effector position to complete the 5-DoF task-space target pose for its IK solver. For an exemplary 3-DoF robot, A1–A2 are free to move, and A3 is under control; a 1-DoF orientation is derived to synthesize the 3-DoF task-space target pose for its IK solver. Generally, the position in space pointed to by the robot end effector can be arbitrarily defined.

[0100] Figure 16 illustrates an example of PnP reverse drive. Here, the user interaction involves dragging the end effector from its position in Figure 16a to a new position indicated by the dashed arrow 1604 in Figure 16b. Joints A1 1601 and A2 1602 are reverse-driven by repositioning the end effector. Joint A3 1603 is actively controlled to point the apparatus at a fixed point in the task space.

[0101] The third mode can be PnT reverse drive. PnT reverse drive is almost identical to PnP reverse drive, except that it can be used to target dynamic targets. Therefore, the reverse drive part is the same as PnP reverse drive, and the target is updated in real time by another module (such as the dynamic aiming described above).

[0102] Reverse drives offer many technical advantages. They can be simple, inexpensive, robust, and high-performance.

[0103] With collaborative robots ( cobot Compared to alternative solutions in [the context of] ...

[0104] Since both reverse-drive and automated motion modes rely purely on kinematics for robot control, they fall within the same framework described in this paper. Mode switching and workflow support can be straightforward and robust.

[0105] Inverse motion is intuitive and effective for user interaction. Physical interaction and force / haptic feedback can be available without imposing any additional fine-point aiming burden on the user (the best of both unconstrained inverse motion and fully automated motion). Inverse motion minimizes the disruption caused by switching back and forth between physical robot and GUI input. Users can choose to primarily use the GUI for visual feedback without removing their hands from the robot / probe or patient. Inverse motion alleviates the 2D constraints of the GUI in designing user interactions. For example, D2R is only used for in-plane manipulation, while PnP inverse motion can be used for any orientation.

[0106] Reverse drive is compatible with other enhancements. For example, it can easily add joint space impedance control, such as gravity / friction / backlash compensation, to further enhance the reverse drive experience. In another example, it can easily add more modes to enhance the user experience in new areas, such as PnT reverse drive to enhance dynamic aiming or PnP reverse drive for out-of-plane operation.

[0107] Inverse kinematics can be used to constrain robot inverse actuation. Three specific constrained inverse actuation modes for different robot designs can be used to preserve both the full precision aiming capability of automated motion and inverse actuation user interaction with force-haptic / tactile feedback. Switching between fully automated motion control and (constrained or unconstrained) inverse actuation modes via IK and trajectory planning can be utilized. "Always manual" operation with full sensory feedback to the user (visual from the GUI and force-haptic / tactile from direct interaction with the robot / probe and / or patient) can be enabled.

[0108] Figure 13 This is an example of a 5-DoF robot design. A0-A4 represent the actuated joints of the robot. These actuated joints can be linear or rotary. L0-L4 represent the robot's links. The RCM reverse drive contact can be located at the proximal end of the instrument track 1303, while the unconstrained reverse drive, PnP reverse drive, and PnT reverse drive contacts can all be the same, near L4 1304.

[0109] Figure 14This is an example of a 3-DoF robot design. A1-A3 represent the actuated joints of the robot. These actuated joints can be linear or rotational. L1-L3 represent the robot's links. As shown, A0 1403 is not active, therefore A1-A3 are the actual 3-DoF. Additionally, a machine track (not shown) may be included. The contact for RCM reverse drive can be at the proximal end of the machine track, while the contacts for unconstrained reverse drive, PnP reverse drive, and PnT reverse drive can all be the same, near L3 1404.

[0110] Figure 17 This is an example of a patient-side robotic system for percutaneous instrumental intervention. The two main subsystems are a patient-side cart 1701 and a handheld robotic instrument guide 1702 with an ultrasound probe 1703. A computer, power supply, USB communication components, and a display are part of the patient-side cart subsystem 1701. The handheld robotic subsystem is inserted into the patient-side cart 1701 via a cable facilitating digital communication and power. The handheld robotic subsystem includes the ultrasound probe 1703, the robotic positioning arm 1702, and the instrument guide (not shown).

[0111] The user holds the ultrasound machine (with the attached robot) in one hand and manipulates the robot's end effector / instrument and graphical user interface with the other. This paradigm allows the robot to aim at objects seen in the ultrasound image by tapping on a target—or other controls—on the display. Ultrasound settings are also available to the user. Once the user is satisfied with the aiming configuration, they are able to insert the instrument into the target via a guide. A graphical representation on the screen shows the path the instrument will take and the position of the instrument tip when it is inserted into the patient.

Claims

1. A system comprising: Robot system, the robot system comprising: A robotic arm, wherein the robotic arm includes a robotic manipulator and a machine guide coupled to the robotic manipulator; and Medical imaging equipment; A system user interface, wherein the system user interface displays multiple medical images and robotic system controls; A remote user interface provided to at least one remote user, wherein the remote user interface includes at least one of the following: at least one element of the system user interface and at least one image from the medical imaging device; and The two-way communication capability between at least one remote user and robot operator, wherein the two-way communication capability includes at least one of the following: audio communication and video communication.

2. The system of claim 1, wherein the robotic system facilitates percutaneous medical intervention.

3. The system of claim 1, wherein the medical imaging device is an ultrasound probe.

4. The system of claim 1, wherein the at least one remote user interacts remotely with elements of the system user interface.

5. The system of claim 1, wherein the remote user is provided with a portal containing telemetry data from the robot system.

6. The system of claim 1, wherein the remote user modifies the robot system software.

7. The system of claim 1, wherein the remote user interface is a copy of the system user interface.

8. The system of claim 1, wherein the at least one remote user provides guidance to the robot operator using remote annotations.

9. The system of claim 1, wherein the system user interface utilizes a graphical overlay.

10. A non-transitory computer-readable storage medium, comprising instructions executable by a processor to perform a method, the method comprising: An imaging device coupled to a robot manipulator receives multiple medical images; The system user interface displays multiple received medical images and robot system controls; Display at least one of the following on the remote user interface: At least one element of the system user interface and At least one image from the medical imaging device The remote user interface is provided to at least one remote user; and Enable bidirectional communication capability between the at least one remote user and the robot operator, wherein the bidirectional communication capability includes at least one of the following: audio communication and video communication.

11. The non-transitory computer-readable storage medium of claim 10, wherein the robotic system facilitates percutaneous medical intervention.

12. The non-transitory computer-readable storage medium of claim 10, wherein the medical imaging device is an ultrasound probe.

13. The non-transitory computer-readable storage medium of claim 10, wherein the at least one remote user remotely interacts with elements of the system user interface.

14. The non-transitory computer-readable storage medium of claim 10, wherein the remote user is provided with a portal containing telemetry data from the robotic system.

15. The non-transitory computer-readable storage medium of claim 10, wherein the remote user modifies the robot system software.

16. The non-transitory computer-readable storage medium of claim 10, wherein the remote user interface is a copy of the system user interface.

17. The non-transitory computer-readable storage medium of claim 10, wherein the at least one remote user provides guidance to the robot operator using remote annotations.

18. The non-transitory computer-readable storage medium of claim 10, wherein the system user interface utilizes a graphical overlay.

Images