Continuous teleoperation with assisted master control

Abstract

The medical system may include a user input device for controlling the medical instrument. The medical system may also include a sensor coupled to the user input device for generating information related to the user input device. The medical system may identify drift of the user input device based on information from the sensor while operating the user input device in an unassisted mode, and switch operation of the user input device from the unassisted mode to an assisted mode according to the identified drift, the assisted mode may enable operation of the user input device based on a set of parameters different from the set of parameters used for the unassisted mode. A method for operating a medical system including a user input device for controlling a medical instrument as also disclosed herein.

Description

[Technical field]

[0001] TECHNICAL FIELD The systems and methods disclosed herein relate to input devices, and more particularly, to input devices for robotic surgical systems. [Background technology]

[0002] Robot-enabled medical systems can perform a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopic procedures (e.g., bronchoscopy, ureteroscopy, gastroscopy, etc.).

[0003] Such a robotic medical system may include a robotic arm configured to control the movement of a medical tool during a given medical procedure. The robotic medical system may also include input devices used to control the positioning and / or actuation of the medical tool during a medical procedure. Summary of the Invention [Means for solving the problem]

[0004] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

[0005] In one aspect, an input device for controlling a robotic surgical tool is provided. The input device can include a first pair of opposing links and a second pair of opposing links. The first pair of opposing links and the second pair of opposing links can be radially symmetrically arranged. The input device can be configured to control the operation of the robotic surgical tool.

[0006] In another aspect, a physician console can be provided. The physician console can include an input device including a first gripper and a second gripper. The input device can be configured to control a surgical tool. At least one of the first gripper and the second gripper can include a four-link radially symmetrical gripper for controlling operation of the surgical tool.

[0007] In another aspect, an input device for controlling a movement of a robotic arm is provided. The input device can include a first finger pad including a first set of two or more electrodes for determining a presence of a user at the first finger pad. The input device can also include a processor for modifying a movement of the robotic arm in response to information from the input device regarding the presence of a user at the first finger pad.

[0008] In another aspect, a medical system is provided having a user input device for controlling a medical instrument. The user input device is operable in an unassisted mode having a first set of parameters for operation of the user input device and an assisted mode having a second set of parameters for operation of the user input device, the second set being different from the first set. The medical system further includes a sensor coupled to the user input device for generating information related to the user input device. The medical system also includes a processor and a memory storing instructions for execution by the processor. The stored instructions include instructions for identifying drift of the user input device based on information from the sensor while operating the user input device in the unassisted mode, and switching operation of the user input device from the unassisted mode to the assisted mode in accordance with identifying the drift.

[0009] According to some embodiments, a method for operating a medical system including a user input device for controlling a medical instrument is performed by one or more processors executing instructions stored in a memory. The method includes detecting drift of the user input device while operating in an unassisted mode, and switching operation of the user input device to an assisted mode in accordance with the detection of the drift, the mode including applying a second parameter set different from the first parameter set. The method also includes determining, while operating in the assisted mode, that the user input device is stable, and switching operation of the user input device to the unassisted mode in accordance with the determination.

[0010] According to some embodiments, a method for operating a medical system including a user input device for controlling a medical instrument is performed by one or more processors executing instructions stored in a memory. The method includes operating the user input device in an unassisted mode, the method including applying a first set of parameters to the user input device. The method further includes, while operating in the unassisted mode, identifying drift of the user input device, and switching operation of the user input device to an assisted mode in accordance with detection of the drift, the switching including applying a second set of parameters different from the first set of parameters. [Brief description of the drawings]

[0011] The disclosed aspects are hereinafter described in conjunction with the accompanying drawings, in which like designations refer to like elements and which illustrate, but do not limit, the disclosed aspects. [Figure 1] 1 illustrates one embodiment of a cart-based robotic system deployed for diagnostic and / or therapeutic bronchoscopy procedures. [Diagram 2] 2 illustrates further aspects of the robotic system of FIG. 1, according to some embodiments. [Diagram 3] 2 illustrates an embodiment of the robotic system of FIG. 1 positioned for ureteroscopy. [Figure 4]2 illustrates one embodiment of the robotic system of FIG. 1 deployed for a vascular procedure. [Diagram 5] 1 illustrates an embodiment of a table-based robotic system positioned for a bronchoscopy procedure. [Figure 6] FIG. 6 is an alternative view of the robotic system of FIG. 5, according to some embodiments. [Figure 7] 1 illustrates an exemplary system configured to accommodate a robotic arm, according to some embodiments. [Figure 8] 1 is an embodiment of a table-based robotic system configured for a ureteroscopy procedure. [Figure 9] 1 illustrates one embodiment of a table-based robotic system configured for laparoscopic procedures. [Figure 10] 10 illustrates an embodiment of the table-based robotic system of FIGS. 5-9 with pitch or tilt adjustment. [Figure 11] FIG. 11 illustrates a detailed view of the interface between the table and column of the table-based robotic system of FIGS. 5-10, according to some embodiments. [Figure 12] 1 illustrates an alternative embodiment of a table-based robotic system. [Figure 13] FIG. 13 illustrates an end view of the table-based robotic system of FIG. 12, in accordance with some embodiments. [Figure 14] FIG. 1 illustrates an end view of a table-based robotic system with an attached robotic arm, according to some embodiments. [Figure 15] 1 illustrates an exemplary instrument driver, according to some embodiments. [Figure 16] 1 illustrates an exemplary medical instrument having paired instrument drivers, according to some embodiments. [Figure 17] 13A-13C show alternative designs for the instrument driver and instrument, where the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument, according to some embodiments. [Figure 18] 1 illustrates an instrument having an instrument-based insertion architecture, according to some embodiments. [Figure 19] 1 illustrates an example controller, according to some embodiments. [Figure 20] FIG. 17 shows a block diagram illustrating a localization system that estimates the location of one or more elements of the robotic system of FIGS. 1-10, such as the location of the instrument of FIGS. 16-18, according to some embodiments. [Figure 21A] 1 illustrates an input device in an open position according to some embodiments. [Figure 21B] 21B illustrates the input device of FIG. 21A in a closed position according to some embodiments. [Figure 22] 21A-21B show exploded views of the input device according to some embodiments. [Figure 23A] 21B illustrates a sensor in the input device of FIG. 21A for sensing the presence of a user, according to some embodiments. [Figure 23B] 21B illustrates a sensor in the input device of FIG. 21A for sensing the presence of a user, according to some embodiments. [Figure 24A] 23C illustrates measurement of the self-capacitance of the electrodes of the sensor shown in FIG. 23B, according to some embodiments. [Figure 24B] 23C illustrates a measurement of capacitance between electrodes of the sensor shown in FIG. 23B, according to some embodiments. [Figure 24C] 23C illustrates a measurement of capacitance between electrodes of the sensor shown in FIG. 23B, according to some embodiments. [Figure 24D] 24B illustrates crosstalk between the electrodes shown in FIG. 24A and other components of the input device of FIG. 21A in accordance with some embodiments. [Figure 24E] 24B illustrates a shield for reducing crosstalk between the electrodes shown in FIG. 24A and other components of the input device of FIG. 21A according to some embodiments. [Diagram 25] 23A and 23B show exemplary measurements for sensing the presence of a user, according to some embodiments. [Figure 26A]23A and 23B to an integrated circuit for sensing the presence of a user, according to some embodiments. [Figure 26B] 23A and 23B to an integrated circuit for sensing the presence of a user, according to some embodiments. [Figure 27A] 21B illustrates an example of a light curtain configuration generated by a light sensor located on the input device of FIG. 21A to sense the presence of a user, according to some embodiments. [Figure 27B] 21B illustrates an example of a light curtain configuration generated by a light sensor located on the input device of FIG. 21A to sense the presence of a user, according to some embodiments. [Figure 28A] 21B shows an example of a finger loop for use in conjunction with a time-of-flight sensor in the input device of FIG. 21A according to some embodiments. [Figure 28B] 21B shows an example of a finger loop for use in conjunction with a time-of-flight sensor in the input device of FIG. 21A according to some embodiments. [Figure 29] 21B illustrates a gimbaled robotic joint that supports the input device of FIG. 21A according to some embodiments. [Figure 30A] FIG. 1 is a state diagram illustrating states of a medical system according to some embodiments. [Figure 30B] 1 illustrates transitions between different states of a medical system according to some embodiments. [Figure 31A] 1 is a flowchart illustrating a method for manipulating a surgical tool via an input device, according to some embodiments. [Figure 31B] 1 is a flowchart illustrating a method for manipulating a surgical tool via an input device, according to some embodiments. [Diagram 32] FIG. 1 is a schematic diagram illustrating electronic components of a medical system including an input device according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1. Overview. Aspects of the present disclosure may be integrated into a robot-enabled medical system capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopy. Among endoscopic procedures, the system may perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0013] In addition to performing a wide range of procedures, the system may provide additional benefits such as enhanced imaging and guidance to assist the physician. Furthermore, the system may provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm movements and positions. Still further, the system may provide the physician with the ability to perform procedures with improved ease of use, such that even a single user may control one or more of the instruments of the system.

[0014] Various embodiments are described below in conjunction with the drawings for illustrative purposes. It should be understood that many other implementations of the concepts of the present disclosure are possible, and that various advantages may be achieved by implementing the concepts of the present disclosure. Headings are included herein for reference and to aid in locating various sections. The headings do not limit the scope of the concepts described therein. Such concepts may be applied throughout the entire specification.

[0015] A. Robotic System - Cart. A robot-enabled medical system may be configured in a variety of ways depending on the particular procedure. FIG. 1 illustrates one embodiment of a cart-based robot-enabled system 10 arranged for diagnostic and / or therapeutic bronchoscopy. During bronchoscopy, the system 10 may include a cart 11 having one or more robotic arms 12 for delivering medical instruments, such as a steerable endoscope 13, which may be a procedure-specific bronchoscope for bronchoscopy, to a natural orifice access point (i.e., the mouth of a patient positioned on a table in this example) for delivering diagnostic and / or therapeutic tools. As shown, the cart 11 may be positioned adjacent to the patient's upper torso to provide access to the access point. Similarly, the robotic arms 12 may be actuated to position a bronchoscope relative to the access point. The arrangement of FIG. 1 may also be utilized when performing a gastrointestinal (GI) procedure with a gastroscope, an endoscope specialized for GI procedures. FIG. 2 depicts an example embodiment of the cart in more detail.

[0016] With continued reference to FIG. 1, once the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient robotically, manually, or a combination thereof. As shown, the steerable endoscope 13 can include at least two nested parts, such as an inner leader section and an outer sheath section, each section coupled to a separate instrument driver from a set of instrument drivers 28, each instrument driver coupled to the distal end of an individual robotic arm. This linear arrangement of the instrument drivers 28, which facilitates coaxial alignment of the leader section with the sheath section, creates a "virtual rail" 29 that can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual rails described herein are depicted in the figures using dashed lines, and thus do not depict any physical structure of the system. Translation of the instrument driver 28 along the virtual rail 29 nests the inner leader section relative to the outer sheath section, or advances or retracts the endoscope 13 from the patient. The angle of the virtual rail 29 may be adjusted, translated, or pivoted based on clinical application or physician preference. For example, in bronchoscopy, the angle and position of the virtual rail 29 shown represents a compromise between providing the physician access to the endoscope 13 and minimizing friction that results from bending the endoscope 13 into the patient's mouth.

[0017] The endoscope 13 can be directed downstream of the patient's trachea and lungs after insertion using precise commands from the robotic system until the target destination or surgical site is reached. The endoscope 13 can be manipulated to telescope the inner leader portion from the outer sheath portion to enhance articulation and increase bend radius to enhance navigation through the patient's pulmonary network and / or reach a desired target. The use of a separate instrument driver 28 also allows the leader and sheath portions to be driven independently of one another.

[0018] For example, the endoscope 13 may be directed to deliver a biopsy needle to a target, such as a lesion or nodule in the patient's lung. The needle may be deployed through a working channel that runs the length of the endoscope to obtain a tissue sample that is analyzed by a pathologist. Depending on the results of the pathology, additional tools may be deployed through the working channel of the endoscope for additional biopsies. After identifying the nodule as malignant, the endoscope 13 may deliver tools endoscopically to ablate the potential cancerous tissue. In some cases, the diagnostic and therapeutic treatments may be delivered in separate procedures. In these situations, the endoscope 13 may also be used to deliver fiducials to "mark" the location of the targeted nodule. In other cases, the diagnostic and therapeutic treatments may be delivered during the same procedure.

[0019] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide support for control, electronics, fluidics, optics, sensors, and / or power to the cart 11. Placing such functionality in the tower 30 may reduce the form factor of the cart 11, allowing the surgeon and his / her staff to more easily adjust and / or reposition the cart 11. Additionally, the division of functionality between the cart / table and the support tower 30 reduces clutter in the operating room and promotes improved clinical workflow. The cart 11 may be positioned near the patient, while the tower 30 may be housed in a remote location so as not to get in the way during the procedure.

[0020] To support the robotic system described above, the tower 30 may include computer-based control system components that store computer program instructions in a non-transitory computer-readable storage medium, such as, for example, a persistent magnetic storage drive, a solid-state drive, or the like. Execution of these instructions, whether execution occurs in the tower 30 or in the cart 11, may control the entire system or subsystems thereof. For example, when executed by a processor in a computer system, the instructions may cause the robotic system components to actuate the carriage and arm mounts, operate the robotic arm, and control medical instruments. For example, in response to receiving control signals, motors in the joints of the robotic arm may position the arm in a particular pose.

[0021] The tower 30 may also include pumps, flow meters, valve controls, and / or fluid access to provide controlled irrigation and aspiration capabilities to a system that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, irrigation and aspiration capabilities may be provided directly to the endoscope 13 via separate cables.

[0022] The tower 30 may include voltage and surge protection designed to provide filtered, protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components within the cart 11 and making the cart 11 smaller and more mobile.

[0023] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras anywhere in the robotic system 10. In combination with a control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles located throughout the system, including in the tower 30. Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on a medical instrument.

[0024] The tower 30 may also include a console 31 in addition to other consoles available to the rest of the system, for example a console mounted on top of a cart. The console 31 may include a user interface and a display screen, such as a touch screen, for the operator physician. The consoles in the system 10 are generally designed to provide both pre-operative and real-time information of the procedure, such as robotic control and navigation and localization information for the endoscope 13. If the console 31 is not the only console available to the physician, the console 31 may be used by a second operator, such as a nurse, to provide procedure-specific data, such as navigation and localization information, as well as monitor the patient's health or vitals and the operation of the system 10. In other embodiments, the console 31 is housed in a body separate from the tower 30.

[0025] The tower 30 may be coupled to the cart 11 and the endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 through only one cable, simplifying and decluttering the operating room. In other embodiments, certain functions may be combined in separate wiring and connections. For example, power may be provided to the cart 11 through a single power cable, while support for control, optics, fluidics, and / or navigation may be provided through separate cables.

[0026] FIG. 2 provides a detailed view of one embodiment of a cart 11 from the cart-based robot-enabled system shown in FIG. 1. The cart 11 generally includes an elongated support structure 14 (often referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more carriages, such as a carriage 17 (alternatively an "arm support") for supporting the deployment of one or more robotic arms 12 (three are shown in FIG. 2). The carriage 17 may include individually configurable arm mounts that rotate along orthogonal axes to adjust the base of the robotic arms 12 for better positioning relative to the patient. The carriage 17 also includes a carriage interface 19 that allows the carriage 17 to translate vertically along the column 14.

[0027] The carriage interface 19 is connected to the column 14 through slots, such as slots 20 positioned on either side of the column 14 to guide the vertical translation of the carriage 17. The slots 20 include vertical translation interfaces for positioning and holding the carriage 17 at various vertical heights relative to the cart base 15. The vertical translation of the carriage 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, an individually configurable arm mount on the carriage 17 allows the robotic arm base 21 of the robotic arm 12 to be angled in various configurations.

[0028] In some embodiments, a slot cover may be added to the slot 20 that is flush and parallel with the slot surface to prevent dirt and fluids from entering the interior chamber of the column 14 and the vertical translation interface as the carriage 17 translates vertically. The slot cover may be deployed through a pair of spring spools positioned near the vertical top and bottom of the slot 20. The cover is coiled within the spools until it deploys to extend and retract from the coiled state as the carriage 17 translates vertically up and down. The spring loading of the spools provides the force to retract the cover into the spool as the carriage 17 translates towards the spool, while also maintaining a seal as the carriage 17 translates away from the spool. The cover may be connected to the carriage 17 using, for example, a bracket at the carriage interface 19 to ensure that the cover extends and retracts properly as the carriage 17 translates.

[0029] Column 14 may contain mechanisms therein, such as gears and motors, designed to use a vertically aligned leadscrew to mechanically translate carriage 17 in response to control signals generated in response to user input, such as input from console 16.

[0030] The robotic arms 12 may generally include a robotic arm base 21 and end effectors 22 separated by a series of links 23 connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arms 12. Each of the robotic arms 12 may have seven joints, thus providing seven degrees of freedom. A large number of joints results in a large number of degrees of freedom, allowing for "redundant" degrees of freedom. Having redundant degrees of freedom allows the robotic arms 12 to position their respective end effectors 22 at specific positions, orientations, and trajectories in space using different link positions and joint angles. This allows the system to position and orient the medical instrument from a desired point in space, while allowing the physician to move the arm joints to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0031] The cart base 15 balances the weight of the column 14, carriage 17, and robotic arm 12 on the floor. Thus, the cart base 15 houses the heavier parts such as electronics, motors, power supplies, and components that allow the cart 11 to be either mobile and / or immobilized. For example, the cart base 15 includes casters 25 in the form of rollable wheels that allow the cart 11 to be easily moved around the room before a procedure. After reaching the proper position, the casters 25 can be immobilized using wheel locks to hold the cart 11 in place during a procedure.

[0032] The console 16 positioned at the vertical end of the column 14 allows for both a user interface and a display screen (or dual-purpose device, such as, for example, a touch screen 26) to receive user input to provide both pre-operative and intra-operative data to the physician user. Potential pre-operative data on the touch screen 26 may include pre-operative planning, navigation and mapping data derived from a pre-operative computed tomography (CT scan), and / or notes from a pre-operative patient interview. Intra-operative data on the display may include optical information provided by tools, sensor and coordinate information from sensors, and vital patient statistics such as respiration, heart rate, and / or pulse. The console 16 may be positioned and angled to allow the physician to access the console 16 from the opposite side of the column 14 from the carriage 17. From this position, the physician can view the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 to aid in maneuvering and stabilizing the cart 11.

[0033] FIG. 3 illustrates an embodiment of the robot-enabled system 10 positioned for ureteroscopy. In a ureteroscopy procedure, the cart 11 can be positioned to deliver a ureteroscope 32, a procedure-specific endoscope designed to follow the patient's urethra and ureters, to the patient's lower abdominal region. In a ureteroscopy, it may be desirable for the ureteroscope 32 to be aligned directly with the patient's urethra to reduce friction and forces on the sensitive anatomical structures in that area. As shown, the cart 11 can be aligned to the table legs to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. From the table legs, the robotic arm 12 can insert the ureteroscope 32 along a virtual rail 33 directly through the urethra and into the patient's lower abdomen.

[0034] After insertion into the urethra using control techniques similar to those in bronchoscopy, the ureteroscope 32 can be navigated to the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications. For example, the ureteroscope 32 can be directed to the ureter and kidney to fragment accumulated kidney stones using a laser lithotriptor or ultrasonic lithotriptor device deployed through the working channel of the ureteroscope 32. After lithotripsy is complete, the resulting stone fragments can be removed using a basket deployed through the ureteroscope 32.

[0035] FIG. 4 shows an embodiment of the robot-enabled system 10 similarly positioned for a vascular procedure. In a vascular procedure, the system 10 may be configured such that the cart 11 can deliver a medical instrument 34, such as a steerable catheter, to an access point in the femoral artery in the patient's leg. The femoral artery presents both a larger diameter for navigation as well as a less circuitous and tortuous path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 may be positioned toward the patient's leg and lower abdomen to allow the robotic arm 12 to provide a virtual rail 35 with direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical instrument 34 may be oriented and inserted by translating the instrument driver 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0036] B. Robot system-table. An embodiment of a robot-enabled medical system may also incorporate a patient table. The incorporation of a table reduces the amount of capital equipment in the operating room by removing the cart and allows better access to the patient. FIG. 5 shows one embodiment of such a robot-enabled system deployed for a bronchoscopy procedure. The system 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") across the floor. Much like a cart-based system, the end effector of the robotic arm 39 of the system 36 includes an instrument driver 42 designed to manipulate an elongated medical instrument, such as the bronchoscope 40 of FIG. 5, through or along a virtual rail 41 formed from the linear alignment of the instrument driver 42. In practice, a C-arm for providing fluoroscopic imaging may be positioned above the patient's upper abdominal region by placing emitters and detectors around the table 38.

[0037] FIG. 6 provides an alternative view of the system 36 without the patient and medical instruments for discussion purposes. As shown, the column 37 may include one or more carriages 43, shown as a ring shape, in the system 36, which may be the base for one or more robotic arms 39. The carriages 43 may translate along a vertical column interface 44 that runs the length of the column 37 to provide different vantage points from which the robotic arms 39 may be positioned to reach the patient. The carriages 43 may rotate about the column 37 using a mechanical motor positioned within the column 37 to allow the robotic arms 39 to access multiple sides of the table 38, such as both sides of the patient. In embodiments having multiple carriages, the carriages may be individually positioned on the column and may translate and / or rotate independently of the other carriages. The carriages 43 need not surround the column 37, or even be circular, although the ring shape as shown facilitates rotation of the carriages 43 around the column 37 while maintaining structural balance. Rotation and translation of carriage 43 allows system 36 to align medical instruments such as endoscopes and laparoscopes to different access points on the patient. In other embodiments (not shown), system 36 may include a patient table or patient bed having an adjustable arm support in the form of a bar or rail extending alongside it. One or more robotic arms 39 may be attached to the adjustable arm support that can be adjusted vertically (e.g., via a shoulder with an elbow joint). By providing vertical adjustment, robotic arms 39 may advantageously be stored compactly under a patient table or patient bed and then raised during a procedure.

[0038] The robotic arm 39 may be mounted to the carriage 43 via a set of arm mounts 45 that include a series of joints that may be individually rotated and / or telescopically extended to provide additional configurability to the robotic arm 39. Additionally, the arm mounts 45 may be positioned on the carriage 43 such that, when the carriage 43 is appropriately rotated, the arm mounts 45 may be positioned either on the same side of the table 38 (as shown in FIG. 6), on opposite sides of the table 38 (as shown in FIG. 9), or on adjacent sides of the table 38 (not shown).

[0039] The column 37 structurally provides support for the table 38 and a path for the vertical translation of the carriage 43. Internally, the column 37 may include a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the translation of the carriage 43 based on the lead screw. The column 37 may also transmit power and control signals to the carriage 43 and a robotic arm 39 mounted thereon.

[0040] The table base 46 functions similarly to the cart base 15 of the cart 11 shown in FIG. 2, housing the heavier components to counterbalance the table / bed 38, column 37, carriage 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during the procedure. The casters extend from the bottom of the table base 46 in opposite directions on either side of the base 46 and may be retracted when the system 36 needs to be moved.

[0041] Continuing to refer to FIG. 6, the system 36 may also include a tower (not shown) that divides the functionality of the system 36 between the table and the tower to reduce the form factor and bulk of the table. As with the previously disclosed embodiments, the tower may provide various support functions to the table, such as processing, computing, and control capabilities, power, fluidics, and / or optical and sensor processing. The tower may also be mobile for positioning away from the patient to improve physician access and keep the operating room clutter-free. Additionally, locating components in the tower allows for more storage space in the table base 46 for potential accommodation of the robotic arm 39. The tower may also include a master controller or console that provides both a user interface for user input, such as a keyboard and / or pendant, and a display screen (or touch screen) for pre-operative and intra-operative information, such as real-time imaging, navigation, and tracking information. In some embodiments, the tower may also include a holder for a gas tank used for venting.

[0042] In some embodiments, the table base may house and store the robot arm when not in use. FIG. 7 illustrates a system 47 for housing the robot arm in one embodiment of a table-based system. In the system 47, the carriage 48 may be translated vertically into the base 49 to house the robot arm 50, arm mount 51, and carriage 48 therein. The base cover 52 may be translated and retracted to open to deploy the carriage 48, arm mount 51, and robot arm 50 about the column 53, and closed to house and protect them when not in use. The base cover 52 may be sealed with a membrane 54 along the edge of its opening to prevent dirt and fluid ingress when closed.

[0043] FIG. 8 illustrates one embodiment of a robotic-enabled table-based system configured for a ureteroscopy procedure. For ureteroscopy, the table 38 may include a swivel 55 for positioning the patient at an off angle from the column 37 and table base 46. The swivel 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) to position the bottom of the swivel 55 away from the column 37. For example, pivoting the swivel 55 may allow a C-arm (not shown) to be positioned over the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the carriage (not shown) about the column 37, the robotic arm 39 may insert the ureteroscope 56 directly into the patient's groin area along the virtual rail 57 to reach the urethra. For ureteroscopy, stirrups 58 may be secured to the swivel 55 of the table 38 to support the position of the patient's legs during the procedure and allow clear access to the patient's groin area.

[0044] In a laparoscopic procedure, minimally invasive instruments may be inserted into a patient's anatomy through small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instruments include an elongated rigid member, such as a shaft, that is used to access anatomy within the patient. After distension of the patient's abdominal cavity, the instruments may be oriented to perform a surgical or medical task, such as grasping, cutting, ablation, suturing, etc. In some embodiments, the instruments may include a scope, such as a laparoscope. FIG. 9 illustrates one embodiment of a robotic-enabled table-based system configured for laparoscopic procedures. As shown in FIG. 9, the carriage 43 of the system 36 may be rotated and vertically adjusted to position a pair of robotic arms 39 on either side of the table 38 such that the instruments 59 may be positioned using the arm mounts 45 to pass through minimal incisions on either side of the patient to reach the patient's abdominal cavity.

[0045] To accommodate laparoscopic procedures, the robotic table system may also tilt the platform to a desired angle. FIG. 10 illustrates one embodiment of a pitch or tilt adjustable robotic medical system. As shown in FIG. 10, the system 36 accommodates the tilt of the table 38 to position one portion of the table higher off the floor than another portion. Additionally, the arm mount 45 may rotate to match the tilt so that the robotic arm 39 maintains the same planar relationship with the table 38. To accommodate steeper angles, the column 37 may also include a nesting portion 60 that allows the column 37 to extend vertically to prevent the table 38 from contacting the floor or colliding with the table base 46.

[0046] FIG. 11 provides a detailed illustrative view of the interface between the table 38 and the column 37. The pitch mechanism 61 may be configured to vary the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch mechanism 61 may be enabled by positioning orthogonal axes 1, 2 at the column-table interface, each actuated by a separate motor 3, 4 in response to an electrical pitch command. Rotation along one screw 5 would allow tilt adjustment in one axis 1, while rotation along the other screw 6 would allow tilt adjustment along the other axis 2. In some embodiments, the use of ball joints allows the pitch angle of the table 38 relative to the column 37 to be varied in multiple degrees of freedom.

[0047] For example, pitch adjustment is particularly useful when attempting to position the table in the Trendelenburg position, i.e., positioning the patient's lower abdomen higher off the floor than the patient's upper abdomen for lower abdominal surgery. The Trendelenburg position allows gravity to slide the patient's internal organs down to the patient's upper abdomen, emptying the abdominal cavity for entry of minimally invasive tools to perform lower abdominal surgical or medical procedures, such as a laparoscopic prostatectomy.

[0048] 12 and 13 show isometric and end views of an alternative embodiment of a table-based surgical robotic system 100. The surgical robotic system 100 includes one or more adjustable arm supports 105 that can be configured to support one or more robotic arms relative to a table 101 (see, e.g., FIG. 14). In the illustrated embodiment, only one adjustable arm support 105 is shown, but additional arm supports can be provided on the opposite side of the table 101. The adjustable arm support 105 can be configured to move relative to the table 101 to adjust and / or change the position of the adjustable arm support 105 and / or any robotic arm attached thereto relative to the table 101. For example, the adjustable arm support 105 can be adjusted with one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides the system 100 with great versatility, including the ability to easily accommodate one or more adjustable arm supports 105 and any robotic arms attached thereto under the table 101. The adjustable arm support 105 can be raised from a stowed position to a position below the top surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from a stowed position to a position above the top surface of the table 101.

[0049] The adjustable arm support 105 can provide multiple degrees of freedom, including lift, lateral translation, tilt, etc. In the illustrated embodiment of FIGS. 12 and 13, the arm support 105 is configured with four degrees of freedom, which are indicated by arrows in FIG. 12. A first degree of freedom allows adjustment of the adjustable arm support 105 in the z-direction ("Z-lift"). For example, the adjustable arm support 105 can include a carriage 109 configured to move up and down along or relative to the column 102 that supports the table 101. A second degree of freedom can allow the adjustable arm support 105 to tilt. For example, the adjustable arm support 105 can include a rotational joint, which can allow the adjustable arm support 105 to align with the bed in the Trendelenburg position. A third degree of freedom allows the adjustable arm support 105 to "pivot up," which can be used to adjust the distance between the side of the table 101 and the adjustable arm support 105. A fourth degree of freedom may allow translation of the adjustable arm support 105 along the longitudinal length of the table.

[0050] The surgical robotic system 100 of Figures 12 and 13 may include a table supported by a column 102 mounted to a base 103. The base 103 and column 102 support the table 101 against a support surface. A bed axis 131 and a support axis 133 are shown in Figure 13.

[0051] An adjustable arm support 105 can be mounted to the column 102. In other embodiments, the arm support 105 can be mounted to the table 101 or the base 103. The adjustable arm support 105 can include a carriage 109, a bar or rail connector 111, and a bar or rail 107. In some embodiments, one or more robot arms mounted to the rail 107 can translate and move relative to each other.

[0052] The carriage 109 may be attached to the column 102 by a first joint 113, which allows the carriage 109 to move relative to the column 102 (e.g., up and down a first or vertical axis 123). The first joint 113 may provide a first degree of freedom ("Z-lift") to the adjustable arm support 105. The adjustable arm support 105 may include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 may include a third joint 117 that may provide a third degree of freedom ("pivot up") to the adjustable arm support 105. A further joint 119 (shown in FIG. 13) may be provided that mechanically constrains the third joint 117 to maintain the orientation of the rail 107 as the rail connector 111 rotates about a third axis 127. The adjustable arm support 105 may include a fourth joint 121 that may provide a fourth degree of freedom (translation) to the adjustable arm support 105 along a fourth axis 129 .

[0053] FIG. 14 illustrates an end view of a surgical robotic system 140A with two adjustable arm supports 105A, 105B mounted on either side of a table 101. A first robotic arm 142A is attached to a bar or rail 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A attached to the rail 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B attached to the rail 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to attach to one or more robotic medical instruments or tools.

[0054] In some embodiments, one or more of the robotic arms 142A, 142B comprise arms with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B can include eight degrees of freedom, including an insertion axis (one degree of freedom including insertion), wrist (three degrees of freedom including wrist pitch, yaw and roll), elbow (one degree of freedom including elbow pitch), shoulder (two degrees of freedom including shoulder pitch and yaw), and base 144A, 144B (one degree of freedom including translation). In some embodiments, the insertion degree of freedom can be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides the insertion via an instrument-based insertion architecture.

[0055] C. Instrument drivers and interfaces. The end effector of the robotic arm of the system may include (i) an instrument driver (alternatively called an "instrument drive mechanism" or "instrument device manipulator") that incorporates electromechanical means for actuating the medical instrument, and (ii) a removable or detachable medical instrument that may lack any electromechanical components such as a motor. This dichotomy may be caused by the need to sterilize the medical instruments used in medical procedures and the inability to adequately sterilize expensive capital equipment due to the complex mechanical assembly and sensitive electronics of the medical instruments. Thus, the medical instruments may be designed to be detached, removed, and replaced from the instrument driver (and thus the system) upon individual sterilization or disposal by the physician or physician's staff. In contrast, the instrument driver does not need to be replaced or sterilized and may be draped for protection.

[0056] FIG. 15 illustrates an exemplary instrument driver. An instrument driver 62 positioned at the distal end of the robotic arm includes one or more drive units 63 arranged with parallel axes to provide a controlled torque to the medical instrument via a drive shaft 64. Each drive unit 63 includes an individual drive shaft 64 for interacting with the instrument, a gearhead 65 for converting motor shaft rotation to a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the rotational speed of the motor shaft and providing feedback to the control circuitry, and a control circuit 68 for receiving control signals to operate the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (e.g., four as shown in FIG. 15) independent drive outputs to the medical instrument. In operation, the control circuit 68 would receive the control signal, send a motor signal to the motor 66, compare the resulting motor rotational speed measured by the encoder 67 to a desired speed, and modulate the motor signal to generate the desired torque.

[0057] In procedures requiring a sterile environment, the robotic system may incorporate a drive interface, such as a sterile adapter connected to a sterile drape, located between the instrument driver and the medical instrument. The primary purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument while maintaining physical separation between the drive shaft and the drive input, and thus sterility. Thus, an exemplary sterile adapter may include a set of rotational inputs and outputs intended to mate with the drive shaft of the instrument driver and a drive input for the instrument. The sterile drape, which is connected to the sterile adapter, is constructed of a thin flexible material, such as a clear or translucent plastic, and is designed to cover the instrument driver, the robot arm, and capital equipment, such as a cart (in a cart-based system) or a table (in a table-based system). The use of the drape allows the capital equipment to be positioned in close proximity to the patient while still being located in an area that does not require sterility (i.e., the non-sterile field). On the other side of the sterile drape, the medical instrument may interface with the patient in an area that requires sterility (i.e., the sterile field).

[0058] D. Medical equipment. 16 illustrates an exemplary medical instrument with a paired instrument driver. Similar to other instruments designed for use with a robotic system, the medical instrument 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its design intended for manual interaction by a physician, may include a rotary drive input 73, e.g., a receptacle, pulley, or spool, designed to mate with a drive output 74 that passes through a drive interface on an instrument driver 75, typically at the distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mated drive input 73 of the instrument base 72 may share an axis of rotation with the drive output 74 on the instrument driver 75 to allow for the transfer of torque from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include a spline designed to mate with a receptacle on the drive input 73.

[0059] The elongated shaft 71 is designed to be delivered through either an anatomical opening or lumen, such as in an endoscope, or a minimally invasive incision, such as in a laparoscopy. The elongated shaft 71 can be either flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or can include a customized combination of both flexible and rigid sections. If designed for laparoscopy, the distal end of the rigid elongated shaft can be connected to an end effector extending from an articulating wrist formed from a clevis having at least one degree of freedom, and to a surgical tool or medical instrument, such as a grasper or scissors, that can be actuated based on a force from a tendon as the drive input rotates in response to a torque received from the drive output 74 of the instrument driver 75. If designed for endoscopy, the distal end of the flexible elongated shaft can include a steerable or controllable bend that can be articulated and bent based on a torque received from the drive output 74 of the instrument driver 75.

[0060] Torque from the instrument driver 75 is transmitted downstream of the elongated shaft 71 using tendons along the elongated shaft 71. These individual tendons, such as pull wires, can be individually secured to individual drive inputs 73 in the instrument handle 72. From the handle 72, the tendons travel down the elongated shaft 71 through one or more pull lumens and are secured to a distal portion of the elongated shaft 71 or are secured to a wrist at the distal portion of the elongated shaft. During a surgical procedure, such as a laparoscopic, endoscopic, or hybrid procedure, these tendons can be coupled to a distally mounted end effector, such as a wrist, grasper, or scissors. In such an arrangement, torque exerted on the drive input 73 would transmit tension to the tendons, thereby actuating the end effector in some manner. In some embodiments, the tendons can rotate a joint about an axis to move the end effector in one direction or another during a surgical procedure. Alternatively, the tendon may be connected to one or more jaws of a grasper at the distal end of the elongate shaft 71, with tension from the tendon causing the grasper to close.

[0061] In endoscopy, the tendons may be coupled to a flex or articulating section positioned along (e.g., at the distal end) of the elongated shaft 71 via adhesive, control rings, or other mechanical fixation. If fixedly attached to the distal end of the flex, torque exerted on the drive input 73 is transferred downstream of the tendon, bending or articulating the softer flex (sometimes called the articulatable section or articulatable region). Along the non-flexing section, it may be convenient to thread or spiral the individual pull lumens that direct the individual tendons along (or inwardly) the wall of the endoscope shaft to counterbalance the radial forces resulting from tension in the pull wires. The angle of the thread and / or spacing between these can be altered or designed for a particular purpose, with a tighter spiral exhibiting less shaft compression under load forces, while a lesser amount of spiral resulting in more shaft compression under load forces but limiting bending. At the other end of the spectrum, orienting the pull lumen parallel to the longitudinal axis of the elongate shaft 71 can allow for controlled articulation at the desired bend or articulation.

[0062] In endoscopy, the elongated shaft 71 houses several components that aid in robotic procedures. The shaft 71 may include a working channel for deploying surgical tools (or medical instruments), irrigating, and / or aspirating the surgical field at the distal end of the shaft 71. The shaft 71 may also house wires and / or optical fibers for transmitting signals to and from an optical assembly at the distal tip, which may include an optical camera. The shaft 71 may also house optical fibers for carrying light from a proximally located light source, such as a light emitting diode, to the distal end of the shaft 71.

[0063] At the distal end of the instrument 70, the distal tip may include a working channel opening for delivering tools to the surgical site for diagnosis and / or treatment, irrigation, and aspiration. The distal tip may also include a port for a camera, such as a fiberscope or digital camera, to capture images of the internal anatomical space. In this regard, the distal tip may also include a port for a light source to illuminate the anatomical space when the camera is in use.

[0064] 16, the drive shaft axis, i.e., the drive input axis, is orthogonal to the axis of the elongated shaft 71. However, this arrangement complicates the roll ability of the elongated shaft 71. Rolling the elongated shaft 71 along its axis while holding the drive input 73 stationary results in undesirable entanglement of the tendons as they exit the drive input 73 and enter the pull lumen within the elongated shaft 71. The resulting entanglement of such tendons can interfere with control algorithms intended to predict the movement of the flexible elongated shaft 71 during an endoscopic procedure.

[0065] FIG. 17 illustrates an alternative design of the instrument driver and instrument, in which the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units with their drive outputs 81 aligned in parallel at the end of a robotic arm 82. The drive units and their respective drive outputs 81 are housed in a rotating assembly 83 of the instrument driver 80 that is driven by one of the drive units in that assembly 83. In response to torque provided by the rotating drive units, the rotating assembly 83 rotates along a circular bearing that connects the rotating assembly 83 to a non-rotating portion 84 of the instrument driver 80. Power and control signals may be transferred from the non-rotating portion 84 of the instrument driver 80 to the rotating assembly 83 via electrical contacts that may be maintained throughout the rotation by a brushed slip ring connection (not shown). In other embodiments, the rotating assembly 83 may be responsive to a separate drive unit that is integrated into the non-rotatable portion 84 and thus not parallel to the other drive units. The rotation mechanism 83 enables the instrument driver 80 to rotate the drive units and their respective drive outputs 81 as a single unit about the instrument driver axis 85 .

[0066] Similar to the previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown with a transparent exterior skin for purposes of discussion) that includes a number of drive inputs 89 (such as receptacles, pulleys, and spools) configured to receive the drive outputs 81 in the instrument driver 80. Unlike the previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87 with an axis that is substantially parallel to the axis of the drive inputs 89, rather than orthogonal as in the design of FIG.

[0067] When coupled to the rotating assembly 83 of the instrument driver 80, the medical instrument 86, including the instrument base 87 and the instrument shaft 88, rotates with the rotating assembly 83 about the instrument driver axis 85. Because the instrument shaft 88 is positioned in the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Rotation of the rotating assembly 83 therefore causes the instrument shaft 88 to rotate about its own longitudinal axis. Also, because the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 at the instrument base 87 do not become entangled during rotation. Thus, the parallelism of the axes of the drive output 81, drive input 89, and instrument shaft 88 allows shaft rotation without entangling any of the control tendons.

[0068] FIG. 18 illustrates an instrument having an instrument-based insertion architecture, according to some embodiments. The instrument 150 can be coupled to any of the instrument drivers discussed above. The instrument 150 includes an elongated shaft 152, an end effector 162 connected to the elongated shaft 152, and a handle 170 coupled to the elongated shaft 152. The elongated shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongated shaft 152 includes one or more channels or grooves 158 along its outer surface. The grooves 158 are configured to receive one or more wires or cables 180. Thus, the one or more cables 180 run along the outer surface of the elongated shaft 152. In other embodiments, the cables 180 can run through the elongated shaft 152. Manipulation of one or more of the cables 180 (e.g., via an instrument driver) results in actuation of the end effector 162.

[0069] The instrument handle 170, sometimes referred to as the instrument base, may typically include a mounting interface 172 having one or more mechanical inputs 174, e.g., receptacles, pulleys or spools, designed to intermate with one or more torque couplers on a mounting surface of the instrument driver.

[0070] In some embodiments, instrument 150 includes a series of pulleys or cables that allow elongated shaft 152 to translate relative to handle 170. In other words, instrument 150 itself constitutes an instrument-based insertion architecture that accommodates the insertion of the instrument, thereby minimizing reliance on a robotic arm to effect the insertion of instrument 150. In other embodiments, the robotic arm may be significantly involved in the insertion of the instrument.

[0071] E. Controller. Any of the robotic systems described herein can include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the controller can be coupled (e.g., communicatively, electrically, wirelessly, and / or mechanically) to the instrument such that manipulation of the controller causes a corresponding manipulation of the instrument, e.g., via master-slave control.

[0072] 19 is a perspective view of one embodiment of the controller 182. In this embodiment, the controller 182 includes a hybrid controller that can include both impedance control and admittance control. In other embodiments, the controller 182 can only utilize impedance control, i.e., passive control. In other embodiments, the controller 182 can only utilize admittance control. Advantageously, being a hybrid controller allows the controller 182 to have a lower perceived inertia during use.

[0073] In the illustrated embodiment, the controller 182 is configured to enable manipulation of two medical instruments and includes two handles 184. Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0074] 19, each positioning platform 188 includes a SCARA arm (Selectively Compliant Assembly Robot Arm) 198 coupled to a column 194 by a prismatic joint 196. The prismatic joint 196 is configured to translate along the column 194 (e.g., along a rail 197) to allow each of the handles 184 to translate in the z-direction, providing a first degree of freedom. The SCARA arm 198 is configured to allow movement of the handles 184 in the xy-plane, providing an additional two degrees of freedom.

[0075] In some embodiments, one or more load cells are positioned within the controller. For example, in some embodiments, a load cell (not shown) is positioned on the body of each of the gimbals 186. Providing load cells allows a portion of the controller 182 to operate under admittance control, thereby advantageously reducing the perceived inertia of the controller during use. In some embodiments, the positioning platform 188 is configured for admittance control while the gimbal 186 is configured for impedance control. In other embodiments, the positioning platform 188 is configured for impedance control while the gimbal 186 is configured for admittance control. Thus, in some embodiments, the translational or positional degree of freedom of the positioning platform 188 can rely on admittance control, while the rotational degree of freedom of the gimbal 186 is determined by impedance control.

[0076] F. Navigation and Control. Traditional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered through a C-arm) and other forms of radiation-based imaging modalities to provide intraluminal guidance to the operator-physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and localization means to reduce the physician's exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "localization" may refer to determining and / or monitoring the position of an object in a reference coordinate system. Techniques such as pre-operative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, pre-operative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to improve the information obtained solely by radiation-based imaging modalities.

[0077] FIG. 20 is a block diagram illustrating a localization system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an exemplary embodiment. The localization system 90 may be a set of one or more computing devices configured to execute one or more instructions. The computing devices may be embodied by a processor (or processors) and computer readable memory in one or more of the components discussed above. By way of example, and not by way of limitation, the computing devices may be in the tower 30 shown in FIG. 1, in the cart 11 shown in FIGS. 1-4, or in the bed shown in FIGS. 5-14.

[0078] 20, the localization system 90 may include a localization module 95 that processes the input data 91-94 to generate position data 96 of the distal tip of the medical instrument. The position data 96 may be data or logic that represents the location and / or orientation of the distal tip of the instrument relative to a frame of reference. The frame of reference may be relative to the patient's anatomy or relative to a known object such as an EM field generator (see discussion of EM field generators below).

[0079] The various input data 91-94 will now be described in more detail. Pre-operative mapping can be accomplished using the acquisition of low-dose CT scans. The pre-operative CT scans are reconstructed into three-dimensional images that are visualized, for example, as cutaway "slices" of the patient's internal anatomy. When analyzed as a whole, image-based models can be developed that cover the anatomical cavities, anatomical spaces, and anatomical structures of the patient's anatomy, such as the patient's pulmonary network. Techniques such as centerline geometry can be determined and approximated from the CT images to create a three-dimensional volume of the patient's anatomy, referred to as model data 91 (also referred to as "pre-operative model data" when developed using only pre-operative CT scans). The use of centerline geometry is discussed in U.S. Patent Application Serial No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topological models can also be derived from CT images and are particularly suited for bronchoscopy.

[0080] In some embodiments, the instrument may be equipped with a camera to provide visual data (or image data) 92. The localization module 95 may process the visual data 92 to enable one or more visual-based (or image-based) location tracking modules or functions. For example, the pre-operative model data 91 may be used in conjunction with the visual data 92 to enable computer vision-based tracking of a medical instrument (e.g., an endoscope or an instrument advancing through a working channel of the endoscope). For example, using the pre-operative model data 91, the robotic system may generate a model-based predicted endoscopic image library based on the expected path of travel of the endoscope, with each image linked to a location in the model. During surgery, this library can be referenced by the robotic system to compare real-time images captured by a camera (e.g., a camera at the distal end of the endoscope) to those in the image library to assist with localization.

[0081] Other computer vision based tracking techniques use feature tracking to determine the motion of the camera and therefore the endoscope. Some features of the localization module 95 may identify a circular geometry in the pre-operative model data 91 that corresponds to an anatomical lumen and track changes in that geometry to ascertain which anatomical lumen has been selected and the relative rotational and / or translational motion of the camera. The use of a phase map may further enhance vision-based algorithms or techniques.

[0082] Optical flow, another computer vision based technique, may analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer camera movement. Examples of optical flow techniques may include motion detection, object segmentation computations, luminance, motion compensated coding, stereo disparity measurements, etc. By comparing multiple frames over multiple iterations, the movement and location of the camera (and therefore the endoscope) may be determined.

[0083] The localization module 95 may use real-time EM tracking to generate a real-time position of the endoscope in a global coordinate system that may be registered to the patient's anatomy represented by the pre-operative model. In EM tracking, an EM sensor (or tracker) including one or more sensor coils embedded in one or more locations and orientations in a medical instrument (e.g., an endoscopic instrument) measures variations in an EM field generated by one or more static EM field generators positioned at known locations. The location information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) may be placed near the patient to produce a low-intensity magnetic field that the embedded sensor can detect. The magnetic field induces small currents in the sensor coils of the EM sensor, which may be analyzed to determine the spacing and angle between the EM sensor and the EM field generator. These distances and orientations may be "registered" intraoperatively to the patient's anatomy (e.g., the pre-operative model) to determine a geometric transformation that aligns a single location in the coordinate system with a location in the pre-operative model of the patient's anatomy. Once registered, an EM tracker embedded at one or more locations on the medical instrument (e.g., the distal tip of an endoscope) can provide a real-time indication of the medical instrument's progression through the patient's anatomy.

[0084] The robotic commands and kinematic data 94 may also be used by a localization module 95 to provide localization data 96 for the robotic system. The device pitch and yaw resulting from the articulation commands may be determined during pre-operative calibration. Intraoperatively, these calibration measurements may be used in combination with known insertion depth information to estimate the position of the instrument. Alternatively, these calculations may be analyzed in combination with EM, vision, and / or phase modeling to estimate the position of the medical instrument within the network.

[0085] As Fig. 20 illustrates, several other input data may be used by the localization module 95. For example, although not shown in Fig. 20, an instrument utilizing shape sensing fibers may provide shape data that the localization module 95 can use to ascertain the position and shape of the instrument.

[0086] The localization module 95 may use a combination of the input data 91-94. In some cases, such combination may use a probabilistic approach in which the localization module 95 assigns confidence weights to the locations identified from each of the input data 91-94. Thus, if the EM data is unreliable (such as in the presence of EM interference), the reliability of the locations identified by the EM data 93 may be reduced, and the localization module 95 may rely more heavily on the vision data 92 and / or the robot command and kinematics data 94.

[0087] As discussed above, the robotic systems discussed herein can be designed to incorporate one or a combination of two or more of the above-mentioned techniques. The computer-based control system of the tower, bed, and / or cart-based robotic system can store, for example, in a non-transitory computer-readable storage medium, such as a persistent magnetic storage drive, solid-state drive, etc., computer program instructions that, when executed, cause the system to receive and analyze sensor data and user commands, generate control signals throughout the system, and display navigational and localization data, such as instrument position within a global coordinate system, anatomical maps, etc.

[0088] 2. Manually operated input device for robotic systems. Embodiments of the present disclosure relate to systems and techniques for input devices for operating a robotic medical system and / or one or more medical instruments comprising such a robotic medical system.

[0089] A robotic medical system, such as the system described above, can include an input device configured to enable an operator (e.g., a physician performing a robot-enabled medical procedure) to manipulate and control one or more instruments. In some embodiments, the robotic medical system can include an input device for manipulating one or more medical tools. In some examples, the input device can remotely manipulate the one or more medical tools, such as via teleoperation or telesurgery.

[0090] Those skilled in the art will appreciate that the input devices described herein can also be applied in non-medical situations. For example, the input device may be useful for manipulating tools involving hazardous materials. Additionally, in some embodiments, the input device described herein may be useful for grasping objects in both physical and virtual environments. In some configurations, the input device may be self-sufficient as a service robot that interacts with a human operator. In some configurations, the input device may be coupled (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) with a medical instrument such that manipulation of the input device causes a corresponding manipulation of the medical instrument. In some configurations, the input device and the medical instrument are arranged in a master-slave pair. In some configurations, the input device may be configured to control the operation of a robotic surgical tool. In some configurations, the input device may be referred to as a manipulator, emulator, master, controller, interface, etc.

[0091] The input device can serve as an input for an operator to control the operation of a medical instrument, such as an endoscopic, intraluminal, laparoscopic, or open surgical instrument. Movement of the input device by the operator can direct the movement of the medical instrument. For example, when the operator translates the input device in three-dimensional space (e.g., up, down, left, right, back, forward), the system can cause a corresponding translation of the medical instrument. Similarly, when the operator rotates the input device (e.g., about any of three orthogonal axes), the system can cause a corresponding rotational movement of the medical instrument. The input device can also include one or more inputs that allow the operator to actuate the medical instrument. As an example, when the medical instrument includes a gripper, the input device can include one or more inputs that allow the operator to open and close the gripper.

[0092] In some embodiments, the robotic medical system includes an input device with seven degrees of freedom that tracks the movement of the operator's hand, including three positional degrees of freedom (e.g., translational movements in x, y, z space), three rotational degrees of freedom (rotational movements about pitch, roll, and yaw axes), and one (or more) instrument actuation degrees of freedom (e.g., angular degrees of freedom). In some embodiments, the instrument actuation degrees of freedom can control the opening and closing of an end effector of a medical instrument, such as a gripper or grasper for holding an object. In some embodiments, the input device may include a greater or lesser number of degrees of freedom. For example, in some embodiments, the input device may include more than three positional degrees of freedom or more than three rotational degrees of freedom to provide one or more redundant degrees of freedom. In some embodiments, the redundant degrees of freedom can provide the input device with additional mechanical flexibility, for example, to avoid singularities caused by the mechanical structure of the input device.

[0093] 19 illustrates an embodiment of an input device or controller 182 that may be used by a user to control one or more instruments. As mentioned above, the controller may include two handles 184 that may be used to control the instruments. Each of the handles 184 may be connected to a gimbal 186. Both of the handles 184 may function as gripping implements.

[0094] The grasper may be part of an input device that an operator (e.g., a physician, a user, etc.) contacts and holds, allowing the operator to control components of the robotic system, such as medical instruments. The grasper may be the operator's primary input to the system during surgery.

[0095] A. Multi-link gripper 21A-21B show an embodiment of a handle or grip 200 that can be used as part of an input system, such as the input system described above with reference to FIG. 19. FIG. 21A shows the grip 200 in an open configuration, and FIG. 21B shows the grip 200 in a closed configuration. FIG. 22 shows the grip of FIGS. 21A-21B in an exploded view. The grip 200 can include multiple links, and in the illustrated embodiment, the grip 200 includes four links 202, 204, 206, 208. In some configurations, the grip 200 can include at least two links. In some configurations, the grip 200 can include at least three links. In some examples, the grip 200 can have any number of links, such as 2-12 links. In some embodiments, the multiple links can be circumferentially arranged around the grip 200. In some configurations, the multiple links can be equally spaced apart from one another. In some configurations, the links may be spaced less than 180 degrees apart from one another about the central axis. The configuration described below includes four links, but any number of links may be included.

[0096] As shown in FIGS. 21A-B, the gripping tool 200 includes four links including a first link 202, a second link 204, a third link 206, and a fourth link 208. The four links 202, 204, 206, 208 may be spaced around the circumference of the gripping tool 200. The four links 202, 204, 206, 208 may be evenly spaced from one another. The four links may be radially symmetrically arranged. In some embodiments, the four links 202, 204, 206, 208 may each be spaced less than 180 degrees from an adjacent link. The four links 202, 204, 206, 208 may be spaced approximately 90 degrees from one another, as shown in the illustrated arrangement. A number of links 202, 204, 206, 208 may be circumferentially disposed about the gripper 200 as shown.

[0097] The links 202, 204, 206, 208 can be arranged in pairs. For example, the gripper 200 can include a first pair of opposing links 202, 204 and a second pair of opposing links 206, 208. In the illustrated arrangement, the first pair of opposing links can include a first link 202 and a second link 204 spaced approximately 180 degrees apart from each other. In the illustrated arrangement, the second pair of opposing links can include a third link 206 and a fourth link 208 spaced approximately 180 degrees apart from each other. In a modified arrangement, the links 202, 204 of the first pair of opposing links can be spaced less than 180 degrees apart from each link 206, 208 of the second pair of opposing links.

[0098] 22, the gripper 200 can include a central shaft 250. The central shaft 250 can be referred to as a longitudinal shaft, longitudinal member, central member, shaft, or member. The central shaft can include a circuit 300, such as a printed circuit board, for connecting to other components of the gripper 200 or other components of a robotic system. In the illustrated arrangement, the circuit 300 can be located inside the central shaft 250.

[0099] The central shaft 250 may support a plurality of links 202, 204, 206, 208. Each of the first pair of opposing links 202, 204 and each of the second pair of opposing links 206, 208 may be coupled to the central shaft 250. The first link 202 may have a proximal end 232 and a distal end 222. The second link 204 may have a proximal end 234 and a distal end 224. The third link 206 may have a proximal end 236 and a distal end 226. The fourth link 208 may have a proximal end 238 and a distal end 228. The multiple links 202 , 204 , 206 , 208 may be connected or operatively connected at their respective proximal ends 232 , 234 , 236 , 238 and / or at their respective distal ends 222 , 224 , 226 , 228 .

[0100] The first pair of opposing links 202, 204 can each include a finger grip or pad 212, 214. The second pair of opposing links 206, 208 can each include a secondary input 216, 218. Each of the plurality of links 202, 204, 206, 208 can include a secondary link 242, 244, 246, 248 for attaching the proximal ends 232, 234, 236, 238 of the links 202, 204, 206, 208 to the central shaft 250.

[0101] Each of the plurality of links can be configured to move from an open position in which the proximal end 232, 234, 236, 238 of each of the plurality of links 202, 204, 206, 208 is disposed radially away from the central shaft 250, to a closed position in which the proximal end 232, 234, 236, 238 of each of the plurality of links 202, 204, 206, 208 is disposed radially closer to the central shaft 250. Referring again to FIG. 21A, the grasper 200 is shown in an open position in which the proximal ends 232, 234, 236, 238 of the plurality of links 202, 204, 206, 208 are disposed radially closer to the central shaft 250 of the grasper 200. Each of the multiple links 202, 204, 206, 208 can be connected at each of its respective distal ends 222, 224, 226, 228 to the gripper 200 such that each link can extend or pivot at an angle away from the central shaft 250. The proximal ends 232, 234, 236, 238 of each of the first pair of opposing links 202, 204 and each of the second pair of opposing links 206, 208 are configured to move radially relative to the central shaft 250.

[0102] 21B, the gripper 200 is shown in a closed position with the proximal ends 232, 234, 236, 238 of the plurality of links 202, 204, 206, 208 disposed proximate to a central shaft 250 of the gripper 200. In the closed position, each of the proximal ends 232, 234, 236, 238 may be positioned proximate to the central shaft 250 such that each of the plurality of links 202, 204, 206, 208 may be parallel in length to the central shaft 250.

[0103] Each of the multiple links 202, 204, 206, 208 can be biased to the open position. In some configurations, each link can be spring loaded in the open position. In some configurations, there are at least two springs (not shown) for each link, with a first spring providing the majority of the force biasing the link to the open position. A second spring can provide slight tactile feedback when the links have reached a degree of closure to indicate to the user when the grasper is closed and that further movement to close the grasper will result in an increase in the clamping force of the surgical instrument.

[0104] For example, the first pair of opposing links 202, 204 and / or the second pair of opposing links 206, 208 may be manipulated in a clamping motion that may translate to movement of a surgical instrument within the body. For example, opening and closing the first pair of opposing links 202, 204 corresponds to opening and closing a scissor tool or jaws of a medical instrument. The facilitated clamping motion may make grasper actuation natural and easy for a user.

[0105] The multiple links 202, 204, 206, 208 on the gripper 200 can measure the input angle of a user's fingers. For example, the angle at which any one or more of the multiple links 202, 204, 206, 208 are positioned relative to the central shaft 250 can be translated into a desired angle of an instrument component, such as one or more jaws of an end effector of the instrument.

[0106] The grasper 200 can have an increased number of links (such as four links as shown in Figures 21A-21B and 22) and / or links spaced closer together. The grasper 200 can also be radially symmetric, especially at the distal-most end.

[0107] By having such a radially symmetric configuration of the gripper, a user may advantageously be able to easily perform certain movements (e.g., roll operations) that would otherwise be difficult. If a user wishes to perform a roll-intensive task (e.g., suturing) with a different gripper than that described herein, the user can only rotate the gripper approximately 180 degrees without repositioning the user's hand before the wrist goes out of range of motion. To continue rotating the gripper, it is necessary to release the gripper's current position, rotate the wrist, and continue gripping the gripper again.

[0108] A radially symmetric grasper with links spaced less than 180° apart allows the clinician to rotate the grasper between his fingertips while maintaining a desired orientation of the grasper (e.g., in a closed position or while maintaining a closed angle.) This is possible because the user's fingers always contact at least two links as the number of links increases and dead zones that may exist between the links are reduced.

[0109] Some users may choose to work on the outside of the finger pads 212, 214 to hold the links 202, 204, 206, 208 closer to the distal ends 222, 224, 226, 228 of the links 202, 204, 206, 208. The gripping tool 200 can advantageously accommodate this, allowing for comfortable use both on and off the finger pads 212, 214. Additionally, when working on the outside of the finger pads 212, 214 (such as at one or more distal ends 222, 224, 226, 228 of the plurality of links 202, 204, 206, 208), the grasper 200 can have radial symmetry such that the physician can close the grasper 200 (such as by closing a first pair of opposing links 202, 204 and / or by closing a second pair of opposing links 206, 208) and then roll the grasper 200 between the fingers. When performing this rolling operation of the grasper 200, it may be desirable to have the grasper 200 remain in the closed position. The first pair of opposing links 202, 204 and / or the second pair of opposing links 206, 208 may be identical and symmetrical at their distal ends 222, 224, 226, 228, allowing a user to close the grasper 200 using any of the multiple links 202, 204, 206, 208.

[0110] The radial symmetry at the distal end can advantageously be more tolerant of misalignment of the user's hand when manipulating the gripping tool 200. Furthermore, with an increased number of links spaced closer together (e.g., less than 180 degrees from each other), a user can more easily position their fingers to maintain contact with one or more of the multiple links when manipulating the gripping tool 200. For example, when working outside the finger pads 212, 214, a user can use any combination of multiple links to actuate the gripping tool 200. Multiple links can increase the number of contact points for a user to actuate the gripping tool 200. This can allow a user to more easily maintain contact with the gripping tool's actuators and reduce the difficulty of positioning and realigning the user's hand. The multiple links and radial symmetry can give a user more degrees of freedom to operate the gripping tool 200.

[0111] The first pair of opposing links 202, 204 may be longer in length than the second pair of opposing links 206, 208. The links may be arranged such that the longer links 202, 204 are opposite one another and the shorter links 206, 208 are located between the two. The two longer links 202, 204 may function as primary gripping links. The longer links 202, 204 may have finger grips, pads, loops, such as finger pads 212, 214 in the illustrated arrangement.

[0112] In other examples, the first pair of opposing links 202, 204 and the second pair of opposing links 206, 208 may be of equal length. In other examples, the second pair of opposing links 206, 208 may be of greater length than the first pair of opposing links 202, 204.

[0113] As described above, when a user manipulates the links of the gripping tool, the user's fingers can adjust the angle of the links. The links can be oriented or angled relative to a central axis or shaft. The links on the gripping tool can measure the input angle of the user's fingers. For example, the angle at which any one or more of the links are positioned relative to a central shaft can be translated into a desired angle of an instrument component, such as one or more jaws of an end effector of the instrument. The gripping tool 200 can include one or more sensors for measuring the angle of the links and thus the input angle of the user's fingers. Also as described above, a secondary input state can be measured by one or more sensors.

[0114] The gripping tool can include one or more sensors in various locations. In some configurations, the one or more sensors can be located on or coupled to one or more of the links. In some configurations, the one or more sensors can be located in or coupled to the central shaft. One or more sensors in the central shaft can be advantageous in that space above each of the links is limited. One or more sensors in the central shaft can also advantageously position the sensor away from movement of the links and from contact by the user, which can reduce the risk of damaging the sensor.

[0115] In some configurations, the one or more sensors can include Hall effect sensors, such as 3D Hall effect sensors. The one or more sensors can include three different sensors oriented orthogonally to each other. These sensor readings can be used to develop algorithms to determine the angles of the multiple links and to detect secondary input conditions. The location of the one or more sensors in the central shaft can also advantageously eliminate the need to run wires on the links and package the sensors.

[0116] The one or more sensors can detect one or more magnets included in the links and / or one or more magnets in the secondary input. For example, each link can include one or more magnets in a fixed location. As the links change angle, one or more sensors can be used to detect changes in the magnetic field due to the movement of these magnets.

[0117] Similarly, the secondary input may include or be operatively connected to one or more magnets such that a change or movement of the secondary input may change the position or orientation of the one or more magnets, which may be detected by a sensor. In some examples, a change in the magnetic field resulting from the secondary input may be coupled to the motion of the gripper or one or more components of the gripper. In some examples, the angle of the multiple links may be decoupled from the secondary input state.

[0118] In some configurations, one or more of the sensors may be a three degree of freedom sensor with multiple links and physical electrical connections to secondary inputs.

[0119] In some cases, the input device (e.g., a gripper) may not always be under the control of the operator. Additionally, the input device may receive inputs that are not intended by the user (e.g., unintended movements), which may be caused by disturbances to the input device by the user or some other personnel in the operating room, or from gravity compensation errors (e.g., the user overcompensating for gravity). In these cases, it may be necessary to modify the teleoperation to inhibit or reduce the movement of the robotic arm and its associated instruments. By detecting the unintended movements, the robotic system can modify the teleoperation to reduce, inhibit, or stop the movement of the robotic arm and associated instruments in response to the user input.

[0120] B. Capacitive Sensor 23A and 23B show a sensor in the grip 200 of FIG. 21A for sensing the presence of a user, according to some embodiments.

[0121] 23A , gripping tool 200 includes one or more sensors 310 configured to generate a signal in response to a user being in proximity to the sensors. In some embodiments, a first sensor of one or more sensors 310 is embedded in finger pad 212. In some embodiments, one or more sensors 310 includes a second sensor embedded in finger pad 214. In some embodiments, gripping tool 200 also includes finger pad 211 that is part of opposing link 206 and finger pad 213 that is part of opposing link 208. Finger pad 213 may include a third sensor of one or more sensors 310, and finger pad 213 may include a fourth sensor of one or more sensors 310.

[0122] In some embodiments, as shown in FIG. 23B, a sensor of the one or more sensors 310 includes one or more electrodes 311 (e.g., metal electrodes, conductive electrodes). In some embodiments, the one or more electrodes 311 are connected to an integrated circuit (e.g., integrated circuit 328 shown in FIG. 28A). The integrated circuit is configured to measure a capacitance that changes when a body part of a user (such as a user's hand, finger, or palm) is in proximity to the one or more electrodes. In some embodiments, the sensor includes the integrated circuit. In some embodiments, the integrated circuit is separate from the sensor (although the integrated circuit is electrically connected to the sensor).

[0123] 24A, at least one sensor (e.g., sensor 310) of the one or more sensors 310 includes (only) one electrode 311, and the integrated circuit includes a circuit 300 for measuring a self-capacitance of the electrode 311 (e.g., a capacitance between the electrode 311 and ground). The self-capacitance changes when a body part of a user is in proximity to the electrode 311. In some embodiments, the circuit 300 includes an excitation signal source 392 (e.g., an oscillator or an AC current source) and an analog-to-digital converter 394 coupled to the (same) electrode 311 via a multiplexer 306.

[0124] In some embodiments, as shown in Figures 24B and 24C, at least one sensor (e.g., sensor 310) of the one or more sensors 310 includes two or more electrodes 311-1 and 311-2, and the integrated circuit includes a circuit 301 for measuring a mutual capacitance between the two electrodes 311-1 and 311-2. As shown in Figures 24B and 24C, when a body part of a user is in proximity to the electrode 311-1 or 311-2 or between the two electrodes 311-1 and 311-2, the mutual capacitance changes. In some embodiments, the circuit 301 includes an excitation signal source 392 (e.g., an oscillator or an AC current source) coupled to a first electrode (e.g., electrode 311-1) of the two or more electrodes, and an analog-to-digital converter 394 coupled to a second electrode (e.g., electrode 311-2) of the two or more electrodes that is different from the first electrode.

[0125] 24D illustrates that in some configurations, the sensor may detect a change in capacitance due to placement of a user's body part in proximity to a component 314 (e.g., a metal component) located adjacent to one or more electrodes 311 of the sensor. Also, if the links 202, 204, 206, and 208 of the gripping tool 200 are movable relative to the central support 250 of the gripping tool 200, the distance between one or more electrodes 311 of the sensor 310 and other components 314 of the gripping tool 200, such as the central support 250 and other components located on the central support, may change during movement of the gripping tool 200. Thus, the capacitance may change even when the user's body part is not in proximity to one or more electrodes 311, which may be erroneously interpreted as indicating that the user's body part is in proximity to one or more electrodes 311.

[0126] 24E shows that in some embodiments, at least one sensor (e.g., sensor 310) of the one or more sensors 310 includes an electrode layer including one or more electrodes 311 and a shielding layer 312 for reducing capacitance between the one or more electrodes 311 and a component 314 of the gripping tool 200. The shielding layer 312 of the sensor 310 is disposed between the one or more electrodes 311 of the sensor 310 and the other component 314 of the gripping tool 200. This reduces (ideally eliminates) changes in measured capacitance due to changes in the position of the one or more electrodes 311 of the sensor 310 relative to the position of the other component 314 of the gripping tool 200. Thus, the shielding layer 312 reduces the effect that the other component 314 of the gripping tool 200 may have on the capacitance measured by a sensor having one or more electrodes 311.

[0127] FIG. 25 illustrates an exemplary measurement for sensing the presence of a user using the sensor illustrated in FIGS. 23A and 23B. FIG. 25 illustrates an exemplary graph showing a signal 318 over time corresponding to a measured capacitance between two electrodes 311-1 and 311-2 of the sensor 310 in response to a user's body part varying its distance to the electrodes 311-1 and 311-2. As shown, the signal 318 increases from a baseline value 316 (e.g., a background capacitance value when the user's body part is not in close proximity to the electrodes 311-1 and 311-2) in response to the user's body part approaching the electrodes 311-1 and 311-2 of the sensor 310. The signal 318 increases above a detection threshold 322 until it reaches a maximum capacitance value 324. The detection threshold 322 corresponds to a capacitance value above which the presence of a user is deemed to be present. In some embodiments, the detection threshold 322 is selected such that the difference between the detection threshold 322 and the baseline value 316 is greater than a representative value of the noise 320 (e.g., 3, 4, 5, 6, 7, 8, 9, or 10 times greater than the root mean square amplitude of the noise 320 detected by electrodes 311-1 and 311-2).

[0128] In some embodiments, the signal 318 is processed (e.g., by integrated circuit 328) before determining the presence of a user. For example, the signal 318 is filtered to remove or reduce noise 320 (e.g., based on rate of change, for example, by using a frequency domain filter such as a Fourier filter or a differential and integral filter).

[0129] 26A illustrates the coupling of the sensor shown in FIGS. 23A and 23B to an integrated circuit 328 for sensing the presence of a user. The sensor includes a flexible connector 330 (corresponding to extension 313) for electrical connection to the integrated circuit. This allows relative movement (e.g., rotational movement) of the sensing portion of the sensor with respect to the integrated circuit 328 while maintaining electrical contact between the sensing portion of the sensor with respect to the integrated circuit 328, which may be located at the tip of the gripper 200.

[0130] In some embodiments, each finger pad of gripping tool 200 includes a sensor, and the sensor of each finger pad is electrically coupled to integrated circuit 328 via flexible connector 330. For example, when gripping tool 200 includes four finger pads (and thus at least four sensors), integrated circuit 328 is electrically coupled to the four sensors via four flexible connectors 330 (e.g., one flexible connector 330 for each sensor). FIG. 26B shows four flexible connectors 330 electrically coupling the four sensors embedded in the four finger pads 211-214 to integrated circuit 328.

[0131] C. Optical sensor 27A and 27B show examples of light curtain configurations generated by light sensors located in the input device of FIG. 21A to sense the presence of a user.

[0132] In FIG. 27A, gripping tool 200 mounted on gimbal 332 is coupled with one or more optical sensors 334 (e.g., optical sensors 334-1 to 334-3). One or more optical sensors 334 detect light signals from a user's body part to determine the presence of the user. In some embodiments, one or more optical sensors 334 are coupled with or include a light source for providing illumination light, and one or more optical sensors detect light reflected or scattered by a user's body part, such as a finger. In some embodiments, one or more optical sensors 334-1 to 334-3 are each coupled with or include a light source for providing light curtains 336-1 to 336-3, respectively. In some embodiments, one or more optical sensors 334 include a time-of-flight sensor for determining a distance from the time-of-flight sensor to an object (e.g., a user's body part) entering the corresponding light curtain. In some embodiments, each of the one or more optical sensors 334 is a time-of-flight sensor. In some embodiments, the one or more optical sensors 334 include other types of optical sensors (e.g., beam-break sensors, LIDAR, cameras, etc.). In some embodiments, the one or more optical sensors 334 are mounted adjacent one end of the gripper 200 toward the gimbal 332 and toward the opposite end of the gripper 200, as shown in FIG. 27A. This configuration allows for the one or more optical sensors 334 to be positioned such that a finger is positioned within the field of view of the one or more optical sensors 334.

[0133] In FIG. 27B, at least a subset of the one or more optical sensors 334 (e.g., optical sensors 334-1 and 334-2) are mounted on the gimbal 332 facing the grip 200 (e.g., an end of the grip 200 that is not directly coupled to the gimbal 332). In some embodiments, the optical sensors 334-1 and 334-2 are positioned to avoid occlusion by a finger pad. In some embodiments, the one or more optical sensors 334 also include at least one optical sensor 334-3 mounted on an end of the grip 200 (facing away from the opposite end of the grip 200 that is coupled to the gimbal 332). This optical sensor may be used to detect the presence of a user's palm, which can provide additional information for determining the presence of the user.

[0134] 27A and 27B show three light sensors, in some embodiments, additional or fewer light sensors may be used (e.g., 1, 2, 4, 5, 6, 7, 8, 9, 10, or more light sensors). In addition, while Figures 27A and 27B are used to illustrate the placement of light sensors, other types of sensors (e.g., ultrasonic sensors, radar, sonar, pressure sensors, etc.) may be used instead of or in addition to the light sensors.

[0135] 28A and 28B show examples of finger loops for use with the optical sensor in the input device of FIG. 21A. FIG. 28A shows finger loops 338-1 and 338-2 (with open ends) and FIG. 28B shows finger loops 340-1 and 340-2 (with closed ends, in which case the finger loops may be referred to as finger cups). The finger loops 338 provide a secure connection between the user and the gripping implement. In some embodiments, the integrated circuit 328 is configured to distinguish the finger loop from a body part of the user (e.g., using a threshold window to distinguish signals from the body part of the user from signals from the finger loop). In some embodiments, the walls of the finger cup 340 facilitate distinguishing the body part of the user from the finger cup 340.

[0136] As described herein, sensors (e.g., capacitance sensors, optical sensors, etc.) are used to determine user presence, which can be used to reduce or eliminate unintentional movement of the robotic arm or surgical tool. It is also useful to determine user control even when user contact is not detected on the gripper 200. For example, when a user uses a finger loop or finger cup (shown in FIGS. 28A and 28B) to control the gripper 200 (e.g., during opening of the gripper 200), the user is still controlling the gripper 200, but as the finger is lifted off the finger pad, the user may not be in contact (or in close proximity) with the sensor in the finger pad.

[0137] D. Improved Determination of User Presence or Absence In some embodiments, user control is determined by combining information from one or more of the sensors described above (e.g., capacitance and / or optical sensors) with additional information (also referred to herein as secondary information). In some embodiments, movement of a robotic arm or a surgical tool is controlled based on information from one or more sensors (e.g., capacitance and / or optical sensors) as well as the secondary information, thereby improving operation of the medical robotic system.

[0138] In some embodiments, the additional information includes the gripper angle (e.g., the angle defined by the gripper's central axis and a link, such as link 202). In some configurations, the gripper is at a fully open angle when no external force is applied (e.g., no user input). Thus, a gripper angle other than or below a predetermined gripper angle threshold (e.g., a fully open angle) indicates that the user is in control.

[0139] In some embodiments, the additional information includes the velocity or movement of one or more joints supporting the input device. FIG. 29 illustrates a robot joint supporting the input device of FIG. 21A, according to some embodiments. For example, as shown in FIG. 29, a gimbal roll axis 342 extends through a roll joint that is rotatably coupled to the gripping tool 200 along the axis of the gripping tool 200. The roll joint is designed to have a passive zero velocity, e.g., there is no gravity compensation, so there cannot be any gravity compensation error. Thus, when there is no user input to the gripping tool 200 (e.g., when the user is not manipulating the gripping tool 200), it does not move (hence zero velocity). Thus, the rotational movement (or rotational velocity) of the roll joint along the gimbal roll axis 342 indicates that the user is in control (e.g., the user is rotating the gripping tool 200 around the gimbal roll axis 342). In another example, a rotational joint extending through a shoulder yaw axis 344 is coupled to the gripping tool 200. When there is no user input to the gripping apparatus 200 (e.g., when the user is not manipulating the gripping apparatus 200), the rotational joint extending through the shoulder yaw axis 344 does not move (and thus has zero velocity). Thus, the rotational movement (or rotational velocity) of the rotational joint extending through the shoulder yaw axis 344 can also indicate that the user is in control (e.g., the user is moving the gripping apparatus 200 about the shoulder yaw axis 344).

[0140] In some embodiments, the additional information includes a period or duration since the most recent detection of a user's presence. By providing the duration as an additional input, the system can advantageously account for when a user may be present but temporarily withdraws the user's hand from the grasper for a short period of time (e.g., .01-.04 seconds). In such a situation, the system may not want to stop the movement of the robotic arm or surgical tool despite the user's presence, as this may cause unnecessary interruptions. Thus, in some embodiments, the system can incorporate different time thresholds to help refine the determination of user presence and control. For example, a first time threshold can be provided as a debounce (e.g., .05-1 second or more), whereby if the user is not detected by the system within the first time threshold, movement can be reduced or stopped and / or monitored for unintended movement that exceeds a distance threshold. A second time threshold (e.g., 60 seconds, 90 seconds, 120 seconds, or more) can be provided as a timeout, whereby if the user is not detected by the system within the second time threshold, movement can be automatically stopped regardless of the distance of the unintended movement. It should be noted that the time periods provided for the first and second thresholds are exemplary and are not meant to be limiting.

[0141] In some embodiments, the additional information includes information indicative of changes in the movement of the robotic arm or surgical tool that are inconsistent with the user input. Such information can be obtained from encoders (e.g., position sensors) and / or from information derived based on changes in velocity, acceleration, jerk, etc. For example, bumps or impulses in the motion profile of the robotic arm or surgical tool, or information from mechatronic sensors such as current, back-EMF, torque, or force, can indicate a movement of the robotic arm or surgical tool that is different from the movement of the robotic arm or surgical tool caused by the user input. In another example, gravity drift is detected based on information from mechatronic sensors (e.g., errors in gravity compensation can result in a constant acceleration in a constant direction).

[0142] E. Unassisted and assisted remote control modes FIG. 30A illustrates a state diagram of a medical robotic system, according to some embodiments. In some embodiments, the medical robotic system includes a user input device (e.g., a tactile input device) and at least one robotic arm. The state diagram of FIG. 30A includes an unassisted teleoperation mode 381, an assisted teleoperation mode 382, ​​and a safe mode 383.

[0143] In some embodiments, the medical robotic system enters the unassisted teleoperation mode 381 as a result of an initiation procedure. In some embodiments, the medical robotic system enters the unassisted teleoperation mode 381 in response to detecting user control. In some embodiments, the medical robotic system enters or remains in the unassisted teleoperation mode 381 following a determination that user control has been detected. In some embodiments, while the medical robotic system is in the unassisted teleoperation mode 381, the medical robotic system moves the robotic arm and / or surgical instrument according to user input using a first set of parameters. In some embodiments, the first set of parameters includes forgoing any damping or using a first damping that is less than a second damping provided while the medical robotic system is in the assisted teleoperation mode 382.

[0144] In some embodiments, if drift (e.g., lack of user control) is detected in conjunction with movement of the robotic arm or surgical tool, the medical robotic system transitions to an assisted teleoperation mode 382. In some embodiments, the medical robotic system modifies control algorithms while in the assisted teleoperation mode 382 to reduce the possibility of dangerous unintended movements. In some embodiments, while in the assisted teleoperation mode 382, ​​the medical robotic system does one or more of the following: (i) changes motion scaling so that input device movements result in smaller tool tip movements; (ii) adjusts damping for the input device to increase the load required to generate input device movements; (iii) provides haptic feedback to maintain the current position of the input device; or (iv) saturates (e.g., caps) command velocities from the input device at values ​​lower than those used for normal operation. In some embodiments, switching between the unassisted teleoperation mode 381 and the assisted teleoperation mode 382 includes temporarily pausing teleoperation command transmission between the user input device and the robotic arm.

[0145] Advantages of operating in assisted teleoperation mode 382, ​​as opposed to disabling teleoperation, include that teleoperation is not unnecessarily interrupted. For example, situations may arise where user presence / control may not be detected, but entering safe mode is not warranted, thereby stopping teleoperation. In such scenarios, the system may, for example, dampen the movement of the input device instead of immediately stopping teleoperation of the system. In some situations, after disabling teleoperation (e.g., entering safe mode 383), the medical robotic system must perform a setup procedure to re-enter teleoperation. The setup procedure requires a certain amount of time, and this delay can be avoided by switching to assisted teleoperation mode 382 instead of stopping teleoperation.

[0146] In some embodiments, the medical robotic system transitions to safe mode 383 if a lack of user control is detected in conjunction with movement of the robotic arm or surgical tool. In some embodiments, while operating in assisted teleoperation mode 382, ​​the medical robotic system detects an unsafe condition and transitions to safe mode 383. Teleoperation is stopped or blocked between the input device and the robotic arm or surgical tool while the medical robotic system is in safe mode 383. In some embodiments, to avoid unnecessary disruptions, the medical robotic system monitors the movement of the input device and transitions to safe mode 383 if the position of the input device has changed by an unsafe distance compared to when the user was last detected to be in active control. In some embodiments, the medical robotic system transitions to safe mode 383 if the movement (e.g., velocity, acceleration, etc.) of the input device is determined to be unsafe (e.g., the velocity of the input device exceeds a velocity threshold or the acceleration of the user acceleration exceeds an acceleration threshold) while the user is not detected to be in active control.

[0147] Thus, following detection of drift and / or absence of user control, the medical robotic system may be configured to switch to a safe mode 383 (e.g., switch from master-side and / or slave-side teleoperation modes) until the user regains control. However, such suspension of teleoperation modes may be unnecessary and disruptive to smooth teleoperation control, especially when the drift detection is a false positive. Thus, rather than switching to safe mode 383 in some circumstances, it may be advantageous to switch to assisted teleoperation mode 382 following detection of drift and / or absence of user control.

[0148] Although the system diagram shown in FIG. 30A includes state transitions between unassisted teleoperation mode 381 and assisted teleoperation mode 382, ​​between assisted teleoperation mode 382 and safe mode 383, and between unassisted teleoperation mode 381 and safe mode 383, in some embodiments, one or more state transitions may be omitted or blocked. For example, in some embodiments, the state transition between assisted teleoperation mode 382 and safe mode 383 is disabled. In addition, while the system diagram shown in FIG. 30A has three states, in some embodiments, the system diagram includes additional or fewer states. For example, the system diagram may have only two states, such as unassisted teleoperation mode 381 and assisted mode 383. In some embodiments, the system diagram may have additional states not shown in FIG. 30A.

[0149] FIG. 30B illustrates transitions between different states of a medical system, according to some embodiments. In some embodiments, the medical system includes a user input device (e.g., a tactile input device) and at least one robotic arm. The states illustrated in FIG. 30B include an unassisted state 370 and an assisted state 371. In some embodiments, the unassisted state 370 corresponds to an unassisted teleoperation mode 381 illustrated in FIG. 30A. In some embodiments, the assisted state 371 corresponds to an assisted mode 383 illustrated in FIG. 30A. According to some embodiments, the unassisted state 370 includes sub-states such as an "undriven" state 372, which indicates that the user is not driving the input device, and a driven state 373, which indicates that the user is actively driving the input device.

[0150] In some embodiments, the medical system transitions (374) from the "undriven" state 372 to the driven state 373 in response to a determination that the user input device (e.g., a gripper or grasper) matches. In some embodiments, the medical system unlocks the haptic input device in conjunction with the transition from the "undriven" state 372 to the driven state 373. In some embodiments, the medical system resets a timer (and / or restarts a timer) in conjunction with the transition to the driven state 373, e.g., a timer used to determine if no user presence is detected. In some embodiments, the medical system detects a user presence while in the driven state 373 (e.g., any sensor detects a user presence) and, in response, remains in the driven state 373 and resets a timer, e.g., to determine if the medical system should transition to the assist state 371.

[0151] In some embodiments, the medical system transitions (375) from the drive state 373 to the assist state 371 following a determination that the robotic arm or surgical tool has drifted. For example, the robotic arm or surgical tool has drifted beyond a distance threshold (e.g., 5, 10, 15, 20, 25, 30, 40, 50, or 60 mm, or within an interval between any two of the aforementioned values). In some embodiments, the medical system identifies drift according to detecting movement of a user input device without detecting the presence of a user. In some embodiments, the medical system identifies drift according to detecting unexpected and / or unsafe movement of a user input device. In some embodiments, detecting drift includes detecting unsafe movement, e.g., the gripper angle is greater than a first threshold angle (e.g., 30°, 40°, 45°, etc.) and the shoulder yaw axis 344 has a rotational speed greater than a first threshold rate (e.g., less than 0.1 rad / s, 0.2 rad / s, 0.3 rad / s, 0.4 rad / s, 0.5 rad / s, 0.6 rad / s, 0.7 rad / s, 0.8 rad / s, 0.9 rad / s, 1 rad / s, 2 rad / s, 3 rad / s, 4 rad / s, 5 rad / s, or 10 rad / s, etc.). In some embodiments, the medical system modifies one or more parameters used to operate the input device in conjunction with transitioning to the assisted state.

[0152] In some embodiments, the medical system transitions (376) from the drive state 373 to the assisted state 371 pursuant to a determination that no user presence is detected, e.g., none of the sensors detect the user presence, or fewer than a certain number of sensors (e.g., two, three, or four) detect the user presence. In some embodiments, the medical system transitions from the drive state 373 to the assisted state 371 pursuant to a determination that a timer has elapsed a threshold duration (e.g., 5, 10, 15, 20, 25, 30, 40, 50, or 60 mm, or within an interval between any two of the aforementioned values). In some embodiments, the medical system modifies one or more parameters for operation of the input device in conjunction with the transition to the assisted state. For example, the medical system adjusts damping, stiffness, motion scaling, and / or braking control pursuant to the transition to the assisted state.

[0153] In some embodiments, the drive state 373 includes a "user detected" substate 380 and a "user not detected" substate 384. In some embodiments, in response to a detected lack of user presence, the medical system transitions from the "user detected" substate 380 to the "user not detected" substate 384, e.g., before transitioning to the assisted state 371. In some embodiments, the medical system transitions from the "user not detected" substate 384 to the assisted state 371 according to a preset time (e.g., 1, 2, or 5 seconds) that elapses while in the "user not detected" substate. In some embodiments, the medical system transitions from the "user not detected" substate 384 to the "user detected" substate 380 according to at least one sensor detecting the presence of a user. Thus, in some embodiments, a temporary absence of user presence may not trigger a transition to the assisted state 371.

[0154] In some embodiments, secondary information (with or without sensor information) is used to determine the criteria for state transitions. In some embodiments, a timer serves as secondary information (e.g., in addition to sensing via capacitance) to help determine when the system can transition appropriately. For example, in some embodiments, when the system is in the "no user detected" substate 384 for more than a predetermined duration (e.g., 500 milliseconds), the system requires certain criteria based on contact sensing and / or secondary information to transition back to the "user detected" substate 380. These criteria may be different from the criteria for re-establishing "user detection" if the user has not been detected for less than that duration. For example, the system may require only one sensor to indicate presence if the system is in the "no user detected" substate 384 for less than 500 milliseconds, and at least two sensors to indicate presence if the system is in the "no user detected" substate 384 for more than 500 milliseconds. As another example, the system could use secondary information such as G6 motion to indicate presence if the system has been in the "user not detected" substate 384 for less than 500 ms, but require a contact sensor to indicate presence if the system has been in the "user not detected" state 384 for more than 500 ms. Providing a timer as secondary information can advantageously help determine which criteria are sufficient to transition.

[0155] In some embodiments, the medical system resets the timer (and / or restarts the timer) in conjunction with the transition to the assisted state 371. In some embodiments, the timer serves as secondary information (e.g., in addition to sensing via capacitance) to help determine when the system can properly return to the unassisted state 370 (e.g., the driving state 373). For example, in some embodiments, when the system is in the assisted state 371, it takes some time beyond the first contact between the user and the user input device and associated capacitance sensor before the system returns to the unassisted state 370. This is because the system may take into account the first contact by the user with the gripping pad before the actual teleoperated driving. In this way, the system may not immediately transition to the driving state 373 upon the user's first contact with the gripping implement. Providing a timer as secondary information in the assisted state 371 can advantageously help determine whether the user's contact with the gripping implement has been made long enough to represent actual driving. In some embodiments, the medical system resets the timer (and / or restarts the timer) in response to detecting the presence of a user such that, for example, a period of time (e.g., 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, or 1 second) may elapse after the presence of a user is detected before allowing the user to drive the robotic arm in unassisted mode.

[0156] In some embodiments, when the system is in the assisted state 371 for more than a predetermined duration (e.g., 1 second), the system requires certain criteria based on contact sensing and / or secondary information to transition back to the unassisted state 370 (e.g., driven state 373). These criteria may be different from the criteria for re-establishing "user detection" if the user has not been detected for less than a predetermined duration. For example, the system may require only one sensor to indicate presence if the system has been in the assisted state 371 for less than 1 second, and at least two sensors to indicate presence if the system has been in the assisted state 371 for more than 1 second. In some embodiments, the system uses secondary information such as the rotational motion (or rotational rate) of a revolute joint extending through the shoulder yaw axis 344 to indicate user presence if the system has been in the assisted state 371 for less than 1 second, but requires a contact sensor to indicate user presence if the system has been in the assisted state 371 for more than 1 second. Providing a timer as secondary information in the assisted state 371 can advantageously help determine which criteria are sufficient to transition back to the unassisted state 370.

[0157] In some embodiments, the medical system transitions (377) from the assisted state 371 to the driven state 373 pursuant to a determination that a preset time has elapsed while in the assisted state 371. In some embodiments, the preset time corresponds to a settling time of the user input device (e.g., the time required for the user input device to transition from uncontrolled to controlled movement). As used herein, the settling time of the user input device is the time associated with the user input device transitioning from uncontrolled to controlled movement (e.g., from an unsteady state to a steady state). In some embodiments, the settling time is based on the configuration of the user input device (e.g., current position, velocity, angle, damping, etc.). In some embodiments, the settling time is stored in a lookup table in a memory of the medical system. In some embodiments, the lookup table specifies the settling time of the corresponding user input device configuration. In some embodiments, the settling time corresponds to a worst case scenario for the user input device (e.g., the time to recover from a maximum velocity drift). In some embodiments, the settling time is determined based on characteristics of the detected drift (e.g., velocity, acceleration, and / or angle).

[0158] In some embodiments, the medical system transitions (378) from the assisted state 371 to the driven state 373 following a determination that a certain number of sensors (e.g., capacitance sensors or optical sensors), such as two or more sensors, detect the presence of a user. In some embodiments, the medical system transitions from the assisted state 371 to the driven state 373 following a determination that any sensor detects the presence of a user and / or a determination that the movement of the user input device is within an acceptable range. In some embodiments, a timer serves as secondary information (e.g., in addition to sensing via capacitance) to help the system determine when to properly transition back to the driven state 373. For example, in some embodiments, when the system is in the assisted state 371, a certain amount of time, even beyond the first contact between the user and the gripper and associated capacitance sensor, occurs before the system transitions back to the driven state 373. In some embodiments, the medical system resets the timer (and / or restarts the timer) in conjunction with the transition from the assisted state 371 to the driven state 373.

[0159] In some embodiments, the medical system transitions (379) from the assist state 371 to the drive state 373 according to the position and movement of the user input device being within an acceptable range. In some embodiments, the position and movement of the user input device being within an acceptable range includes the user input device having a velocity below a pre-defined velocity threshold and / or the user input device having an acceleration below a pre-defined acceleration threshold. For example, the gripper angle is closed (e.g., below a second threshold angle of 15°, 10°, 5°, etc.) or the shoulder yaw axis 344 has a rotational velocity of a threshold velocity (e.g., below 0.1 rad / s, 0.2 rad / s, 0.3 rad / s, 0.4 rad / s, 0.5 rad / s, 0.6 rad / s, 0.7 rad / s, 0.8 rad / s, 0.9 rad / s, 1 rad / s, 2 rad / s, 3 rad / s, 4 rad / s, 5 rad / s, or 10 rad / s, etc.).

[0160] In some embodiments, the secondary information includes information indicative of the end effector pose or a change in the end effector pose. For example, in some embodiments, the secondary information includes (i) information indicative of the end effector pose or a change in the end effector pose, and (ii) a rotational motion (or rotational velocity) of a rotational joint extending through the shoulder yaw axis 344, such that user control is determined based on the change in the end effector pose and the rotational motion even when user contact is not detected (e.g., using a capacitive sensor). In some embodiments, the change in the end effector pose is used to select a criterion for a state transition. For example, following a determination that an end effector movement associated with a user control has been detected (e.g., a continuous or semi-continuous movement of the end effector, a movement of the end effector at a velocity below a threshold velocity, a movement of the end effector including a change in the direction of the end effector), a first criterion (e.g., a less stringent criterion such as requiring detection of contact by a single sensor) is used for the state transition. Upon determination of end effector movement not associated with user control (e.g., gravity drift or bump), a second criterion (e.g., more stringent criteria, such as requiring detection of contact by multiple sensors) is used for state transitions.

[0161] In addition, although the system diagram shown in Figure 30B has two states and four substates, in some embodiments the system includes additional or fewer states (and / or additional or fewer substates). For example, the system may have three states, such as an unassisted state, an assisted state, and a safe state.

[0162] 31A and 31B are flow charts illustrating a method 400 for operating a medical system including a user input device for controlling a medical instrument, according to some embodiments. The method 400 is executed in a computing system (e.g., the medical system 36 or a computer in communication with the medical system 36) having one or more processors (e.g., the processor 280) and a memory (e.g., the memory 282). In some embodiments, the memory (e.g., the memory 282) stores instructions for execution by the one or more processors (e.g., the processor 280). In some embodiments, the method 400 is executed by a medical system including an input device (e.g., the controller 182) for controlling (e.g., moving, functioning) the operation (e.g., teleoperation) of a robotic arm. In some embodiments, the robotic arm is coupled to the medical instrument. In some embodiments, movement of the user input device causes a corresponding movement of the robotic arm and / or the medical instrument. In some embodiments, the input device includes a gripping implement (eg, gripping implement 200) that includes one or more electrodes (eg, electrodes 311-1 and 311-2) for determining the presence of a user at the input device.

[0163] The computing system operates the user input device in an unassisted mode (402), including applying a first set of parameters to the user input device. For example, the computing system operates the user input device in an unassisted teleoperation mode 381 (e.g., unassisted state 370). In some embodiments, the unassisted mode is a user-driven mode. For example, a user of the user input device manipulates the user input device to control movement of the robotic arm (e.g., robotic arm 12) and / or a medical instrument mounted on the robotic arm. In some embodiments, the first set of parameters includes damping parameters, motion scaling parameters, stiffness parameters, and brake control parameters.

[0164] The computing system detects (404) drift of the user input device while operating in the unassisted mode. In some embodiments, the computing system includes or is coupled to various sensors (e.g., capacitance sensors, time-of-flight sensors, etc.) that can be used to detect drift of the user input device (e.g., a tactile input device). In some embodiments, detecting the drift includes detecting unintentional user-driven movement of the user input device. In some embodiments, detecting the drift includes detecting a lack of user presence. In some embodiments, detecting the drift includes detecting a lack of user presence while also detecting movement of the user input device. In some embodiments, the lack of user presence is detected based on the capacitance of one or more electrodes (e.g., in FIGS. 24A-24E) of the user input device (e.g., via the integrated circuit 328). In some embodiments, determining the user presence is based on a comparison of a duration that the sensor does not detect the user presence to a preset time threshold. In some embodiments, the sensor is a capacitance-based sensor or a light-based sensor.

[0165] In some embodiments, detecting drift of the user input device includes detecting motion instability (e.g., inefficiency) in the user input device, hi some embodiments, the motion instability is indicative of an increase in the total energy of the user input device.

[0166] In some embodiments, the electrode includes a second electrode interdigitated (e.g., coiled) with the first electrode. In some embodiments, the electrode includes a first electrode disposed adjacent to the second electrode. In some embodiments, the first electrode and the second electrode extend in substantially the same direction. In some embodiments, the presence of a user is determined based on mutual capacitance (e.g., mutual capacitance above a threshold indicates the presence of a user). In some embodiments, the user input device includes an integrated circuit for measuring mutual capacitance between two or more electrodes, and the presence of a user is determined based on the mutual capacitance. In some embodiments, the user input device includes a plurality of finger pads (e.g., finger pad 212), and the presence of a user is determined based on the presence of a finger on one or more of the plurality of finger pads.

[0167] In some embodiments, the computing system receives sensor information from a sensor coupled to (e.g., embedded in) the user input device regarding a user's presence at the user input device, and detects the drift based on the sensor information, hi some embodiments, the sensor is one of a capacitance-based sensor or an optical-based sensor.

[0168] In some embodiments, detecting drift of the user input device includes tracking (406) movement of the user input device and determining that the tracked movement corresponds to unintentional movement. In some embodiments, the computing system includes a sensor (e.g., a motion sensor), where the sensor tracks movement of the user input device. In some embodiments, the determination that the tracked movement corresponds to unintentional movement is based on historical data (e.g., historical movement data associated with a medical procedure being performed and / or an operator of the user input device). In some embodiments, the determination that the tracked movement corresponds to unintentional movement is based on pattern recognition and / or machine learning.

[0169] In some embodiments, the computing system obtains secondary information associated with the user input device, and the detection of drift is based on the secondary information. In some embodiments, the input device includes a joint, and the secondary information includes a position and / or velocity of the joint. In some embodiments, the joint is a passively stationary joint of the input device. In some embodiments, the joint is a roll joint of the input device. In some embodiments, the secondary information includes a time threshold for comparison with a duration over which the sensor detects a lack of user presence. In some embodiments, the secondary information includes a configuration of the user input device, such as an angle or velocity, a motion of a passively stationary joint, and / or a time of last movement or last detected user presence. In some embodiments, the secondary information includes information indicative of a change in the configuration (e.g., position and / or orientation) of the input device at a first time and a second time subsequent to the first time (e.g., the current time).

[0170] In some embodiments, detecting the drift includes comparing the change in the configuration of the input device to a configuration change threshold (e.g., a predetermined distance and / or angle difference). In some embodiments, the information indicative of the change in the configuration of the input device is obtained by determining a configuration (e.g., a position and / or orientation) of the input device at a first time, determining a configuration of the input device at a second time, and determining the change in the configuration of the input device based on the configuration of the input device at the first time and the configuration of the input device at the second time.

[0171] In some embodiments, the secondary information includes information indicative of a change in a configuration (e.g., position and / or orientation) of the medical instrument at a first time and a second time subsequent to the first time. In some embodiments, the information indicative of the change in the configuration of the medical instrument is obtained by determining the configuration (e.g., position and / or orientation) of the medical instrument at the first time, determining the configuration of the medical instrument at the second time, and determining the change in the configuration of the medical instrument based on the configuration of the medical instrument at the first time and the configuration of the medical instrument at the second time.

[0172] The computing system switches (408) operation of the user input device to an assisted mode, including applying a second set of parameters different from the first set of parameters, pursuant to detection of drift (e.g., transition 375 described above with respect to FIG. 30B). For example, the computing system switches operation to an assisted teleoperation mode 382 (e.g., assisted state 371). In some embodiments, the assisted mode is a stabilization mode (e.g., the computing system limits movement of the user input device to accelerate stabilization of the user input device). In some embodiments, the computing system continues to operate in an unassisted mode pursuant to the absence of detected drift.

[0173] In some embodiments, the second set of parameters includes modifications to one or more parameters of the first set of parameters, the modifications to the one or more parameters including adjustment of one or more of a damping of the user input device, a stiffness of the user input device, a motion scaling between the user input device and the medical instrument, or a brake control of a joint of the user input device (410). In some embodiments, adjusting the motion scaling includes reducing motion scaling between the input device and the robot arm. For example, the robot arm may move a first distance in response to movement of the input device by a particular input distance before the adjustment, and the robot arm may move a second distance less than the first distance in response to movement of the input device by a particular input distance after the adjustment. In some embodiments, adjusting the motion scaling includes temporarily disabling motion updates from the user input device to the robot arm. In some embodiments, the modifications to the one or more parameters include providing haptic feedback to the user (e.g., via the user input device) to maintain a position of the user input device. In some embodiments, adjusting the damping and / or stiffness of the user input device includes increasing kinetic energy dissipation in the user input device. In some embodiments, adjusting the braking control of the joint comprises increasing friction within the joint.

[0174] In some embodiments, switching operation of the user input device to the assisted mode includes one or more of: performing a position control operation or performing a velocity control operation (412). In some embodiments, performing the position control operation includes moving the user input device to a known good position (e.g., a last known good position). In some embodiments, moving the user input device to the known good position also moves the robot arm to the known good position. In some embodiments, performing the velocity control operation includes reducing a velocity of the robot arm. In some embodiments, the velocity control operation includes setting a velocity of the user input device to zero. In some embodiments, setting the velocity of the user input device to zero also sets the robot arm velocity to zero. In some embodiments, switching operation of the user input device to the assisted mode includes temporarily ceasing teleoperation command transmission between the user input device and the robot arm.

[0175] In some embodiments, the second set of parameters includes modifications to one or more parameters of the first set of parameters, and the modifications to the one or more parameters include applying weak feedback control 414. As used herein, weak feedback control is feedback control that can be overcome by a user (e.g., by applying more force to a user input device).

[0176] In some embodiments, the modification to the one or more parameters includes modifying 416 one or more parameters of the first set of parameters within a predetermined limit, for example, setting a damping parameter to a maximum damping parameter at a predetermined limit, in some configurations, the maximum damping parameter is selected such that the user can continue to move the user input device despite damping being applied to the user input device at the maximum damping parameter.

[0177] In some embodiments, the computing system continues to operate the user input device in the assisted mode (418) pursuant to a determination that the user input device is not stable. For example, after a preset amount of time, the computing system determines that the movement and / or position of the user input device is still not stable and continues to operate in the assisted mode. In some embodiments, pursuant to a determination that the user input device is not stable after a preset amount of time, the computing system switches operation from the assisted mode to a safe mode (e.g., safe mode 383).

[0178] The computing system determines that the user input device is stable while operating in the assisted mode (420). In some embodiments, determining that the user input device is stable includes tracking a time that the user input device is in the assisted mode and determining that the user input device is stable according to determining that the time meets a predetermined criterion.

[0179] In some embodiments, determining that the user input device is stable includes determining that the user input device has been operating in the assisted mode for a predetermined time (422). In some embodiments, determining that the user input device is stable includes determining that the user input device has been operating in the assisted mode for a predetermined time and that a user presence has been detected at the user input device.

[0180] In some embodiments, the predetermined time corresponds to a settling time of the user input device (424). In some embodiments, the settling time depends on the configuration of the user input device (e.g., position, velocity, and acceleration). In some embodiments, the configuration of the user input device includes the configuration of the user input device while in the assisted mode (e.g., amount of damping, stiffness, damping, etc.). In some embodiments, the settling time is predetermined and stored in a memory of the computing device. In some embodiments, the settling time is stored in a lookup table that specifies respective settling times for corresponding user input device configurations. In some embodiments, the settling time corresponds to a worst case scenario for the user input device (e.g., a scenario in which the user input device is moving at maximum speed). In some embodiments, the settling time is based on one or more characteristics of the detected drift (e.g., velocity or acceleration of the drift).

[0181] In some embodiments, determining that the user input device is stable includes determining (426) that a user's presence at the user input device meets one or more predetermined criteria.

[0182] In some embodiments, determining that the user input device is stable includes determining (428) that movement of the user input device meets a predetermined criterion.

[0183] The computing system switches operation of the user input device to an unassisted mode (e.g., transitions 377, 378, or 379 described above with respect to FIG. 30B) (430) in accordance with the determination that the user input device is stable. In some embodiments, the computing system switches operation of the user input device to an unassisted mode based on the sensor information and secondary information. In some embodiments, the computing system uses pattern recognition and / or machine learning to determine that the user input device is stable. For example, the computing system compares the movement and / or position of the user input device to historical movement / position data to determine that the user input device is stable. In some embodiments, the second information includes information indicating a duration that the medical system has been in the assisted mode. For example, the medical system may wait a certain period of time before switching operation of the user input device back to the unassisted mode (e.g., even if a user's presence is detected before the certain period of time has elapsed). In some embodiments, switching operation of the user input device to an unassisted mode includes temporarily ceasing teleoperation command transmission between the user input device and the robotic arm.

[0184] In some embodiments, while operating the user input device in the assisted mode, the computing system tracks movements of the user input device and switches operation of the user input device from the assisted mode to the unassisted mode pursuant to a determination that the movements of the user input device satisfy a predetermined criterion. In some embodiments, the predetermined criterion includes a criterion that a velocity of the user input device is below a predetermined threshold. In some embodiments, the predetermined criterion includes both a time threshold and one or more motion criteria for the user input device.

[0185] In some embodiments, while operating the user input device in the assisted mode, the computing system monitors a user's presence at the user input device and switches operation of the user input device from the assisted mode to a non-assisted mode pursuant to a determination that the user's presence satisfies a predetermined criterion, hi some embodiments, the predetermined criterion includes that the user's presence is detected for at least a threshold time.

[0186] 3. Implementation Systems and Terminology. 32 is a schematic diagram showing electronic components of a medical system including an input device. The robotic medical system includes one or more processors 280 in communication with a memory 282 (e.g., computer memory devices such as random access memory, read only memory, static random access memory, and non-volatile memory, as well as other storage devices such as hard drives, optical disks, magnetic tape recordings, or any combination thereof) computer-readable storage medium that stores instructions for performing any of the methods described herein (e.g., the operations described with respect to FIGS. 30A-30B and 31A-31B). The one or more processors 280 also communicate with an input / output controller 284 (via a system bus or any suitable electrical circuitry). The input / output controller 284 receives user input from an input device 286 (e.g., the controller 182, or the gripper 200 in particular), receives sensor data from one or more sensors (e.g., sensors 287, such as capacitance sensors, optical sensors, gripper angle sensors as described herein, coupled to the input device 286), and relays the sensor data to the one or more processors 280. In some embodiments, the sensors 287 include integrated circuits 328. In some embodiments, the integrated circuits 328 are integrated with the one or more processors 280 (or vice versa). The input / output controller 284 also receives instructions and / or data from the one or more processors 280 and relays the instructions and / or data to one or more actuators, such as motors 292-1 and 292-2. In some embodiments, the input / output controller 284 is coupled to one or more actuator controllers 290 and provides instructions and / or data to at least a subset of the one or more actuator controllers 290, which in turn provide control signals to selected actuators 292. In some embodiments, one or more actuator controllers 290 are integrated with the input / output controller 284, and the input / output controller 284 provides control signals directly to one or more actuators 292 (without a separate actuator controller).FIG. 32 shows that there is one actuator controller 290 (e.g., one actuator controller for the entire medical platform), however, in some embodiments, additional actuator controllers (e.g., one actuator controller for each actuator, etc.) may be used.

[0187] It should be noted that, as used herein, the terms "couple," "coupled," "coupled," or other variations of the word coupled, may indicate either an indirect connection or a direct connection. For example, when a first component is "coupled" to a second component, the first component may be either indirectly connected to the second component through another component, or directly connected to the second component.

[0188] The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example, and not limitation, such media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disk read-only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be noted that computer-readable media may be tangible and non-transitory. As used herein, the term "code" may refer to software, instructions, code, or data that is executable by a computing device or processor.

[0189] The methods disclosed herein include one or more steps or actions for achieving the described method. The method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is required for the proper operation of the described method, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0190] As used herein, the term "plurality" refers to two or more. For example, a plurality of components refers to two or more components. The term "determining" covers a wide variety of acts, and thus "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., consulting a table, database, or another data structure), ascertaining, and the like. Also, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Also, "determining" can include resolving, selecting, electing, establishing, and the like.

[0191] The phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" refers to both "based only on" and "based at least on."

[0192] The foregoing description of the disclosed implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these implementations will be readily apparent to those skilled in the art, and the general principles set forth herein may be applied to other implementations without departing from the scope of the present invention. For example, those skilled in the art will recognize that many corresponding alternatives, equivalent structural details, such as similar manners of fastening, mounting, coupling, or engaging tool components, equivalent mechanisms for producing specific actuation motions, and equivalent mechanisms for delivering electrical energy, may be employed. Thus, the present invention is not intended to be limited to the implementations shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0193] 4. Examples of subject technology clauses. The following clauses describe several embodiments or implementations. Article 1. A health care system, 1. A user input device for controlling a medical instrument, the user input device comprising: an unassisted mode having a first set of parameters for operation of the user input device; a user input device operable in an assisted mode having a second set of parameters for operation of the user input device, the second set of parameters being different from the first set of parameters; a sensor coupled to the user input device for generating information related to the user input device; A processor; and a memory for storing instructions for execution by the processor, the stored instructions comprising: Identifying drift of the user input device based on information from the sensor while operating the user input device in an unassisted mode; The medical system includes instructions for switching operation of a user input device from an unassisted mode to an assisted mode in response to identifying the drift. Clause 2. The stored instructions, while operating the user input device in the assistance mode, Tracking the time that the user input device is in an assistive mode; 13. The medical system of claim 1, further comprising instructions for switching operation of the user input device from an assisted mode to an unassisted mode in accordance with a determination that the time satisfies a predetermined criterion. Clause 3. The medical system of clause 2, wherein the predetermined criterion is a time threshold corresponding to a settling time of a user input device. Clause 4. The stored instructions, while operating the user input device in the assistance mode, Monitoring for user presence at a user input device; A medical system as described in any one of clauses 1 to 3, further comprising instructions for switching operation of the user input device from an assisted mode to an unassisted mode in accordance with a determination that the presence of a user satisfies one or more predetermined criteria. Clause 5. The stored instructions, while operating the user input device in the assistance mode, Tracking the movement of a user input device; 5. The medical system of any one of clauses 1 to 4, further comprising instructions for switching operation of the user input device from an assisted mode to an unassisted mode in accordance with a determination that movement of the user input device satisfies a predetermined criterion. Clause 6. A medical system described in any one of clauses 1 to 5, wherein the sensor comprises a capacitance-based sensor or an optical-based sensor. Clause 7. The medical system of any one of clauses 1 to 6, wherein the sensor tracks movement of a user input device and identifying drift of the user input device includes determining that the tracked movement corresponds to unintentional movement. Clause 8. The medical system of any one of clauses 1 to 7, wherein identifying drift of the user input device comprises detecting movement instability in the user input device. Clause 9. The second set of parameters includes modifications to one or more parameters of the first set of parameters, the modifications being: Adjusting the attenuation of a user input device; Adjusting the stiffness of a user input device; Coordinating motion scaling between the user input device and the medical instrument; Adjusting brake control of a joint of a user input device; Performing a position control operation; or The medical system of any one of clauses 1 to 8, further comprising one or more of the following: Clause 10. A medical system as described in any one of clauses 1 to 9, wherein the second parameter set includes modifications to one or more parameters of the first parameter set, and the modifications to the one or more parameters include applying weak feedback control. Clause 11. The medical system of clause 10, wherein applying weak feedback control comprises modifying one or more parameters of the first parameter set within predetermined limits. Clause 12. A medical system as described in any one of clauses 1 to 11, further comprising a robotic arm coupled to the medical instrument for moving the medical instrument according to one or more instructions from the processor. Clause 13. A method for operating a medical system including a user input device for controlling a medical instrument, comprising: operating the user input device in an unassisted mode, the unassisted mode including applying a first set of parameters to the user input device; Detecting drift of a user input device while operating in an unassisted mode; switching operation of the user input device to an assisted mode in response to detecting the drift, the assisted mode including applying a second set of parameters different from the first set of parameters; determining that a user input device is stable while operating in an assisted mode; and switching operation of the user input device to an unassisted mode according to the determination. Clause 14. The method of clause 13, wherein determining that the user input device is stable includes determining that the user input device has been operating in an assisted mode for a predetermined period of time. Clause 15. The method of clause 14, wherein the predetermined time corresponds to a settling time of a user input device. Clause 16. The method of any one of clauses 13 to 15, wherein determining that the user input device is stable comprises determining that the presence of a user at the user input device satisfies one or more predetermined criteria. Clause 17. The method of any one of clauses 13 to 16, wherein determining that the user input device is stable comprises determining that movement of the user input device satisfies a predetermined criterion. Clause 18. Detecting drift of a user input device Tracking movement of a user input device; and determining that the tracked movement corresponds to unintentional movement. Clause 19. The second set of parameters includes modifications to one or more parameters of the first set of parameters, the modifications to the one or more parameters being: Adjusting the attenuation of a user input device; Adjusting the stiffness of a user input device; Adjusting the motion scaling between the user input device and the medical instrument; or and adjusting a brake control of a joint of the user input device. Clause 20. Switching the operation of a user input device to an assistive mode Performing a position control operation; or 20. The method of any one of clauses 13 to 19, comprising one or more of: performing a speed control operation. Clause 21. The method of any one of clauses 13 to 20, wherein the second set of parameters includes modifications to one or more parameters of the first set of parameters, and the modifications to the one or more parameters include applying weak feedback control. Clause 22. The method of clause 21, wherein applying weak feedback control comprises modifying one or more parameters of the first parameter set within predetermined limits. Clause 23. A method comprising: 23. The method of any one of clauses 13 to 22, further comprising continuing to operate the user input device in the assisted mode in accordance with determining that the user input device is not stable while operating in the assisted mode. Clause 24. A method for operating a medical system including a user input device for controlling a medical instrument, comprising: operating the user input device in an unassisted mode, the unassisted mode including applying a first set of parameters to the user input device; identifying drift of a user input device while operating in an unassisted mode; and switching operation of the user input device to an assisted mode in accordance with detecting the drift, the assisted mode including applying a second set of parameters different from the first set of parameters.

[0194] [Embodiment] (1) A health care system, comprising: 1. A user input device for controlling a medical instrument, the user input device comprising: an unassisted mode having a first set of parameters for operation of the user input device; a user input device operable in an assisted mode having a second set of parameters for operation of the user input device that is different from the first set of parameters; a sensor coupled to the user input device for generating information related to the user input device; A processor; and a memory for storing instructions for execution by the processor, the stored instructions comprising: identifying drift of the user input device based on the information from the sensor while operating the user input device in the unassisted mode; and instructions for switching operation of the user input device from the unassisted mode to the assisted mode in response to identifying the drift. (2) while operating the user input device in the assistance mode, the stored instructions Tracking the time that the user input device is in the assistance mode; 2. The medical system of claim 1, further comprising instructions for switching operation of the user input device from the assisted mode to the unassisted mode in accordance with a determination that the time meets a predetermined criterion. (3) The medical system of embodiment 2, wherein the predetermined criterion is a time threshold corresponding to a settling time of the user input device. (4) while operating the user input device in the assistance mode, the stored instructions monitoring for a user presence at the user input device; The medical system of embodiment 1, further comprising instructions for switching operation of the user input device from the assisted mode to the unassisted mode in accordance with a determination that the presence of the user meets one or more predetermined criteria. (5) while operating the user input device in the assistance mode, the stored instructions Tracking the movement of the user input device; 2. The medical system of claim 1, further comprising instructions for switching operation of the user input device from the assisted mode to the unassisted mode in accordance with a determination that the movement of the user input device meets a predetermined criterion.

[0195] (6) The medical system of embodiment 1, wherein the sensor includes a capacitance-based sensor or an optical-based sensor. (7) The medical system of embodiment 1, wherein the sensor tracks movement of the user input device, and identifying the drift of the user input device includes determining that the tracked movement corresponds to unintentional movement. (8) The medical system of embodiment 1, wherein identifying the drift of the user input device includes detecting movement instability in the user input device. (9) The second parameter set includes modifications to one or more parameters of the first parameter set, the modifications being: adjusting the attenuation of the user input device; adjusting the stiffness of the user input device; adjusting motion scaling between the user input device and the medical instrument; adjusting a brake control of a joint of the user input device; Performing a position control operation; or The medical system of embodiment 1, further comprising one or more of the following: performing a speed control operation. (10) The medical system of embodiment 1, wherein the second parameter set includes modifications to one or more parameters of the first parameter set, and the modifications to the one or more parameters include applying weak feedback control.

[0196] (11) The medical system of embodiment 10, wherein applying the weak feedback control includes modifying the one or more parameters of the first parameter set within predetermined limits. (12) The medical system of embodiment 1, further comprising a robotic arm coupled to the medical instrument for moving the medical instrument according to one or more instructions from the processor. (13) A method for operating a medical system including a user input device for controlling a medical instrument, comprising: operating the user input device in an unassisted mode, the unassisted mode including applying a first set of parameters to the user input device; detecting drift of the user input device while operating in the unassisted mode; switching operation of the user input device to an assisted mode in response to detecting the drift, the assisted mode including applying a second parameter set different from the first parameter set; determining that the user input device is stable while operating in the assisted mode; and switching the operation of the user input device to the unassisted mode in accordance with the determination. (14) The method of claim 13, wherein determining that the user input device is stable includes determining that the user input device has been operating in the assisted mode for a predetermined time. (15) The method of claim 14, wherein the predetermined time corresponds to a settling time of the user input device.

[0197] (16) The method of embodiment 13, wherein determining that the user input device is stable includes determining that a user's presence at the user input device satisfies one or more predetermined criteria. (17) The method of embodiment 13, wherein determining that the user input device is stable includes determining that movement of the user input device meets a predetermined criterion. (18) Detecting the drift of the user input device includes: tracking movement of the user input device; determining that the tracked movement corresponds to unintentional movement. (19) The second parameter set includes modifications to one or more parameters of the first parameter set, the modifications to the one or more parameters being: adjusting the attenuation of the user input device; adjusting the stiffness of the user input device; adjusting motion scaling between the user input device and the medical instrument; or 14. The method of claim 13, further comprising one or more of: adjusting a brake control of a joint of the user input device. (20) Switching the operation of the user input device to the assistance mode includes: Performing a position control operation; or 14. The method of embodiment 13, comprising one or more of: performing a speed control operation.

[0198] (21) The method of embodiment 13, wherein the second parameter set includes modifications to one or more parameters of the first parameter set, and the modifications to the one or more parameters include applying weak feedback control. (22) The method of embodiment 21, wherein applying the weak feedback control includes modifying the one or more parameters of the first parameter set within predetermined limits. (23) The method of claim 13, further comprising continuing to operate the user input device in the assisted mode in response to determining that the user input device is not stable while operating in the assisted mode. (24) A method for operating a medical system including a user input device for controlling a medical instrument, comprising: operating the user input device in an unassisted mode, the unassisted mode including applying a first set of parameters to the user input device; identifying drift of the user input device while operating in the unassisted mode; and in response to detecting the drift, switching operation of the user input device to an assisted mode that includes applying a second set of parameters different from the first set of parameters.

Claims

1. 1. A health care system comprising:

1. A user input device for controlling a medical instrument, the user input device comprising: an unassisted mode having a first set of parameters for operation of the user input device; a user input device operable in an assisted mode having a second set of parameters for operation of the user input device that is different from the first set of parameters; a sensor coupled to the user input device for generating information related to the user input device; A processor; and a memory for storing instructions for execution by the processor, the stored instructions comprising: identifying drift of the user input device based on the information from the sensor while operating the user input device in the unassisted mode; and instructions for switching operation of the user input device from the unassisted mode to the assisted mode in response to identifying the drift.

2. The stored instructions, while operating the user input device in the assistance mode, Tracking the time that the user input device is in the assistance mode; The medical system of claim 1 , further comprising instructions for switching operation of the user input device from the assisted mode to the unassisted mode in accordance with a determination that the time meets a predetermined criterion.

3. The medical system of claim 2 , wherein the predetermined criterion is a time threshold corresponding to a settling time of the user input device.

4. The stored instructions, while operating the user input device in the assistance mode, monitoring for a user presence at the user input device; The medical system of claim 1 , further comprising instructions for switching operation of the user input device from the assisted mode to the unassisted mode in accordance with a determination that the user's presence meets one or more predetermined criteria.

5. The stored instructions, while operating the user input device in the assistance mode, Tracking the movement of the user input device; The medical system of claim 1 , further comprising instructions for switching operation of the user input device from the assisted mode to the unassisted mode in accordance with a determination that the movement of the user input device meets a predetermined criterion.

6. The medical system of claim 1 , wherein the sensor comprises a capacitance-based sensor or an optical-based sensor.

7. 10. The medical system of claim 1, wherein the sensor tracks movement of the user input device, and identifying the drift of the user input device comprises determining that the tracked movement corresponds to unintentional movement.

8. The medical system of claim 1 , wherein identifying the drift of the user input device comprises detecting movement instability in the user input device.

9. The second parameter set includes modifications to one or more parameters of the first parameter set, the modifications comprising: adjusting the attenuation of the user input device; adjusting the stiffness of the user input device; adjusting motion scaling between the user input device and the medical instrument; adjusting a brake control of a joint of the user input device; Performing a position control operation; or and performing a rate control operation.

10. 2. The medical system of claim 1, wherein the second parameter set includes modifications to one or more parameters of the first parameter set, and the modifications to the one or more parameters include applying weak feedback control.

11. The medical system of claim 10 , wherein applying the weak feedback control comprises modifying the one or more parameters of the first parameter set within predetermined limits.

12. 10. The medical system of claim 1, further comprising a robotic arm coupled to the medical instrument for moving the medical instrument according to one or more instructions from the processor.

13. 1. A method for operating a medical system including a user input device for controlling a medical instrument, comprising: operating the user input device in an unassisted mode, the unassisted mode including applying a first set of parameters to the user input device; detecting drift of the user input device while operating in the unassisted mode; switching operation of the user input device to an assisted mode in response to detecting the drift, the assisted mode including applying a second set of parameters different from the first set of parameters; determining that the user input device is stable while operating in the assisted mode; and switching the operation of the user input device to the unassisted mode in accordance with the determination.

14. The method of claim 13 , wherein determining that the user input device is stable comprises determining that the user input device has been operating in the assisted mode for a predetermined amount of time.

15. The method of claim 14 , wherein the predetermined time corresponds to a settling time of the user input device.

16. The method of claim 13 , wherein determining that the user input device is stable comprises determining that a user's presence at the user input device meets one or more predetermined criteria.

17. The method of claim 13 , wherein determining that the user input device is stable comprises determining that movement of the user input device meets a predetermined criterion.

18. Detecting the drift of the user input device includes: tracking movement of the user input device; and determining that the tracked movement corresponds to unintentional movement.

19. the second parameter set includes modifications to one or more parameters of the first parameter set, the modifications to the one or more parameters being: adjusting the attenuation of the user input device; adjusting the stiffness of the user input device; adjusting motion scaling between the user input device and the medical instrument; or and adjusting a brake control of a joint of the user input device.

20. Switching the operation of the user input device to the assistance mode includes: Performing a position control operation; or and performing a speed control operation.

21. 14. The method of claim 13, wherein the second parameter set includes modifications to one or more parameters of the first parameter set, and the modifications to the one or more parameters include applying weak feedback control.

22. 22. The method of claim 21, wherein applying the weak feedback control comprises modifying the one or more parameters of the first parameter set within predetermined limits.

23. The method of claim 13 , further comprising: continuing to operate the user input device in the assisted mode in response to determining that the user input device is not stable while operating in the assisted mode.

24. 1. A method for operating a medical system including a user input device for controlling a medical instrument, comprising: operating the user input device in an unassisted mode, the unassisted mode including applying a first set of parameters to the user input device; identifying drift of the user input device while operating in the unassisted mode; and in response to detecting the drift, switching operation of the user input device to an assisted mode that includes applying a second set of parameters different from the first set of parameters.

Images