De-matching / matching UID to robotic gripper to engage teleoperation

Abstract

The present invention provides a surgical robotic system having a robotic grasper, a user interface device (UID), and one or more processors communicatively coupled to the UID and the robotic grasper. The system detects instructions to engage or reengage a teleoperation mode, determines that the system is in a non-teleoperation mode, receives a user action sequence via the UID, determines that the UID matches the jaw angle or gripping force of the robotic grasper, and switches to a teleoperation mode. Other embodiments are also described and claimed.

Description

[0001] One embodiment of the invention relates to a user input device, a robot or a robotic gripper, and remote operation of a robotic gripper. Other embodiments are also described. Background Art

[0002] In a surgical robotic system, a surgeon remotely operates a robotic arm with surgical tools attached to its distal end, which is also referred to as teleoperation. Application scenarios include endoscopic surgery, which involves viewing a patient's body and performing surgery, for example, within the abdominal cavity, using an endoscope and other surgical tools attached to the ends of several robotic arms. The system provides the surgeon with a close-up view of the surgical site while enabling the surgeon to remotely operate the tools attached to the arm by manipulating a handheld user interface device (UID) throughout the process in real time. The tool may be a clamp with jaws, a cutter, a camera, or an energy transmitter such as a laser for coagulation. Therefore, the system controls the tool with precision and high dexterity, depending on the surgeon's manipulation of the handheld UID.

[0003] The core function of a UID is to act as an input device to control translation, rotation, and other actions, such as grasping, of an end effector of a surgical tool. One type of tool is a grasper, which has two jaws that are controlled by the opening and closing movement of a grasping link of the UID or a deformable surface that can be squeezed or pressed to close the jaws. Other UIDs have other types of links or mechanisms. Summary of the invention

[0004] Various aspects of a surgical robot system are described herein that implements a method for engaging a user interface device (UID) to a surgical robot grasper for remote operation. These embodiments help solve the problem of when and how to initially engage remote operation at the beginning of a surgical session, and when and how to reengage remote operation from a pause mode or from a disengagement mode (where the system becomes paused or disengaged during remote operation of an ongoing surgical session) to reduce the risk of the robotic grasper damaging patient tissue, damaging equipment, or accidentally releasing tissue during surgery. The UID has a grasper part such as a connecting rod, a deformable surface that can be squeezed or pressed, or other finger actuation mechanisms whose position or orientation is sensed as an electronic user input signal (user input). When engaging remote operation, the user input is processed by the control system, generally to proportionally control the angle and / or force of the surgical robot grasper during remote operation.

[0005] In one embodiment, a surgical robotic system has: a robotic grasper having two jaws; a user interface device (UID) having grasper parts, sensing electronics generating sensing signals (user inputs) for the grasper parts; and one or more processors (generally referred to herein as "processors" for convenience) communicatively coupled to the UID sensing electronics and the robotic grasper. The processors are configured to execute a process for deciding when to allow engagement or reengagement of a teleoperation mode in which the robotic grasper is controlled by user inputs from the UID.

[0006] In one embodiment, the processor may first determine that the surgical robotic system is in a non-teleoperation mode (e.g., pause or "clamp" mode, disengagement mode, or waiting to initiate teleoperation at the start of a surgical session). The processor receives a detected sequence of user actions via a UID (also referred to herein as a user input). Based on the received user input, the processor determines when the UID (its gripper part) matches: i) the jaw angle of the robotic gripper (the angle between the two jaws of the robotic gripper); or ii) the gripping force affected by the two jaws of the robotic gripper. In response to determining that the UID matches the jaw angle or gripping force of the robotic gripper, the processor transitions the surgical robotic system to a teleoperation mode.

[0007] In another embodiment, a surgical robotic system performs a method for reengaging teleoperation as described below. First, the system's processor determines that the surgical robotic system has transitioned to a disengaged mode during teleoperation, and in response begins executing a finite state machine having multiple states. While executing the state machine, the processor receives a sequence of detected user actions (user inputs), the user actions comprising a user's finger actuating a grasper part of the UID. This detected sequence of user inputs is interpreted by the system, causing the state machine to transition between several states of a so-called dematching / matching algorithm. When the state machine reaches a state of the algorithm corresponding to a UID user input matching the jaw angle or gripping force of the robotic grasper, the surgical robotic system reengages the robotic grasper in teleoperation.

[0008] The above summary does not include an exhaustive list of all aspects of the invention. It is contemplated that the invention includes all systems and methods that can be practiced by all suitable combinations of the various aspects summarized above and disclosed below in the detailed description and specifically pointed out in the claims filed with this patent application. Such combinations have specific advantages not specifically recited in the above summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments of the present invention are shown in the figures of the accompanying drawings by way of example and not by way of limitation, wherein like reference numerals indicate like elements. It should be noted that references to "one" or "an" embodiment of the present invention in this disclosure are not necessarily the same embodiment, and that they refer to at least one. In addition, for the sake of brevity and to reduce the total number of drawings, a given drawing may be used to illustrate features of more than one embodiment of the present invention, and all elements in the drawings may not be required for a given embodiment.

[0010] Figure 1 is a pictorial view of an exemplary surgical robotic system in a surgical setting.

[0011] Figure 2A An example of a user input device (UID) is shown.

[0012] Figure 2B An example of a surgical robotic grasper attached to a surgical robotic arm is shown.

[0013] Figure 3 is a block diagram of a surgical robot system including a UID, a surgical robot arm, a surgical robot grasper, and a control computer system for implementing remote operation through the UID matched with the surgical robot grasper.

[0014] Figure 4A is a graph of exemplary mappings from UID values ​​(user input) to surgical robot grasper angles and forces for two different tools.

[0015] Figure 4B is a graph of another exemplary mapping from UID values ​​to surgical robot grasper angles and forces for two different tools.

[0016] Figure 5A is a state diagram for matching a UID to a surgical robotic grasper, in one embodiment.

[0017] Figure 5B is another state diagram for matching a UID to a surgical robotic grasper in another embodiment.

[0018] Fig. 6A is a flow chart of a method for engaging teleoperation of a surgical robotic system.

[0019] Figure 6B is a flow chart of another method for engaging teleoperation of a surgical robotic system. DETAILED DESCRIPTION

[0020] Several embodiments of the present invention are now explained with reference to the accompanying drawings. Whenever the shapes, relative positions and other aspects of the parts described in the embodiments are not clearly defined, the scope of the present invention is not limited to the parts shown, which are shown for illustration purposes only. In addition, although many details are set forth, it should be understood that some embodiments of the present invention can be practiced without these details. In other cases, well-known circuits, structures and technologies are not shown in detail to avoid obscuring the understanding of this specification.

[0021] The surgical robotic system and algorithm described herein ensure that teleoperation is not engaged unless the interface device (UID) matches the surgical robotic grasper based on the jaw angle and / or gripping force of the surgical robotic grasper. In one embodiment, if teleoperation is reengaged from a disengaged (inactive) mode, a mismatch (or "de-match") must first be intentionally created between the UID and the robotic grasper (via user action on a finger-actuated UID grasper) before matching. In contrast, when attempting to reengage teleoperation from a pause mode (also known as a clamping mode, in which the control system freezes the robotic grasper in response to a user actuating a pause or clamping button in the UID or the user initiating another form of a pause or clamping command), the algorithm does not need to look for an initial de-match, but instead may allow reengagement of teleoperation when it detects that the UID user input matches the jaw angle and / or gripping force of the surgical robotic grasper being maintained.

[0022] See also Figure 1 , which is a pictorial view of an exemplary surgical robotic system 1 in a surgical setting. The robotic system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic platform 5 (e.g., a table, bed, etc.). The system 1 may incorporate any number of devices, tools, or accessories for performing a surgical procedure on a patient 6. For example, the system 1 may include one or more surgical tools 7 for performing a surgical procedure. The surgical tool 7 may be an end effector attached to the distal end of the surgical arm 4 for performing a surgical procedure.

[0023] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during a surgical procedure. For example, the surgical tool 7 can be a tool used to access, view, or manipulate the internal anatomy of the patient 6. In one embodiment, the surgical tool 7 is a grasper that can grasp the patient's tissue. The surgical tool 7 can be manually controlled by a bedside operator 8; or it can be robotically controlled via actuated movement of the surgical robotic arm 4 to which it is attached. The robotic arm 4 is shown as a table mounted system, but in other configurations, the arm 4 can be mounted on a cart, ceiling or sidewall, or mounted in another suitable structural support.

[0024] Generally, a remote operator 9 (such as a surgeon) may use the user console 2 to remotely manipulate the arm 4 and / or attached surgical tools 7, e.g., teleoperation. The user console 2 may be located in the same operating room as the rest of the system 1, e.g., Figure 1 As shown. However, in other environments, the user console 2 may be located in an adjacent or nearby room, or it may be located in a remote location, for example, in a different building, city, or country. The user console 2 may include a seat 10, foot controls 13, one or more handheld user input devices UID 14, and at least one user display 15, which is configured to display a view of a surgical site within a patient 6, for example. In the exemplary user console 2, the remote operator 9 sits in the seat 10 and views the user display 15, while manipulating the foot controls 13 and the handheld UID 14 to remotely control the arm 4 and the surgical tool 7 (which is mounted on the distal end of the arm 4).

[0025] In some embodiments, the UID 14 can be a portable, handheld user input device or controller that is ungrounded relative to another component of the surgical robotic system. For example, the UID 14 can be ungrounded, whether tethered or untethered to the user console 2. The term "ungrounded" is intended to refer to the following specific implementation: for example, one or more UIDs are neither mechanically nor kinematically constrained relative to the user console 2. For example, a user can hold the UID 14 in his hand and move freely to any possible position and orientation within a space limited only by, for example, the tracking mechanism of the user console. In contrast, a grounded UID can maintain its position / orientation / grip.

[0026] In some variations, the bedside operator 8 can also operate the system 1 in an "in bed" mode, where the bedside operator 8 (user) is now located on the side of the patient 6 and manipulates the robot-driven tools (the end effector attached to the arm 4) at the same time, for example, holding the handheld UID 14 and the manual laparoscopic tools with one hand. For example, the left hand of the bedside operator can manipulate the handheld UID to control the robotic components, while the right hand of the bedside operator can manipulate the manual laparoscopic tools. Therefore, in these variations, the bedside operator 8 can perform both robot-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0027] During the exemplary procedure (surgical operation), the patient 6 is prepared and covered with a sterile drape in a sterile manner to achieve anesthesia. When the arms of the robotic system 1 are in a stowed configuration or a withdrawn configuration, the initial access to the surgical site (to facilitate access to the surgical site) can be performed manually. Once access is completed, the initial positioning or preparation of the robotic system 1 including its arms 4 can be performed. Next, the surgical operation is performed as the remote operator 9 at the user console 2 uses the foot control 13 and the UID 14 to manipulate various end effectors and possible imaging systems to perform the surgical operation. Manual assistance can also be provided at the operating bed or operating table by a bedside person (e.g., a bedside operator 8) wearing a sterile surgical gown, who can perform tasks on one or more of the robotic arms 4, such as retracting tissue, manually performing repositioning, and tool changes. Non-sterile personnel may also be present to assist the remote operator 9 at the user console 2. When a procedure or surgical operation is completed, the system 1 and user console 2 may be configured or set to a state to facilitate completion of post-operative procedures, such as cleaning or sterilization, and importing or printing medical records via the user console 2 .

[0028] In one embodiment, the teleoperator 9 holds and moves the UID 14 to provide input commands to move the robot arm actuator 17 in the robot system 1. The UID 14 can be communicatively connected to the rest of the robot system 1, for example, via the console computer system 16. The UID 14 can generate a spatial state signal corresponding to the movement of the UID 14, such as the position and orientation of the handheld housing of the UID, and the spatial state signal can be an input signal to control the movement of the robot arm actuator 17. The robot system 1 can use the control signal derived from the spatial state signal to control the proportional movement of the actuator 17. In one embodiment, the console processor of the console computer system 16 receives the spatial state signal and generates a corresponding control signal. Based on these control signals that control how the actuator 17 is powered to move the segments or connectors of the arm 4, the movement of the corresponding surgical tool attached to the arm can simulate the movement of the UID 14. Similarly, the interaction between the teleoperator 9 and the UID 14 can generate, for example, a grasping control signal that causes the jaws of the grasper of the surgical tool 7 to close and grasp the tissue of the patient 6.

[0029] The surgical robotic system 1 may include several UIDs 14, wherein respective control signals are generated for each UID controlling the actuators and surgical tools (end effectors) of the respective arms 4. For example, the teleoperator 9 may move the first UID 14 to control the movement of an actuator 17 located in the left robotic arm, wherein the actuator responds by moving a link, gear, etc. in the arm 4. Similarly, the movement of the second UID 14 by the teleoperator 9 controls the movement of another actuator 17, which in turn moves other links, gears, etc. of the robotic system 1. The robotic system 1 may include a right arm 4 fixed to a bed or table on the right side of a patient, and a left arm 4 located on the left side of the patient. The actuator 17 may include one or more motors that are controlled so that they drive the joint of the arm 4 to rotate, for example, to change the orientation of an endoscope or grasper of a surgical tool 7 attached to the arm relative to the patient. The movement of several actuators 17 in the same arm 4 may be controlled by a spatial state signal generated from a particular UID 14. The UID 14 may also control the movement of the respective surgical tool grasper. For example, each UID 14 may generate a corresponding grasping signal to control the movement of an actuator (eg, a linear actuator) that opens or closes the jaws of a grasper at the distal end of the surgical tool 7 to grasp tissue within the patient 6 .

[0030] In some aspects, communication between the platform 5 and the user console 2 can be through a control tower 3, which can convert user commands received from the user console 2 (and more specifically from the console computer system 16) into robotic control commands transmitted to the arm 4 on the robotic platform 5. The control tower 3 can also transmit status and feedback from the platform 5 back to the user console 2. The robotic platform 5, the user console 2, and the control tower 3 can be communicatively connected via wired and / or wireless links using any suitable data communication protocol among a variety of data communication protocols. Any wired connection can optionally be built into the floor and / or walls or ceiling of the operating room. The robotic system 1 can provide video output to one or more displays, including displays within the operating room and remote displays accessed via the Internet or other networks. The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0031] One problem to be solved in the surgical robotic system 1 is how or when to safely engage or resume (i.e., reengage) teleoperation without causing unintended movement of the surgical tool that could cause tissue damage. Without the solution described herein, there is a risk of reengaging teleoperation, particularly when teleoperation is interrupted for some reason (the system enters a disengaged mode), which could cause the surgical robotic grasper to suddenly open, suddenly close, forcefully clamp tissue, or loosen previously grasped tissue, etc.

[0032] The system of the present invention determines when user input from a user interface device (UID) matches the gripper angle or gripper force of a surgical robot gripper, at which point teleoperation of the robot gripper can be engaged. The UID is not mechanically constrained to maintain the same position, orientation, and gripper angle or force as the surgical tool. Therefore, an algorithm is described that, when executed, detects when the gripper part of the UID matches the surgical tool gripper (in terms of angle and force, i.e., the UID and the surgical robot gripper are within an acceptable threshold of being consistent with each other) before the system engages or resumes (i.e., reengages) teleoperation. The algorithm may be particularly useful for ungrounded UIDs that do not maintain their position or orientation after being released by the user. In this context, two devices that match mean that the positions and orientations or other related values ​​of the two devices are within a threshold of being consistent with each other. The matching process means the process of bringing the positions and orientations or other related values ​​of the two devices within a threshold of being consistent with each other. User input from the gripper part of the UID is converted into angles and gripping forces for the surgical robot gripper. In one embodiment, the system first looks for a situation where the UID and the robotic grasper have not yet matched (e.g., the system waits until the user input from the UID has changed enough to exceed a threshold that is consistent with the current angle and / or force of the robotic grasper). At some point thereafter, the user's finger manipulation of the UID causes the user input to match the current resting angle and / or force of the surgical grasper. Upon detecting this match, the system may respond by alerting the user and engaging teleoperation of the surgical robotic grasper.

[0033] In one embodiment, the user is expected to open the UID, exceed the UID open threshold, or start with the UID in an open position or orientation. For example, the user may set the grip crank of the UID to an angle greater than the UID open threshold. The user then closes the UID. For example, the user may close the UID beyond a certain UID closed threshold. During this time, the algorithm monitors user input reflecting the change of the grasper part of the UID from open to closed. After closing the grasper part of the UID, the user opens it while the algorithm still monitors the user input until the algorithm determines that the user input matches the surgical grasper in response to engaging teleoperation. Matching the UID to the tool grasper before engaging teleoperation helps prevent damage to surgical tools and tissue injury. The need to create a mismatch by first opening and closing the UID and then matching the UID to the surgical robotic grasper eliminates guesswork, thereby helping to ensure a safe transition from a disengaged mode (in which teleoperation of the robotic grasper is disabled) to a teleoperation enabled mode (see Figure 5A ).

[0034] Figure 2A An example of a UID grasper part (UID grasper 22) is shown, which may be part of the UID 14, see Figure 1 The UID grasper 22 has one or more movable grasping cranks 23 and associated electronic circuitry (not shown) that can sense the position of the grasping cranks 23. As part of the UID 14, the UID grasper 22 is held in the user's hand in an open configuration with the grasping cranks 23 in the fully open position as shown. The UID grasper 22 is designed so that the user's fingers can manipulate the grasping cranks 23 within their range of positions (angle 26 between the grasping cranks 23 and the axis 25) through a finger squeezing action, which is sensed by the associated electronic circuitry and then controlled by the controller 39 (see Figure 3 ) is converted into a corresponding position (angle) command to actuate the jaws of the grasper of the surgical tool 7. The user's hand can manipulate the UID grasper 22 in various ways. In addition, when the user's hand pinches or squeezes the grasping crank 23 inward toward the axis 25 to reduce the angle 26 (i.e., "close" the UID 14), the grasping crank 23 can provide tactile feedback to the user or surgeon. Therefore, such a UID 14 provides enhanced mobile dexterity and precision for controlling surgical robotic tools. Other UIDs can be used in other embodiments of the surgical robotic system, including, for example, whose UID grasper 22 is different from Figure 2A UID of the type shown.

[0035] Figure 2B An example of a surgical robotic grasper 30 is shown attached to a surgical robotic arm 4. An actuator 35 moves two jaws 31, 32, labeled Jaw 2 and Jaw 1, through an angle 33 between the two jaws 31, 32, which is sensed by a sensor 37. In a surgical robotic system, the jaws 31, 32 are used to grasp tissue, and the actuator 35 is driven to control the angle 33 based on measurements made by the sensor 37. The sensor 37 also measures the grasping force applied by the jaws 31, 32. Other surgical robotic graspers or end effectors and tools may be used in additional embodiments of the surgical robotic system.

[0036] Figure 31 is a block diagram of a surgical robot system including a UID 14 (with a UID gripper 22), a surgical robot arm 4, a surgical robot gripper 30, and a computer system for implementing remote operation through a UID matched to the surgical robot gripper. The computer system has one or more processors (processor 42) that execute software (stored in memory). These include a matching / dematching controller 39, which includes a state machine 45. In addition, based on user input from the UID 14, a remote operation controller 43 controls the remote operation of the surgical robot arm 4 and the surgical robot gripper 30. The remote operation controller 43 has an input command generator 47 that converts or transforms the sensor measurements of the UID 14 (including translation, rotation, and grasping) into input commands for remote operation of the surgical robot arm 4 and the surgical robot gripper 30. Figure 3 The computer system shown may be implemented as part of a control tower 3, see Figure 1 .

[0037] In some embodiments, the UID grasper 22 includes a gripping link or deformable surface and one or more sensors that are part of the user console 2 and generate gripping signals as user inputs (based on user actions, such as squeezing the gripping link or deformable surface) to control the jaw angle and gripping force of the robotic grasper 30. These gripping signals are examples of detected user actions. For example, the gripping signal may include sensor measurements of the angle 26 (or the degree of deformation of the UID surface) caused by the user's finger actuation of the gripping link or deformable surface. The gripping signal is input into the control computer system to generate a control command that causes the two jaws of the robotic grasper to close / open at a certain jaw angle and / or affect a certain force. The purpose of the matching / de-matching process is that when the user squeezes the gripping link or deformable surface, the sensor measurements (and therefore the gripping signal) will correspond to (i.e., match) the current jaw angle and force of the robotic grasper at that time. Therefore, when the robotic system engages / reengages to teleoperation at that time, the robotic grasper will maintain the same jaw angle and gripping force as it did when it was disengaged or paused, without any sudden movements.

[0038] The matching / dematching process may be performed by processor 42 executing state machine 45 (in software). Figure 5A and Figure 5BThe functionality of the state machine 45 is discussed in greater detail. More generally, the state machine 45 keeps track of the teleoperation mode and dematch / match status of the UID and the surgical robotic grasper 30. When (or in response to) the user input from the UID matches the angle and / or force of the surgical robotic grasper, the state machine outputs an indication 51 of the matching completion status. This indication 51 may be presented to the user in the form of an audio message, such as a ringtone, sound, or voice; a visual alert, such as a light turning on or flashing; a text or graphical message displayed on the display 15 of the user console 2; a tactile alert, such as a vibration, e.g., of the UID 14, armrest, or seat vibrator in the user console 2; or other notification to the user at the user console 2. Alternatively, in various embodiments, the indication 51 may trigger engagement of teleoperation with or without additional notification.

[0039] Figure 4A is a graph for converting from a UID user input value to a surgical robot grasper control command, which is a value specifying a surgical robot grasper angle and / or force. Graphs are shown for two different tools, namely graphs 52 and 53. The UID state (e.g., open, closed, matched with a robotic grasper, or not matched with a robotic grasper) can be derived from an analysis of the UID value and an analysis of the sensed surgical robot grasper angle and force. During teleoperation, it is assumed (due to the operation of the control system) that the UID continuously matches the surgical robot grasper, which means that the measurement pair (UID value, robotic grasper value) at any given moment defines a point that is above or within a threshold band around graph 52 (for that particular tool). However, when the measurement pair defines a point that is not within the threshold band around graph 52, the UID is considered to be mismatched with the robotic grasper. The latter case should only occur during non-teleoperation mode.

[0040] Graph 52 depicts the linear relationship between the UID value (angle) on the x-axis and the surgical robot grasper angle on the y-axis. Graph 52 extends the range of the UID minimum angle to the maximum angle and the surgical robot grasper minimum angle to the maximum angle from the grasper angle control to the grasper force control. Generally speaking, a portion of the x-axis (UID variable) is converted to the grasper angle and another portion of the UID variable is converted to the grasper force as shown. When the UID value is less than or equal to g u,open and thus in g u,min The UID is considered open when it is within the protection band threshold of g. At the opposite end of the range, when the UID value is greater than or equal to g u,closed and thus in g u,maxThe UID is considered closed when it is within the protection band threshold of . Another graph 53 (dashed line) for another tool shows a similar linear relationship between the UID angle and the surgical robot grasper angle, but offset by a different range from the minimum angle to the maximum angle of the surgical robot grasper. Other graphs, including graphs with curved or segmented relationships between angle and force for various tools, are readily developed and understood based on the teachings herein.

[0041] Figure 4B Graphs of mappings from UID values ​​to surgical robot grasper angles and forces for two different tools in another embodiment. One graph 55 (solid line) for one tool shows a linear relationship between the value of the UID (in this case, a measurement of the angle) and the angle of the surgical robot grasper for grasper angle control, which is converted to different slopes for grasper force control. Another graph 56 (dashed line) for another tool shows a linear relationship between the value of the UID (angle) and the angle of the surgical robot grasper, extending with a slope that does not change from grasper angle control to grasper force control. At the same angle of the UID (e.g., zero), both graphs 55, 56 are converted from grasper angle control to grasper force control. It is easy to develop and understand other graphs, including other graphs showing other conversion points, other slopes, and other angle-to-force relationships for various tools.

[0042] The unmatch / match algorithm may consist of a state machine that requires the UID to transition between a series of states before engaging teleoperation from a disengaged (inactive) mode in order to unmatch and then match the UID to the robotic gripper. In one embodiment, when engaging teleoperation from a pause mode (rather than a disengaged or inactive mode), the algorithm does not need to detect a specific unmatch condition, but instead looks directly for a match condition; in other words, in that case, there is no need to first determine whether the UID and the robotic gripper are unmatched.

[0043] The robot gripper is characterized by the opening angle and the gripping force, which will be denoted as g r =[θ g , f g ]. Use a numerical range to represent the value indicated as g u During teleoperation, the UID input is converted into the opening angle and gripping force (control commands) for the robot gripper, e.g. Figure 4A and Figure 4B The function that converts the UID value into the robot gripper value will be expressed as: g r =f(g u ). The lower and upper limits of the UID gripper value are respectively u,min and gu,max The lower limit g is given. u,min corresponds to the fully open position of the UID, and the upper limit value g u,max Corresponds to the fully closed position of the UID. Regarding the robot gripper angle, its lower and upper limits are respectively given by θ g，min and θ g，max The robot gripper force is given by (0, f g，max ). Function f converts fully opened UID to maximum opening angle and zero force of the robotic grasper. Full closure of UID is converted to maximum force applied by the robotic grasper, which may occur within a range of robotic grasper opening angles, depending on the width of the tissue being grasped.

[0044] The robot gripper value to be compared with the current UID value by the dematching / matching algorithm can be calculated by the inverse mapping: The inverse function converts the current robot gripper angle and force into UID units so that the UID and robot gripper values ​​can be compared in consistent units by the unmatching / matching algorithm. The inverse conversion is represented by the following formula:

[0045]

[0046]

[0047]

[0048]

[0049] Parameter k angle_to_force is a negative parameter, and the quantity 1 / k angle_to_force Convert the force to a negative angle to convert both the gripper angle and the force to UID units. The variable α is the scaling factor between UID units and the robot gripper angle. The parameter ε f >0 can be set based on the estimated accuracy of the measured clamp force.

[0050] Note that the above conversion allows for the minimum clamp angle to be greater than zero. Given that in most cases the minimum clamp angle is zero, the formula can be simplified as described below for this case.

[0051]

[0052]

[0053]

[0054]

[0055] The unmatching / matching algorithm can use three parameters that can be tuned according to the given UID and the characteristics of the robot gripper. We will g u,open (≥g u,min ) is defined as the clamp opening threshold, and g u,closed (≤g u,max ) is defined as the clamp closing threshold. The matching threshold is denoted as δg match .if The UID and the robotic gripper are then considered to be "matched".

[0056] The dematching / matching algorithm can be composed of the following four states:

[0057] Status description Status Name Waiting for UID grabber to open UNMATCHING_OPEN Waiting for the UID gripper to close UNMATCHING_CLOSE Waiting for UID grasper to match surgical robot grasper MATCHING UID grasper and surgical robot grasper match MATCHED

[0058] The algorithm state machine logic is depicted in Figure 5A and Figure 5B . Figure 5A is a diagram of various teleoperation modes and algorithms used to mate a UID with a surgical robotic grasper in one embodiment. Figure 4A and Figure 4B A range of UID values ​​(in this case angles) versus surgical robot grasper angles and forces are shown, where the UID is matched to the surgical robot grasper during teleoperation. Figure 5A , the left column indicates the remote operation mode {DISENGAGED 57,PAUSED 59,ENGAGED 60}. Figure 5A In the figure, the right column indicates the state of the unmatching / matching algorithm {UNMATCHING_OPEN 61, UNMATCHING_CLOSE 62, MATCHING 65, MATCHED 67}. The arrows indicate the allowed transitions between states and modes. Note that Figure 5ADetails of other operations that may be required to transition between remote operation modes are not shown, as these are not part of the dematching / matching algorithm presented herein. The remote operation mode DISENGAGED 57 directs the dematching / matching algorithm to the state UNMATCHING_OPEN 61 (the system is waiting for the UID to open). In response to the user opening the UID grasper 22, the algorithm transitions to the state UNMATCHING_CLOSE 62 (the system is waiting for the UID to close). The user then closes the UID grasper 22, and the algorithm detects and responds to it by transitioning to the state MATCHING 65 (the system is waiting for the UID to match the surgical robot grasper). The user then opens the UID grasper 22 until the algorithm detects that the UID value (user input) is consistent with the robot grasper angle and / or force within a certain threshold, and responds by making a transition from MATCHING 65 to MATCHED 67 (the system has determined that the UID and the surgical robot grasper match). When the MATCHED 67 state is achieved, the remote operation mode becomes ENGAGED 60. That is, in response to determining that the UID is being matched with the surgical robotic grasper, teleoperation of the surgical robotic grasper 30 is engaged via the UID 14, as indicated by the MATCHED 67 state.

[0059] Figure 5B is another state diagram for matching a UID to a surgical robotic grasper in another embodiment. While engaging teleoperation, a user, such as a surgeon, may need to pause (or clamp) the surgical robotic system, such as to take a break, observe the patient or the progress of the surgical procedure, obtain additional surgical strategies or instructions, consult with other surgical personnel, etc. Pausing the surgical robotic system may be performed by voice command, interaction with a user interface, a button or other control on the UID, etc., any of which may place the surgical robotic system in PAUSED 59 mode in various embodiments. When the user is ready to resume teleoperation, it may be the case that the UID no longer matches the frozen position of the surgical robotic grasper because the UID may have been set aside, moved, opened or closed, etc. Figure 5B The state diagram in shows how the UID can be matched with the surgical robot grasper again so that the remote operation can be resumed from the PAUSED 59 mode. In this state diagram, two possibilities are explicitly considered. One possibility is that when the robot grasper force > F min (value 71), for example, the surgical robot grasper 30 clamps the tissue and the system is paused. Another possibility is to set the robot grasper force <F min (value 73), i.e. the robot gripper is open but not holding anything, pause the system.

[0060] The teleoperation mode PAUSED 59 directs a transition from the dematching / matching algorithm state MATCHING 65 to the state MATCHED 67, depending on the value of the robotic grasper force. If the system has been paused while the robotic grasper force is greater than the threshold Fmin, the transition from MATCHING 65 to MATCHED 67 occurs in response to the UID grasper 22 closing (from an open state). This makes it easier to maintain the surgical robotic grasper force, thereby clamping the tissue, as the UID approaches and matches the angle of the surgical robotic grasper. Under this condition, it is expected that the user will open the UID grasper 22 and then (e.g., gradually) close the UID grasper 22 until the user input from the UID 14 matches the surgical robotic grasper, at which point the system can indicate "matching complete" and thus resume teleoperation (at which point the user input from the UID is mapped to the surgical robotic grasper).

[0061] However, if the system is paused when the robot gripper force is less than a minimum threshold Fmin (e.g., essentially zero, and where the gripper 30 is open but not gripping anything), a transition from MATCHING 65 to MATCHED 67 occurs when the UID gripper 22 is opened (from a closed state) or when the UID gripper 22 is closed somewhat (from an open state). This depends on how large the matching threshold is and on the values ​​of the robot gripper variables and the UID variables (e.g., angles). For example, if the robot gripper is half open and the UID gripper is fully open, the user is expected to close the UID gripper somewhat in order to match (unless the matching threshold is half of the full range). If the robot gripper is half open and the UID gripper is fully closed, the user is expected to open the UID gripper somewhat in order to match. After entering the MATCHED 67 state, the algorithm will indicate that the ENGAGED 60 teleoperation mode can be resumed. In this way, as the UID approaches and mates with the surgical robot grasper, the surgical robot grasper force is less likely to increase suddenly, which could damage patient tissue or damage the device.

[0062] See also Figure 5A, in order to go from DISENGAGED 57 to ENGAGED 60 teleoperation mode, the de-matching / matching algorithm state is initialized to UNMATCHING_OPEN 61 and then goes through the following sequence {UMMATCHING_CLOSE 62, MATCHING 65, MATCHED 67}. The UNMATCHING_OPEN 61 state and the UNMATCHING_CLOSE 62 state require the UID grasper to cover the full range from open to closed, thereby ensuring that a mismatch between the UID grasper 22 and the surgical robot grasper 30 always occurs initially. This is because when the system is disengaged, the surgical robot grasper 30 cannot be controlled to follow the UID 14, and there is no guarantee that the UID matches the surgical robot grasper. When the system is paused or disengaged, even the slightest manipulation of the UID grasper 22 will result in a mismatch (mismatch) between the UID and the surgical robot grasper. To reengage teleoperation from the DISENGAGED 57 mode, the user is expected to open the UID grasper 22 (e.g., release and rely on a spring loaded mechanism to automatically open), then close the UID grasper 22, then open the UID grasper 22 until the user input from the UID 14 matches the sensed angle and / or force of the surgical robotic grasper 30. This provides a consistent set of required user inputs regardless of the initial values ​​of the user inputs and the initial values ​​of the angles and forces of the surgical robotic grasper 30, resulting in a simple user interface that requires minimal training and no guesswork.

[0063] Once the dematching is complete, the state transitions to MATCHING 65, and the algorithm waits for the UID to match the surgical robotic grasper 30 before finally transitioning to the MATCHED 67 state. Once in the MATCHED 67 state, teleoperation can be engaged, with the teleoperation mode being ENGAGED 60. The UID action used to perform the dematching can be selected to be open and then closed (or vice versa) so that matching generally occurs in the opening direction of the UID grasper 22 (e.g., loosening), and is therefore less likely to result in the user inadvertently applying high grasper forces to the tissue. For situations where the robotic grasper is applying high forces while tightly grasping the tissue, matching is allowed to occur in the closing direction (e.g., squeezing). This behavior is desirable because matching the UID to the surgical robotic grasper 30 in the opening direction will result in the user inadvertently dropping the grasped tissue.

[0064] The robot gripper range (expressed in units of pressure, where matching can occur in the closing direction) can be determined by the algorithm parameters as Parameter g u,noise is some small value greater than 0, which may be determined by the noise characteristics of the sensor, since it may not occur at a UID value of exactly g.u,closed When the closing threshold is exceeded g u,closed .

[0065] In contrast to the case where the system is disengaged from teleoperation, consider the case where the system is paused (and thus enters PAUSED 59 mode). There, the algorithm waits for the UID to "re-match" the last retained robotic grasper value (corresponding to the robotic grasper grasped or frozen by the control system when paused) because after being in the PAUSED state, the robotic grasper automatically retains its last posture. In order to enter the ENGAGED 60 teleoperation mode from PAUSED 59, the algorithm state is initialized to MATCHING 65 and waits for the UID 22 to match the surgical robotic grasper 30 in order to transition to the MATCHED 67 state, thereby allowing engagement of teleoperation (see Figure 5B ). It is assumed that in pause remote operation mode, the user is alert and no accidental engagement occurs.

[0066] 1) Alternative embodiment for converting from surgical robot grasper values ​​to corresponding UID units:

[0067] The surgical robot grasper g r =[θ g , f g ] to the corresponding UID unit The mapping may result in different ranges of UID values ​​being mapped to surgical robot grasper angles 33 (see Figure 2B ) and the forces for different instruments (see Figure 4A and Figure 4B ). It may be desirable that the UID range converted to surgical robot grasper angle 33 and the UID range converted to surgical robot grasper force be consistent between different surgical instruments. In this case, the inverse conversion will be given below

[0068]

[0069]

[0070]

[0071]

[0072] Parameter g u,zero is the UID pressure value equivalent to the surgical robot grasper at minimum clamp angle and zero force.

[0073] 2) Alternative implementation for estimating surgical robot grasper values ​​in UID units :

[0074] Two methods for extracting the surgical robot grasper gr =[θ g , f g ]Mapped to the corresponding UID unit However, considering the difficulty in accurately estimating the grasper force, another alternative embodiment can be used. During remote operation (ENGAGED 80 mode), according to the conversion function g r =f(g u ) commands the surgical robot grasper 30 to follow the UID g u Assuming reasonable tracking performance, the surgical robotic grasper will closely approximate the commanded surgical robotic grasper variable, which is represented by the UID g u Therefore, whenever remote operation is effective (i.e., in ENGAGED 60 mode), the corresponding UID unit is The surgical robotic grasper will be very close to the UID g u When teleoperation switches from ENGAGED 60 to PAUSED 59 mode or DISENGAGED 57 mode, teleoperation is invalid and the surgical robotic grasper 30 will remain in the last commanded posture. Given that the surgical robotic grasper 30 only changes posture when teleoperation is valid (in ENGAGED 60 mode) and provided that the surgical robotic grasper 30 accurately follows the commands, whenever teleoperation is valid (ENGAGED 60), the surgical robotic grasper (corresponding to the UID unit) Can be set to UID g u When remote operation is invalid (becomes DISENGAGED 57 or PAUSED 59), The last g of ENGAGED 60 will be latched u value.

[0075] Fig. 6A is a flow chart of a method for engaging teleoperation of a surgical robotic system, which may be practiced in or by an embodiment.

[0076] In action 81, the surgical robotic system detects an instruction to engage teleoperation mode. For example, the system may detect that: i) the UID is selected by the user or otherwise removed from the system's taskbar; ii) the user's gaze is focused on a display of the system's user console; or iii) a pedal action occurs at the system's foot pedal. These or other detected actions or events are interpreted by the system as instructions to engage teleoperation. In one embodiment, the detected action or event is part of a clamp release process that indicates the user's intent to unpause the system (resume teleoperation).

[0077] In act 83, the system determines that it is in a non-teleoperation mode. For example, the system may be in a disengaged mode. The system may alternatively be in a paused mode. In both modes, the UID cannot control the robotic gripper. However, in the disengaged mode, the system needs to perform a complex process or series of checks to allow the dematching / matching algorithm to reengage in teleoperation. In contrast, in the paused mode, the system performs a simple process to allow reengagement in teleoperation, for example, detecting that the user has "released" the grip.

[0078] In act 85, the system has been directed to reengage teleoperation and a detected sequence of user actions is received via a user interface device (UID). In one version, the sequence of user actions includes opening the UID, closing the UID, and then opening the UID again until the system detects that the UID and the robotic grasper match as a result of the sequence of user actions (act 87). Next, in act 89, in response to the match detected in act 87, the surgical robotic system transitions to teleoperation mode.

[0079] To extend the above method, one embodiment of the surgical robotic system provides guidance on a sequence of user actions. For example, the system may display instructions, diagrams, or images on a display of a user console, give audio instructions, sequence a series of lighted arrows on a UID, or otherwise provide visual, audio, or tactile guidance to the user on how to generate a sequence of user actions. In another extension of the above method, one embodiment of the surgical robotic system displays an indication of switching to a remote operation mode. For example, the system may generate a visible alarm, display a message, diagram, or image on a display of a user console, activate a light, or otherwise display an indication to the user that remote operation is being engaged.

[0080] Figure 6B is a flow chart of a method for reengaging teleoperation of a surgical robotic system, which may be practiced in or by an embodiment. In act 92, the surgical robotic system detects an instruction to engage teleoperation. For example, the system may detect that: the UID is selected or otherwise removed from the system's taskbar; the user's gaze is focused on a display of the system's user console; or a pedal action occurs at a foot pedal of the system. These or other detected actions or events are interpreted by the system as instructions to reengage teleoperation.

[0081] In act 94, the system determines that it is in a non-remote operating mode.

[0082] In act 96, the system executes a state machine. In one version, the state machine has a matching state for waiting for the UID to match the robotic gripper, and a matching complete state for when the UID matches the robotic gripper. In response to the UID matching the robotic gripper, the system transitions from the matching state for waiting for the UID to match the robotic gripper to the matching complete state for when the UID matches the robotic gripper. When in the matching complete state, the system engages teleoperation.

[0083] In another version, the state machine has a Dematching Open state during which the state machine waits for the UID gripper to fully open and then transitions to a Dematching Closed state during which the state machine waits for the UID gripper to fully close and then transitions to the Matching state.

[0084] In action 98, the system receives a user action sequence through a user interface device (UID). This user action sequence causes the state machine to transition between states. In one form, the user action sequence includes opening the UID, and closing the UID until the UID and the robot gripper match.

[0085] In act 99 , in response to determining that the state machine has reached a state corresponding to the UID matching the robotic grasper jaw angle or grasping force of the surgical robotic system, the surgical robotic system engages the UID and the robotic grasper in teleoperation.

[0086] Although certain embodiments have been described and shown in the drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art. Figure 2A and Figure 2B A specific example of a UID grasper 22 and a surgical robot grasper 30 is depicted, but other UIDs, graspers, and tools are possible. The UID grasper 22 may be part of the UID 14, such as Figure 1 As shown in the example of , the UID is an ungrounded device that is not mechanically connected to a base or bracket; alternatively, the UID gripper 22 can be a part of the UID 14 that is grounded (ie, mechanically connected to a base or bracket). Therefore, this description should be regarded as illustrative rather than restrictive.

Claims

1. A surgical robot system, comprising: a robotic grasper having two jaws; User interface device; and one or more processors communicatively coupled to the user interface device and the robotic gripper and configured to: detecting a command to engage or re-engage a teleoperation mode in which the robotic grasper is controlled by user input received from the user interface device; determining that the surgical robotic system is in a non-teleoperation mode; receiving, by the user interface device, a detected sequence of user actions; determining a mismatch in jaw angles between the user interface device and the two jaws or a gripping force effected by the two jaws of the robotic gripper as a result of the detected sequence of user actions and then matching; and In response to determining that the user interface device and the jaw angle or the grip force are mismatched and then matched, the surgical robotic system is transitioned to a teleoperation mode.

2. A surgical robotic system according to claim 1, wherein the instruction to engage or re-engage the remote operation mode is indicated by at least one of the following: the user deletes the user interface device from the task bar, the user focuses on the display of the user console, or the user performs a pedal action.

3. The surgical robotic system of claim 1, wherein: The non-remote operation mode includes one of a disengaged mode or a pause mode; wherein the disengaged mode is when the user interface device disengages control of the robotic grasper prior to a teleoperation session; and The pause mode is when the user interface device disengages control of the robotic gripper to pause a teleoperation session.

4. The surgical robotic system of claim 1, wherein the detected sequence of user actions comprises opening the user interface device, closing the user interface device, and opening the user interface device until the user interface device and the robotic grasper mate.

5. The surgical robotic system of claim 1 , wherein to determine that the user interface device matches and to transition the surgical robotic system to the teleoperation mode, the one or more processors are further configured to operate a state machine, the state machine comprising: The dematching open state is used to wait for the user interface device to be opened, and then switch to the dematching closed state in which the user interface device is opened and starts to be closed; The unmatching closed state is used to wait for the user interface device to be closed, and then switch to the matching state in which the user interface device is closed and begins to open; The matching state is used to wait for the user interface device to match with the robot gripper, and is converted to a matching completion state in which the user interface device matches the robot gripper; and The matching completed state is used when the user interface device is matched with the robotic gripper and when switching to the remote operation mode.

6. The surgical robotic system of claim 1, wherein the one or more processors are further configured to: determining that the surgical robotic system is in a pause mode, the pause mode being when the user interface device disengages control of the robotic grasper to pause a teleoperation session; receiving user actions to match the user interface device and the robotic gripper; and The surgical robotic system is converted to the teleoperation mode.

7. The surgical robotic system of claim 1, wherein the one or more processors are further configured to: Guidance is provided regarding the detected sequence of user actions.

8. The surgical robotic system of claim 1, wherein the one or more processors are further configured to: In response to determining that the user interface device matches the jaw angle or the grip force, an indication of a matching status is displayed.

9. The surgical robotic system of claim 5, wherein the operating state machine comprises, in response to the user interface device being matched to the robotic grasper, transitioning from a first state for waiting for the user interface device to be matched to the robotic grasper to a second state for when the user interface device is matched to the robotic grasper.

10. The surgical robotic system of claim 9, wherein the detected sequence of user actions that results in transitioning from the first state to the second state is when a user is squeezing a user interface device grasper link or a deformable surface.

11. The surgical robotic system of claim 1 , wherein the one or more processors are further configured to: An indication of a transition of the surgical robotic system to the teleoperation mode is displayed.

Images