Method and system for haptic feedback for a surgical robotic system

Abstract

A method performed by a surgical robotic system. The system determines a pose of a HID, determines a target pose of a surgical instrument based on at least one of the pose of the HID and one or more parameters of the robotic arm. The system provides a control command to adjust at least one of several joints of the robotic arm to move the surgical instrument into the target pose. The system determines a resulting pose of the surgical instrument and determines whether at least a portion of the robotic arm has encountered a limit or has collided with an object based at least partially on the pose of the HID, the target pose, or the resulting pose. The system provides haptic feedback through the HID based on whether the robotic arm has encountered the limit or has collided with the object.

Description

Atorney Docket No. : AUR6374WOPCT1Electronically FiledMethod and System for Haptic Feedback for a Surgical Robotic System RELATED APPLICATIONS

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 707,181, filed October 14, 2024 and U.S. Provisional Patent Application No. 63 / 707,187, filed October 14, 2024, which are herein incorporated by reference in their entirety.FIELD

[0002] Various embodiments of the disclosure relate generally to surgical systems, and more specifically to a surgical system for discrepancy-based haptic feedback through an input device. Other embodiments are also described.BACKGROUND

[0003] Minimally-invasive surgery, MIS, such as laparoscopic surgery, uses techniques that are intended to reduce tissue damage during a surgical procedure. Laparoscopic procedures typically call for creating a number of small incisions in the patient, e.g., in the abdomen, through which several surgical tools such as an endoscope, a blade, a grasper, and a needle, are then inserted into the patient. A gas is injected into the abdomen which insufflates the abdomen thereby providing more space around the tips of the tools, making it easier for the surgeon to see (via the endoscope) and manipulate tissue at the surgical site. MIS can be performed faster and with less surgeon fatigue using a surgical robotic system in which the surgical tools are operatively attached to the distal ends of robotic arms, and a control system actuates the arm and its attached tool. The tip of the tool will mimic the position and orientation movements of an input device as the latter is being manipulated by the surgeon. The surgical robotic system may have multiple surgical arms, one or more of which has an attached endoscope, and others have attached surgical instruments for performing certain surgical actions.Atorney Docket No. : AUR6374WOPCT1Electronically FiledSUMMARY

[0004] According to one embodiment of the disclosure, a method performed by at least on programmed processor of a surgical robotic system that includes a human interface device (HID) arranged to control a robotic arm including several joints and a surgical instrument, the method including: determining a pose of the HID; determining a pose of the surgical instrument based on at least one of the pose of the HID and one or more motion limits of the robotic arm; providing a control command based on the pose of the surgical instrument to adjust at least one of the joints of the robotic arm to move the surgical instrument; determining an actual pose of the surgical instrument resulting from the movement; determining whether a portion of the robotic arm has encountered a motion limit or has collided with an object based at least partially on the pose of the HID, the pose of the surgical instrument, or the actual pose of the surgical instrument; and providing haptic feedback through the HID based on whether the robotic arm has encountered the motion limit or a collision with the object.

[0005] In one embodiment, the method further including determining a target pose of the surgical instrument based on the pose of the HID, where determining the pose of the surgical instrument includes determining a limited target pose of the surgical instrument that is different than the target pose of the surgical instrument based on the target pose of the surgical instrument and the one or more motion limits. In another embodiment, determining the limited target pose of the surgical instrument includes adjusting the target pose of the surgical instrument such that the movement of the surgical instrument satisfies the one or more motion limits. In some embodiments, the one or more motion limits of the robotic arm includes a position limit of the robotic arm, a velocity limit of the robotic arm, an acceleration limit of the robotic arm, a velocity limit of a joint of the robotic arm, or a joint limit of the joint. According to another embodiment, determining whether the at least a portion of the robotic arm has encountered the motion limit or has collided with the object includes: responsive to determining that a difference between the target pose of the surgical instrument and the limited target pose of the surgical instrument exceeds a first threshold, determining that the robotic arm has encountered the motion limit; and responsive to determining that a difference between the limited target pose of the surgical instrument and the actual pose of the surgical instrument exceeds a second threshold, determining that the robotic armAtorney Docket No. : AUR6374WOPCT1Electronically Filed has encountered the collision with the object. In some embodiments, the first threshold is different than the second threshold.

[0006] In one embodiment, providing haptic feedback through the HID includes: responsive to the determining that the robotic arm has encountered the motion limit, causing one or more actuators of the HID to produce a first haptic effect; and responsive to the determining that the robotic arm has encountered the collision, causing the one or more actuators of the HID to produce a second haptic effect that is different than the first haptic effect. In another embodiment, the one or more motion limits includes a velocity limit, an acceleration limit, or a position limit, where determining the pose of the surgical instrument includes: determining a target pose of the surgical instrument based on the pose of the HID; determining that movement of a joint of the robotic arm to cause the surgical instrument to move from a current pose to the target pose is to cause at least one of the velocity limit, the acceleration limit, or the position limit to be exceeded; and adjusting the target pose such that movement of the joint ensures that the velocity limit, the acceleration limit, or the position limit is maintained.

[0007] In one embodiment, the method further including receiving position data from one or more encoders of at least one joint of the joints, where the actual pose is determined based on the position data. In another embodiment, the method further including receiving sensor data from one or more sensors of the surgical robotic system, where the actual pose is determined based on the sensor data. In some embodiments, the HID includes a grounded input device coupled to a surgeon console of the surgical robotic system, the grounded input device including an end user control coupled to the joints. In another embodiment, the pose of the HID includes a pose of the end user control.

[0008] According to another embodiment of the disclosure, a method performed by at least one programmed processor of a surgical robotic system that includes a HID arranged to control a robotic arm that includes one or more joints, the method including: receiving, using the HID, a target pose of the robotic arm; determining a limited target pose of the robotic arm based on the target pose and one or more motion limits of the robotic arm; causing the robotic arm to perform a movementAtorney Docket No. : AUR6374WOPCT1Electronically Filed based on the limited target pose; responsive to the movement of the robotic arm, receiving position data of the robotic arm, and determining an actual pose of the robotic arm; and providing haptic feedback through the HID based on at least one of a first discrepancy between the target pose and the limited target pose, or a second discrepancy between the limited target pose and the actual pose.

[0009] In one embodiment, providing haptic feedback through the HID includes: applying a first force through the HID when the first discrepancy is greater than a first threshold; and applying a second force through the HID when the second discrepancy is greater than a second threshold. In another embodiment, providing haptic feedback through the HID further includes applying a third force through the HID when first and second discrepancies exceed the first and second thresholds, respectively. In some embodiments, the third force includes a combination of the first and second forces. In another embodiment, the first threshold is different than the second threshold.

[0010] In one embodiment, determining the limited target pose of the robotic arm includes: receiving joint commands for the one or more joints, where the joint commands are produced to cause the robotic arm to perform the movement; and generating the limited target pose of the robotic arm by performing a forward kinematics function using the joint commands. In another embodiment, determining the limited target pose of the robotic arm includes: determining that the target pose of the robotic arm is to exceed the one or more motion limits; and generating the limited target pose of the robotic arm by adjusting the target pose of the robotic arm to satisfy the one or more motion limits.

[0011] In one embodiment, the one or more motion limits of the robotic arm includes a velocity limit, an acceleration limit, a position limit, a joint velocity limit, a joint acceleration limit, or a joint position limit.

[0012] According to another embodiment of the disclosure, a method including: detecting a movement of a HID configured to control a robotic arm; determining a command for moving the robotic arm based on the movement of the HID; causing the robotic arm to move based on the command; receiving position data of the robotic arm; and determining whether at least a portion of the robotic arm has encountered a knownAtorney Docket No. : AUR6374WOPCT1Electronically Filed limit or has encountered an unknown limit based at least partially on the movement of the HID, the command, or the position data of the robotic arm.

[0013] In one embodiment, the known limit includes at least one of a velocity limit, an acceleration limit, a position limit, or a joint limit of a joint of the robotic arm, and the unknown limit includes a collision with an object. In another embodiment, the method further including: determining, responsive to determining that the at least the portion of the robotic arm has encountered the known limit, that a motion limit of a several motion limits of the robotic arm has been exceeded based on the command; and in response, providing haptic feedback force through the HID according to the exceedance of the motion limit. In some embodiments, the method further including providing a notification indicating that the motion limit has been exceeded along with the provided haptic feedback force.

[0014] In one embodiment, determining the command includes: determining a target pose of the robotic arm based on the movement of the HID; and determining a limited target pose of the robotic arm as output of a movement model responsive to input based on the target pose of the robotic arm and the known limit. In another embodiment, determining whether includes: determining that the portion of the robotic arm has encountered the known limit responsive to a first discrepancy between the target pose and the limited target pose exceeding a first threshold; and determining that the portion of the robotic arm has encountered the unknown limit responsive to a second discrepancy between the limited target pose and an actual pose of the robotic arm based on the position data exceeding a second threshold. In some embodiments, the method further including determining an actual pose of the robotic arm based on the position data, where determining whether the at least the portion of the robotic arm has encountered the known limit or has encountered the unknown limit uses the movement of the HID, the command, and the actual pose of the robotic arm.

[0015] In another embodiment, the HID includes a grounded input device coupled to a surgeon console, the grounded input including an end user control coupled to several joints, where the command includes a pose of the HID resulting from the movement. In one embodiment, the method further including: providing, using an actuator, a first haptic force through the HID responsive to determining that the portionAtorney Docket No. : AUR6374WOPCT1Electronically Filed of the robotic arm has encountered the known limit; or providing, using the actuator, a second haptic force that is stronger than the first haptic force through the HID responsive to determining that the portion of the robotic arm has encountered the unknown limit or force.

[0016] According to another embodiment of the disclosure, a method including: determining, based on input of a human interface device (HID) configured to manipulate a robotic arm that includes a surgical tool and several joints, a target pose of the surgical tool; determining a first discrepancy based on a first difference between tool motion for moving the surgical tool in three-dimensional (3D) space into the target pose and a tool-level motion limit; determining a second discrepancy based on a second difference between joint motion of at least one joint that is to cause the surgical tool to move towards the target pose and a joint-level motion limit; determining a third discrepancy based on a combination of the first discrepancy and the second discrepancy; and providing haptic feedback through the HID based on whether at least one of the first, second, or third discrepancy exceeds a different threshold.

[0017] In one embodiment, providing haptic feedback includes providing different haptic feedback based on which one of the first, second, or third discrepancy exceeds a respective different threshold. In another embodiment, the method further including determining, for each discrepancy, a different haptic parameter, where providing the different haptic feedback includes providing, using one or more actuators of the HID and based on which of the first, second, or third discrepancy exceeds the respective different threshold, a different haptic force according to a respective different haptic parameter. In some embodiments, at least one different threshold or at least one different haptic parameter is user defined.

[0018] In one embodiment, the tool-level motion limit includes at least one of a velocity limit, an acceleration limit, or a position limit, where determining the first discrepancy includes determining whether a velocity for moving surgical tool into the target pose exceeds the velocity limit, an acceleration for moving surgical tool into the target pose exceeds the acceleration limit, or a position for the tool at the target pose exceeds the position limit. In another embodiment, the position limit includes a user- defined boundary within 3D space. In some embodiments, the first discrepancy is alongAtorney Docket No. : AUR6374WOPCT1Electronically Filed a direction in which the tool motion is to occur, where providing the haptic feedback includes providing a haptic force responsive to the first discrepancy exceeding a threshold along the direction.

[0019] In one embodiment, the at least one joint includes a first joint and the joint-level motion limit includes a first joint-level motion limit, where determining the second discrepancy includes: determining a first joint discrepancy based on a difference between a first joint motion of the first joint and the first joint-level motion limit; and determining a second joint discrepancy based on a difference between a second joint motion of a second joint of the joints and a second joint-level motion limit, where providing haptic feedback includes providing, using an actuator of the HID, a haptic force based on at least one of the first joint discrepancy orthe second joint discrepancy exceeding one or more thresholds. In another embodiment, the first joint-level motion limit is different than the second joint-level motion limit.

[0020] In one embodiment, the method further including that the second discrepancy exceeds a joint motion threshold, while the first discrepancy and the third discrepancy remain within respective threshold, where providing haptic feedback includes providing, using an actuator of the HID, a haptic force based on an exceedance of the second discrepancy and on a haptic parameter associated with the at least one joint. In another embodiment, the method further including: determining a limited target pose of the surgical tool based on the third discrepancy; causing the robotic arm to perform a movement based on the limited target pose; determining, based on sensor data of the robotic arm, an actual pose of the robotic arm responsive to the movement; and determining a fourth discrepancy based on the limited target pose and the actual pose, where providing haptic feedback includes, providing, using an actuator of the HID, at least one of: a first haptic force indicating that at least one of 1) the tool motion is limited due to the tool-level motion limit or 2) the joint motion is limited due to the joint-level motion limit, when the third discrepancy exceeds a first threshold, or a second haptic force indicating that the robotic arm has collided with an object when the fourth discrepancy exceeds a second threshold.

[0021] According to another embodiment of the disclosure, a system, an apparatus, or an electronic device as shown and as described herein. According toAtorney Docket No. : AUR6374WOPCT1Electronically Filed another embodiment of the disclosure, a method substantially as herein described. According to another embodiment of the disclosure, includes a processor configured to perform one or more operations as described herein. According to another embodiment of the disclosure, includes a non-transitory machine-readable medium that includes instructions which when executed by at least one processor causes a system to perform one or more operations as described herein.

[0022] The above summary does not include an exhaustive list of all embodiments of the disclosure. It is contemplated that the disclosure includes all systems and methods that can be practiced from all suitable combinations of the various embodiments summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the claims. Such combinations may have particular advantages not specifically recited in the above summary.Atorney Docket No. : AUR6374WOPCT1Electronically FiledBRIEF DESCRIPTION OF THE DRAWINGS

[0023] The embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to "an" or “one” embodiment of this disclosure are not necessarily to the same embodiment, and they mean at least one. Also, in the interest of conciseness and reducing the total number of figures, a given figure may be used to illustrate the features of more than one embodiment, and not all elements in the figure may be required for a given embodiment.

[0024] Fig. 1 shows a pictorial view of an example surgical system in an operating arena.

[0025] Fig. 2 shows an example of a user-side human interface device (HID) of a user console with which an operator may use to control a robotic component of the surgical system according to one embodiment of the disclosure.

[0026] Fig. 3 shows a patient-side robotic arm that includes a surgical tool of the surgical system according to one embodiment of the disclosure.

[0027] Fig. 4 is a block diagram of the surgical system for providing haptic force feedback based on modeled and unmodeled discrepancy between poses of the surgical tool according to one embodiment.

[0028] Fig. 5 is a flowchart of one embodiment of a process for providing discrepancy-based the haptic feedback force.

[0029] Fig. 6 illustrates modeled discrepancy verses modeled discrepancy between a target tool pose, a limited (or commanded) target tool pose, and an actual tool pose according to one embodiment.

[0030] Fig. 7 is a flowchart of one embodiment of a process for providing discrepancy-based haptic feedback force.

[0031] Fig. 8 is a flowchart of another embodiment of a process for providing discrepancy-based haptic feedback force.Atorney Docket No. : AUR6374WOPCT1Electronically FiledDETAILED DESCRIPTION

[0032] Several embodiments of the disclosure with reference to the appended drawings are now explained. Whenever the shapes, relative positions and other embodiments of the parts described in a given embodiment are not explicitly defined, the scope of the disclosure here is not limited only to the parts shown, which are meant merely for the purpose of illustration. Also, while numerous details are set forth, it is understood that some embodiments may be practiced without these details. In other instances, well-known circuits, structures, and techniques have not been shown in detail so as not to obscure the understanding of this description. Furthermore, unless the meaning is clearly to the contrary, all ranges set forth herein are deemed to be inclusive of each range’s endpoints.

[0033] Fig. 1 shows a pictorial view of an example (e.g., laparoscopic) surgical system (which hereafter may be referred to as “system”) 100 in an operating arena.The system 100 includes a user console 120, a control tower 131, and one or more surgical robotic arms 110 at a surgical robotic table (surgical table or surgical platform) 151. In one embodiment, the arms 110 may be mounted to a table or bed on which the patient rests as shown in the example of Fig. 1. In one embodiment, at least some of the arms 110 may be configured differently. For example, at least some of the arms may be mounted on a ceiling, sidewall, or in another suitable structural support, such as a cart separate from the table. The system 100 can incorporate any number of devices, tools, or accessories used to perform surgery on a patient 161. For example, the system 100 may include one or more surgical tools (instruments) 171 used to perform surgery (surgical procedure). A surgical tool 171 may be an end effector that is attached to a distal end of a surgical arm 110, for executing a surgical procedure.

[0034] Each surgical tool 171 may be manipulated manually, robotically, or both, during the surgery. For example, the surgical tool 171 may be a tool used to enter, view, or manipulate an internal anatomy of the patient 161. In an embodiment, the surgical tool 171 may include a grasper that can grasp tissue of the patient. In another embodiment, the surgical tool may include one or more cameras (e.g., an endoscopic camera), which may be configured to capture images of a surgical site in which one or more other surgical tools 171 may be used to perform one or moreAtorney Docket No. : AUR6374WOPCT1Electronically Filed surgical tasks. For instance, the camera may be arranged to have a field of view that includes a surgical site with one or more other surgical tools, which may be manipulated by an operator. The surgical tool 171 may be controlled manually by a bedside operator 180; or it may be controlled robotically, via actuated movement of the surgical robotic arm 110 to which it is attached. For example, when manually controlled an operator may (e.g., physically) hold a portion of the tool (e.g., a handle), and may manually control the tool by moving the handle and / or pressing one or more input controls (e.g., buttons) on the (e.g., handle of the) tool. In another embodiment, when controlled robotically, the surgical system may manipulate the surgical tool-based user input (e.g., received via the user console 120, as described herein).

[0035] Generally, a remote operator 190, such as a surgeon or other operator, may use the user console 120 to remotely manipulate the arms 110 and / or the attached surgical tools 171, e.g., during a teleoperation. The user console 120 may be located in the same operating room as the rest of the system 100, as shown in Fig. 1. In other environments however, the user console 120 may be located in an adjacent or nearby room, or it may be at a remote location, e.g., in a different building, city, or country. The user console 120 may include one or more components, such as a seat 119, one or more foot-operated controls (or foot pedals) 130, one or more human interface devices (HIDs) 140, and at least one display 150. In one embodiment, the user console may include less components. For example, the seat 119 may be separate from the user console. This may allow the user to sit on the seat and to move into a position that allows the user to use the console. In particular, the seat may be on casters that allow the seat to move on the floor.

[0036] The display is configured to display, for example, a view of the surgical site inside the patient 161. The display may be configured to display image data (e.g., still images and / or video). In one embodiment, the display may be any type of display, such as a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) display, etc. In some embodiments, the display may be a three- dimensional (3D) immersive display that is for displaying 3D (surgical) presentations. For instance, during a surgical procedure one or more endoscopic cameras may be capturing image data of a surgical site, which the display presents to the user in 3D. The display may include a viewer that includes one or two screens that may beAtorney Docket No. : AUR6374WOPCT1Electronically Filed configured to display stereoscopic images. In this case, the viewer of the display may include a contoured portion that may be arranged to come into contact with and contour to at least a portion of the operator’s face when the remote operator 190 moves towards and / or comes into contact with the display. Coming into contact and contouring with the operator’s face may prevent ambient light from interfering with the displayed surgical presentation, while the screen(s) of the display 150 may give the remote operator 190 a perception of a 3D space captured by one or more cameras. In one embodiment, the 3D display may be an autostereoscopic display that provides 3D perception to the user without the need for special glasses. As another example, the 3D display may be a stereoscopic display that provides 3D perception with the use of glasses (e.g., via active shutter or polarized).

[0037] In another embodiment, the display 150 may be configured to display at least one graphical user interface (GUI) that may provide informative and / or interactive content, to thereby assist a user in performing a surgical procedure with one or more instruments in the surgical system 100. For example, some of the content displayed may include image data captured by one or more endoscopic cameras, as described herein. In another embodiment, the GUI may include selectable UI items, which when manipulated by the user may cause the system to perform one or more operations. For instance, the GUI may include a UI item as interactive content to switch control between robotic arms. In one embodiment, to interact with the GUI, the system may include input devices, such as a keyboard, a mouse, etc. In another embodiment, the user may interact with the GUI using the HID 140. For instance, the user may manipulate the HID to navigate through the GUI, (e.g., with a cursor), and to make a selection may hover the cursor over a UI item and manipulate the HID (e.g., selecting a control or button). In some embodiments, the display may be a touch-sensitive display screen. In this case, the user may perform a selection by navigating and selecting through touching the display. In some embodiments, any method may be used to navigate and / or select a UI item.

[0038] As shown, the remote operator 190 is sitting in the seat 119 and viewing the user display 150 while manipulating a foot-operated control 130 and a handheld HID 140 in order to remotely control one or more of the arms 110 and the surgical tools 171 (that are mounted on the distal ends of the arms 110.)Atorney Docket No. : AUR6374WOPCT1Electronically Filed

[0039] In some variations, the bedside operator 180 may also operate the system 100 in an “over the bed” mode, in which the bedside operator 180 (user) is now at a side of the patient 161 and is simultaneously manipulating a robotically-driven tool (end effector as attached to the arm 110), e.g., with a handheld HID 140 held in one hand, and a manual laparoscopic tool. For example, the bedside operator’s left hand may be manipulating the handheld HID to control a robotic component, while the bedside operator’s right hand may be manipulating a manual laparoscopic tool. Thus, in these variations, the bedside operator 180 may perform both robotic-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 161.

[0040] During an example procedure (surgery), the patient 161 is prepped and draped in a sterile fashion to achieve anesthesia. Initial access to the surgical site may be performed manually while the arms of the system 100 are in a stowed configuration or withdrawn configuration (to facilitate access to the surgical site.) Once access is completed, initial positioning or preparation of the system 100 including its arms 110 may be performed. Next, the surgery proceeds with the remote operator 190 at the user console 120 utilizing the foot-operated controls 130 and the HIDs 140 to manipulate the various end effectors and perhaps an imaging system, to perform the surgery. Manual assistance may also be provided at the procedure bed or table, by sterile- gowned bedside personnel, e.g., the bedside operator 180 who may perform tasks such as retracting tissues, performing manual repositioning, and tool exchange upon one or more of the robotic arms 110. Non-sterile personnel may also be present to assist the remote operator 190 at the user console 120. When the procedure or surgery is completed, the system 100 and the user console 120 may be configured or set in a state to facilitate post-operative procedures such as cleaning or sterilization and healthcare record entry or printout via the user console 120.

[0041] In one embodiment, the remote operator 190 holds and moves the HID 140 to provide an input command to drive (move) one or more robotic arm actuators 170 (or driving mechanism) in the system 100 for teleoperation. The HID 140 may be communicatively coupled to the rest of the system 100, e.g., via a console computer system 160 (or host). The HID 140 can generate spatial state signals corresponding to movement of the HID 140, e.g., position and orientation of the handheld housing of the HID, and the spatial state signals may be input signals to control motions of the roboticAtorney Docket No. : AUR6374WOPCT1Electronically Filed arm actuators 170. The system 100 may use control signals derived from the spatial state signals, to control proportional motion of the actuators 170. In one embodiment, a console processor of the console computer system 160 receives the spatial state signals and generates the corresponding control signals. Based on these control signals, which control how the actuators 170 are energized to drive a segment or link of the arm 110, the movement of a corresponding surgical tool that is attached to the arm may mimic the movement of the HID 140. Similarly, interaction between the remote operator 190 and the HID 140 can generate, for example, a grip control signal that causes a jaw of a grasper of the surgical tool 171 to close and grip the tissue of patient 161.

[0042] The system 100 may include one or more user-side (or surgeon-side) HIDs 140, where respective control signals are generated for each HID that control the actuators and the surgical tool (end effector) of a respective arm 110. For example, the user console 120 may include two HIDs 140, a first (or left) HID arranged to be held and controlled by the operator’s left hand and a second (or right) HID arranged to be held and controlled by the operator’s right hand. In which case, the remote operator 190 may move the left HID 140 to control the motion of an actuator 170 that is in a one robotic arm, where the actuator responds by moving linkages, gears, etc., in that arm 110. Similarly, movement of the right HID 140 by the remote operator 190 controls the motion of another actuator 170, which in turn drives other linkages, gears, etc., of (e.g., another robotic component, such as a robotic arm of) the system 100. The system 100 may include a right arm 110 that is secured to the bed or table to the right side of the patient, and a left arm 110 that is at the left side of the patient. An actuator 170 may include one or more motors that are controlled so that they drive the rotation of a joint of the arm 110, to for example change, relative to the patient, an orientation of an endoscope or a grasper of the surgical tool 171 that is attached to that arm. Motion of several actuators 170 in the same arm 110 can be controlled by the spatial state signals generated from a particular HID 140. The HIDs 140 can also control motion of respective surgical tool graspers. For example, each HID 140 can generate a respective grip signal to control motion of an actuator, e.g., a linear actuator that opens or closes jaws of the grasper at a distal end of surgical tool 171 to grip tissue within patient 161.

[0043] In some embodiments, the communication between the surgical robotic table 151 and the user console 120 may be through a control tower 131, which mayAtorney Docket No. : AUR6374WOPCT1Electronically Filed translate user commands that are received from the user console 120 (and more particularly from the console computer system 160) into robotic control (or joint) commands that transmitted to the arms 110 on the surgical table 151. The control tower 131 may also transmit status and feedback from the surgical table 151 back to the user console 120. The communication connections between the surgical table 151, the user console 120, and the control tower 131 may be via wired (e.g., optical fiber) and / or wireless links, using any suitable one of a variety of wireless data communication protocols, such as BLUETOOTH protocol. Any wired connections may be optionally built into the floor and / or walls or ceiling of the operating room. The system 100 may provide video output to one or more displays, including displays within the operating room as well as remote displays that are accessible via the Internet or other networks. The video output or feed may also be encrypted to ensure privacy and all or portions of the video output may be saved to a server or electronic healthcare record system.

[0044] As described herein, this shows an example of the operator 190 using the user console 120 to control one or more robotic components, such as a robotic arm 110 and / or a surgical tool 171 coupled to a distal end of the robotic arm, of the surgical system 100. Specifically, this figure shows the operator 190 seated on the seat 119 in front of the user console 120, while controlling (or moving) one or more robotic components by manipulating control inputs, such as one or both of the HIDs 140 and / or the foot pedal(s) 130.

[0045] The user console 120 includes a base 201 to which the foot pedals 130 may be attached and a support structure 202 that may be mounted on the base 201 and extends vertically upward from the base. Coupled to the support structure 202 includes the display 150 and the HIDs 140. As shown, the HIDs may be coupled (mounted) on a bottom side of a structure (e.g., an arm rest) that is coupled to the support structure 202. In which case, the HID may be a wired or “grounded” input device that may be connected to the user console via a wired connection (e.g., to exchange data), which is in contrast to an “ungrounded’ input device that may be wirelessly connected to the user console (e.g., exchanging data via a wireless connection). As described herein, the HID 140 may include one or more joints that couple one or more links, which allow the HID to have one or more rotational degrees of freedom. In one embodiment, the HID may be grounded such that the joints of the HID may be arranged to hold their positionAtorney Docket No. : AUR6374WOPCT1Electronically Filed and / or orientation absent to an external applied force upon the HID. More about the configuration of the HID is described herein.

[0046] In another embodiment, the HIDs may be coupled to the user console in other configurations. For example, the HIDs may be separate from the user console, but may be communicatively coupled (e.g., through wired-connection). In which case, the HIDs may be mounted on another device (e.g., a surgical cart), which may be coupled to the user console.

[0047] As described herein, the operator 190 may be using the user console to control a robotic arm during a teleoperation. As shown, the user may be viewing a surgical workspace through the display 150, while operating one or more components of the surgical system by manipulating one or both of the HIDs 140 and / or the foot pedals 130.

[0048] Fig. 2 shows an example of the HID 140 of the user console 120 with which the operator 190 may control a robotic arm of the surgical system 100 during a surgical procedure. In particular, this figure shows the HID 140 that includes a support arm 200 and a gimbal 205, which may include one or more links coupled together by one or more joints. As described herein, the HID 140 may be a low -inertia such that as the user manipulates the HID, one or more joints may move (or be adjusted) such that the HID may be moved between positions (or poses) within a workspace surrounding the user console 120. The support arm 200 may be configured to couple the gimbal 205 to the user console 120, where the gimbal may be arranged to rotate about one or more axes and / or the arm support may be arranged to rotate about one or more axes to allow the user to move an end user control 220 of the gimbal within three-dimensional (3D) space. Movement of the end user control may allow the user to control (e.g., movement of) a robotic component, such as a robotic arm, of the surgical system. In one embodiment, each of the joints may be capable of rotating about and / or translating along one or more axes.

[0049] As shown, the HID 140 includes seven joints 215a-215g, which may provide the HID 140 with seven (rotational) degrees of freedom (DoF). In particular, the support arm 200 includes three links 210a-210c and three joints 215a-215c. In particular, link 210a may be coupled to link 210b, via joint 215b, and link 210b may beAtorney Docket No. : AUR6374WOPCT1Electronically Filed coupled to link 210c, via joint 215c. In one embodiment, the HID 140 may be arranged to couple to the support structure 202 at the joint 215a. The gimbal 205 may be coupled to the support arm 200 via joint 215d. The gimbal includes three links 210d-210f and four joints 215d-215g. Link 210d may be coupled to joint 215d and joint 215e. Link 210d may be coupled to link 210e, via joint 215e, and link 210e may be coupled to link 210f, via joint 215f. The gimbal 205 may also include an end user control 220 that may be coupled to link 21 Of via joint 215g.

[0050] As described herein, the HID may include seven DoFs. In one embodiment, one or more DoF may be redundant with respect to other DoF. For example joint 215d may be redundant with respect to joint 215f, since both joints allow the end user control 220 to rotate about a same axis (e.g., a Z-axis). In which case, the system 100 may be configured to provide null space motion in which one or more joints of the HID may be moved, while a pose of an end effector (e.g., the end user control 220) may remain unaffected. As described herein, this redundancy may allow the system to perform one or more secondary tasks by manipulating one or more joints, without affecting the end user control 220. For instance, as described herein, the gimbal 205 may include four joints 215d, 215e, 215f, and 215g, where the last three may provide a 3D range of motion, and the first may be redundant by having a redundant range of motion as joint 215f, whereby motion of joint 215d may occur without affecting (or causing motion) to the other three joints. In one embodiment, a secondary task may include an adjustment of one or more of redundant joints in order to provide the user with a more ergonomic position of the end user control 220, without affecting the pose of the end user control.

[0051] In one embodiment, the HID 140 may include more or fewer links and / or joints, which may provide more or fewer DoFs for the HID. For example joint 215a may be arranged to rotate the entire HID about an axis (e.g., the Z-axis) with respect to the link 210a. Conversely, joint 135b may be arranged to rotate link 210b (and the links / joints to which the link 210b may be coupled) about an orthogonal axis (e.g., Y-axis) to the axis that runs through link 210a and joint 215b.

[0052] In one embodiment, at least some of the joints may include one or more motors (or actuators) that may allow the joint to rotate around and / or translate alongAtorney Docket No. : AUR6374WOPCT1Electronically Filed one or more axes. In this way, the HID may provide a user with assisted movement in response to user input, such as a user applying an external force upon the end user control 220. In another embodiment, the HID 140 may be arranged to provide haptic feedback to the user. As described herein, the HID may be configured to control movement of a robotic component, such as a surgical tool coupled to a distal end of a robotic arm. As a result of this movement, external forces may be applied onto the surgical tool when the tool presses up against an object, such as pressing up against an abdominal wall of a patient during a surgical procedure. This force applied by the object may be haptically applied by the HID such that the operator may perceive the force. In one embodiment, the system 100 may be configured to determine whether an external force is being applied to the robotic arm, such as through one or more sensors (e.g., force sensors). In another embodiment, the system may determine that the robotic arm is pressing onto an object, based on a comparison between control commands from the HID and encoder data from one or more encoders of the robotic arm. Upon determining this external force, the system may be configured to provide haptic feedback to the user by controlling one or more motors of the HID to relay an applied force through the HID onto the user. In one embodiment, this applied force may be an opposite (and / or proportional) external force applied onto the robotic arm. More about providing haptic feedback force is described herein.

[0053] Fig. 3 shows a patient-side robotic arm 110 that includes the surgical tool 171 of the surgical system 100 according to one embodiment of the disclosure. The surgical arm includes ten links 303a-303j that are coupled together through nine joints 302a-302i, where each joint may be arranged to move a correspondingly coupled link by at least one of rotating the link about at least one axis and / or translating the link along at least one axis. For instance, each joint may include an actuator, which may move its corresponding joint based on user input (e.g., user commands) received through movement of the HID 140. In one embodiment, the robotic arm may include more or less joints and / or links.

[0054] The robotic arm may be coupled to a structure (such as the surgical table 151 or a surgical cart) at the most proximal joint 302a. Coupled to the most distal link 303j includes a tool drive 302 configured to control a surgical tool 171. In another embodiment, any type of end effector may be coupled to the tool drive, such as anAtorney Docket No. : AUR6374WOPCT1Electronically Filed endoscopic camera. Coupled to the tool drive 302 is a cannula 304 that may be inserted into a patient’s cavity (e.g., abdominal region), where the (e.g., end effector of the) surgical tool may be received through the cannula.

[0055] As described herein, the system 100 may be configured to control movement of the robotic arm based on user input through the HID 140. In particular, the system may be configured to cause one or more joints of the robotic arm to move in order to match motion of the HID. To match motion, the system may attempt to align (or match) a pose of the (e.g., end user control 220 of the) HID 140 with that of the surgical tool 171 of the robotic arm with respect to at least one reference frame. For instance, the system 100 may be configured to receive user input through the HID, where the user input may indicate one or more spatial state signals corresponding to movement of the HID, as described herein. From this input, the system may determine a pose of the HID, or more specifically a pose of the end user control of the HID, and may be configured to generate one or more control commands to cause one or more joints of the robotic arm to actuate in order for a pose of the surgical tool coupled to the robotic arm to match the pose of the end user control, as perceived by the user through the display 150 of the user console 120. The system may perform one or more conversions (or transformations) of the HID pose with respect to one or more reference frames into the pose of the tool 171 with respect to the reference frames. One reference frame may be with respect to the display. As described herein, the user may control the surgical instrument by manipulating the HID, while viewing the surgical instrument through the display 150 that may receive video images from one or more cameras. In which case, the system may define tool motion with respect to the camera to follow the HID motion with respect to the display. Thus, the system may convert a user command (e.g., HID pose based on position data from sensors 400) that may with respect to the display 150 to a target (or desired) tool pose with respect to a camera of the system 100. In one embodiment, the system may be configured to adjust a pose of the surgical tool according to movement (or changing poses of the HID) in real-time, whereby motion of the tool may track motion of the HID in real-time and during a surgical procedure. In one embodiment, the system may be configured to adjust a pose of the surgical tool according to movement (or changing poses of the HID) in real time (e.g.,Atorney Docket No. : AUR6374WOPCT1Electronically Filed during a surgical procedure). More about the HID controlling movement of the robotic arm is described herein.

[0056] Some surgical systems may be configured to provide haptic feedback force through a user interface device, such as the HID 140, based on forces exerted onto a robotic component, such as the robotic arm 110, that is being controlled with the HID. Haptic feedback may help a user feel the robot interacting with the environment, such as when a collision event occurs in which the robotic arm may collide or bump into an object that may be due to arm movement or movement of the object. Feedback force may also be provided when a limiting event occurs by pushing back against a user when the robotic arm is reaching (or has reached) a system limit in order to haptically alert the user of that limit. A limiting event may be due to one or more system motion limits (or parameters) being exceeded, such as when the user moves the HID faster than a velocity limit of robotic arm, thereby preventing the robotic arm from efficiently tracking the movements of the HID. For instance, once a velocity limit is reached, the motion of the robotic arm may trail the motion of the HID, since the user may continue to move the HID faster than the robotic arm can be moved. Another example limit may be a joint-level motion limit (or joint limit), which may be reached when the user attempts to move the robotic arm beyond its limits (or constraints), which may be defined by a specification of the joint or by actual physical limitations. One way for a system to determine whether to provide a haptic feedback force may include considering a total discrepancy (or difference) between a received user command through the HID and a pose of the robotic arm that results from movement due to the user command. In other words, the system may use the discrepancy between a user commanded (or target) pose of the surgical instrument coupled to the robotic arm and an actual pose resulting from the user command, calculate the haptic force based on stiffness and damping parameters, project force to one or more joints of the HID, and implement the haptic force on the HID.

[0057] Looking at this total discrepancy between the user-desired pose (or target pose) and the actual pose for calculating haptic force may have several drawbacks. For instance, as described herein, haptic force may be provided as a result of the robotic arm encountering a collision event or a limiting event (e.g., speed limit, joint limit, acceleration limit). Each event, however, may be the result of or causeAtorney Docket No. : AUR6374WOPCT1Electronically Filed different discrepancies between the target pose and the actual resulting pose. When using the total discrepancy, however, the discrepancies required for the different events may be combined together, and the system may use the same haptic parameters, even though the source of the haptic force may be different. These haptic parameters may include stiffness, damping, and / or a dead band (or threshold) whereby the system may not react or provide haptic force until the total discrepancy exceeds this threshold. As a result, the system may be incapable of differentiating between a collision event and a limiting event, since a determination of whether to exert haptic force may be based on whether the total discrepancy exceeds one dead band. Moreover, since the system would use the same haptic parameters regardless of the event, the user would feel the same haptic force for both events, which would not help the user differentiate between the events. Although using one dead band, regardless of which event is causing the haptic feedback force, may provide a simpler design, this may cause the system to be more or less sensitive to different events. For instance, a large shared dead band may be more preferrable for a speed limit but may reduce the sensitivity for providing haptic force from collision. For example, if the system were to use a shared dead band, such as 20 mm, and the discrepancy were due to a collision event, the surgical instrument would have to accumulate 20 mm of discrepancy (e.g., the user would have to move the HID 20 mm beyond the dead band) before the operator can feel the haptics caused by the collision. But if the dead band were to be decreased in order to be more sensitive to a collision, the user would feel a lot more haptic force during regular use because the speed required before haptic force is applied, due to the system reaching a speed limit, would be less due to the smaller dead band. Therefore, there is a need for a discrepancy-based haptic force feedback algorithm that is capable of differentiating between collision and limiting events and providing different haptic forces when those events occur.

[0058] The present disclosure provides a method and a system for providing haptic feedback force based on the source of discrepancy, as opposed to the total discrepancy. The system may determine a pose of the HID, which may be based on a user command resulting from the user moving the end user control of the HID. The pose of the HID may correspond to a user-desired target pose for the surgical instrument. The system may determine a “limited” target pose of the surgicalAtorney Docket No. : AUR6374WOPCT1Electronically Filed instrument, whereby the system may limit or adjust a desired pose of the surgical instrument based on the pose of the HID and the parameters of the robotic arm. These parameters may include motion limits, such as a velocity limit of the robotic arm, an acceleration limit of the robotic arm, and / or a joint limit of a joint of the robotic arm. In other words, if the pose of the HID would result in the robotic arm reaching or exceeding a speed limit, the system may automatically reduce the speed of the robotic arm and therefore may adjust the resulting pose of the surgical instrument due to the reduced speed. As a result, the system may be configured to produce a “modeled” discrepancy between the user-desired target pose and the limited target pose, by determining (or modeling) the effect of the system parameters (or limits) towards the user-desired target pose. The system may cause the robotic arm to move the surgical instrument according to the limited target pose. The system may determine an actual pose of the surgical instrument resulting from the movement. This determination may be based on position data of one or more sensors of the robotic arm, such as encoders. The system may be configured to produce an “unmodeled” discrepancy between the actual pose and the limited target pose, whereby this discrepancy may be due to external unknown (or unforeseeable) forces acting upon the tool, such as object collisions. The system may determine whether the robotic arm has collided with an object or has a limited target pose. In one embodiment, to make this determination, the system may determine whether the modeled discrepancy exceeds one threshold and / or whether the unmodeled discrepancy exceeds another threshold. The system may provide haptic feedback through the HID based on whether the robotic arm has encountered a limit and / or has collided with an object. As a result, the system may be capable of differentiating between known (or modeled) limits and unknown (or unmodeled) forces when providing haptic feedback, which may allow the system to use different parameters, such as different thresholds and therefore different sensitivities for both events.

[0059] Fig. 4 is a block diagram of the surgical system for providing haptic feedback force based on modeled and unmodeled discrepancy between one or more poses of the surgical tool according to one embodiment. The system 100 includes the user console 120, the control tower 131, and a robotic arm 110. In one embodiment, the system may include more or less elements, such as those shown in Fig. 1. The userAtorney Docket No. : AUR6374WOPCT1Electronically Filed console 120 includes the HID 140 and a controller 415, which may be a part of the console computer system 160. Although shown as being a part of the control tower 131, the controller 420 may be a part of another electronic device. In another embodiment, one or both controller may be a part of an electronic server, which may be configured to perform at least some of the operations described herein. In yet another embodiment, at least some of the operations described herein may be performed by one or both controllers. For example, the operations performed by the controller 420 may be performed by the controller 415 to perform the discrepancy-based haptic feedback. In which case, the user console may be configured to produce joint commands according to a limited target pose, where the joint commands may be provided to one or more robotic arms, as described herein.

[0060] The HID 140 includes one or more sensors 400 and one or more actuators 405. As described herein, at least some joints of the HID may include an actuator that may be arranged to rotate and / or translate a corresponding joint in response to one or more control signals (or joint commands). In another embodiment, the actuators may also be arranged to provide (or exert) a haptic or tactile force feedback in response to one or more haptic signals. Haptic feedback force may include any type of force that may be felt by the user such as an applied resistance, a vibration, a movement away from the user motion (e.g., a pull), a movement towards a user motion (e.g., a push), etc. The sensors 400 may be arranged to provide position data of the HID. For instance, the sensors may include encoders that may be a part of one or more joints of the HID, which may produce encoder data (as position data), responsive to movement of at least a portion of the HID, such as the end user control 220.

[0061] The robotic arm 110 includes one or more actuators 170, a tool 171, and one or more sensors 425. As described herein, the actuators may be a part of and / or cause one or more joints of the arm to move based on joint commands. This movement may cause corresponding joints to rotate and / or translate about one or more axes. The sensors may be configured to produce position data of the robotic arm 110. For instance, the sensors may include encoders, each of which may be configured to measure a joint position of a corresponding joint. As described herein, this position data may be used to determine an actual tool pose of the tool 171 coupled to the robotic arm 110.Atorney Docket No. : AUR6374WOPCT1Electronically Filed

[0062] Turning back to the user console 120, the controller 415 of the console may include several operational blocks, such as the HID / target tool pose generator 430, the velocity and acceleration limiter 435, the range limiter 440, the gravity compensation 455, the limited target tool pose generator 460, the haptics generator 470, and the actual tool pose generator 465. In one embodiment, one or more of these operational blocks may be optional, such as the gravity compensation and / or the range limiter.

[0063] The HID / target tool pose generator 430 may be configured to receive position data from the HID 140 responsive to the HID receiving user input. In particular, the position data may be produced by one or more of the sensors 400, responsive to the user moving the (e.g., pose of the end user control 220 of the) HID. The position data may be produced by one or more sensors 400 of the HID 140 as the user moves the end user control 220. The generator 430 may be configured to receive a user command for moving the robotic arm based on the movement of the HID. The user command may include to a pose of the HID that is the result of the user movement. The HID pose may be a six degrees of freedom (6 DoF) pose of the end user control 220 (e.g., in three-dimensional (3D) Cartesian space) that is being held by the user while moving the HID. In one embodiment, the generator 430 may be configured to convert the position data into the end user control pose by using a forward kinematics algorithm.

[0064] The HID / target tool pose generator 430 may be configured to determine a target (or user-desired) tool pose of the surgical tool 171 that may be coupled to the robotic arm 110. For instance, the generator 430 may be configured to convert the HID pose to a target tool pose for the surgical tool 171. The system may perform one or more transformations such that the pose of the tool 171 and the HID pose are the same with respect to one or more reference frames. As described herein, the user may control the surgical instrument by manipulating the HID, while viewing the surgical instrument through the display 150 that may receive video images from one or more cameras. In which case, the system may convert the HID pose into the desired tool pose such that tool motion with respect to the camera follows the HID motion with respect to the display. Thus, the system 100 may convert a user command (or HID pose based on position data from the HID) that may with respect to the display 150 to a desired toolAtorney Docket No. : AUR6374WOPCT1Electronically Filed pose with respect to a camera of the system 100. The HID / tool pose generator 430 may be configured to determine the actual (current) tool pose of the (tool 171 of the) robotic arm 110 based on sensor data from the sensors 425. For instance, the generator 430 may perform a forward kinematics process to convert the sensor data into a 6 DoF tool pose of the tool 171. In another embodiment, the generator 430 may receive the current tool pose as output of the generator 465. More about generator 465 is described herein.

[0065] The system 100 may be configured to cause the (e.g., tool 171 of the) robotic arm 110 to perform at least one movement based on the user command, which may be based on the target tool pose, and one or more motion limits of the robotic arm. In particular, the system may provide the user command to one or more limiters in order to determine whether the user command exceeds one or more motion limits. If so, the system may adjust the user command in order to satisfy those limits, and may provide the adjusted user command to the robotic arm to change the arm’s pose.

[0066] Turning to the velocity and acceleration limiter 435, this limiter may be configured to receive the target tool pose estimate from the generator 430 and may be configured to enforce velocity and / or acceleration limits on the target tool pose. In particular, the limiter may receive a current tool pose of the surgical instrument and the target tool pose, and may be configured to modify the target tool pose to ensure that one or more parameters of the robotic arm are satisfied. In one embodiment, the parameters, which may be tool-level motion limits that indicate motion limits upon the surgical tool 171 (and / or robotic arm 110) due to commanded movement from its current position to the target tool pose. These motion limits may include a velocity limit and / or an acceleration limit of the surgical instrument (e.g., in Cartesian space). In one embodiment, the limiter may modify the target tool pose by simulating the surgical tool’s movement between the current pose to the target pose, while velocity and acceleration is saturated. The simulated movement may be used to determine the tool’s trajectory from the current pose to a limited target pose that stays within the velocity and / or acceleration limits of the tool. For example, this limited target pose may move a shorter distance than the target pose when the user command through the HID occurs at a velocity that exceeds the tool’s velocity limit. Similarly, the target tool pose may be adjusted by adjusting one or more distances along which the arm is to move from its current pose when an acceleration exceeds the tool’s acceleration limit.Atorney Docket No. : AUR6374WOPCT1Electronically Filed

[0067] The range limiter 440 may be configured to receive the limited target tool pose from the velocity and acceleration limiter 435 and may be configured to limit movement the surgical tool based on the target tool pose. In one embodiment, the range limiter 440 may receive the target tool pose without having been limited by the limiter 435 when the velocity and / or acceleration limits are to be maintained. The range limiter 440 may be configured to determine whether the target tool pose exceeds a position limit of the surgical tool and / or robotic arm 110. As described herein, this position limit may be predefined (e.g., based on a specification of the system 100), and / or may be based on a user-specific setting that may define a boundary in which the user wishes the robotic arm to stay within. For instance, the system may be configured to define the boundary as a workspace that the surgical tool may operate within a surgical site. This boundary may be a 3D region of motion. This may ensure that motion is limited within a confined region, which may be the case during a constrained surgical procedure.

[0068] An end effector’s overall reachable workspace and position / orientation capabilities within 3D space may depend upon the range of motions (or joint motion limits) of the joints that articulate the movement of the surgical tool Some instruments may include one or more joints that allow the tool to actuate within a joint space range. Some motion, such as a pure roll motion, may cause a tool’s range of motion to vary due to different joints having different ranges of motion rotating in space, thereby changing their axes of rotation with respect to a global reference. As a result, the range limiter 440 may define an area with respect to a center of one of the joints in which movement of the joints must stay within, where the area may define a range of motion for the surgical tool. In one embodiment, this area may be cone shaped, where a joint may be at the apex (or vertex) of the cone. As a result, the limiter may be configured to determine whether the target tool pose may cause the surgical tool to extend beyond this cone-shaped area based on the target tool pose. If so, the range limiter 440 may be configured to further limit the movement of the tool, by limiting the target tool pose to stay within this confined area. In one embodiment, the range limiter may be an operational block, which may only be employed with specific types of surgical tools, such as a surgical grasper (e.g., needle driver) that may include one or more joints that allow a grasper to rotate about one or more axes. For instance, a grasper may include at least three joints, thereby allowing the grasper to actuate within three dimensions (e.g.,Atorney Docket No. : AUR6374WOPCT1Electronically Filed yaw, pitch, and roll), where the predefined range may be arranged about a most distal joint (e.g., a yaw joint) of the tool (e.g., the joint before the grasper end effector).

[0069] The controller 415 may be configured to generate a limited target pose of the target pose of the surgical tool 171 by applying limiting operations described herein. For instance, the limiters 435 and / or 440 may determine that the target pose of the robotic arm is to exceed one or more motion limits, and may be configured to generate a limited target pose by adjusting the target pose to satisfy the motion limits. This limited target pose may be used by the system to move the robotic arm, as described herein.

[0070] The controller 420 may be configured to receive control data from the user console 120. This control data may include a limited target tool pose, based on any modification by either (or both) limiters 435 and 440. In this case, however, when the target tool pose would not result in a modification by either limiter, e.g., due to the target pose not offending or exceeding a velocity limit, acceleration limit, and / or a position limit of the tool, the controller 420 may receive the original target tool pose. The controller 420 includes an inverse kinematics 445 and a joint limiter 450. The inverse kinematics 445 may be configured to convert the tool pose joint tool space (e.g., Cartesian space) into joint space. In particular, the inverse kinematics may be configured to determine one or more joint motions of one or more joints of the robotic arm as output of an inverse kinematics function responsive to input based on the (limited) tool pose. In one embodiment, the joint motions may include joint positions, joint velocities, and / or joint accelerations. Thes joint motions may be with respect to time. For instance, joint positions may indicate joint movement from the joint’s current positions to new joint positions, which may be based on the limited target tool pose, with respect to time.

[0071] The joint limiter 450 may be configured to receive the one or more joint motions (e.g., joint positions) from the inverse kinematics, and may be configured to determine whether the joint motions will cause one or more joints to exceed one or more joint-level motion limits. In one embodiment, these limits may be predefined (e.g., due to a specification of the joint and / or physical limits). In another embodiment, one or more limits may be user-defined. In one embodiment joint limits may include aAtorney Docket No. : AUR6374WOPCT1Electronically Filed joint position limit that may be a limit to the joint’s motion range. In which case, the joint limiter may determine whether a joint position towards which a joint is to move based on the received control data may cause a joint to exceed its joint limit (or threshold). For example, a joint of the robotic arm may only be capable of rotating one radian from its current position. The new joint position received from the inverse kinematics may result in the joint moving more than one radian. As a result, the joint limiter 450 may be configured to limit joint movement when the joint position exceeds the joint limit. In one embodiment, the joint limiter may set the joint position to the joint limit when it is to be exceeded. In another embodiment, the joint limiter may set the joint position to be less than the joint limit (e.g., by a threshold amount) in order to prevent the joint from reaching its limit.

[0072] As another example, the joint limiter may determine whether movement of a joint from its current joint position to the new joint position may exceed a joint velocity limit that may limit the speed (and / or direction) towards which a joint may move and may exceed a joint acceleration limit. For instance, the system 100 may be configured to determine the velocity at which the joint is to move based on the distance the joint is to travel from its current position to the new position over a period of time. If the joint limiter determines that the joint is to exceed that joint velocity limit, the limiter may adjust the new joint position to accommodate the velocity limit. As another example, the joint limiter may determine whether the acceleration at which the joint is to move may exceed a joint acceleration limit that may limit the acceleration at which a joint may begin to move towards a desired joint position and / or the deceleration at which a joint may begin to slow down when reaching the desired joint position. In particular, to reduce the acceleration, the system may adjust the desired joint position, which may therefore adjust the acceleration at which the joint is to move from its current position to the adjusted joint position.

[0073] The control tower 131 may be configured to provide the joint motions (e.g., joint positions), of which some may be modified (limited) by the joint limiter 450, as joint commands to the robotic arm to cause one or more of the actuators 170 to move their corresponding joints. The sensors 425 may be encoders that are configured to measure the position of one or more joints of the robotic arm 110 and provide the position data to the user console 120. As a result, the system may repeat at least someAtorney Docket No. : AUR6374WOPCT1Electronically Filed of the operations described herein in order to allow user movement of the HID 140 to cause corresponding movement of the tool 171 of the robotic arm 110, in real time.

[0074] Turning back to the user console 120, the actual tool pose generator 465 may be configured to receive the position data from the sensors 425 and may be configured to generate the actual tool pose of the tool 171 that is responsive to the received user command. In particular, the system 100 may receive the position data responsive to movement of the surgical tool 171 based on the user command and / or the motion limits, as described herein. In one embodiment, the actual tool pose may be with respect to the camera frame, to which the target and limited target tool poses may be with respect as well. As described herein, the user may want the resulting pose of the tool 171 to be the same as the HID pose (with respect to one or more reference frames), but the actual tool pose may be different. This difference may be the result of foreseeable limits (e.g., velocity limits) and / or unforeseeable forces, such as collisions, which may prevent the target pose of the tool matching the pose of the HID. As a result, the final actual pose of the tool may not correspond to the HID pose. Therefore, the actual tool pose generator 465 may use the position data to generate the final pose of the tool that may be affected by these limits and / or forces. In one embodiment, the actual tool pose generator may generate the actual pose output of a forward kinematics function that may be responsive to input based on the position data.

[0075] As described herein, the system may be configured to provide haptic feedback force based on sources of discrepancy between the target tool pose of the tool 171 and its actual tool pose. The system may differentiate between a collision event and a limiting event due to different discrepancies in order to provide haptic forces. To perform at least some of the operations, the controller 415 may include a limited target tool pose generator 460 and a haptics generator 470. The limited target tool pose generator 460 may be configured to receive data from the controller 420 and may be configured to determine a limited target tool pose. This data may include one or more of the joint commands from the joint limiter 450, which may be provided to the robotic arm 110 in order to cause one or more joints to move, as described herein. The generator 460 may use the joint commands to produce the limited target pose using forward kinematics. In another embodiment, the generator 460 may use other data toAtorney Docket No. : AUR6374WOPCT1Electronically Filed produce the limited pose, such as data from one or more of the other limiters 435 and / or 440.

[0076] In another embodiment, the pose generator 460 may be configured to receive the target tool pose from the generator 430 and may be configured to determine a limited target tool pose based on the target tool pose and / or a current tool pose (e.g., based on data from the actual tool pose generator 465) of the tool 171. In particular, the generator 460 may be configured to determine a limited target pose of the surgical instrument based on the target tool pose and the one or more motion limits. For instance, the generator 460 may be configured to adjust the target tool pose such that movement of the surgical instrument (e.g., by one or more joints of the robotic arm 110) satisfies at least one of the motion limits described herein. In particular, the generator 460 may estimate modifications that may be performed onto the target tool pose by one or more of the limiters 435, 440, and / or 450. In which case, the generator 460 may be configured to determine the limited target pose by modifying the target tool pose based on one or more parameters of the robotic arm 110, which may include a velocity limit of the robotic arm, an acceleration limit of the robotic arm, a position limit of the robotic arm, a velocity limit of a joint of the robotic arm, and / or a joint position limit of a joint of the robotic arm. In one embodiment, the generator 460 may include a (predefined) model that may be configured to estimate the limited target tool pose by taking into account the parameters. In some embodiments, the model may be a machine learning (ML) model that may be trained using the parameters. As a result, the generator 460 may determine the limited target pose, which may be different than the target pose of the tool determined by the generator 430 based on the original target pose and one or more parameters. More specifically, the limited target pose may be different if the movement of the robotic arm in order to reach the target pose exceeds a threshold of a corresponding parameter.

[0077] The haptics generator 470 may be configured to determine whether the (e.g., tool 171 of the) robotic arm 110 may be encountering a collision event and / or a limiting event based on discrepancies of the tool pose. The generator may be configured to receive the target tool pose from the generator 430, the limited target tool pose from the generator 460, and the actual tool pose from the generator 465, and may be configured to determine whether at least one of a collision event, a limiting event, orAtorney Docket No. : AUR6374WOPCT1Electronically Filed a combination of both events, is occurring based on discrepancies between the poses. For instance, the generator 470 may determine the modeled discrepancy based on a difference between the target tool pose and the limited target tool pose. In one embodiment, this difference may include a vector that indicates a difference between the two poses. For example, the vector may be a 3D vector that indicates differences in translational (e.g., Cartesian) coordinates or in rotational coordinates (e.g., roll, pitch, yaw). In another embodiment, the vector may be a 6D vector that indicates differences in translational and rotational coordinates. As described herein, this difference may be modeled or foreseeable, since the difference between the two poses may be based on one or more limits that may be known or foreseeable by the system 100 based on the physical or defined characteristics of the system. Although these limits may be know, the user may inadvertently exceed the limits during normal use, such as attempting to move the surgical tool 171 faster than a velocity limit due to the speed at which the user is moving the HID. In one embodiment, this modeled discrepancy may be derived by a predefined algorithm (stored in memory of the system 100), which may be based on a mathematical model that may be configured to output discrepancy (due to a limited tool pose) based on system requirements and physical limits, as described herein. In one embodiment, the modeled discrepancy may include a relative transformation (e.g., 6 DoF pose) based on the difference. The generator 470 may determine the unmodeled discrepancy based on a difference between the limited target tool pose and the actual tool pose. This discrepancy may be unmodeled since any difference between the actual pose and the limited pose was not considered during the production of the limited pose, and may therefore be based on external forces that were unforeseeable, such as a collision event.

[0078] The haptic generator 470 may determine which event may be occurring based on whether either discrepancy exceeds one or more thresholds. As described herein, some systems may look at the total discrepancy to determine whether to provide haptic force. To do this, a system compares the total discrepancy between a desired tool pose and the resulting tool pose to a threshold. The threshold may be a deadband, whereby the system may not provide haptic feedback force until the discrepancy exceeds the dead band. This threshold, however, may result in the system being overly sensitive to some events or being less sensitive to others, which may result in to littleAtorney Docket No. : AUR6374WOPCT1Electronically Filed haptic force or too much. The present system overcomes such issues by including multiple thresholds for the discrepancies. For instance, the haptics generator 470 may determine whether a limiting event is occurring based on the modeled discrepancy exceeding a first threshold, or whether a collision event is occurring based on the unmodeled discrepancy exceeding a second threshold, which may be different than the first threshold.

[0079] In one embodiment, the thresholds may be distances that may be compared to one or more distances associated with the discrepancies. For example, each discrepancy may be a relative transformation that indicates one or more translational and / or rotational distances between two poses, as described herein. As a result, the threshold may be a translational threshold or a rotational threshold. In another embodiment, the haptics generator may combine one or more translational and / or rotational components of the discrepancies to determine a single error value (or metric), which may be compared to a corresponding threshold.

[0080] As described herein, the thresholds used to determine whether a collision event or a limiting event occurs may be different. For instance, since a collision event may result in the robotic arm stopping completely, the collision threshold may be relatively small compared to the limit threshold. As a result, a small collision threshold allows the system to be sensitive to collisions, while the limit threshold may be higher in order to avoid being overly sensitive during regular use.

[0081] The haptics generator 470 may be configured to provide different haptic feedback force based on which event has occurred. In particular, the generator may be configured to apply similar or different haptic parameters based on the detected event. Haptic parameters may include stiffness and damping, as described herein. Thus, the generator may be configured to use a first stiffness value for a collision event and use a second, different, stiffness value for a limiting event, which may provide different haptic forces. The different forces may alert the user of which event has occurred, and as a result the user may act accordingly. For instance, in the case of a collision event the user may pause HID motion, or in the case of a limiting event the user may move the HID in an opposite direction to avert the limiting event. Thus, the system may be configured to provide, using one or more actuators 405 of the HID 140, a first hapticAtorney Docket No. : AUR6374WOPCT1Electronically Filed force through the HID responsive to determining that a limiting event has occurred (e.g., that a portion of the robotic arm has encountered a known limit), or may provide a second haptic force that may be different than the first force (e.g., stronger than the first force) responsive to determining that a collision event has occurred (e.g., that a portion of the arm has encountered an unknown limit or force). The second force may be stronger to alert the user to stop movement of the tool or to reverse the movement in order to resolve the collision event. In one embodiment, the stiffness of for the collision event may be higher, while the collision limit threshold may be smaller than the limiting event threshold, which may make haptic force more sensitive to collisions, while also providing the user better (or more apparent) force feedback when a collision occurs.

[0082] In one embodiment, the thresholds and / or haptic parameters may be predefined. In another embodiment, the system may be configured to adapt one or more thresholds and / or adapt one or more haptic parameters. For instance, the system may keep track of how many events occur during a surgical procedure, and may be configured to adapt thresholds in order to make collision and / or limiting events more sensitive. In other words, upon determining that a number of collision events have exceeded a threshold, the system may reduce the collision threshold in order to alert the user sooner that a collision has occurred, thereby reducing the amount of unmodeled discrepancy needed before haptics force is provided to the user. Similarly, the system may adjust haptic parameters by increasing the stiffness as the number of collisions increase. In another embodiment, the thresholds and / or haptic parameters may be user- defined. For example, the user may provide user input through a GUI displayed on the display 150 for setting the thresholds and / or haptic parameters, such that the user may define the sensitivity at which haptics may be provided based on whether a limiting event occurs or a collision event occurs.

[0083] As described thus far, the haptics generator 470 may be configured to provide haptics based on whether modeled and / or unmodeled discrepancies exceed one or more thresholds. With respect to modeled discrepancy, the haptics generator 470 may be further configured to differentiate between one or more exceeded motion limits, and therefore provide different haptics parameters for those limits. In particular, the haptics generator may be configured to determine whether a limiting event occursAtorney Docket No. : AUR6374WOPCT1Electronically Filed based on the modeled discrepancy, and may be further configured to determine which of one or more motion limits are exceeded based on the target motion of the robotic arm, and may be configured to apply different haptics parameters based on the exceeded limits. Each one of these motion limits may be associated with a particular haptics parameter (e.g., stiffness value) and / or a limit threshold. As a result, upon determining that a velocity limit discrepancy exceeds a threshold, the system may apply haptics according to a corresponding stiffness, which may be different than when an acceleration limit discrepancy is exceeded. As a result, the system may provide different haptic forces (e.g., force sensitivities) according to different limit exceedances based on applied corresponding stiffness values.

[0084] The gravity compensator 455 may be configured to provide additional commands (or control signals) to one or more actuators 405 to provide gravity compensation to one or more joints. For instance, the gravity compensator may be configured to determine gravity-compensating torque that is to be applied by one or more joints based on the joint’s current position and / or based on other robotic characteristics, such as the arm’s mass and kinematics. In which case, the commands from the gravity compensator may be added to those commands produced by the haptics generator to drive one or more actuators to exert force.

[0085] As a result of the haptic feedback force operations described herein, the system may improve the sensitivity for detecting collisions without changing the feeling for regular driving (e.g., by having the dead band for modeled discrepancy being greater than that of the unmodeled discrepancy). In addition, the decision of whether a collision event or a limiting event has occurred may be based on existing sensors (or encoders) of the surgical system, without the need of using any additional (or external) sensors to sense the collision. For instance, the system may be configured to detect a collision event without any sensors, such as proximity sensors and image sensors (e.g., cameras).

[0086] Fig. 5 is a flowchart of one embodiment of a process 500 performed by the system 100 for providing discrepancy-based haptic feedback force based on whether a collision event or a limiting event has occurred. The process 500 may be at least partially performed by one or more components of the system 100, such as theAtorney Docket No. : AUR6374WOPCT1Electronically Filed controller 415 and / or the controller 420. The system 100 begins with determining a pose of the HID (at block 505). Specifically, the system may receive a user command through the HID as one or more detected movements of the HID for controlling one or more robotic components, such as the robotic arm 110. In particular, the movement may be of the end user control 220, while being held by the user, where the system may detect the movement based on sensor data of one or more sensors 400 that may be arranged to detect changes to one or more joints of the HID. For instance, the system may receive position data of one or more sensors of the HID that may changes to joint positions of one or more joints of the HID, and may use the position data to determine the pose of the HID, as described herein.

[0087] The system determines a target pose of the surgical instrument based on the pose of the HID (at block 510). The system 100 may set the target pose of the surgical instrument to be the pose of the HID, with respect to one or more reference frames. In particular, the system may define the pose of the instrument with respect to a frame of reference to a camera of the system 100 to correspond to a determined pose of the end user control 220 of the HID with respect to a frame of reference to the display 150. In which case, the target tool pose may be the same as the HID pose. The system determines a limited target pose of the surgical instrument based on the target pose of the surgical instrument and one or more parameters of the robotic arm (at block 515). As described herein, the parameters may include limits of the tool, such as a velocity limit, an acceleration limit, and / or a range limit. For instance, the limiter 435 may modify the target pose to ensure that the movement of the tool between a current pose and the target pose satisfies velocity and / or acceleration limits (e.g., in Cartesian space). For the ranging limit, the limiter 440 may be configured to ensure that a roll motion of the tool is feasible within a bounded area with respect to one or more joints of the tool. The parameters may also include joint-motion limits of one or more joints of the robotic arm 110, such as a joint position limit, a joint velocity limit, and / or a joint acceleration limit. Joint commands associated with the target (or limited target) pose may be modified to ensure that the motion limits, as described herein, are maintained (e.g., not exceeded). As a result, the system 100 may cause the robotic arm 110 to perform a movement, by moving one or more joints according to joint motion commands, based on a limited target pose due to adjustments performed by one orAtorney Docket No. : AUR6374WOPCT1Electronically Filed more limiters, as described herein. In which case, the system 100 may determine whether a movement of a joint of the robotic arm to cause the surgical instrument to move from a current pose to a desired pose is to cause one or more joint limits to be exceeded, and in response may adjust the desired tool pose such that movement of the joint ensures that the limits are not exceeded. As a result, when at least one of the limits is to be exceeded due to the target pose of the surgical instrument, the limited target pose may be different than the target pose, due to one or more modifications, such as a velocity modification, an acceleration modification, and / or a position modification.

[0088] The system determines one or more joint commands to adjust at least one joint of the robotic arm to move the surgical instrument into the limited target pose (at block 520). The system 100 provides the joint commands to one or more actuators of the robotic arm to cause the surgical instrument to move (at block 525). The system determines an actual pose of the surgical instrument (at block 530). In particular, the system 100 may receive sensor data from one or more sensors 425 and may use the sensor data to determine the pose of the surgical instrument resulting from the joint commands.

[0089] The system 100 determines whether a difference between the target pose of the surgical instrument and the limited target pose of the surgical instrument exceed a first threshold (at decision block 535). In particular, the system determines whether a modeled discrepancy (e.g., first discrepancy) due to modeled modifications of the target tool pose according to one or more parameters exceeds a threshold or dead band. For example, when the parameters include a velocity limit of the surgical instrument, the system determines whether a velocity of the instrument from a current pose to the target pose would exceed the velocity limit. If so, the system 100 may modify (or limit) the target pose in order for the velocity of the surgical instrument to satisfy the velocity limit. This modification may result in the discrepancy, as described herein. If so, the system 100 may provide a first haptic feedback force through the HID (at block 540). Thus, responsive to determining that the robotic arm has encountered a limiting event, the system causes one or more actuators 405 of the HID 140 to produce a first haptic effect.Atorney Docket No. : AUR6374WOPCT1Electronically Filed

[0090] The system 100 determines whether a difference between the limited target pose of the surgical instrument and the actual pose of the surgical instrument exceed a second threshold (at decision block 545). The system determines whether an unmodeled discrepancy (e.g., a second discrepancy) due to unforeseeable external forces has adversely affected the commanded pose of the surgical instrument. If so, the system may provide a second haptic feedback force through the HID (at block 550). Thus, responsive to determining that the robotic arm has encountered a collision, the system causes the actuators of the HID to produce a second haptic effect, which may be different to the effect provided when the limiting event was detected. In one embodiment, the haptic feedback forces may be provided to exert force in one or more opposite directions along which the discrepancies lie. For instance, when the discrepancy is in a direction along a horizontal axis (e.g., X-axis) of the surgical instrument, the system 100 may provide the haptic feedback force in an opposite direction along a corresponding axis of the end user control 220. In another embodiment, the exerted haptic feedback force may be based on the discrepancy. For example, the haptic force may increase as either difference exceeds one or more threshold. As an example, haptic force may be equal to a stiffness parameter multiplied by how much a discrepancy exceeds one or more thresholds. Thus, the applied force may be a function of one or more stiffness parameters and one or more corresponding discrepancies. This may ensure that the system applies increasing haptic force in order to alert the user.

[0091] As described thus far, the system 100 may provide different haptic feedback based on whether a limiting event or a collision event occurs. As a result, the system may be configured to be sensitive to one or both events by adjusting one or more haptic parameters, such as stiffness. In one embodiment, when both events occur, the applied force may be greater than that of whether the individual events occur. In which case, upon determining that a limiting and collision event occurs, the system may apply a haptic force as a combination of the first and second haptic feedback forces.

[0092] As shown, upon determining that the difference between the target and limited target pose exceeds the first threshold, the system 100 may provide a first haptic feedback force. In another embodiment, the applied feedback force may be based onAtorney Docket No. : AUR6374WOPCT1Electronically Filed which of the motion limits have been exceeded to cause the difference to exceed the first threshold. For example, upon determining that a velocity limit is exceeded by a velocity threshold, the system may apply a haptic force according to a respective stiffness value. Conversely, upon determining that an acceleration limit is exceeded by an acceleration threshold, the system may apply a different haptic force according to a different stiffness value. Thus, upon determining that a limiting event has occurred, the system 100 may determine which of the motion limits has been exceeded based on the limited target tool pose, and, in response, may provide haptic feedback according to the exceedance of the motion limit (and / or based on a stiffness associated with that motion limit).

[0093] The system 100 may be configured to provide a notification to alert the user of which motion limit is exceeded as a result of the modeled discrepancy. For instance, upon determining that the velocity limit is exceeded by a velocity threshold, the system may provide a visual notification through the display 150 and / or may output an audible notification through one or more speakers. The notification may alert the user that the arm has encountered the motion limit. Thus, the notification may indicate which motion limit has been exceeded along with the provided haptic feedback force.

[0094] In one embodiment, the system 100 may perform a variation of the process 500. In particular, one or more operational blocks may be optional, or may be repeated. In particular, at least one of the decision blocks 535 and 545 may be optional, such that the system may only be concerned with either collision events or limiting events. For example, the system may omit the operations of blocks 535 and 540 to totally eliminate an analysis of the modeled discrepancy in order to only provide haptic feedback force for collisions.

[0095] As described thus far, the system 100 may provide discrepancy-based haptic feedback for one or more surgical tools 171. Those tools may include a grasper, for example. In another embodiment, the system may provide similar operations for other robotic components, such as a surgical camera that may be used to capture a surgical site within its field of view.

[0096] As shown herein, the system may be configured to provide one or more haptic feedback forces based on whether the modeled discrepancy or the unmodeledAtorney Docket No. : AUR6374WOPCT1Electronically Filed discrepancy exceed one or more thresholds. As a result, haptic feedback may be provided when one or both of a collision event and a limiting event occurs.

[0097] Fig. 6 illustrates modeled discrepancy verses modeled discrepancy between a target tool pose, a limited (or commanded) target tool pose, and an actual tool pose according to one embodiment. This figure illustrates the end user control 220 of the HID 140 and the surgical tool 171, which includes a pair of graspers 603 that are coupled to shaft 602 via a joint 601. In one embodiment, the shaft may be coupled to the tool drive 302 of the robotic arm 110. The HID pose 600 may result from a movement of the end user control 220, where the HID pose may be the pose of the end user control 220. Based on this user-desired HID pose, the system determines the target tool pose 610. In one embodiment, the HID pose 600 and the target tool pose 610 may be the same with respect to a global reference point.

[0098] The limited target tool pose 620 is different than the target tool pose 610, which is now illustrated with a dashed boundary. As described herein, this limited pose may be the result of movement of the end user control 220 causing the tool to exceed one or more limits. For instance, the user controlling the HID may be moving the end user control 220 faster than the velocity limit of the surgical tool 171. Thus, as a result, the limited target pose may not be the same as the target tool pose 610. This difference is the modeled discrepancy, as described herein. In one embodiment, when this modeled discrepancy is below a threshold (e.g., a dead band), the system may not provide haptic feedback force to the user, since the difference may be negligible.

[0099] Also shown is the actual tool pose 630, which differs from both tool poses 610 and 620 that are illustrated with dashed and dotted boundaries, respectively. As described herein, this actual tool pose may be the result of external forces being applied to the grasper 603, such as colliding with an object (and the target tool pose exceeding one or more limits). The difference between the limited target pose 620 and the actual tool pose 630 may be defined as an un-modeled discrepancy, as described herein.

[0100] In one embodiment, the actual tool pose 630 may be the same as the limited target tool pose 620. In this case, the system 100 may limit the target tool pose, and the robotic arm may be moved according to the limited pose without experiencingAtorney Docket No. : AUR6374WOPCT1Electronically Filed any external unknown forces. As a result, the system may determine that a limiting event has occurred based on a discrepancy between the target pose and the limited target (actual) pose being greater than a threshold, but a discrepancy between the limited target and the actual being negligible because they are approximately the same.

[0101] As described herein, the system 100 may be configured to determine whether the robotic arm has collided with an object or has encountered a limit based on encoder data of the robotic arm and / or the HID. In another embodiment, the system 100 may include other sensors, which may be used to determine whether there is a collision. For instance, the system may receive sensor data, such as proximity data and / or image data, from one or more sensors, and may use the data to determine the actual pose of the tool, responsive to the joint commands. Thus, the sensors may include a proximity sensor and / or an image sense, for example, which may be communicatively coupled with the controller 415. The sensors may be a part of separate electronic devices that may be positioned around an operating room for capturing sensor data.

[0102] Fig. 7 is a flowchart of one embodiment of a process 700 for providing discrepancy-based haptic feedback force. The process 700 begins with the system 100 determining a pose of the HID (at block 710). The system may detect movement of the HID, and may determine the pose based on this movement. The system determines a (e.g., limited target) pose of the surgical instrument based on at least one of the pose of the HID, which may correspond to a target pose of the surgical instrument, and one or more motion limits of the robotic arm (at block 720). The system provides a control command based on the pose of the surgical instrument to adjust one or more joints of the robotic arm to move the surgical instrument (at block 730). As a result, these commands may be for moving the robotic arm based on the movement of the HID. For instance, the control command may include one or more joint commands which may be sent to one or more actuators 170 of the robotic arm 110 to cause the arm to move. The control command may be intended to move the surgical instrument into the limited target pose. The system 100 determines an actual pose of the surgical instrument resulting from the movement (at block 740). This determination may be based on position data of the robotic arm, which may be received responsive to the movement. The system determines whether at least a portion of the robotic arm has encountered a motion limit or has collided with an object based at least partially on the pose of theAtorney Docket No. : AUR6374WOPCT1Electronically FiledHID, the pose of the surgical instrument, or the actual pose of the surgical instrument (at block 750). In particular, the system determines whether the robotic arm has encountered a known limit or has encountered an unknown limit based at least partially on the movement of the HID, the control commands and / or the position data of the robotic arm. In particular, when the parameters include a velocity limit of the surgical instrument, the limit in which the robotic arm may encounter may be the velocity limit, when the movement of the HID may result in the surgical instrument moving too fast. The velocity limit may be one known limit. Other known limits may include acceleration limits, position limits, or joint limits. Conversely, unknown limits may be the result of the robotic arm colliding with an object. Thus, the system 100 provides haptic feedback through the HID based on whether the robotic arm has encountered the motion limit or a collision with the object (at block 760).

[0103] As described herein, the system 100 may be configured to differentiate between a limiting event in which a discrepancy of a target pose of the surgical tool 171 and a limited pose due to the target pose exceeding one or more limits exceeds one threshold, and a collision event in which a discrepancy between the limited pose and the actual (resulting) pose of the tool due to movement exceeds another threshold in order to provide respective haptics. In another embodiment, the system 100 may provide haptic feedback based on whether a tool-level motion limit or a joint-level motion limit is exceeded. For example, providing haptic feedback when a tool-level motion is exceeded, such as velocity, may provide a user with an indication that the user is moving the HID too fast, whereas providing haptic feedback when a joint-level motion is exceeded may help the user know whether portions of the robotic arm have reached their limit and how to move the arm to avoid such limits. Fig. 8 is a flowchart of another embodiment of a process 800 for providing discrepancy-based haptic feedback force.

[0104] The process 800 begins with determining, based on input of the HID configured to manipulate the arm 110 that includes the tool 171 and several joints, a target pose of the tool 171 (at block 810). For instance, the HID 140 may receive user input to move the HID in order cause the arm 110 to perform a similar (or matching) movement. The HID / target tool pose generator 430 may be configured to use sensor data from one or more sensors 400 of the HID to generate a target tool pose based onAtorney Docket No. : AUR6374WOPCT1Electronically Filed the data and based on a current tool pose. The system 100 determines a first discrepancy based on a first difference between tool motion for moving the tool 171 in 3D space into the target pose and a tool-level motion limit (at block 820). In this case, the tool-level motion limit may include at least one of a velocity limit, an acceleration limit, or a position limit of the surgical tool 171 (and / or robotic arm 110). The tool motion may be a target tool motion that may correspond to a movement of the HID. For instance, the tool motion may be a velocity for moving the tool into the target pose, which may correspond to a velocity at which the user is moving the HID. As a result, the system may determine the first discrepancy by determining whether a velocity for moving the tool into the target pose exceeds the velocity limit, an acceleration for moving the surgical tool into the target pose exceeds the acceleration limit, or a position for the tall at the target pose exceeds the position limit.

[0105] The system 100 determines a second discrepancy based on a second difference between joint motion of at least one joint that is to cause the surgical tool to move towards the target pose and a joint-level motion limit (at block 830). The jointlevel limits may include a velocity limit, an acceleration limit, and / or a position limit for one or more joints of the robotic arm 110. For this operation, the system determines whether joint motion of one or more joints of the robotic arm for moving the surgical tool 171 into the target pose will result an exceedance of one or more joint motion limits. If so, the system may produce the second discrepancy as a difference between the two.

[0106] The system 100 determines a third discrepancy based on a combination of the first and second discrepancies (at block 840). As described thus far, the system may determine whether there is one or more tool-level discrepancies and whether there are one or more joint-level discrepancies. Each discrepancy represented as one or more vectors. Upon determining all joint-level and tool-level discrepancies, the system may produce the modeled discrepancy as the third discrepancy, represents all of the resulting discrepancy between the target tool pose and the limited target tool pose, as described herein.

[0107] The system 100 produces haptic feedback through the HID based on whether at least one of the first, second, or third discrepancy exceeds a differentAtorney Docket No. : AUR6374WOPCT1Electronically Filed threshold (at block 850). In particular, the system determines whether the first discrepancy exceeds a first threshold, the second discrepancy exceeds a second threshold, and the third discrepancy exceeds a third threshold, where each threshold may be different. In one embodiment, each haptic feedback may be different. For example, the system may provide different haptic feedback based on which one of the first, second, or third discrepancies exceeds its respective different threshold. Specifically, the system may provide different force based on whether a tool-level limit has been exceeded by its threshold, a joint-level limit has been exceeded by its threshold, or the modeled discrepancy has exceeded its threshold. For example, the system may determine, for each discrepancy, a different haptic parameter, where providing the haptic feedback includes providing, using actuators of the HID and based on which discrepancy exceeds it threshold, a different force according to a respective haptic parameter.

[0108] As described thus far, the process 800 may determine discrepancies for a tool-level motion and a joint-level motion, based on whether they exceed respective limits. In one embodiment, at least some of these operations may be performed for multiple tool-level motions and / or joint-level motions. For example, when the robotic arm includes two joints, the system 100 may determine whether joint motion for each joint exceeds a joint-level motion limit. If so, the system may determine corresponding discrepancies, and determine whether those discrepancies exceed one or more thresholds in order to determine whether to provide haptic feedback. As a result, the system can provide specific haptic feedback for specific joint-level exceedances of one or more joint limits, which may differ between joints.

[0109] In one embodiment, one or more operations of process 800 may be optional. For example, responsive to receiving a target tool pose, the system may determine that there are no tool motions (e.g., in tool space) that exceed tool-level motion limits, or if there are, the system may determine that corresponding discrepancies do not exceed their respective thresholds. The system, however, may identify that one or more joint motions exceed one or more joint-level motion limits. As a result, the system may determine whether any of these discrepancies exceed one or more thresholds, and if so, may provide haptic feedback based on the exceedance ofAtorney Docket No. : AUR6374WOPCT1Electronically Filed that discrepancy and a corresponding haptic parameter, which may be associated with that particular joint.

[0110] As described thus far, haptic feedback may be provided responsive to a discrepancy exceeding a threshold. This feedback may be provided based on other criteria. For example, in addition to discrepancy, the system may determine whether that discrepancy occurs along a particular direction (e.g., in Cartesian space). For example, with the tool-level discrepancy, the system may determine whether this discrepancy is along a predefined direction, and if so, the system may provide feedback responsive to the discrepancy exceeding a threshold along that direction. In another embodiment, the direction may be in which the tool motion is to occur. In another embodiment, the direction may be a different direction in which the motion is to occur.

[0111] In one embodiment these joint-level and tool-level motion limits, thresholds, and haptic parameters may be user defined. For example, the user may specify that some joints are to be more sensitive to joint motion than others by setting their corresponding velocity limits to below a threshold, while other joints’ velocity limits may be less sensitive by being set above the threshold. In another embodiment, the position limit of the tool may be user-defined. In which case, the position limit may be a boundary within 3D space, which may be smaller than the actual (physical) limits of the robot. As a result, the user may set a smaller boundary in order to keep the robotic arm 110 within a more constrained workspace.

[0112] As previously explained, an embodiment of the disclosure may be a non-transitory machine-readable medium (such as microelectronic memory) having stored thereon instructions, which program one or more data processing components (generically referred to here as a “processor”) to (automatically) perform discrepancybased haptic feedback operations, as described herein. In other embodiments, some of these operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed data processing components and fixed hardwired circuit components.

[0113] To aid the Patent Office and any readers of any patent issued on this application in interpreting the claims appended hereto, applicants wish to note that theyAtorney Docket No. : AUR6374WOPCT1Electronically Filed do not intend any of the appended claims or claim elements to invoke 35 U.S.C. 112(f) unless the words “means for” or “step for” are explicitly used in the particular claim.

[0114] While certain embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad disclosure, and that the disclosure is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art. The description is thus to be regarded as illustrative instead of limiting.

[0115] In some embodiments, this disclosure may include the language, for example, “at least one of [element A] and [element B].” This language may refer to one or more of the elements. For example, “at least one of A and B” may refer to “A,” “B,” or “A and B.” Specifically, “at least one of A and B” may refer to “at least one of A and at least one of B,” or “at least of either A or B.” In some embodiments, this disclosure may include the language, for example, “[element A], [element B], and / or [element C] ” This language may refer to either of the elements or any combination thereof. For instance, “A, B, and / or C” may refer to “A,” “B,” “C,” “A and B,” “A and C,” “B and C,” or “A, B, and C.”

Claims

Atorney Docket No. : AUR6374WOPCT1Electronically FiledCLAIMSWhat is claimed is:

1. A method performed by at least on programmed processor of a surgical robotic system that comprises a human interface device (HID) arranged to control a robotic arm comprising a plurality of joints and a surgical instrument, the method comprising: determining a pose of the HID; determining a pose of the surgical instrument based on at least one of the pose of the HID and one or more motion limits of the robotic arm; providing a control command based on the pose of the surgical instrument to adjust at least one of the plurality of joints of the robotic arm to move the surgical instrument; determining an actual pose of the surgical instrument resulting from the movement; determining whether a portion of the robotic arm has encountered a motion limit or has collided with an object based at least partially on the pose of the HID, the pose of the surgical instrument, or the actual pose of the surgical instrument; and providing haptic feedback through the HID based on whether the robotic arm has encountered the motion limit or a collision with the object.

2. The method of claim 1 further comprising determining a target pose of the surgical instrument based on the pose of the HID, wherein determining the pose of the surgical instrument comprises determining a limited target pose of the surgical instrument that is different than the target pose of the surgical instrument based on the target pose of the surgical instrument and the one or more motion limits.

3. The method of claim 2, wherein determining the limited target pose of the surgical instrument comprises adjusting the target pose of the surgical instrument such that the movement of the surgical instrument satisfies the one or more motion limits.

4. The method of claim 3, wherein the one or more motion limits of the robotic arm comprises a position limit of the robotic arm, a velocity limit of the robotic arm, anAtorney Docket No. : AUR6374WOPCT1Electronically Filed acceleration limit of the robotic arm, a velocity limit of a joint of the robotic arm, or a joint limit of the joint.

5. The method of claim 3, wherein determining whether the at least a portion of the robotic arm has encountered the motion limit or has collided with the object comprises: responsive to determining that a difference between the target pose of the surgical instrument and the limited target pose of the surgical instrument exceeds a first threshold, determining that the robotic arm has encountered the motion limit; and responsive to determining that a difference between the limited target pose of the surgical instrument and the actual pose of the surgical instrument exceeds a second threshold, determining that the robotic arm has encountered the collision with the object.

6. The method of claim 5, wherein the first threshold is different than the second threshold.

7. The method of claim 1, wherein providing haptic feedback through the HID comprises: responsive to the determining that the robotic arm has encountered the motion limit, causing one or more actuators of the HID to produce a first haptic effect; and responsive to the determining that the robotic arm has encountered the collision, causing the one or more actuators of the HID to produce a second haptic effect that is different than the first haptic effect.

8. The method of claim 1, wherein the one or more motion limits comprises a velocity limit, an acceleration limit, or a position limit, wherein determining the pose of the surgical instrument comprises: determining a target pose of the surgical instrument based on the pose of the HID; determining that movement of a joint of the robotic arm to cause the surgical instrument to move from a current pose to the target pose is to cause at least one of the velocity limit, the acceleration limit, or the position limit to be exceeded; andAtorney Docket No. : AUR6374WOPCT1Electronically Filed adjusting the target pose such that movement of the joint ensures that the velocity limit, the acceleration limit, or the position limit is maintained.

9. The method of claim 1 further comprising receiving position data from one or more encoders of at least one joint of the plurality of joints, wherein the actual pose is determined based on the position data.

10. The method of claim 1 further comprising receiving sensor data from one or more sensors of the surgical robotic system, wherein the actual pose is determined based on the sensor data.

11. The method of claim 1 , wherein the HID comprises a grounded input device coupled to a surgeon console of the surgical robotic system, the grounded input device comprising an end user control coupled to the plurality of joints.

12. The method of claim 11, wherein the pose of the HID comprises a pose of the end user control.

13. A method performed by at least one programmed processor of a surgical robotic system that comprises a human interface device (HID) arranged to control a robotic arm that comprises one or more joints, the method comprising: receiving, using the HID, a target pose of the robotic arm; determining a limited target pose of the robotic arm based on the target pose and one or more motion limits of the robotic arm; causing the robotic arm to perform a movement based on the limited target pose; responsive to the movement of the robotic arm, receiving position data of the robotic arm, and determining an actual pose of the robotic arm; and providing haptic feedback through the HID based on at least one of a first discrepancy between the target pose and the limited target pose, or a second discrepancy between the limited target pose and the actual pose.Atorney Docket No. : AUR6374WOPCT1Electronically Filed14. The method of claim 13, wherein providing haptic feedback through the HID comprises: applying a first force through the HID when the first discrepancy is greater than a first threshold; and applying a second force through the HID when the second discrepancy is greater than a second threshold.

15. The method of claim 14, wherein providing haptic feedback through the HID further comprises applying a third force through the HID when first and second discrepancies exceed the first and second thresholds, respectively.

16. The method of claim 15, wherein the third force comprises a combination of the first and second forces.

17. The method of claim 14, wherein the first threshold is different than the second threshold.

18. The method of claim 13, wherein determining the limited target pose of the robotic arm comprises: receiving joint commands for the one or more joints, wherein the joint commands are produced to cause the robotic arm to perform the movement; and generating the limited target pose of the robotic arm by performing a forward kinematics function using the joint commands.

19. The method of claim 13, wherein determining the limited target pose of the robotic arm comprises: determining that the target pose of the robotic arm is to exceed the one or more motion limits; and generating the limited target pose of the robotic arm by adjusting the target pose of the robotic arm to satisfy the one or more motion limits.Atorney Docket No. : AUR6374WOPCT1Electronically Filed20. The method of claim 13, wherein the one or more motion limits of the robotic arm comprises a velocity limit, an acceleration limit, a position limit, a joint velocity limit, a joint acceleration limit, or a joint position limit.

21. A method comprising : detecting a movement of a human interface device (HID) configured to control a robotic arm; determining a command for moving the robotic arm based on the movement of the HID; causing the robotic arm to move based on the command; receiving position data of the robotic arm; and determining whether at least a portion of the robotic arm has encountered a known limit or has encountered an unknown limit based at least partially on the movement of the HID, the command, or the position data of the robotic arm.

22. The method of claim 21, wherein the known limit comprises at least one of a velocity limit, an acceleration limit, a position limit, or a joint limit of a joint of the robotic arm, and the unknown limit comprises a collision with an object.

23. The method of claim 22 further comprising: determining, responsive to determining that the at least the portion of the robotic arm has encountered the known limit, that a motion limit of a plurality of motion limits of the robotic arm has been exceeded based on the command; and in response, providing haptic feedback force through the HID according to the exceedance of the motion limit.

24. The method of claim 23 further comprising providing a notification indicating that the motion limit has been exceeded along with the provided haptic feedback force.

25. The method of claim 21, wherein determining the command comprises: determining a target pose of the robotic arm based on the movement of the HID; andAtorney Docket No. : AUR6374WOPCT1Electronically Filed determining a limited target pose of the robotic arm as output of a movement model responsive to input based on the target pose of the robotic arm and the known limit.

26. The method of claim 25, wherein determining whether comprises: determining that the portion of the robotic arm has encountered the known limit responsive to a first discrepancy between the target pose and the limited target pose exceeding a first threshold; and determining that the portion of the robotic arm has encountered the unknown limit responsive to a second discrepancy between the limited target pose and an actual pose of the robotic arm based on the position data exceeding a second threshold.

27. The method of claim 21 further comprising determining an actual pose of the robotic arm based on the position data, wherein determining whether the at least the portion of the robotic arm has encountered the known limit or has encountered the unknown limit uses the movement of the HID, the command, and the actual pose of the robotic arm.

28. The method of claim 21, wherein the HID comprises a grounded input device coupled to a surgeon console, the grounded input comprising an end user control coupled to a plurality of joints, wherein the command comprises a pose of the HID resulting from the movement.

29. The method of claim 21 further comprising: providing, using an actuator, a first haptic force through the HID responsive to determining that the portion of the robotic arm has encountered the known limit; or providing, using the actuator, a second haptic force that is stronger than the first haptic force through the HID responsive to determining that the portion of the robotic arm has encountered the unknown limit or force.

30. A method comprising:Atorney Docket No. : AUR6374WOPCT1Electronically Filed determining, based on input of a human interface device (HID) configured to manipulate a robotic arm that comprises a surgical tool and a plurality of joints, a target pose of the surgical tool; determining a first discrepancy based on a first difference between tool motion for moving the surgical tool in three-dimensional (3D) space into the target pose and a tool-level motion limit; determining a second discrepancy based on a second difference between joint motion of at least one joint that is to cause the surgical tool to move towards the target pose and a joint-level motion limit; determining a third discrepancy based on a combination of the first discrepancy and the second discrepancy; and providing haptic feedback through the HID based on whether at least one of the first, second, or third discrepancy exceeds a different threshold.

31. The method of claim 30, wherein providing haptic feedback comprises providing different haptic feedback based on which one of the first, second, or third discrepancy exceeds a respective different threshold.

32. The method of claim 31 further comprising determining, for each discrepancy, a different haptic parameter, wherein providing the different haptic feedback comprises providing, using one or more actuators of the HID and based on which of the first, second, or third discrepancy exceeds the respective different threshold, a different haptic force according to a respective different haptic parameter.

33. The method of claim 32, wherein at least one different threshold or at least one different haptic parameter is user defined.

34. The method of claim 30, wherein the tool-level motion limit comprises at least one of a velocity limit, an acceleration limit, or a position limit, wherein determining the first discrepancy comprises determining whether a velocity for moving surgical tool into the target pose exceeds the velocity limit, an acceleration for moving surgical toolAtorney Docket No. : AUR6374WOPCT1Electronically Filed into the target pose exceeds the acceleration limit, or a position for the tool at the target pose exceeds the position limit.

35. The method of claim 34, wherein the position limit comprises a user-defined boundary within 3D space.

36. The method of claim 30, wherein the first discrepancy is along a direction in which the tool motion is to occur, wherein providing the haptic feedback comprises providing a haptic force responsive to the first discrepancy exceeding a threshold along the direction.

37. The method of claim 30, wherein the at least one joint comprises a first joint and the joint-level motion limit comprises a first joint-level motion limit, wherein determining the second discrepancy comprises: determining a first joint discrepancy based on a difference between a first joint motion of the first joint and the first joint-level motion limit; and determining a second joint discrepancy based on a difference between a second joint motion of a second joint of the plurality of joints and a second joint-level motion limit, wherein providing haptic feedback comprises providing, using an actuator of the HID, a haptic force based on at least one of the first joint discrepancy or the second joint discrepancy exceeding one or more thresholds.

38. The method of claim 37, wherein the first joint-level motion limit is different than the second joint-level motion limit.

39. The method of claim 30 further comprising that the second discrepancy exceeds a joint motion threshold, while the first discrepancy and the third discrepancy remain within respective threshold, wherein providing haptic feedback comprises providing, using an actuator of the HID, a haptic force based on an exceedance of the second discrepancy and on a haptic parameter associated with the at least one joint.

40. The method of claim 30 further comprising:Atorney Docket No. : AUR6374WOPCT1Electronically Filed determining a limited target pose of the surgical tool based on the third discrepancy; causing the robotic arm to perform a movement based on the limited target pose; determining, based on sensor data of the robotic arm, an actual pose of the robotic arm responsive to the movement; and determining a fourth discrepancy based on the limited target pose and the actual pose, wherein providing haptic feedback comprises, providing, using an actuator of the HID, at least one of: a first haptic force indicating that at least one of 1) the tool motion is limited due to the tool-level motion limit or 2) the joint motion is limited due to the joint-level motion limit, when the third discrepancy exceeds a first threshold, or a second haptic force indicating that the robotic arm has collided with an object when the fourth discrepancy exceeds a second threshold.

Images