Inverse kinematics of a surgical robot for remote operation with hardware constraints

Abstract

Various approaches for solving inverse kinematics can be used for the remote operation of surgical robot systems. In one approach, an iterative solver solves the linear component of motion independently of solving the angular component of motion. A single solver can be used to solve for both together. In another approach, all constraints (e.g., position, velocity, and acceleration) are handled in a single solution. When a constraint is reached, the constraint is used as a boundary in the intermediate solution, enabling a solution even when the boundary is reached. In another approach, a ratio of position constraints is used to create deceleration regions near the boundaries to control motion more naturally. In yet another approach, medical-based remote operation uses a bounded Gaussian-Seidel solver with sequential acceleration relaxation, for example.

Description

Background Technology

[0001] The present embodiment relates to a robotic system for minimally invasive surgery (MIS). MIS may be performed by a robotic system comprising one or more robotic arms for manipulating surgical instruments based on commands from a remote operator. The robotic arms may support various surgical end effectors, for example, including scalpels, imaging devices (e.g., endoscopes), clamps, and scissors at their distal ends. Using the robotic system, a surgeon controls the robotic arms via teleoperation during MIS.

[0002] Remote operation is an inverse kinematic problem that converts user commands into robot arm and surgical tool joint commands. Depending on the end effector used, the inverse kinematic problem may need to handle different numbers of degrees of freedom (DOF). For example, a clamp has 6 DOF in remote operation. Once one or more joints positioned for operation are braked, 6 active joints and corresponding 6 DOFs are provided by 3 joints on the surgical tool (e.g., rotation, pitch, and yaw) and 3 joints on the robot arm (e.g., spherical rotation, spherical pitch, and tool translation). As another example, staplers have 5 DOF in remote operation. 5 active joints and corresponding DOFs include 2 joints on the instrument (e.g., rotation and articulation) and 3 joints on the robot arm (e.g., spherical rotation, spherical pitch, and tool translation). In another example, an endoscope or ultrasonic scalpel has 4 DOFs in remote operation. The four active joints and corresponding DOFs include one joint on the instrument (e.g., rotation) and three joints on the robot arm (e.g., spherical rotation, spherical pitch, and tool translation).

[0003] During remote operation, user inputs with 6 DOF are converted into joint movements, and thus the mechanism follows the user commands with controlled accuracy. To convert user commands into joint commands, the solution of the inverse kinematics function is obtained. An example is given as follows:

[0004] ,

[0005] Here, is the current time i It is a user command matrix (6x1) in the workspace, and is the current time i It is the Jacobian matrix (6xn) at, and is the current time i This is the joint motion matrix (nx1) in. Due to physical constraints on the robot arm and surgical tool (e.g., joint position limits, joint velocity limits, and joint acceleration limits), an accurate solution may not be achievable when one or more of these constraints are violated by the solution.

[0006] As an introduction, the preferred embodiments described below include a method, system, instructions, and a computer-readable medium for remote operation of a surgical robot system. Various approaches may be used to find solutions for inverse kinematics. In one approach, an iterative solver finds a solution for the linear component of motion independently of finding a solution for the angular component of motion. A single solver may be used to find solutions for both together. In another approach, all constraints (e.g., position, velocity, and acceleration) are handled in a single solution. When a constraint is reached, the constraint is used as a boundary in the intermediate solution to enable a solution even when the boundary is reached. In another approach, a ratio of position constraints is used to create deceleration regions near the boundaries to control motion more naturally. In yet another approach, medical-based remote operation uses a bounded Gauss-Seidel solver with, for example, successive-over-relaxation. Any of these approaches may be used together or individually.

[0007] In the first embodiment, a method for remote operation of a surgical robot system is provided. A user command to move a robot arm or a surgical tool of the robot arm is received during remote operation. A solution for the linear motion of the movement is obtained in a first stage. A solution for the angular motion of the movement is obtained in a second stage separate from the first stage. The robot arm or the surgical tool of the robot arm moves based on the solutions for the linear motion and the angular motion.

[0008] In a second embodiment, a method for remote operation of a surgical robot system is provided. A user command to move a robot arm or a surgical tool of the robot arm is received during remote operation. A solution for the motion of the robot arm or the surgical tool of the robot arm is obtained as an iterative solution, wherein the iterative solution includes a solution that considers all constraints on the motion. The robot arm or the surgical tool of the robot arm moves based on the iterative solution.

[0009] In the third aspect, a surgical robot system for medical remote manipulation is provided. A surgical instrument is connected to a robot arm. A controller is configured to find a solution for the motion of the robot arm and / or surgical instrument during medical remote manipulation of a patient and in response to user input of a motion command. The solution uses a bounded Gauss-Seidel solver with sequential acceleration relaxation.

[0010] The present invention is limited by the following claims, and nothing in this section shall be construed as a limitation to such claims. Any teaching for one claim type (e.g., method) may be applicable to another claim type (e.g., computer-readable storage medium or system). Additional aspects and advantages of the present invention are discussed below in relation to preferred embodiments and may be claimed later, either independently or in combination. Brief explanation of the drawing

[0011] The components and drawings are not necessarily drawn to a fixed scale, but instead focus on illustrating the principles of the invention. Furthermore, in the drawings, the same reference numerals throughout the different drawings indicate corresponding parts. FIG. 1 is an example of an example of an operating room environment having a surgical robot system according to one embodiment. Figure 2 illustrates an exemplary surgical robot arm and surgical tool. FIG. 3 is a flowchart of an embodiment of a method for remote operation of a surgical robot system. Figure 4 illustrates an exemplary robotic arm and surgical tool for 6 DOF. Figure 5 illustrates an exemplary robotic arm and surgical tool for 4 DOF. FIG. 6 is a flowchart of an example of nested control loops for iterative solutions of inverse kinematics in remote operation. Figure 7 is a block diagram of one embodiment of a surgical robot system. Specific details for implementing the invention

[0012] Iterative inverse kinematics provides an integrated framework for solving 6, 5, and 4 DOF problems. Joint position limits, joint velocity limits, and joint acceleration limits are handled in a single solution. Hardware-based joint limits, velocity limits, and acceleration limits are integrated. 6, 5, and 4 DOF inverse kinematics can use iterative solutions in a decoupled manner where linear motion and angular motion commands are resolved independently. If there are no limit violations, an exact solution (within tolerance) is provided. If there are limit violations, a suboptimal solution may still be provided. The values ​​of the violated limits are used in the solution to clamp the position, velocity, and / or acceleration to the limits in at least the intermediate solutions in the iterative inverse kinematics.

[0013] FIGS. 1 and 2 illustrate an exemplary surgical robot system. Approaches to iterative inverse kinematics solutions in remote operation are discussed below with reference to this exemplary system. Other surgical robot systems and surgical robots, or non-surgical robot systems and robots, may use these approaches.

[0014] FIGS. 3 to 6 relate to iterative inverse kinematics solutions in remote operation. FIG. 7 relates to a system for using an iterative inverse kinematics solver with a medical robot system for remote operation.

[0015] FIG. 1 is a diagram illustrating an exemplary operating room environment having a surgical robot system (100) in which commands from a user are converted into motion of surgical robot arms (122) using repetitive inverse kinematics. The surgical robot system (100) comprises a user console (110), a control tower (130), and a surgical robot (120) having one or more surgical robot arms (122) mounted on a surgical platform (124) (e.g., a table or a bed, etc.), wherein surgical tools having end effectors are attached to the distal ends of the robot arms (122) to perform surgical procedures. Additional, different, or fewer components may be provided, such as combining the control tower (130) with the console (110) or the surgical robot (120). The robot arms (122) are shown as table-mounted, but in other configurations, the robot arms (122) can be mounted on a cart, ceiling, side wall, or other suitable support surfaces.

[0016] Generally, a user, such as a surgeon or other operator, may sit at the user console (110) to remotely operate the robot arm (122) and / or surgical instrument (e.g., remote operation). The user console (110) may be located within the same operating room as the robot system (100), as illustrated in FIG. 1. In other environments, the user console (110) may be located in an adjacent or nearby room, or may be remotely operated from a remote location in a different building, city, or country. The user console (110) may include a seat (112), pedals (114), one or more handheld user interface devices (UIDs) (116), and an open display (118) configured to display, for example, a view of the surgical site inside the patient and a graphical user interface. As illustrated in the exemplary user console (110), a surgeon seated in a seat (112) and viewing an open display (118) may operate a pedal (114) and / or a handheld user interface device (116) to remotely and directly control a robotic arm (122) and / or a surgical instrument mounted on the distal end of the arm (122). The user inputs commands for the movement of the surgical arm (122) and / or end effector. Such user control determines the position of the robotic arm (122), the speed of movement, and changes in the speed of movement. The speed and changes in speed cause dynamic torque expected to be provided by the robotic arm (122). The surgeon seated in the seat (112) may view and interact with the display (118) to input commands for the movement of the robotic arm (122) and / or surgical instrument in remote operation during surgery.

[0017] In some variations, the user may also operate the surgical robot system (100) in "Over-the-Bed" (OTB) mode, in which the user is at the patient's side and simultaneously operates a robot-driven tool / end effector attached thereto (e.g., a handheld user interface device (116) held in one hand) and a manual laparoscopic tool. For example, the user's left hand may be operating the handheld user interface device (116) to control the robotic surgical component, while the user's right hand may be operating the manual laparoscopic tool. Accordingly, in these variations, the user may perform both robot-assisted MIS and manual laparoscopic surgery on the patient.

[0018] During an exemplary procedure or surgery, the patient is surgically prepared and draped in a sterile manner to achieve anesthesia. Initial access to the surgical site may be performed manually with the robotic system (100) in a stowed configuration or withdrawn configuration to facilitate access to the surgical site. Once access is complete, initial positioning and / or preparation of the robotic system may be performed. During the procedure, the surgeon at the user console (110) may perform the surgery using remote operation by manipulating various end effectors and / or imaging systems using pedals (114) and / or user interface devices (116). Movements may be specific to the surgeon, the patient, and / or situation, and thus may vary. Manual assistance may also be provided at the procedure table by a sterile-gown-wearing personnel capable of performing tasks including, but not limited to, retracting tissue or performing manual repositioning or tool exchange involving one or more robotic arms (122). Some surgical operations, such as traction, suturing, or other tissue manipulation, may instead be performed by one or more robotic arms (122) (e.g., a third or fourth arm). Non-sterile personnel may also be present to assist the surgeon at the user console (110). When the procedure or surgery is completed, the robotic system (100) and / or the user console (110) may be configured or set to a state that facilitates one or more postoperative procedures, including but not limited to cleaning and / or sterilizing the robotic system (100) and / or inputting healthcare records or outputting, whether electronic or hard copy, through the user console (110), for example.

[0019] In some cases, communication between the surgical robot (120) and the user console (110) may be made through a control tower (130), which can convert user input commands from the user console (110) into robot control commands and transmit the control commands to the surgical robot (120). The control tower (130) performs repetitive inverse kinematics. The control tower (130) can also transmit status and feedback from the robot (120) back to the user console (110). The connection between the surgical robot (120), the user console (110), and the control tower (130) may be made via wired and / or wireless connections, and may be proprietary or using any of various data communication protocols. Any wired connection may optionally be established on the floor and / or walls or ceiling of the operating room. The surgical robot system (100) may provide video output to one or more displays, including a display within the operating room as well as a remote display accessible via the Internet or another network. The video output or feed may also be encrypted to ensure privacy, and all or part of the video output may be stored on a server or electronic healthcare record system.

[0020] Before initiating surgery using the surgical robot system, the surgical team may perform a pre-operative setup. During the pre-operative setup, the main components of the surgical robot system (e.g., table (124) and robotic arm (122), control tower (130), and user console (110)) are located, connected, and powered within the operating room. The table (124) and robotic arm (122) may be in a fully retracted configuration where the arm (122) is located beneath the table (124) for storage and / or transport purposes. The surgical team may extend the arms (122) from their retracted positions for sterile draping. After draping, the arms (122) may be partially retracted until needed for use. A number of conventional laparoscopic steps, including trocar placement and insertion, may need to be performed. For example, each sleeve may be inserted into a small incision and through the body wall with the help of a occlusion port. The sleeve and occlusion port allow optical entry for visualization of tissue layers during insertion to minimize the risk of damage during placement. Typically, an endoscope is placed first to provide hand-held camera visualization for the placement of other trocars. After insertion, if required, manual instruments can be inserted through the sleeve to perform any laparoscopic steps by hand.

[0021] Next, the surgical team can position the robotic arms (122) over the patient and attach each arm (122) to a corresponding sleeve. The surgical robot system (100) has the ability to uniquely identify each tool (endoscope and surgical instrument) upon attachment and to display the tool type and arm position on an open or immersive display (118) on the user console (110) and on a touchscreen display on the control tower (130). Corresponding tool functions can be enabled and activated using a master UID (116) and a foot pedal (114). A patient-side assistant can attach and detach the tools as required throughout the procedure. A surgeon seated at the user console (110) can begin performing surgery as a remote operation using tools controlled by two master UIDs (116) and foot pedals (114). The system converts the surgeon's hand, wrist, and finger movements into precise, real-time movements of the surgical tools via the master UID (116). Accordingly, in direct remote operation, the system continuously monitors all surgical operations of the surgeon and stops instrument movement if the system cannot accurately reflect the surgeon's hand movements. When the endoscope is moved from one arm to another during surgery, the system can adjust the master UID (116) for instrument alignment and continue instrument control and movement. The foot pedal (114) can be used to activate various system modes, such as endoscope control and various instrument functions including unipolar and bipolar cauterization, without the surgeon's hand being removed from the master UID (116).

[0022] FIG. 2 is a schematic diagram illustrating one exemplary design of a robot arm, a tool actuator, and a cannula loaded with a robot surgical tool, according to an aspect of the present art. As illustrated in FIG. 2, the exemplary surgical robot arm (122) may include a plurality of links (e.g., links (202)), and a plurality of actuable joint modules for acting the plurality of links relative to each other (e.g., joints (204), also see joints (J1 to J8)). The joint modules may include various types, such as pitch joints or roll joints, which can substantially restrict the movement of adjacent links around certain axes relative to each other. A tool actuator (210) attached to the distal end of the robot arm (122) is also illustrated in the exemplary design of FIG. 2. The tool actuator (210) may include a cannula (214) coupled to its end to receive and guide a surgical instrument or end effector (220) (e.g., endoscope, stapler, scalpel, scissors, clamp, retractor, etc.). The surgical instrument (or “tool”) (220) may include an end effector (222) at the distal end of the tool. Multiple joint modules of the robot arm (122) may be actuated to position and orient the tool actuator (210) to actuate the end effector (222) for robotic surgery. The end effector (222) is located at the end of the tool shaft. In another embodiment, the end of the tool shaft is a needle or the tip of another object.

[0023] In the example of FIG. 2, joint (J0) is a table pivot joint and is located beneath the surgical table top. Joint (J0) is nominally maintained in place during surgery. Joints (J1 through J5) form a setup or Cartesian arm and are nominally maintained in place during surgery, thus not contributing to movement during remote surgical operation. Joints (J6 and J7) form a spherical arm that can move actively during surgery or remote operation. Joint (J8) is part of the tool actuator and translates the tool (220), such as the end effector (222). Joint (J8) can move actively during surgery. Joints (J6 through J8) actively position the tool shaft end (i.e., end effector (222)) during surgery while maintaining the point of entry into the patient at a fixed or stable position (i.e., center of remote movement) to avoid stress on the patient's skin. During setup, any of the joints (J0 to J8) may be moved. During surgery, the joints (J6 to J8) may be moved subject to hardware or safety limits for position, velocity, acceleration, and / or torque. The surgical tool (220) may not include any joints, may include one joint, or may include more than one (e.g., three) joints, e.g., joints for tool rotation + any number of additional joints (e.g., wrists, rotation around a longitudinal axis, or other types of movement). Any number of degrees of freedom may be provided, such as three degrees of freedom from the joints (J6 to J8) and zero, one, or more degrees of freedom from the surgical tool (220).

[0024] FIG. 3 is a flowchart of an embodiment of a method for remote operation of a surgical robot system. Iterative inverse kinematics is used to convert user commands into solutions for the movements of the robot arm and surgical tool. Separate and independent linear and angular solutions, including all constraints in a single solution, limiting intermediate solutions to one or more constraints when a constraint is reached, using the ratio of position boundaries for step-down movements near a constraint, and / or the use of a bounded Gaussian-Seidel solver in medical remote operation may be used.

[0025] The method of FIG. 3 is implemented by a control processor, such as a control tower (130), a computer, a workstation, a server, or other processor. Any computer of the surgical robot system (100) may be used. In an act (300), a user interface provides movement commands from the received user. In an act (320), the robot arm (122) and / or surgical tool (220) are moved using commands or controls from the control processor. In any of the acts, other devices may be performed and / or used.

[0026] The acts are performed in the order described or in a different order. Various acts (311 to 316) that are part of the solution of act (310) may be performed in any order and / or simultaneously (e.g., as part of the same solution converting user commands into robot arm and surgical tool movements).

[0027] Additional, different, or fewer acts may be used. For example, any of the acts (311 to 316) may be performed alone in the solution of act (310). Any combination of two or more of any of the acts (311 to 316) may be performed without the others of the acts. In other examples, an act for initially positioning the surgical instrument (220) within the patient, an act for planning the surgery, and / or an act for removing the surgical instrument (220) from the patient may be provided.

[0028] In Act (300), the control processor receives a user command to move the robot arm (122) or the surgical tool (220) of the robot arm (122) during remote operation. The user input is received from a user console (110), such as a pedal (114) or a user interface device (116), via a wireless or wired interface by the control processor. In another embodiment, the user command is received by loading from memory or by transmitting via a computer network.

[0029] When preparing for remote operation, the user sits at the surgeon console (110). After positioning the robotic arm (122) for remote operation, one or more joints are locked in place with a fixed remote motion center (RCM) located at the patient skin or the point of entry into the incision. For example, joints (J0 to J5) (see FIG. 2) are locked. Locking is achieved by avoiding the brake and / or power supply to the motors for the joints. These joints remain locked during remote operation.

[0030] During remote operation, the user inputs commands to move the robot arm (122) and / or surgical tool (220). The commands are for movement. Different commands may be provided for different movements. The commands may be for the movement of the end effector (222). These commands may not be for the movement of specific joints. The control processor converts the movement commands into control of specific joints of the robot arm (122) and / or surgical tool (220).

[0031] In one embodiment, user movement is tracked using a sensor. For example, the user holds a device such as a pen or UID (116). A magnetic position sensor and / or an inertial measurement unit may be used to determine the position of the pen or UID (116) and / or changes in position. As another example, the user holds a marker having a structure that allows visual tracking, such as optical patterns or structures on one or more parts of the marker. A stereo camera and / or a depth camera track the movement of the marker.

[0032] User commands may be for movements at any number of degrees of freedom. FIG. 4 illustrates a portion of a robot arm (122) and a surgical tool (220) that provides 6 degrees of freedom. 6 degrees of freedom correspond to the movement of six active joints during remote operation. The active joints include three joints on the surgical tool (220)—rotation at joint (J9), pitch at the wrist joint (J10), and yaw at the wrist joint (J11). The active joints include three joints on the robot arm (122)—spherical rotation joint (J6), spherical pitch joint (J7), and tool translation joint (J8). Other joints providing 6 degrees of freedom may be used.

[0033] User commands may be for movements with fewer than 6 degrees of freedom. Five, four, or fewer active joints may be provided. FIG. 5 illustrates a robotic arm (122) and a surgical tool (220) providing 4 DOF, for example, where the surgical tool (220) is an endoscope or an ultrasonic scalpel. The active joints include three joints on the robotic arm (122)—a spherical rotation joint (J6), a spherical pitch joint (J7), and a tool translation joint (J8). The active joints include one joint on the surgical tool (220)—rotation at joint (J9). In the example of 5 DOF, the active joints include three joints on the robotic arm (122)—a spherical rotation joint (J6), a spherical pitch joint (J7), and a tool translation joint (J8)—and two active joints on the surgical tool—rotation at joint (J9) and articulation as other joints. Other active joint arrangements may be used, such as providing 2 or fewer degrees of freedom on the robot arm during remote operation.

[0034] In Act (310), the control processor converts movement commands from the user into movements of the joints. The control processor iteratively finds a solution for the movement of the robot arm or the surgical tool of the robot arm. An iterative inverse kinematic solution is found.

[0035] Any control process may be used. The control process may be the input of user commands, an iterative solution using inverse kinematics with a given termination check for iterations, and the output of the final result of joint commands. The iterative inverse kinematics uses one or more of the following: distinct and independent linear and angular solutions; including all constraints in a single solution; limiting intermediate solutions to one or more constraints when a constraint is reached; using the ratio of position boundaries for step-down movements near a constraint; and / or the use of a bounded Gaussian-Seidel solver in medical remote operation.

[0036] FIG. 6 illustrates an example of a nested loop control process. The outer control loop (600) is the main iterative inverse kinematics loop, and the inner control loop (602) is the iterative solver loop. The outer control loop (600) includes the inner control loop (602). The outer control loop (600) also includes the forward kinematic determination (604) of the position of the surgical tool (220), the generation of a twist command (606) of the movement and position of the surgical tool from a user command, a termination check (608), and a convergence check (610). A user pose command or movement (605) is input to the outer control loop. The twist command (606) is input to the solver (614), and the final solution is output to the termination check (608). The forward kinematics (604) calculates the end effector (222) pose (i.e., position and orientation). These poses are input into a twist command generator (606). The twist command generator (606) calculates the twist motion from the current end effector (222) pose to the pose command (605). The twist is a 6 DOF command (e.g., 3 DOF linear or translational and 3 DOF angular). For better convergence, the linear twist is linearly interpolated by the step size ratio, and the angular twist is interpolated by spherical linear interpolation (SLERP). Other interpolations may be used.

[0037] The termination check (608) of the outer control loop (600) checks for the completion of joint commands (611) to be output to the robot arm (122) and the surgical tool (220). If not satisfied, the outer control loop (600) repeats through forward kinematic determination (604), twist command generation (606), and inner control loop (602). The termination check (608) for the outer control loop (600) uses one or more criteria. For example, three termination checks are performed, where any one of them terminates the outer loop (600). The checks may include (1) the commanded pose and the current pose are compared against a convergence check (610), (2) the maximum number of steps or iterations is reached, and (3) the timer expires. The convergence check (610) determines any error between the calculated pose command and the input pose command (605). The error must be below a threshold level (i.e., the calculated pose output by the solution from the solver (614) matches the input pose command (605) within the tolerance).

[0038] Various approaches to finding a solution, including separate and independent linear and angular solutions, including all constraints in each solution, limiting intermediate solutions to one or more constraints when a constraint is reached, using the ratio of position boundaries for step-down movement near a constraint, and / or using a bounded Gaussian-Seidel solver in medical remote operation, are discussed below in the context of an inner control loop (602) or in the nested control process of FIG. 6 in this specification. Any of these approaches may be used in other control processes, for example, having a single control loop for iterating through the solutions.

[0039] The iterative inverse kinematics solver (614) is part of the inner control loop (602). The inner control loop (602) also includes a boundary generator (612) and a termination check (616). The inner control loop (602) iterates through the solutions of the solver (614) until the termination check (616) is satisfied. The solver (614) optimizes the solutions. Additional, different, or fewer process units or software modules may be provided in the control process.

[0040] In the inner control loop (602), the boundary generator (612) sets boundary limits. The limits relate to hardware, safety, and / or other remote operation requirements that limit the position, velocity, acceleration, torque, and / or other motion characteristics of the robot arm (122) and / or surgical tool (220).

[0041] In one embodiment, all limits or boundaries for the linear component of motion, for the angular component of motion, and / or for both the linear and angular components of motion are included in a single solution as expressed by Act (311) (see FIG. 3). This single solution is for all motions of the robot arm (122) and surgical tool (220), for all linear motions, or for all angular motions. For example, all limits or boundaries for linear motion are provided in a single solution for linear motion, and all limits or boundaries for angular motion are provided in a single solution for angular motion. When repeatedly finding solutions for the conversion of user motion commands of the end effector (222) into joint motions, all limits are considered or dealt with in the solution. All limits are those used, such as position, velocity, and acceleration, rather than torque, where torque is not used to limit motion. In alternative embodiments, different solutions and solutions are provided for different sub-sets of the limits considered.

[0042] In one embodiment, the limits include positive and negative limits for each of the position, velocity, and acceleration boundaries. Positive and negative limits may be provided for torque boundaries. The magnitudes of the positive and negative limits may be the same or different. Limits are provided for each active joint and may be the same or different for different joints.

[0043] Two joint position limits for each active joint There is. The position limit ratio can be used to define a zone of deceleration to approach any of the position limits for any of the active joints in Act (312) (see FIG. 3). To decelerate motion when approaching the limits, two additional limits that form buffer zones by the position limit This is defined, and therefore is as follows.

[0044] [Equation 1]

[0045]

[0046] The offset can be any value, such as 5 mm.

[0047] These additional constraints are used to determine the joint position boundary ratios. Two joint position boundary ratios r are calculated for each joint, one for each constraint:

[0048] [Equation 2]

[0049] , and When,

[0050] When, , or

[0051] , when

[0052] From these two ratios, two joint motion boundaries are calculated:

[0053] [Equation 3]

[0054] , when or,

[0055] , when

[0056] Based on location limit ratios, It is used as a motion position boundary due to joint limit constraints. Position boundaries define deceleration zones for velocity and acceleration deceleration. Other ratios for deceleration and / or position limits or boundaries may be used.

[0057] For each joint, the joint velocity ω is a range configurable maximum speed limit to be within ... exists. Using the joint ratios from Equation 2, the motion boundaries resulting from the joint velocity constraints are determined as follows:

[0058] [Equation 4]

[0059]

[0060] It is used as a motion boundary due to joint velocity constraints.

[0061] For each joint, the joint position is When within, joint acceleration is within the range configurable maximum acceleration limit to remain within exists. Using the joint ratios from Equation 2, the motion boundaries resulting from joint acceleration constraints are determined as follows:

[0062] [Equation 5]

[0063]

[0064] It is used as a motion boundary due to acceleration limitation.

[0065] Joint torque limits can be incorporated into this framework. There are two force (torque) limits: a deceleration force / torque limit (F_slowdown) where motion slows down when passing this limit, and a stopping force / torque limit (F_stop) where motion stops when reaching this limit. The force ratio is calculated as follows:

[0066] When F < F_slowdown, force_ratio = 1.0

[0067] When F >= F_slowdown, force_ratio = (F - F_slowdown) / (F_stop - F_slowdown).

[0068] Other limits, calculations of limits, and / or ratios may be used. Ratios for stepping down motion in terms of position, velocity, and / or acceleration may also be applied to torque, or may not be used for one or more (e.g., all) of the limits.

[0069] To account for all limits in a single solution, the limits are combined. The combined boundaries are formed from joint position boundaries, joint velocity boundaries, and joint acceleration boundaries where force or torque is not limited. One exemplary combination is given by the following:

[0070] [Equation 6]

[0071]

[0072] These are used as the final combined boundaries generated by the boundary generator (612) for input to the solver (614). These combined boundaries are stored in memory and can be explored or generated as needed.

[0073] Where torque or force limits are included, the combined limits are scaled by the force ratio given by the following:

[0074] ,

[0075] Here, is calculated from Equation 6, and These are the final boundaries for the solver (614). Other approaches including force or torque limits may be used. Other combinations of limits may be used.

[0076] The solver (614) receives boundaries such as all boundaries to be used in a single solution, and motion commands from the generator (606). The solver (614) determines the joint motion as optimization or iterative fitting.

[0077] Any optimization, such as linear programming or quadratic programming, may be used. In one embodiment illustrated as Act (313) of FIG. 3, a bounded Gaussian-Seidel solver is used. A bounded Gaussian-Seidel solver may include sequential accelerated relaxation (Bounded Gaussian-Seidel Solver with Sequential Accelerated Relaxation (BGSR)), but may be provided without sequential accelerated relaxation. A BGSR is an iterative solver having the additional feature of clamping intermediate solutions to the boundaries. The main iterative step of a typical BGSR solver may be indicated as follows:

[0078] [Equation 7]

[0079]

[0080] In the case of,

[0081]

[0082] Here, r is the Sequential Acceleration Relaxation (SOR) ratio. am. k is the repetition index, and x i are the variables for which a solution will be found, and [ a ij ] and [ b i ] are linear coefficients. BGSR is [ a ij Convergence is guaranteed if ] is a positive definite matrix. In this implementation, a transformation is performed on the Jacobian matrix to satisfy this condition.

[0083] [Equation 8]

[0084]

[0085] Here, J is the Jacobian matrix (6xn) for 6 DOF, and is in Equation 7 x i As, these are the joint velocities (nx1) for which a solution is obtained, and is the command rate (6x1). "Joint" refers to active joints such as J6 to J11 in remote operation.

[0086] BGSR provides intermediate and final solutions through iterations. If one of the solutions requires position, velocity, acceleration, or other parameters passing through limits, the values ​​for the limits or boundaries are used in the solution in Act (314) (see FIG. 3). This clamps the intermediate or final solution to the boundaries in position, velocity, and / or acceleration. Such clamping may be provided in optimizations using solvers other than BGSR. Through clamping in one or more iterations, the solution can continue to discover joint motions as the final solution. The limits are used in the final solution or can be replaced by values ​​within the limits based on the optimization. The final solution is possible even when the limits are clamped to the boundaries.

[0087] The inner loop termination check (616) checks one or more criteria to terminate when one of the criteria is satisfied or to repeat when none of the criteria are satisfied. In one example, three criteria are used to terminate the repetition of the inner control loop (602): (1) (2) the number of iterations is greater than the maximum number of iterations, and (3) the error reduction in BGSR is too small between adjacent steps or iterations (e.g., 1.0 e-3 * 허용오차 ).

[0088] The solver (614) can be implemented as a one-stage solver that solves linear and angular motions together. In an alternative embodiment indicated as Act (315) (see FIG. 3), the linear and angular motions are solved separately. Two solving stages are provided. For example, the control process of FIG. 6 is performed separately for the linear motion stage and the angular motion stage, respectively. As another example, the inner control loop (602) is performed separately for the linear motion stage and the angular motion stage, and their outputs are then used together in the outer control loop (600). The linear motion and angular motion stages have independent iterative solutions. The linear motion is solved before the angular motion, but the reverse order may be used.

[0089] The separation of kinetic components involves independently finding solutions for the kinetic components using BGSR or another solver. The two functions for which solutions are obtained can be expressed as follows:

[0090] [Equation 12]

[0091]

[0092] Here, 6, 7, and 8 are for the active joints of the robot arm (122), and 9, 10, and 11 are for the active joints of the surgical tool (220). lower, upper, left, and right are related to parts of the Jacobian matrix for the corresponding joints.

[0093] By separating the linear and angular stages, linear motion and angular motion are solved independently. Any constraint violation affects linear motion and angular motion independently due to the separation. A constraint violation in linear motion may not occur in the angular motion component. When there is no constraint violation, an exact solution that satisfies the tolerance requirements is provided. Due to clamping at the boundary, a suboptimal solution that still satisfies the tolerance requirements is provided even when there is a constraint violation.

[0094] The solution is obtained for 6 DOF. If the joint commands (611) are commands for each of the 6 joints, the solver is applied directly. If the robot arm (122) and the surgical instrument (220) have fewer DOFs, zero padding may be used to still provide a solution in Act (316) of FIG. 3. Zeros are used in the terms for linear and / or angular motion that are missing from the 6 DOF. For 5 DOF surgical instruments, a linear or angular motion solver may be applied directly (e.g., 3 DOF for linear), and the Jacobian matrix for angular or linear motion has only 2 DOFs (e.g., 2 DOF for angular). The Jacobian matrix (for angular motion in the example below) is extended to 3 DOFs, as expressed by:

[0095]

[0096]

[0097] For 4 DOF mechanisms, linear or angular motion solvers can be applied directly, and the Jacobian matrix for angular or linear motion has only 1 DOF. The Jacobian matrix (for angle in the example below) is extended to 3 DOF, as expressed by:

[0098]

[0099]

[0100] In Act (320) of FIG. 3, the control processor causes the movement of the robot arm (122) and / or surgical tool (220). Output movement commands (611) for the active joints during remote operation cause the joints to change their position in velocity and / or acceleration. Results from iterative inverse kinematics control the movement of the joints (e.g., joints (J6 to J11)). In separate linear and angular solutions, the linear and angular movements control the movement of the corresponding joints.

[0101] Even when a suboptimal solution is provided, real-time control of the robot arm (122) and surgical tool (220) is provided. The iterative solution converts user commands into robot movements.

[0102] FIG. 7 illustrates a block diagram of an embodiment of a surgical robot system for medical remote operation. The system performs the method of FIG. 3, the control process of FIG. 6, or other methods. Any one or more (e.g., all) of the solution approaches of FIG. 3, acts (311 to 316) may be used. An iterative inverse kinematic solution is provided for the control of the robot arm (122).

[0103] A surgical robot system comprises one or more robot arms (122), said

[0104] Each of the robot arms (122) includes one or more links and joints. The joints may be pitch or roll joints. Tool actuators and cannulas for receiving and guiding surgical tools may be provided on each of the robot arms (122). Different combinations of links and joints may define or form different parts of the robot arms (122), e.g., different degrees or types of movement (e.g., translation and / or rotation). Any currently known or later-developed robot arm (122) having motors, sensors, links, joints, controllers, surgical instruments, and / or other structures may be used.

[0105] One or more robot arms are provided. For example, three or four robot arms (122) are provided. The robot arms (122) are mounted on a table, for example, the base of a surgical table. Alternatively, a cart, floor, ceiling, or other mounting point may be used. The robot arms (122) include a cable or wireless transceiver for communication with a processor (206) or an intermediate (for example, a control tower (130)).

[0106] The robotic surgical instrument (220) is one or more graspers, retractors, scalpels, endoscopes, staplers, scissors, or other surgical devices for manipulating the patient's tissue. The tissue manipulation may be direct, such as cutting or grasping. The tissue manipulation may be indirect, such as pressing or contacting the tissue when the endoscope is guided to image or view the patient's internal parts. Different or identical types of instruments (220) may be mounted on different arms among the robotic arms (122). For example, two robotic arms (122) may have graspers, a third robotic arm (122) may have a scalpel, and a fourth robotic arm (122) may have an endoscope.

[0107] The robotic surgical instrument (220) is connected to the distal end of the robotic arm (122), but may be connected at other locations. The connection provides an actuator to enable the tool to operate, such as closing a gripper or scissors.

[0108] The user counsel (110) is a graphical user interface for interaction between a surgeon and a processor for controlling a surgical robot system, e.g., a robot arm (122). The user interface includes a user input section (708) and a display (118). The user input section (708) and / or the display (118) are provided at the user console (110) and / or the control tower (130), but may be located at other locations.

[0109] The user input unit (708) is a button, keyboard, rocker, joystick, trackball, voice recognition circuit, mouse, touchpad, touchscreen, slider, switch, UID (116), foot pedal (114), a combination thereof, or any other input device for inputting into a surgical robot. The display (118) is a monitor, liquid crystal display (LCD), projector, plasma display, CRT, printer, or any other device currently known or later developed for outputting visual information. In an alternative embodiment, the display (118) is a head-mounted display. The user input unit (708) may be a sensor or sensors for detecting eye movements and / or blinking. In other embodiments, the user input unit (708) is a microphone for voice-based input. A speaker for outputting audio information may be provided instead of or in addition to the display (118).

[0110] An optional camera (706) is a digital camera for optical tracking of user movements, such as tracking during the use of the UID (116) to control the robot arm (122). In some embodiments, the camera (706) may be a stereo camera and / or a depth camera. The camera (706) is positioned relative to the user and the target pattern or user counselor (110) to track human movements in controlling the robot arm (122) or tracking the target pattern with the user input unit (708).

[0111] The controller (702) is a controller that drives and / or models the robot arm (122) and / or surgical instrument (220). The controller (702) is a general processor, central processing unit, control processor, graphics processor, graphics processing unit, digital signal processor, application integrated circuit, field programmable gate array, digital circuit, analog circuit, artificial intelligence processor, combination thereof, or other device currently known or to be developed later for converting user commands into joint commands for the robot arm (122) and / or surgical instrument (220). The controller (702) is a single device or multiple devices that operate serially, in parallel, or individually. The controller (702) may be the main processor of a computer such as a laptop, server, workstation, or desktop computer, or a processor for handling several tasks in a larger system. Based on hardware, software, firmware, or a combination thereof, the controller (702) is configured to implement commands or perform acts.

[0112] The controller (702) is configured to find a solution for the motion of the robotic arm (122) and / or surgical instrument (220) during medical remote manipulation of the patient and in response to user input of motion commands via user counselor (110). The controller (702) is configured to implement a solver such as BGSR. An iterative solution is provided using a bounded Gaussian-Seidel solver with sequential acceleration relaxation. The solution includes all constraints on the position, velocity, and acceleration of the robotic arm and surgical instrument in a single solution. All constraints used in translation are handled by the solver to determine a single solution for inverse kinematics. The controller (702) may be configured to find a solution for linear and angular components, wherein the solution for linear components and the solution are independent of the solution for angular components. The solution handles constraint violations without failure. For example, the intermediate solution of the bounded Gaussian-Seidel solver sets the position, velocity, or acceleration to the boundaries of the robot arm (122) and / or surgical instrument (220). The solver determines the joint motion without violating the boundaries.

[0113] The controller (702) is configured to control the robot arm (122) and the surgical tool (220). Based on the solution, one or more joints move in response to user commands. The iterative inverse kinematic solution controls the joints.

[0114] Memory (704) or other memory is a non-transient computer-readable storage medium that stores data representing instructions executable by a programmed controller (702). Instructions for implementing the processes, methods, and / or techniques discussed herein are provided on a computer-readable storage medium or memory, e.g., a cache, a buffer, RAM, a removable medium, a hard drive, or other computer-readable storage medium. Computer-readable storage media include various types of volatile and non-volatile storage media. A function, act, or task illustrated in the drawings or described herein is executed in response to one or more sets of instructions stored on or on a computer-readable storage medium. A function, act, or task may be performed by software, hardware, integrated circuits, firmware, microcode, etc., which are independent of a specific type of instruction set, storage medium, processor, or processing strategy and operate alone or in combination. Likewise, a processing strategy may include multiprocessing, multitasking, parallel processing, etc.

[0115] In one embodiment, the instruction is stored on a removable media device for reading by a local or remote system. In another embodiment, the instruction is stored at a remote location for transmission over a computer network or a telephone line. In yet another embodiment, the instruction is stored within a given computer, CPU, GPU, or system.

[0116] Instructions are provided for any of the acts discussed in this specification. Instructions are for sizing, for example, for designing and / or testing a surgical robot. Instructions are for generating and / or generating a relationship between velocity and acceleration, generating a synthetic trajectory, or generating a virtual trajectory.

[0117] Although the present invention has been described above with reference to various embodiments, it should be understood that many changes and modifications can be made without departing from the scope of the invention. Accordingly, the foregoing detailed description should be regarded as illustrative rather than limiting, and it is intended to be understood that the following claims, which include all equivalents, are intended to limit the spirit and scope of the invention.

Claims

Claim 1 A computer-readable recording medium storing commands for teleoperation of a surgical robot system, wherein, when the commands are executed by a controller, the controller receives a user command to move a surgical tool mounted on a robotic arm during the teleoperation; wherein, by a first optimization of a solver, a solution for the linear motion of the movement of the surgical tool is repeatedly obtained in a first stage of kinematics, the solution obtained in the first stage provides a solution from the first optimization for the linear motion without obtaining a solution for the angular motion; wherein, by a second optimization of the solver, a solution for the angular motion of the movement is repeatedly obtained in a second stage separate from the first stage of kinematics, the solution obtained in the second stage provides a solution from the second optimization for the angular motion without obtaining a solution for the linear motion, and the solution obtained in the second stage and the corresponding solution for the angular motion are the first A computer-readable recording medium for moving a robot arm and / or surgical tool based on the solutions for the linear motion and the solutions for the angular motion, wherein the solution for the linear motion and the corresponding second optimization are separated as a solution for the linear motion and the corresponding first optimization, and any violation of a restriction affects the linear motion and the angular motion independently due to the separation; and a computer-readable recording medium for moving the robot arm and / or surgical tool based on the solutions for the linear motion and the angular motion. Claim 2 A computer-readable recording medium according to claim 1, wherein receiving during the remote operation includes receiving when the robot arm has a remote center of motion fixed at the patient entry point and has one or more joints locked in place. Claim 3 A computer-readable recording medium according to claim 1, wherein receiving comprises receiving the user command when the remote operation is for less than 6 degrees of freedom, and solving for the linear motion and / or solving for the angular motion comprises solving with a Jacobian matrix for 6 degrees of freedom having zero terms for the linear and / or angular motion not in less than 6 degrees of freedom. Claim 4 A computer-readable recording medium according to claim 1, wherein finding a solution for the linear motion comprises finding a solution with all restrictions on the linear component of the motion of the robot arm in the first single solution, and finding a solution for the angular motion comprises finding a solution with all restrictions on the angular component of the motion of the robot arm in the second single solution. Claim 5 A computer-readable recording medium according to claim 4, wherein the first single solution comprises a first claim including positive and negative position, velocity, and acceleration boundaries as all of the limits, and the second single solution comprises a second claim including positive and negative position, velocity, and acceleration boundaries as all of the limits. Claim 6 A computer-readable recording medium according to claim 1, wherein finding a solution for the linear motion and / or finding a solution for the angular motion comprises finding a solution with respect to a limitation violation that independently affects the linear motion and the angular motion due to the separate first and second stages. Claim 7 A computer-readable recording medium according to claim 1, wherein finding a solution for the linear motion and / or finding a solution for the angular motion comprises finding a solution with a position, velocity, or acceleration limit used as a result in the solution. Claim 8 A computer-readable recording medium according to claim 1, wherein solving for the linear motion and solving for the angular motion each include solving with a first control loop nested in a second control loop, the first control loop includes a boundary generator operating with a solver and a termination check for iterative solving by the solver, and the second control loop includes forward kinematic determination of the position of the surgical tool, command generation of the movement and the position of the surgical tool from the user command, the movement provided to the second control loop, and a second termination check receiving termination from the first control loop. Claim 9 A computer-readable recording medium according to claim 1, wherein finding a solution for the linear motion and / or finding a solution for the angular motion comprises finding a solution with a position limit ratio that defines a region of deceleration to approach the position limit of the joint of the robot arm. Claim 10 A computer-readable recording medium according to claim 1, wherein finding a solution for the linear motion and / or finding a solution for the angular motion comprises finding a solution using a solver including a bounded Gauss-Seidel solver. Claim 11 A computer-readable recording medium according to claim 10, wherein obtaining a solution with the bounded Gauss-Seidel solver includes obtaining a solution with successive-over-relaxation. Claim 12 In paragraph 10, the bounded Gauss-Seidel solver clamps intermediate solutions to boundaries in the position, velocity, or acceleration of the robot arm, a computer-readable recording medium. Claim 13 A computer-readable recording medium storing commands for remote operation of a surgical robot system, wherein the commands, when executed by a controller, cause the controller to receive an input command to move a surgical tool mounted on the distal end of a robot arm during the remote operation; to obtain a solution for the motion of the robot arm or the surgical tool using an inverse kinematic solver as an iterative solution, wherein the iterative solution includes a solution that considers all constraints on the motion within an optimization corresponding to the iterative solution for obtaining the solution, wherein the constraints on the motion include position constraints, velocity constraints, and acceleration constraints, wherein the position constraints, velocity constraints, and acceleration constraints each include hardware, safety, or operation boundaries for the motion; and a computer-readable recording medium that moves the surgical tool and / or the robot arm based on the iterative solution. Claim 14 A computer-readable recording medium according to claim 13, wherein obtaining the solution comprises obtaining the solution using a bounded Gaussian-Seidel solver having sequential acceleration relaxation. Claim 15 A computer-readable recording medium according to claim 13, wherein obtaining the above solution comprises clamping an intermediate solution of the above iterative solution to one of the above limitations. Claim 16 A computer-readable recording medium according to claim 13, wherein obtaining the solution comprises obtaining a solution for linear motion in one stage and obtaining a solution for angular motion in another stage, each of said stages having the iterative solution independent of the other stage. Claim 17 A surgical robot system for remote operation comprising: a robot arm; and a surgical instrument coupled to the distal end of the robot arm; A surgical robot system for remote operation comprising a controller configured to obtain a solution for the motion of the robot arm and / or surgical instrument during the remote operation and in response to a motion command, wherein the solution uses a bounded Gauss-Seidel solver having sequential acceleration relaxation in iterative solving, the controller is configured to clamp an intermediate solution of the iterative solving of the bounded Gauss-Seidel solver to a position, velocity, or acceleration boundary of the robot arm and / or surgical instrument, the clamping enables the bounded Gauss-Seidel solver to determine the motion without violating the position, velocity, or acceleration boundary in additional iterations of the iterative solving, and the intermediate solution is a solution for the position, velocity, or acceleration prior to at least the last iteration of the bounded Gauss-Seidel solver, wherein the position, velocity, or acceleration of the intermediate solution is set as the position, velocity, or acceleration boundary by the clamping. Claim 18 A surgical robot system for remote operation, wherein, in paragraph 17, the controller is configured to obtain a solution using the bounded Gauss-Seidel solver, and the solution includes all constraints on the position, velocity, and acceleration of the robot arm and surgical instrument in the solution. Claim 19 A surgical robot system for remote operation according to claim 17, wherein the controller is configured to obtain solutions for linear and angular components of the motion, and the solution for the linear components is independent of the solution for the angular components. Claim 20 delete

Images