COOPERATIVE SURGICAL SYSTEM HAVING OPTICAL SCANNER FOR USE WITH SURGICAL INSTRUMENTS TO PERFORM LAPAROSCOPIC SURGERY - Patent application

Abstract

Described herein is a collaborative robotic system that can be used to assist in laparoscopic surgical procedures. The collaborative robotic system allows the surgeon to use commercially available surgical tools while providing the benefits associated with surgical robotics. Advantageously, the surgical tools can be seamlessly coupled to the robotic arms using disposable couplers while the reusable portions of the robotic arms remain in the sterile drape. Additionally, the collaborative robotic system can operate in multiple modes, improving usability and safety while allowing the surgeon to position the instruments directly with the instrument handles and still maintain the desired position of the instruments using the robotic arms.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to PCT Patent Application No. PCT / IB2022 / 052989, filed March 30, 2022, European Patent Application No. 21306904.0, filed December 22, 2021, European Patent Application No. 21306905.7, filed December 22, 2021, and European Patent Application No. 21305929.8, filed July 5, 2021, the contents of each of which are incorporated herein by reference in their entirety.

[0002] The present disclosure is directed to a collaborative robotic system having an optical scanner for assisting laparoscopic surgical procedures. [Background technology]

[0003] Managing visibility and access during a laparoscopic procedure is a challenge. Surgical assistant paradigms are inherently incomplete because the assistant is required to anticipate and understand from the surgeon's perspective without standing where the surgeon stands, as well as anticipate and adjust the degree to which the surgeon desires the tissue of interest to be exposed throughout the procedure. For example, during a laparoscopic procedure, one assistant may be required to hold a retractor device and expose tissue for the surgeon, while another assistant may be required to hold a laparoscopic device and provide the surgeon with a view of the surgical space within the patient during the procedure, one of whom may be required to hold an individual tool in a non-practical position, such as between the surgeon's arms while the surgeon is actively operating additional surgical instruments.

[0004] Various attempts have been made at solving this problem. For example, a rail-mounted orthopedic retractor, which is a purely mechanical device that is mounted to the patient bed / table, may be used to hold a laparoscopic device in place during a laparoscopic procedure, and another rail-mounted orthopedic retractor may be used to hold a retractor device in place during a laparoscopic procedure. However, rail-mounted orthopedic retractors require extensive manual interaction to unlock, reposition, and lock the tools in place.

[0005] Complex robotic-assisted systems, such as the Da Vinci Surgical System (marketed by Intuitive Surgical, Sunnyvale, California), are being used by surgeons to enhance laparoscopic surgical procedures by allowing them to perform the procedure remotely from a surgeon console remote from a patient console where the surgeon holds the surgical instruments. Such complex robotic-assisted systems are very expensive and have a very large footprint, occupying much of the space in the operating room. Furthermore, such robotic-assisted systems typically require unique, system-specific surgical instruments to be compatible with the system, and therefore, surgeons are unable to use the standard off-the-shelf surgical instruments that they are accustomed to. Thus, surgeons are required to learn how to perform an entirely different laparoscopic procedure.

[0006] In light of the aforementioned shortcomings of previously known systems and methods, there is a need for a system that provides surgeons with the ability to seamlessly position and manipulate a variety of surgical instruments as needed, thus avoiding the workflow limitations inherent in both human and mechanical solutions. Summary of the Invention [Means for solving the problem]

[0007] The present disclosure overcomes the shortcomings of previously known systems and methods by providing a co-operative surgical system for assisting in laparoscopic surgery performed using a surgical instrument having a handle, a working end, and an elongated shaft therebetween. The co-operative surgical system may include a robotic arm having a proximal end, a distal end that may be removably coupled to the surgical instrument, a number of connections, and a number of joints between the proximal and distal ends. The co-operative surgical system may further include a controller operably coupled to the robotic arm. The controller may be programmed to cause the robotic arm to automatically switch between a passive mode, where in response to determining that movement of the robotic arm due to movement at the surgical instrument handle is less than a predetermined amount for at least a predetermined dwell period, the controller may be programmed to cause the robotic arm to maintain a static position in the passive mode, and a collaborative manipulation mode, where in response to determining that a force applied at the robotic arm due to a force applied at the surgical instrument handle exceeds a predetermined threshold, the controller may be programmed to allow the robotic arm to be freely movable in response to movement at the surgical instrument handle in the collaborative manipulation mode to perform laparoscopic surgery using the surgical instrument, and the controller may be programmed to apply a first impedance to the robotic arm in the collaborative manipulation mode to account for the weight of the surgical instrument and the robotic arm. The controller may further be programmed to automatically switch the robotic arm to a haptic mode in response to determining that at least a portion of the robotic arm is outside a predefined haptic barrier, and in the haptic mode, the controller may be programmed to apply a second impedance to the robotic arm that is greater than the first impedance, thereby making movement of the robotic arm more viscoelastic in the haptic mode than in the collaborative manipulation mode in response to movement at the handle of the surgical instrument.

[0008] Additionally, the collaborative surgical system may include a base rotatably coupled to a proximal end of the robot arm such that the robot arm may move relative to the base. For example, the base may be rotatable about a first axis such that rotation of the base causes rotation of the robot arm about the first axis. Thus, the system may further include a first motor disposed within the base and operably coupled to the base such that a controller is operably coupled to the first motor and programmed to cause the first motor to apply an impedance to the base. Furthermore, a proximal end of a shoulder link of the multiple links may be rotatably coupled to the base at a shoulder joint of the multiple joints such that rotation of the shoulder link causes rotation of a link of the multiple links distal to the shoulder link about a second axis of the shoulder joint. Thus, the system may further include a second motor disposed within the base and operably coupled to the shoulder joint such that a controller is operably coupled to the second motor and programmed to cause the second motor to apply an impedance to the shoulder joint. For example, the second axis may be perpendicular to the first axis.

[0009] Further, a proximal end of the elbow link of the plurality of links may be rotatably coupled to a distal end of the shoulder link at the elbow joint of the plurality of joints such that rotation of the elbow link causes rotation of the link of the plurality of links distal to the elbow link about a third axis of the elbow joint. Thus, the system may further include a third motor disposed within the base and operably coupled to the elbow joint such that the controller is operably coupled to the third motor and programmed to cause the third motor to apply an impedance to the elbow joint. The shoulder link may include a proximal shoulder link rotatably coupled to the base and a distal shoulder link rotatably coupled to the elbow link. The distal shoulder link may be rotatable relative to the proximal shoulder link such that rotation of the distal shoulder link relative to the proximal shoulder link causes rotation of the link of the plurality of links distal to the distal shoulder link to rotate about a fourth axis parallel to the longitudinal axis of the shoulder link.

[0010] The system may further include an actuator that may be actuated to allow rotation of the distal shoulder link relative to the proximal shoulder link, and in an unactuated state, the actuator prevents rotation of the distal shoulder link relative to the proximal shoulder link. Additionally, a proximal end of a wrist link of the plurality of links may be rotatably coupled to a distal end of the elbow link at a proximal wrist joint of the plurality of joints such that the wrist link may be rotated relative to the elbow link about a fifth axis of the proximal wrist joint. The system may further include an actuator that may be actuated to allow rotation of the wrist link relative to the elbow link, and in an unactuated state, the actuator prevents rotation of the wrist link relative to the elbow link. The wrist link may include a proximal wrist link rotatably coupled to a distal end of the elbow link, a central wrist link rotatably coupled to the proximal wrist link about a sixth axis, and a distal wrist link rotatably coupled to the central wrist link about a seventh axis. The distal wrist link may be removably coupled to a surgical instrument.

[0011] The system may further include a platform coupled to the base. The platform may enable vertical and horizontal movement of the base relative to the platform, thereby causing vertical and horizontal movement of the robotic arm relative to the platform. The platform may include a number of wheels that may enable mobility of the platform, the wheels having a braking mechanism that may be actuated to prevent mobility of the platform. Additionally, the controller may be programmed to receive information associated with a surgical instrument coupled to a distal end of the robotic arm, the information including at least one of an instrument type, weight, center of mass, length, or instrument shaft diameter.

[0012] The system may further include a database having information associated with a plurality of surgical instruments, and the controller is programmed to access the database and retrieve information associated with a surgical instrument coupled to the distal end of the robotic arm. In addition, the system may include an optical scanner that can measure depth data such that the controller is programmed to identify a surgical instrument coupled to the distal end of the robotic arm based on the measured depth data. Furthermore, the controller may be programmed to be calibrated to the surgical instrument when the surgical instrument is coupled to the distal end of the robotic arm.

[0013] The system may further include a base housing at a proximal end of the robot arm such that the motors for the robot arm are all disposed within the base housing, and motors for controlling the robot arm. For example, the system may further include a base rotatably coupled to the proximal end of the robot arm such that the robot arm may move relative to the base, and a plurality of motors disposed within the base operably coupled to at least some of the plurality of joints such that a controller is operably coupled to the plurality of motors and programmed to measure currents in the plurality of motors.

[0014] The controller may further be programmed to calculate a force applied to the distal end of the robotic arm based on the measured currents of the multiple motors. Additionally, the controller may be programmed to determine in real time an entry point of the surgical instrument into the patient based on a longitudinal axis of the surgical instrument when the surgical instrument is coupled to the distal end of the robotic arm. For example, the controller may be programmed to determine in real time an entry point of the surgical instrument into the patient by determining an intersection point of multiple imaginary lines parallel to the longitudinal axis of the surgical instrument as the surgical instrument moves relative to the entry point. In addition, the controller may be programmed to calculate a force applied to the working end of the surgical instrument based on the force applied to the distal end of the robotic arm, the length of the surgical instrument, the center of mass of the surgical instrument, and the entry point. In addition, the controller may be programmed to calculate a force applied to the patient at the entry point of the surgical instrument into the patient based on the force applied to the distal end of the robotic arm, the center of mass of the surgical instrument, and the entry point. The controller may be further programmed to detect a fault condition of the collaborative surgical system, and if a critical fault condition is detected, the controller may cause actuation of braking of the multiple motors. Additionally, the controller may be programmed to apply a third impedance to the robotic arm to resist movement of the robotic arm if a force applied to a distal end of the robotic arm exceeds a predetermined force threshold within a predetermined period of time.

[0015] The system may further include a plurality of encoders disposed on at least some of the joints, the plurality of encoders may measure angles of corresponding links of the plurality of links at at least some of the joints such that the controller may be programmed to determine a position of the distal end of the robot arm in 3D space based on angle measurements by the plurality of encoders. Additionally, the system may include one or more indicators disposed on at least one link of the plurality of links of the robot arm, the one or more indicators may illuminate a plurality of colors, each color indicating a state of the collaborative surgical system. For example, a first color of the plurality of colors may indicate that the robot arm is in a passive mode, a second color of the plurality of colors may indicate that the robot arm is in a collaborative mode, and a third color of the plurality of colors may indicate that the robot arm is in a haptic mode. Additionally, a fourth color of the plurality of colors may indicate that a fault condition of the collaborative surgical system has been detected by the controller. Additionally, a fifth color of the plurality of colors may indicate that a surgical instrument is not coupled to the distal end of the robot arm.

[0016] A predefined force sense barrier may be used to guide a surgical instrument coupled to a distal end of the robotic arm and assist in laparoscopic surgery. For example, the predefined force sense barrier may be a force sense funnel that may guide a surgical instrument coupled to a distal end of the robotic arm into a trocar. The controller may be programmed to apply a third impedance to the robotic arm and account for the weight of the robotic arm when a surgical instrument is not coupled to the distal end of the robotic arm. Additionally, in a passive mode, the controller may be programmed to apply a third impedance to the robotic arm and account for the force applied to the distal end of the robotic arm due to the weight of the surgical instrument, the weight of the robotic arm, and an external force applied to the surgical instrument, and cause the robotic arm to maintain a static position.

[0017] The system may further include a graphical user interface that may display information associated with a surgical instrument coupled to a distal end of the robotic arm. The graphical user interface may allow a user to adjust at least one of a predetermined amount of movement or a predetermined dwell period at the handle of the surgical instrument to automatically switch the robotic arm to a passive mode, a predetermined threshold of force applied at the handle of the surgical instrument to automatically switch the robotic arm to a collaborative manipulation mode, a position of a predefined force sense barrier, an identification of a surgical instrument coupled to a distal end of the robotic arm, a vertical height of the robotic arm, or a horizontal position of the robotic arm.

[0018] The system may further include a coupler body that may be removably coupled to the coupler interface disposed at the distal end of the robotic arm. The coupler body may have a lumen therethrough that is sized and shaped to receive the elongated shaft of a surgical instrument and may transition between an open state, in which the elongated shaft is slidably movable within the lumen, and a closed state, in which longitudinal movement of the elongated shaft relative to the coupler body is prevented, while rotational movement of the elongated shaft relative to the coupler body may be permitted in response to movement at the handle of the surgical instrument. For example, when the coupler body is coupled to the coupler interface in the closed state, the robotic arm may be permitted to be freely movable in response to movement at the handle of the surgical instrument to perform a laparoscopic surgical procedure if the force applied at the handle of the surgical instrument exceeds a predetermined threshold due to a force applied at the robotic arm. In the closed state, longitudinal movement of the elongation shaft relative to the coupler body may be prevented due to frictional forces between the lumen of the coupler body and the elongation shaft of the surgical instrument, while rotational movement of the elongation shaft relative to the coupler body is permitted in response to movement at the handle of the surgical instrument.

[0019] Additionally, the coupler body may be removably coupled to the coupler interface via a magnetic connection. The controller may be programmed to determine an orientation of the surgical instrument relative to the distal end of the robotic arm based on alignment of the magnetic connection when the coupler body is coupled to the coupler interface. The system may further include a sterile drape that may be placed between the coupler body and the coupler interface to prevent contact between the surgical instrument and the robotic arm during laparoscopic surgery. The distal end of the robotic arm may be removably coupled to at least one of a laparoscope, a retractor tool, a grasper tool, or a surgical cutting tool. For example, when the distal end of the robotic arm is coupled to the laparoscope, the controller may be programmed to optically track end effectors of one or more surgical instruments within a field of view of the laparoscope and automatically switch the robotic arm to a robot-assisted mode in response to determining that the end effectors of the one or more surgical instruments are not within a predefined boundary within the field of view of the laparoscope. Additionally, the controller may be programmed to cause the robotic arm to move the laparoscope and adjust the laparoscope's field of view so that the end effectors of one or more surgical instruments are within predefined boundaries within the laparoscope's field of view.

[0020] The collaborative surgical system may not be remotely operated via user input received at the remote surgeon console. Additionally, the collaborative surgical system may be structured such that the surgeon performing the laparoscopic surgery does not touch any part of the collaborative surgical system and move the surgical instrument while performing the laparoscopic surgery. Additionally, the system may include an optical scanner, e.g., a LiDAR device, to measure depth data. For example, the controller may be programmed to determine whether a movement applied to a surgical instrument coupled to a distal end of the robotic arm is by an intended user. Additionally, the controller may be programmed to identify a surgical instrument coupled to a distal end of the robotic arm based on the depth data.

[0021] Additionally, the system may include a second robotic arm having a distal end, multiple connections, and multiple joints between the proximal and distal ends that may be removably coupled to a second surgical instrument having a proximal end, a handle, a working end, and an extension shaft therebetween. Thus, the controller may be operatively coupled to the second robotic arm and programmed to cause the second robotic arm to maintain a static position in a passive mode, in which in response to determining that movement of the second robotic arm due to movement at the handle of the second surgical instrument is less than a predetermined amount for at least a predetermined dwell period associated with the second robotic arm, the controller causes the second robotic arm to maintain a static position in the passive mode, and in response to determining that a force applied at the second robotic arm due to a force applied at the handle of the second surgical instrument exceeds a predetermined threshold associated with the second robotic arm, the controller causes the second robotic arm to move the handle of the second surgical instrument to perform a laparoscopic surgical procedure using the second surgical instrument in the collaborative manipulation mode. The second robotic arm may be programmed to allow the second robotic arm to be freely movable in response to movement in the handle of the second surgical instrument, and the controller may be programmed to apply a third impedance to the second robotic arm in the collaborative manipulation mode to take into account the weight of the second surgical instrument and the robotic arm, and optionally a haptic mode, in response to determining that at least a portion of the second robotic arm is outside a predefined haptic barrier, the controller may be programmed to apply a fourth impedance to the second robotic arm that is greater than the third impedance in the haptic mode, thereby making movement of the second robotic arm more viscoelastic in the haptic mode than in the collaborative manipulation mode in response to movement in the handle of the second surgical instrument.

[0022] According to another aspect of the present disclosure, a collaborative robotic surgical device for manipulating an instrument is provided. The device may include a base portion, a first arm coupled to the base portion, a motor coupled to the first arm that can rotate the first arm relative to the base portion, an instrument coupled to an end portion of the first arm, and a controller that can be programmed to control the first arm according to at least two of the following operating modes: a passive assistance mode, a collaborative assistance mode, a robotic assistance mode, and a haptic mode. For example, in the passive assistance mode, the first arm is static. In the collaborative assistance mode, the first arm may be freely movable by the operator, while the motor at least partially simultaneously moves the first arm to improve the position and / or orientation of the instrument coupled to the end portion of the first arm and / or at least compensate for gravity on the first arm and the instrument coupled to the end portion of the first arm. In the robotic-assisted mode, the motor may move the first arm and reposition an instrument coupled to an end portion of the first arm. In the haptic mode, the first arm may be movable by an operator, while the motor compensates for gravity on at least the first arm and / or an instrument coupled to the end portion of the first arm, guides the instrument along at least a predefined trajectory, prevents undesired movement of the first arm and / or an instrument coupled to the end portion of the first arm, prevents movement of the first arm outside of a particular space and / or prevents movement of the first arm into a particular space.

[0023] In one embodiment, the controller may be switchable between any of at least three of the operating modes. Alternatively, the controller may be switchable between any of four of the operating modes. The co-operated robotic surgical device may be programmed to automatically identify a particular instrument coupled to the first arm end portion using an RFID transmitter chip, a barcode, a near-field communication device, a Bluetooth transmitter, and / or the weight of the instrument coupled to the first arm end portion. Additionally, the co-operated robotic surgical device may be programmed to automatically change to a predetermined one of the operating modes when a particular instrument is coupled to the first arm end portion without any additional input from the operator. For example, the co-operated robotic surgical device may be programmed to change to a passive assistance mode when a particular instrument is coupled to the first arm end portion without any additional input from the operator.

[0024] According to another aspect of the present invention, another cooperatively manipulated surgical system is provided for assisting in laparoscopic surgery performed using a surgical instrument having a handle, a working end, and an elongated shaft therebetween. The cooperatively manipulated surgical system may include a robotic arm having a proximal end, a distal end that may be removably coupled to the surgical instrument, a number of connections, and a number of joints between the proximal and distal ends. The distal end of the robotic arm may include a coupler interface. The system may further include a coupler body that may be removably coupled to the coupler interface. The coupler body may include a lumen therethrough that is sized and shaped to receive the elongated shaft of the surgical instrument, and may transition between an open state, in which longitudinal movement of the elongated shaft relative to the coupler body is prevented, while rotational movement of the elongated shaft relative to the coupler body may be permitted in response to movement in the handle of the surgical instrument. For example, when the coupler body is coupled to the coupler interface in a closed state, the robotic arm is permitted to be freely movable in response to movements in the handle of a surgical instrument to perform laparoscopic surgery.

[0025] The coupler body may be removably coupled to the coupler interface via a magnetic connection. Thus, the controller may be programmed to determine an orientation of the surgical instrument relative to the distal end of the robotic arm based on an alignment of the magnetic connection when the coupler body is coupled to the coupler interface. The system may further include a sterile drape that may be placed between the coupler body and the coupler interface to prevent contact between the surgical instrument and the robotic arm during laparoscopic surgery. The coupler body may be disposable after a single laparoscopic surgery.

[0026] According to another aspect of the present invention, a device for coupling an instrument, e.g., a laparoscopic surgical instrument or an endoscope, to the arm of a surgical robot is provided. The device may include a body sized and shaped to selectively couple with an instrument for use in a surgical procedure, and an interface that may selectively couple with the body and may couple with an end portion of the robot arm. For example, the device may allow the instrument to rotate about its longitudinal axis relative to the device, and may further prevent longitudinal movement of the instrument relative to the device. The body may be crimped around a portion of the outer surface of the instrument. For example, the body may include a first portion coupled with a second portion using a hinge, and the first portion may rotate about the hinge relative to the second portion to selectively crimp the instrument into a recess formed in the body.

[0027] In addition, the body may crimp around a portion of the outer surface of the instrument, preventing rotational movement of the instrument relative to the body under normal operating conditions. For example, the interface may include a recess sized and shaped to removably receive the body therein. The recess in the interface may prevent longitudinal movement of the body relative to the interface and allow rotational movement of the body relative to the interface. Additionally, the device may move between a first state in which the instrument is removable from the device and a second state in which the instrument is not removable from the device. The body may have one or more protrusions extending away from the surface of the body, and the interface may have one or more recesses for receiving the one or more protrusions and aligning the body and the interface.

[0028] According to yet another aspect of the present invention, a collaborative surgical robot system for performing a surgical procedure is provided. The system may include a first surgical robot having a base, an arm coupled to the base, and a motor coupled to the arm that may move the arm relative to the base, a controller programmed to control the arm, and an optical scanner that may collect depth data. For example, the optical scanner may collect depth data related to a position and orientation of an instrument relative to the collaborative surgical robot. The system may be programmed to use the depth data to determine whether an instrument is coupled to the first surgical robot. Additionally, the system may be programmed to determine an identity of the instrument based at least in part on the depth data.

[0029] The optical scanner may collect depth data associated with the position and movement of an instrument, which may be held freely by the surgeon and not coupled to the surgical robot. Additionally, the optical scanner may collect depth data associated with a trocar inserted into the patient. Thus, the system may be programmed to move an arm and / or a base of the first surgical robot if the position of the trocar changes above a threshold amount. The system may further include a second surgical robot having a second base, a second arm coupled to the second base, and a second motor coupled to the second arm and capable of moving the second arm relative to the second base. The optical scanner may have an accuracy of at least 5 mm at a range of 10 meters. The optical scanner may further collect depth data associated with the surgeon's hand during the surgical procedure.

[0030] Further, the controller may be programmed to control the first surgical robot arm according to at least one of the following operating modes: a passive-assist mode, a collaborative-assist mode, a robot-assist mode, and a haptic mode, as described above. The optical scanner may use the depth data to identify a potential inadvertent collision between the first surgical robot arm and a patient, a support platform supporting at least the first surgical robot, another surgical robot, and / or another object in the operating room, and to warn a user of the potential inadvertent collision and / or block movement of the first surgical robot arm to avoid such a collision. In addition, the first surgical robot may be supported by a support platform, and the collaborative surgical robot system may be programmed to move the first surgical robot relative to the support platform and optimize the position of the first surgical robot on the support platform based on the depth data collected by the optical scanner. In addition, the optical scanner may collect depth data that is used to record the movement of the surgeon's hand during the surgical procedure.

[0031] According to another aspect of the present invention, another cooperative surgical robotic system for performing a surgical procedure is provided. The system may include a surgical robot having a base, an arm coupled to the base, and a motor coupled to the arm, an optical scanner capable of tracking movement of one or more objects around a patient, and a controller programmed to collect data regarding the movement of the one or more objects from the optical sensor and move the arm of the surgical robot in response to the movement of the one or more objects.

[0032] According to another aspect of the present invention, a cooperative robotic surgical system for assisting in the manipulation of an instrument is provided. The system may include a base, an arm coupled to the base, the arm having a plurality of arm sections and a plurality of articulation joints, a plurality of motors coupled to the arm, the plurality of motors being capable of rotating the plurality of arm sections about the plurality of articulation joints, and a controller programmed to control at least the plurality of motors. For example, the arm may be movable by a user applying a force directly on the arm and / or directly on an instrument coupled to the arm. Furthermore, the system may be programmed to collect data related to a first motion characteristic of the arm and / or the instrument coupled to the arm. In addition, the controller may be programmed to analyze the data related to the first motion characteristic, detect whether a first condition exists, and modify a first motion parameter of the arm if the first condition is detected.

[0033] The system may be programmed to compare data collected during the surgical procedure with historical data related to the same surgical procedure for the same user using the instrument to detect whether a first condition exists. The system may further include an optical scanner, one or more sensors positioned on the arm, and / or an endoscope to collect data related to a first operating characteristic of the arm and / or an instrument coupled to the arm. The controller may be programmed to automatically change the position and / or orientation of an imaging device supported by the arm to a preferred or optimal position and / or orientation if the position and / or orientation of the imaging device is not in a preferred or optimal position for the camera to capture an image of the instrument. Additionally, the controller may be programmed to detect whether an instrument coupled to the arm has been replaced.

[0034] Additionally, the system may be programmed to detect the magnitude and duration of one or more forces applied to the first robotic arm, and further detect that a first condition exists when a change in the force applied to the arm meets or exceeds a first predetermined value over a threshold duration. The system may be further programmed to calculate an actual or approximate actual direction in which an end effector at a distal end of the arm is pointing and a calculated or approximate calculated direction in which the end effector would point if an instrument were coupled to the end effector, compare the actual or approximate actual direction to the calculated or approximate calculated direction, and determine whether the actual or approximate actual direction and the calculated or approximate calculated direction differ. The controller may be programmed to update a data file associated with the second instrument when the first instrument coupled to the arm is replaced by a second instrument, the data file including at least a center of gravity of the second instrument and a viscoelastic parameter of the second instrument.

[0035] Additionally, the controller may be programmed to detect whether a magnitude of force imparted at a distal end of an instrument coupled to the arm is greater than or equal to a first value and / or whether a magnitude of force imparted on a trocar through which an instrument passes is greater than or equal to a second value, and provide an alert to a user of the arm if a magnitude of force imparted at a distal end of an instrument coupled to the arm is greater than or equal to the first value and / or if a magnitude of force imparted on a trocar through which an instrument passes is greater than or equal to the second value. Further, the controller may be programmed to detect whether a dwell time of the arm and / or an instrument coupled to the arm is greater than or equal to a threshold dwell time, and further to change a motion state of the arm to a static hold state if the dwell time of the arm and / or an instrument coupled to the arm is greater than or equal to the threshold dwell time, the dwell time being an amount of time that the arm and / or an instrument coupled to the arm is held in a static position.

[0036] In the static hold state, the system may be programmed to hold the arm in a static position and prevent movement of the arm from the static position of the arm unless a force applied by a user of the system to the arm and / or an instrument held by the arm is equal to or greater than a predefined threshold release force value. The arm and / or an instrument coupled to the arm may be considered to be held in a static position when the arm is not moved in any direction for more than 5 mm for a dwell time. In some embodiments, the threshold dwell time may be less than 0.5 seconds. Additionally, the controller may be programmed to detect whether the user is attempting to remove the first instrument from the arm such that the controller may be programmed to reduce the coupling force applied by the arm to the first instrument if the controller detects that the user is attempting to remove the first instrument from the arm.

[0037] The system may further include a support platform for supporting at least the base. Thus, the controller may be programmed to detect whether a surgical procedure has been initiated, and if the controller detects that a surgical procedure has been initiated, to move the support platform supporting the base to an initial position and / or move the arms to an initial position and / or orientation for the particular surgical procedure before the surgical procedure begins. According to yet another aspect of the present invention, another collaborative robotic surgical system for assisting in manipulation of an instrument is provided.

[0038] The system may include a base, an arm coupled to the base, the arm having multiple arm sections and multiple articulation joints, multiple motors coupled to the arm, the multiple motors being capable of rotating the multiple arm sections about the multiple articulation joints, and a controller programmed to control at least the multiple motors. For example, the arm may be movable by a user applying a force directly on the arm and / or directly on an instrument coupled to the arm. In response to identifying a first user, the system may be programmed to automatically load a data file associated with the first user, the data file including at least a first operating parameter configured to modify an operating characteristic of the collaborative robotic surgical system. Thus, the controller may be programmed to control the multiple motors according to at least the first operating parameter.

[0039] The first operating parameters of the data file associated with the first surgeon may be based at least in part on data collected during a previous surgical procedure performed by the first user. Additionally, the first operating parameters of the data file associated with the first user may be based at least in part on manually entered preferences for the first user. The system may be programmed to automatically identify the first user using an optical scanner. Additionally, the collaborative system may be programmed to automatically load the data file associated with the first user in response to manual input of the identification of the first user. The data file associated with the first user may include a threshold dwell time value based on dwell time data collected from the procedure performed by the first user and / or manually entered preferences for the first user. Additionally, the data file associated with the first user may include a dwell speed value based on data collected from the procedure performed by the first user and / or manually entered preferences for the first user.

[0040] In addition, the data file associated with the first user may include laparoscopic viewing parameters based on laparoscopic viewing data collected from procedures performed by the first user such that the controller may be programmed to automatically change the position and / or orientation of the laparoscope according to laparoscopic viewing data collected from procedures performed by the first user. The data file associated with the first user may include set joint parameters based on set joint position data collected from past procedures performed by the first user. In addition, the data file may include instrument calibration parameters based on instrument calibration values ​​entered by the first user. The first operating parameters may be based on at least one of a posture of the first user, a height of the first user, or a handedness of the first user.

[0041] Further, the controller may be programmed to automatically detect when an instrument coupled to the arm is not in an optimal or preferred location based on data collected from a procedure performed by the first user, and to move the arm so that the instrument is in an optimal or preferred location. Additionally, the system may be programmed to detect when the first user desires to change the operating mode of the system to a static holding mode, even when the dwell time of the arm and / or the instrument coupled to the arm is less than a threshold dwell time. The data file may be communicable from a network database in communication with the collaborative surgical robotic system. Additionally, the first operating parameter of the data file associated with the first user may be based, at least in part, on data collected during previous surgical procedures performed by multiple users.

[0042] According to another aspect of the present disclosure, a collaborative surgical robot system for performing a surgical procedure is provided. The system may include a first surgical robot having a base, an arm coupled to the base, and a motor coupled to the arm and configured to move the arm relative to the base. The system may further include a controller programmed to control the arm and an optical scanner to collect depth data. The optical scanner may collect depth data related to a position and orientation of an instrument relative to the collaborative surgical robot system. For example, the controller may be programmed to determine whether an instrument is coupled to the first surgical robot based on the depth data. Additionally, the controller may be programmed to identify a type of instrument based at least in part on the depth data.

[0043] Additionally, the optical scanner may collect depth data associated with the position and movement of an instrument, which is held freely by the surgeon and is not coupled to the surgical robot. Additionally, the optical scanner may collect depth data associated with a trocar inserted into the patient. The system may be configured to move an arm and / or a base of the first surgical robot if the position of the trocar changes above a threshold amount. Additionally, the system may further include a second surgical robot having a second base, a second arm coupled to the second base, and a second motor coupled to the second arm and configured to move the second arm relative to the second base. The optical scanner may have an accuracy of at least 5 mm at a range of 10 meters. Additionally, the optical scanner may collect depth data associated with the surgeon's hand during the surgical procedure.

[0044] The controller may be programmed to control the arm of the first surgical robot according to at least one of the following operating modes: passive assistance mode, collaborative assistance mode, robotic assistance mode, and haptic mode. For example, in the passive assistance mode, the arm may be static. In the collaborative assistance mode, the arm may be freely movable by the operator while the motors at least partially simultaneously move the arm to improve the position and / or orientation of an instrument coupled to the end portion of the arm and / or compensate for gravity on at least the arm and the instrument coupled to the end portion of the arm. In the robotic assistance mode, the motors may move the arm to reposition an instrument coupled to the end portion of the arm. In the haptic mode, the arm may be movable by the operator while the motor compensates for gravity on at least the arm and / or an instrument coupled to the end portion of the arm, guides at least the instrument along a predefined trajectory, prevents undesired movement of the arm and / or an instrument coupled to the end portion of the arm, prevents movement of the arm outside of a particular space and / or prevents movement of the arm into a particular space.

[0045] The optical scanner uses the depth data to identify a potential inadvertent collision between the arm of the first surgical robot and at least one of a patient, a support platform supporting at least the first surgical robot, another surgical robot, and / or another object in the operating room, and to warn a user of the potential inadvertent collision and / or block movement of the arm of the first surgical robot to avoid such a collision. Additionally, the first surgical robot may be supported by a support platform such that the collaborative surgical robot system may be configured to move the first surgical robot relative to the support platform and optimize the position of the first surgical robot on the support platform based on the depth data collected by the optical scanner. The optical scanner may collect depth data that is used to record the movement of the surgeon's hand during the surgical procedure.

[0046] Further, the first surgical robot may be supported by a support platform having a plurality of wheels that enable mobility of the support platform. The plurality of wheels may include a braking mechanism that is engaged and configured to prevent mobility of the support platform. The system may further include one or more additional optical scanners disposed on the support platform and configured to collect depth data. For example, the optical scanner or at least one of the one or more additional optical scanners may include at least one of a depth camera, a stereoscopic RGB camera, a LiDAR device, or an electromagnetic, capacitive, or infrared proximity sensor. Additionally, the system may include a display operably coupled to the controller. The controller may be programmed to generate a map of an area surrounding the support platform based on the depth data collected from the optical scanner or at least one of the one or more optical scanners, and cause the display to display the map. The map generated by the controller may include a graphical representation of the support platform relative to one (person) or more objects and / or persons within the area surrounding the support platform.

[0047] Additionally, the controller may be programmed to generate an alert if the map indicates that the support platform is within a predetermined distance of one (a person) or more objects and / or persons in an area surrounding the support platform. The system may further include an actuator operably coupled to the braking mechanism. The actuator may be actuated to cause the controller to automatically disengage the braking mechanism and allow mobility of the support platform such that when the braking mechanism is disengaged, the controller automatically causes the display to display the map.

[0048] According to yet another aspect of the present disclosure, a cooperatively manipulated surgical robot system for performing a surgical procedure is provided. The system may include a surgical robot having a base, an arm coupled to the base, and a motor coupled to the arm. The system may further include an optical scanner configured to track movement of one or more objects around the patient, and a controller programmed to collect data regarding the movement of the one or more objects from the optical sensor and move the arm of the surgical robot in response to the movement of the one or more objects. Additionally, the controller may be programmed to move the base in at least one degree of freedom.

[0049] According to another aspect of the present disclosure, a cooperative surgical system for assisting in laparoscopic surgery performed using a surgical instrument having a handle, a working end, and an extension shaft therebetween is provided. The system may include a robotic arm, a platform configured to support the robotic arm, a plurality of optical sensors coupled to the platform and configured to collect depth data, a display operably coupled to the platform, and a controller programmed to enable the robotic arm to be freely movable in response to movements at the handle of the surgical instrument to perform the laparoscopic surgery using the surgical instrument. The robotic arm may include a proximal end, a distal end configured to be removably coupled to the surgical instrument, a plurality of links, and a plurality of joints. The platform may include a plurality of wheels enabling mobility of the platform. Additionally, the controller may be programmed to receive the depth data collected by the plurality of optical sensors, generate a map of an area surrounding the platform based on the depth data, and cause the display to display the map.

[0050] The plurality of optical sensors may be at least one of a depth camera, a stereoscopic RGB camera, a LiDAR device, or an electromagnetic, capacitive, or infrared proximity sensor. The map generated by the controller may include a graphical representation of the platform relative to at least one of one or more objects or one or more persons in an area surrounding the platform. The controller may further generate an alert if the map indicates that the platform is within a predetermined distance from at least one of one or more objects or one or more persons in an area surrounding the support platform. In addition, the system may include a braking mechanism configured to engage and prevent mobility of the support platform, and an actuator operably coupled to the braking mechanism. The actuator may be actuated to disengage the braking mechanism and allow mobility of the support platform such that the controller may automatically cause the display to display the map when the braking mechanism is disengaged. Additionally, the controller may be programmed to move the robotic arm relative to the platform in at least one degree of freedom, for example, the controller may cause vertical and horizontal movement of the robotic arm relative to the platform.

[0051] According to another aspect of the present disclosure, a method is provided for assisting in laparoscopic surgery using a robotic arm having a proximal end supported by a platform having multiple links, multiple joints, and multiple wheels enabling mobility of the platform, and a distal end configured to be removably coupled to a surgical instrument. The method may include collecting depth data from multiple optical scanners coupled to the platform, generating a map of an area surrounding the platform based on the depth data, the map including a graphical representation of the platform relative to at least one of one or more objects or one or more people in the area surrounding the platform, and causing a display to display the map while the platform is moving and guiding movement of the platform within the operating room.

[0052] According to yet another aspect of the present disclosure, a cooperative surgical system for assisting in laparoscopic surgery performed using a surgical instrument having a handle, a working end, and an extension shaft therebetween is provided. The system may include a robotic arm, a base operably coupled to a proximal end of the robotic arm, a plurality of motors disposed within the base, and a controller operably coupled to the plurality of motors and programmed to enable the robotic arm to be freely movable relative to the base in response to movements at the handle of the surgical instrument to perform laparoscopic surgery using the surgical instrument. The robotic arm may include a proximal end, a distal end configured to be removably coupled to the surgical instrument, a plurality of links, and a plurality of joints between the proximal end and the distal end. Additionally, the plurality of motors may be operably coupled to at least some of the plurality of joints.

[0053] The controller may be programmed to measure a position of the distal end of the robot arm, determine an entry point of the surgical instrument into the patient by determining an intersection point of a plurality of imaginary lines parallel to a longitudinal axis of the surgical instrument as the position of the distal end of the robot arm moves relative to the entry point, calculate a compensation force required to compensate for the gravity of the surgical instrument based on the position of the distal end of the robot arm, the entry point and one or more instrument parameters stored in a memory of the controller, and apply torques via the plurality of motors to at least some of the plurality of joints of the robot arm based on the compensation forces during operation of the cooperatively manipulated surgical system to compensate for the gravity of the surgical instrument. Further, the controller may be programmed to cause the robot arm to maintain a static position in a passive mode in response to determining that movement of the robot arm due to movement at the handle of the surgical instrument is less than a predetermined amount for at least a predetermined dwell period.

[0054] The controller may be further programmed to measure a change in position of the distal end of the robot arm in response to the external force, calculate a holding force required to maintain a static position of the robot arm based on the position of the distal end of the robot arm, the entry point, and one or more instrument parameters, and apply torques via the multiple motors to at least some of the multiple joints of the robot arm based on the compensation force and the holding force to maintain the static position of the robot arm in a passive mode. The controller may be programmed to calculate a force applied by the surgical instrument to the patient at the entry point based on the compensation force, the holding force, the one or more instrument parameters, and the entry point. Additionally, the controller may be programmed to calculate a force applied to the working end of the surgical instrument based on the compensation force, the holding force, the one or more instrument parameters, and the entry point.

[0055] The controller may be programmed to switch from the passive mode to the collaborative mode in response to determining that the holding force exceeds a predetermined threshold. For example, the controller may be programmed to enable the robotic arm to be freely movable in the collaborative mode in response to a movement at the handle of the surgical instrument to perform a laparoscopic surgery using the surgical instrument while compensating for the gravity of the surgical instrument. The system may further include a graphical user interface operably coupled to the controller. The graphical user interface may display at least one of the force applied by the surgical instrument to the patient at the entry point, the applied force at the working end of the surgical instrument, the predetermined threshold, or one or more instrument parameters. In addition, the graphical user interface may enable a user to adjust the predetermined threshold.

[0056] The one or more instrument parameters may include at least one of an instrument type, mass, center of mass, length, or instrument shaft diameter. Additionally, the controller may be programmed to calculate the mass of the surgical instrument by executing a calibration routine. The controller may be programmed to move the base in at least one degree of freedom. For example, the system may further include a platform coupled to the base such that the controller can cause vertical and horizontal movement of the base relative to the platform. Additionally, the system may further include one or more sensors operably coupled to one or more of the multiple joints of the robot arm. The one or more sensors may be configured to measure a position of the one or more joints and generate data indicative of a position of the one or more joints such that the controller can be programmed to measure a position of a distal end of the robot arm based on the data. For example, the one or more sensors may be one or more encoders disposed on one or more of the multiple joints of the robot arm. The multiple encoders may be configured to measure angles of corresponding links of the multiple links at at least some of the joints such that the controller can be programmed to measure a position of the distal end of the robot arm in 3D space based on angle measurements by the multiple encoders.

[0057] According to yet another aspect of the present disclosure, a method is provided for assisting in laparoscopic surgery using a robotic arm comprising a proximal end, a distal end configured to be removably coupled to a surgical instrument, a plurality of links, and a plurality of joints between the proximal and distal ends. The method may include measuring a position of the distal end of the robotic arm via a controller operably coupled to a plurality of motors operably coupled to at least some of the plurality of joints, determining an entry point of the surgical instrument into a patient by determining an intersection point of a plurality of imaginary lines parallel to a longitudinal axis of the surgical instrument as the position of the distal end of the robotic arm moves relative to the entry point, calculating a compensation force required to compensate for the gravity of the surgical instrument based on the position of the distal end of the robotic arm, the entry point, and one or more instrument parameters stored in a memory of the controller, and applying a torque based on the compensation force via the plurality of motors to at least some of the plurality of joints of the robotic arm to compensate for the gravity of the surgical instrument during the laparoscopic surgery. The controller may be programmed to enable the robotic arm to be freely movable relative to a base, operably coupled to a proximal end of the robotic arm, in response to movements in the handle of the surgical instrument to perform laparoscopic surgery using the surgical instrument while compensating for the gravity of the surgical instrument.

[0058] The method may further include causing the robot arm to maintain a static position in a passive mode in response to determining that movement of the robot arm due to movement at the handle of the surgical instrument is less than a predetermined amount for at least a predetermined dwell period. Additionally, the method may include measuring a change in a position of a distal end of the robot arm in response to an external force, calculating a holding force required to maintain a static position of the robot arm based on a position of the distal end of the robot arm, an entry point, and one or more instrument parameters, and applying a torque to at least some of the multiple joints of the robot arm based on the compensation force and the holding force via the multiple motors to maintain the static position of the robot arm in the passive mode.

[0059] Additionally, the method may include calculating a force applied by the surgical instrument to the patient at the entry point based on the compensation force, the holding force, the one or more instrument parameters, and the entry point. Further, the method may include calculating a force applied to a working end of the surgical instrument based on the compensation force, the holding force, the one or more instrument parameters, and the entry point. The method may further include switching from a passive mode to a co-operative mode in response to determining that the holding force exceeds a predetermined threshold, wherein the robotic arm is freely movable in the co-operative mode in response to movements at the handle of the surgical instrument to perform laparoscopic surgery using the surgical instrument while compensating for the gravity of the surgical instrument. [Brief description of the drawings]

[0060] [Figure 1] 1A and 1B illustrate a conventional laparoscopic procedure performed by a surgeon and one or more assistants.

[0061] [Diagram 2] FIG. 2 illustrates an exemplary cooperative surgical system constructed in accordance with the principles of the present disclosure.

[0062] [Figure 3A] 3A-3D illustrate an example robotic arm of the system of FIG. 2 constructed in accordance with the principles of the present disclosure. [Figure 3B] 3A-3D illustrate an example robotic arm of the system of FIG. 2 constructed in accordance with the principles of the present disclosure. [Figure 3C] 3A-3D illustrate an example robotic arm of the system of FIG. 2 constructed in accordance with the principles of the present disclosure. [Figure 3D] 3A-3D illustrate an example robotic arm of the system of FIG. 2 constructed in accordance with the principles of the present disclosure.

[0063] [Figure 4A] 4A and 4B illustrate an exemplary wrist portion of the robotic arm of FIGS. 3A-3D constructed in accordance with the principles of the present disclosure. [Figure 4B] 4A and 4B illustrate an exemplary wrist portion of the robotic arm of FIGS. 3A-3D constructed in accordance with the principles of the present disclosure.

[0064] [Figure 4C] FIG. 4C is a close-up view of an exemplary surgical instrument coupling mechanism of the wrist portion of FIGS. 4A and 4B.

[0065] [Figure 4D] 4D is a close-up view of an exemplary robot arm coupler interface of the surgical instrument coupling mechanism of FIG. 4C, constructed in accordance with the principles of the present disclosure.

[0066] [Figure 5A] 5A and 5B illustrate an exemplary surgical instrument coupler body of the surgical instrument coupling mechanism of FIG. 4C, constructed in accordance with the principles of the present disclosure. [Figure 5B] 5A and 5B illustrate an exemplary surgical instrument coupler body of the surgical instrument coupling mechanism of FIG. 4C, constructed in accordance with the principles of the present disclosure.

[0067] [Figure 6A] FIG. 6A illustrates an alternative exemplary surgical instrument coupler body constructed in accordance with the principles of the present disclosure.

[0068] [Figure 6B] 6B-6D illustrate the attachment of the coupler body of FIG. 6A to a surgical retractor device in accordance with the principles of the present disclosure. [Figure 6C] 6B-6D illustrate the attachment of the coupler body of FIG. 6A to a surgical retractor device in accordance with the principles of the present disclosure. [Figure 6D] 6B-6D illustrate the attachment of the coupler body of FIG. 6A to a surgical retractor device in accordance with the principles of the present disclosure.

[0069] [Figure 7A] FIG. 7A illustrates another alternative exemplary surgical instrument coupler body constructed in accordance with the principles of the present disclosure.

[0070] [Figure 7B] 7B-7D illustrate the attachment of the coupler body of FIG. 7A to a laparoscopic surgical device in accordance with the principles of the present disclosure. [Figure 7C] 7B-7D illustrate the attachment of the coupler body of FIG. 7A to a laparoscopic surgical device in accordance with the principles of the present disclosure. [Figure 7D] 7B-7D illustrate the attachment of the coupler body of FIG. 7A to a laparoscopic surgical device in accordance with the principles of the present disclosure.

[0071] [Figure 8] 8A and 8B illustrate the robotic arm in a sterile drape ready configuration.

[0072] [Figure 9] 9A and 9B illustrate the robotic arm covered in a sterile drape.

[0073] [Figure 10A]10A-10D illustrate rotation of a shoulder joint of a robotic arm in accordance with the principles of the present disclosure. [Figure 10B] 10A-10D illustrate rotation of a shoulder joint of a robotic arm in accordance with the principles of the present disclosure. [Figure 10C] 10A-10D illustrate rotation of a shoulder joint of a robotic arm in accordance with the principles of the present disclosure. [Figure 10D] 10A-10D illustrate rotation of a shoulder joint of a robotic arm in accordance with the principles of the present disclosure.

[0074] [Figure 11A] FIG. 11A illustrates an exemplary cooperative surgical system having an optical scanner in accordance with the principles of the present disclosure, and FIG. 11B illustrates the optical scanner of FIG. 11A. [Figure 11B] FIG. 11A illustrates an exemplary cooperative surgical system having an optical scanner in accordance with the principles of the present disclosure, and FIG. 11B illustrates the optical scanner of FIG. 11A.

[0075] [Figure 11C] FIG. 11C illustrates an exemplary cooperative surgical system having multiple optical scanners in accordance with the principles of the present disclosure.

[0076] [Figure 12] FIG. 12 illustrates a user operating the cooperative surgical system of FIG. 11A in accordance with the principles of the present disclosure.

[0077] [Figure 13] FIG. 13A illustrates the field of view of an optical scanner during a laparoscopic surgical procedure, and FIG. 13B illustrates a depth map of the field of view of the optical scanner of FIG. 13A.

[0078] [Figure 14] FIG. 14 illustrates some example components that may be included within a collaborative robotic platform in accordance with the principles of the present disclosure.

[0079] [Figure 15] FIG. 15 is a flow chart illustrating the operation of a cooperative surgical system in accordance with the principles of the present disclosure.

[0080] [Figure 16] FIG. 16 is a flow chart illustrating surgical instrument calibration of a cooperative surgical system in accordance with the principles of the present disclosure.

[0081] [Figure 17] FIG. 17 is a flow chart illustrating the operation of a robotic arm in accordance with the principles of the present disclosure.

[0082] [Figure 18] 18A and 18B are free body diagrams illustrating the forces applied to a surgical instrument coupled to a robotic arm during a laparoscopic surgical procedure.

[0083] [Figure 19] FIG. 19 is a table of example values ​​associated with several arrangements of passive modes for a robotic arm in accordance with the principles of the present disclosure.

[0084] [Figure 20] FIG. 20 illustrates an exemplary overview of some features and capabilities of a cooperative surgical system in accordance with the principles of the present disclosure.

[0085] [Figure 21] FIG. 21 is a schematic overview of some of the electrical components and connectivity of a cooperative surgical system in accordance with the principles of the present disclosure.

[0086] [Figure 22] FIG. 22 is a flow chart illustrating an exemplary process of obtaining and processing data from an optical scanner and an exemplary use of the data in accordance with the principles of the present disclosure.

[0087] [Diagram 23]FIG. 23 is a schematic overview of data flow for a collaborative surgical system in accordance with the principles of the present disclosure.

[0088] [Figure 24] FIG. 24 is another schematic overview of data flow for a collaborative surgical system in accordance with the principles of the present disclosure.

[0089] [Diagram 25] FIG. 25 is a schematic overview of the data flow and output control of a cooperative surgical system in accordance with the principles of the present disclosure.

[0090] [Figure 26] FIG. 26 is a schematic overview of data flow within a network of collaborative surgical systems in accordance with the principles of the present disclosure.

[0091] [Figure 27A] 27A-27D illustrate vertical and horizontal movement of a robotic arm in accordance with the principles of the present disclosure. [Figure 27B] 27A-27D illustrate vertical and horizontal movement of a robotic arm in accordance with the principles of the present disclosure. [Figure 27C] 27A-27D illustrate vertical and horizontal movement of a robotic arm in accordance with the principles of the present disclosure. [Figure 27D] 27A-27D illustrate vertical and horizontal movement of a robotic arm in accordance with the principles of the present disclosure.

[0092] [Figure 28A] 28A-28D illustrate an exemplary graphical user interface of a collaborative surgical system. [Figure 28B] 28A-28D illustrate an exemplary graphical user interface of a collaborative surgical system. [Figure 28C] 28A-28D illustrate an exemplary graphical user interface of a collaborative surgical system. [Figure 28D]28A-28D illustrate an exemplary graphical user interface of a collaborative surgical system.

[0093] [Figure 29] FIG. 29 is a schematic diagram of an alternative cooperative surgical system constructed in accordance with the principles of the present disclosure.

[0094] [Diagram 30] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Figure 31-1] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Figure 31-2] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 32] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 33] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 34] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Figure 35-1] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Figure 35-2] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 36] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Figure 37] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Figure 38]30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Figure 39] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 40] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 41] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 42] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. [Diagram 43] 30A-43 illustrate various alternative surgical instrument coupling mechanisms constructed in accordance with the principles of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0095] Detailed Description Disclosed herein is a cooperatively operated surgical robotic system and method of use for assisting an operator, e.g., a surgeon, in performing a surgical procedure, e.g., a laparoscopic procedure. Currently, a laparoscopic procedure typically requires a surgeon and one or more assistants. For example, as shown in FIG. 1A, during a laparoscopic procedure, an assistant A1 may be required to hold a retractor device 12 and expose tissue for the surgeon S, while another assistant A2 may be required to hold a laparoscopic device 10 during the procedure and provide the surgeon S with a view of the surgical space within the patient via a display (not shown). As shown in FIG. 1A, the assistant A2 may be required to hold the laparoscopic device 10 in a non-practical position, e.g., between the arms of the surgeon S, while the surgeon actively operates additional surgical instruments, e.g., surgical instruments 14 and 16. As further shown in FIG. 1A, the surgeon S may need to let go of the surgical instruments 16 to guide / reposition the laparoscopic device 10 held by the assistant A2 to achieve the view desired by the surgeon.

[0096] As shown in FIG. 1B, rail-mounted orthopedic retractors 18 may be used to hold one or more surgical instruments in place during a laparoscopic procedure in an attempt to free the surgeon's and / or assistant's hands for other tasks and for stability. As shown in FIG. 1B, a first rail-mounted orthopedic retractor 18a may include a retractor end 20a to engage and hold in place the laparoscopic device 10 in response to actuation of a lock 22a. For example, the lock 22a may be disengaged such that the retractor 18a may be manually positioned at a desired location relative to the patient and reengaged to lock the retractor 18a, and thus the laparoscopic device 10 coupled thereto, in the desired position. 1B, a second rail-mounted orthopedic retractor 18b having a retractor end 20b may also be used to engage and hold in place another surgical instrument during the procedure, upon actuation of locks 22b. Thus, retractors 18a and 18b require extensive manual interaction with locks 22a and 22b, and with retractors 18a and 18b themselves, to reposition and lock individual tools in place.

[0097] The co-operative surgical robotic system described herein provides superior control and stability so that the surgeon and / or assistant may seamlessly position a variety of off-the-shelf surgical instruments as needed, thus avoiding workflow limitations inherent in both human and mechanical solutions. For example, the robotic arms of the co-operative surgical robotic system may provide surgical assistance by holding a first surgical instrument, e.g., a laparoscope, via a first robotic arm, and a second surgical instrument, e.g., a retractor, via a second robotic arm, stable throughout the procedure, providing optimal views of the surgical site, and reducing variability in the forces applied by the surgical instruments to the body wall at the trocar point. As will be understood by one of ordinary skill in the art, the robotic arms of the collaborative surgical robotic systems described herein may preferably hold any surgical instrument having an elongated instrument shaft used for surgical procedures, such as laparoscopic procedures, including, for example, endoscopes / laparoscopes, retractors, graspers, surgical scissors, needle holders, clamps, suturing instruments, cautery tools, staplers, clip appliers, and the like.

[0098] A collaborative surgical robotic system further allows the surgeon to easily manipulate both tools when necessary, providing superior control and stability and overall safety throughout the procedure. Any implementation of the system described herein allows the surgeon to collaboratively manipulate instruments directly at the patient bedside while remaining sterile. For example, the system may include two robotic arms that may be used by the surgeon to hold both a laparoscope and a retractor. During the surgical procedure, the system may seamlessly reposition either of the instruments to provide optimal visualization and exposure of the surgical field. Both instruments may be directly coupled to the robotic arms of the system, and the system may constantly monitor and record the position of the two instruments and / or the two robotic arms throughout the procedure. Additionally, the system may record information such as the location of surgical access ports, the position and movement of the surgeon's hands, the position and orientation of the surgical instruments, the position and orientation of the surgical instruments attached to the robotic arms including whether they are attached to the robotic arms, the patient position, and the orientation and height of the patient table, sensor readings related to the forces applied at the proximal and distal ends of the surgical instruments attached to the robotic arms, the forces required to hold each instrument in place, endoscopic video streams, algorithm parameters, operating room 3D streams captured using optical scanning devices, etc.

[0099] Such data may be used to develop a database of historical data that, in some implementations, may be used to develop algorithms used to control one or more aspects of the operation of the system. In addition, such data may be used to control one or more aspects of the operation of the system / one or more algorithms of the system during a procedure. For example, data may be used to assess the level of fatigue of a user of the system.

[0100] As an operator manipulates the robotic arm of a collaborative surgical robotic system by applying movements to a surgical instrument coupled to the robotic arm, the system may automatically transition the robotic arm between various operating modes in response to determining a predefined condition. For example, the system may transition the robotic arm to a passive mode in response to determining that movement of the robotic arm due to movement at the handle of a surgical instrument is less than a predetermined amount for at least a predetermined dwell period, such that in the passive mode, the robotic arm maintains a static position, e.g., to prevent damage to equipment and / or injury to a patient. Additionally, the system may transition the robot arm to the collaborative manipulation mode in response to determining that a force applied at the robot arm exceeds a predefined threshold due to a force applied at the handle of the surgical instrument such that in the collaborative manipulation mode, a first impedance is applied to the robot arm to account for the weight of the surgical instrument and the robot arm, while in the collaborative manipulation mode, the robot arm is allowed to move freely in response to movement at the handle of the surgical instrument to perform laparoscopic surgery using the surgical instrument. Further, the system may transition the robot arm to the haptic mode in response to determining that at least a portion of the robot arm is outside a predefined haptic barrier such that in the haptic mode, a second impedance that is greater than the first impedance is applied to the robot arm, thereby making movement of the robot arm more viscoelastic in the haptic mode than in the collaborative manipulation mode in response to movement at the handle of the surgical instrument. The system may further transition the robotic arm into a robot-assisted mode in response to detecting various conditions that warrant automated movement of the robotic arm, for example, to guide a surgical instrument attached thereto along a planned trajectory or to avoid collision with another object or person in the surgical space.

[0101] Referring now to Figure 2, a collaborative surgical robotic system 200 is provided. As shown in Figure 2, the system 200 may include a platform 100, e.g., a surgical cart, sized and shaped to support one or more robotic arms 300, e.g., robotic arm 300a and robotic arm 300b (each of the robotic arms 300 having a surgical instrument coupler interface 400 for removably coupling to a surgical instrument), and a computing system operably coupled to the platform 100 and the robotic arms 300. As shown in Figure 2, the system 200 may further include a graphical user interface display 110 for displaying operational information and receiving user input.

[0102] Additionally, each robotic arm 300 may further include an indicator 334 to visually indicate in real time the operational mode associated with the respective robotic arm. For example, the indicator 334 may be located at least on the elbow joint of the robotic arm. Additionally or alternatively, the indicator 334 may be located anywhere on the system 200, such as on the platform 100, on the display 110, etc. Furthermore, the indicator 334 may include a light, such as an LED light, that may illuminate in a variety of distinct colors and in distinct patterns, such as solid or flashing. For example, each operational mode of the system 200 may be associated with a uniquely colored light, such as red, yellow, blue, green, purple, white, orange, etc. Thus, the indicator 334 may indicate a transition from one operational mode to another.

[0103] 2, the platform 100 may include a vertical extension 106 for independently moving the robotic arms 300a and 300b vertically relative to the platform 100, and a horizontal extension 108 for independently moving the robotic arms 300a and 300b horizontally relative to the platform 100, thereby allowing operator flexibility in positioning the robotic arms 300 relative to the patient. Additionally, the platform 100 may include a number of wheels 104, e.g., caster wheels, to provide mobility of the platform 100, and thus the robotic arms 300, within the operating room. Each of the wheels 104 may include a braking mechanism, which may be actuated to prevent movement of the platform 100 via the wheels 104. Thus, the platform 100 may independently move each of the robotic arms 300a and 300b in any direction, including a first or vertical direction toward and away from the floor, a second or horizontal direction toward and away from the patient, and / or a third or horizontal direction along the patient's height. In some embodiments, the platform 100 may simultaneously move the robotic arms 300a and 300b in the same direction and may also cause rotational movement of the robotic arms 300a and 300b. Once ready for operation, the platform 100 may be moved to a desired location on the side of the patient's bed via the wheels 104 and locked in place, and the vertical and horizontal positions of the robotic arms 300a and 300b may be adjusted to an optimal position relative to the patient for the procedure via the vertical extensions 106 and horizontal extensions 108 in response to user input received by the graphical user interface display 110. As described in further detail below, platform 100 may automatically move robotic arms 300a and 300b in response to detection of a potential collision with other objects and / or people in the operating room, for example, during a laparoscopic procedure.

[0104] The surgical robotic system 200 is configured for cooperative manipulation such that the system 200 may assist a user or operator, e.g., a surgeon and / or a surgical assistant, by allowing the user to freely move the robotic arm 300a and / or the robotic arm 300b due to manipulation of one or more surgical instruments coupled to the robotic arm in response to force inputs provided by the user to the surgical instruments. Thus, the system 200 may be configured such that the robotic arm 300 moves in response to direct operator movements of the surgical instruments coupled thereto, while not being remotely controlled, to compensate for the mass of the surgical instruments and the individual robotic arms and provide localized impedance along the robotic arms, thereby increasing the accuracy of the operator's movements or actions as the operator manipulates the surgical instruments.

[0105] System 200 may be particularly useful in laparoscopic and / or other surgical procedures utilizing elongated instruments that may be inserted into a patient's body, for example, via a cannula, to enable surgical intervention. As will be appreciated by one of ordinary skill in the art, system 200 may be used for any desired or suitable surgical procedure. Additionally, system 200 may be used in conjunction or cooperation with video surveillance provided by one or more cameras and / or one or more endoscopes, such that an operator of system 200 may view and monitor the use of instruments coupled to robotic arm 300 via coupler interface 400. For example, robotic arm 300a may be removably coupled to and manipulate an endoscope, while robotic arm 300b may be removably coupled to and manipulate a surgical instrument.

[0106] 3A-3D, a surgical support arm, e.g., a robotic arm 300, is provided. As described above, the system 200 may include multiple robotic arms, e.g., robotic arm 300a and robotic arm 300b. However, because each robotic arm may be constructed similarly, only a single robotic arm will be described with respect to FIGS. 3A-3D collectively as robotic arm 300 for brevity. Aspects of the robotic arm described herein may utilize structures from U.S. Pat. No. 10,118,289 to Louveau, the entire contents of which are incorporated herein by reference. The robotic arm 300 may include multiple arm sections / links and multiple articulation joints 106 extending from a base portion. For example, the robotic arm 300 may include a base portion, a shoulder portion, an elbow portion, and a wrist portion, thereby mimicking the kinematics of a human arm. 3A, the robot arm 300 may include a base, which includes a base portion 302 that is rotatably coupled to a shoulder portion 304 at a base joint 303. For example, the shoulder portion 304 may sit on top of the base portion 302 and may be rotated relative to the base portion 302 about axis Q1 at the base joint 303. In some embodiments, the robot arm 300 may be swapped, replaced, or coupled to a base in any desired arrangement.

[0107] The robotic arm 300 may further include a shoulder link 305, which includes a proximal shoulder link 306 rotatably coupled to a distal shoulder link 308. A proximal end of the proximal shoulder link 306 may be rotatably coupled to the base shoulder portion 304 at a shoulder joint 318 such that the proximal shoulder link 306 may be rotated relative to the shoulder portion 304 at a shoulder joint 318 about an axis Q2. As shown in FIG. 3A, the axis Q2 may be perpendicular to the axis Q1. A distal end of the proximal shoulder link 306 may be rotatably coupled to a proximal end of the distal shoulder link 308 at a joint 320 such that the distal shoulder link 308 may be rotated relative to the proximal shoulder link 306 at a joint 320 about an axis Q3. As shown in FIG. 3A, the axis Q3 may be parallel to the longitudinal axis of the shoulder link 305. Additionally, the robotic arm 300 may include an actuator 330, e.g., a lever, button, or switch, operably coupled to the distal shoulder link 308 and / or the proximal shoulder link 306 such that the distal shoulder link 308 may only be rotated relative to the proximal shoulder link 306 in response to actuation of the actuator 330. Thus, axis Q3 may be a "set" axis such that the distal shoulder link 308 may be rotated and fixed relative to the proximal shoulder link 306 during a set phase prior to an operation phase in which the robotic arm 300 is used in a surgical procedure, as will be described in further detail with respect to FIGS.

[0108] In some embodiments, the distal shoulder link 308 may be manually rotated relative to the proximal shoulder link 306 in predefined increments in response to actuation of the actuator 330. Alternatively, in response to actuation of the actuator 330, the distal shoulder link 308 may be automatically rotated relative to the proximal shoulder link 306 until the actuator 330 is released. For example, the actuator 330 may be a button or switch operably coupled to a motor that is operably coupled to the distal shoulder link 308 and / or the proximal shoulder link 306 such that in response to actuation of the actuator 330, the associated motor rotates the distal shoulder link 308 relative to the proximal shoulder link 306. Preferably, the motor is located within the base of the robot arm 300, or alternatively, the motor may be located on the shoulder link 305. Thus, the actuator 330 may be a button or switch that allows for dual actuation, for example, a first actuation to rotate the distal shoulder link 308 in a first direction relative to the shoulder link 306, and a second actuation to rotate the distal shoulder link 308 in a second direction opposite the first direction. In some embodiments, the button or switch may be located on a graphical user interface, such as the display 110.

[0109] The robotic arm 300 may further include an elbow linkage 310. A proximal end of the elbow linkage 310 may be rotatably coupled to a distal end of the distal shoulder linkage 308 at elbow joint 322 such that the elbow linkage 310 may be rotated relative to the distal shoulder linkage 308 about axis Q4 at elbow joint 322. The robotic arm 300 may further include a wrist portion 311, which may include a proximal wrist linkage 312 rotatably coupled to a distal end of the elbow linkage 310 at wrist joint 324, a central wrist linkage 314 rotatably coupled to the proximal wrist linkage 312 at joint 326, and a distal wrist linkage 316 rotatably coupled to the central wrist linkage 314 at joint 328, as further shown in FIGS. Thus, wrist section 311 may be rotated relative to elbow link 310 about axis Q5 at wrist joint 324, central wrist section 314 may be rotated relative to proximal wrist link 312 about axis Q6 at joint 326, and distal wrist link 316 may be rotated relative to central wrist link 314 about axis Q7 at joint 328. Additionally, as shown in FIG 4B, the robotic arm 300 may include an actuator 332, e.g., a lever, button, or switch, operably coupled to elbow link 310 and / or proximal wrist link 312 such that proximal wrist link 312 may only be rotated relative to elbow link 310 in response to actuation of actuator 332. Thus, axis Q5 may be a "set" axis such that the proximal wrist joint 312 may be rotated and fixed relative to the elbow joint 310 during a set up phase prior to an operation phase in which the robotic arm 300 is used in a surgical procedure. In some preferred embodiments, in response to actuation of the actuator 332, the proximal wrist joint 312 may be manually rotated relative to the elbow joint 310 in predefined increments, thereby eliminating the need to have additional motors and / or electronics in the distal region of the robotic arm 300. Alternatively, in response to actuation of the actuator 330, the proximal wrist joint 312 may be automatically rotated relative to the elbow joint 310 until the actuator 332 is released.

[0110] 3A, the robotic arm 300 may include multiple motors, e.g., motors M1, M2, M3, all of which may be located within the base of the robotic arm 300. The motors M1, M2, M3 may each be operatively coupled to a respective joint of the robotic arm 300, e.g., the base joint 303, the shoulder joint 318, and the elbow joint 322, thereby applying a localized impedance at the respective joint. For example, the motors M1, M2, M3 may each produce an impedance at either the base joint 303, the shoulder joint 318, and the elbow joint 322, thereby in effect applying an impedance at the distal end of the robotic arm, e.g., at the point of attachment with a surgical instrument, to improve the sensation experienced by an operator during manipulation of the surgical instrument and the operator's performance during a surgical procedure. For example, impedance may be applied to the distal end of the robotic arm 300, and thus the surgical instrument coupled thereto, to provide a sense of viscoelasticity, stiffness, and / or inertia to an operator manipulating the surgical instrument. Additionally, the applied impedance may simulate tissue density or stiffness, communicate the surgical boundary to the operator, and may be used to direct the surgical instrument along a desired path or otherwise. In some embodiments, motors may actuate individual joints, thereby causing movement of the robotic arm 300 about the individual joints. Thus, axis Q1, axis Q2, and axis Q4 may each be a "motor" axis, such that motors M1, M2, M3 may apply impedance / torque to base joint 303, shoulder joint 318, and elbow joint 322, respectively, to prevent or actuate rotation about the individual axes. As described in more detail below, motors M1, M2, M3 may be controlled by a processor of the co-operative robotic platform. By using three motor axes, some implementations of the robotic arm 300 may apply force / torque in three directions at the distal end of the robotic arm 300, thereby moving a surgical instrument coupled to the distal end of the robotic arm 300 in three degrees of freedom.

[0111] Axis Q6 and axis Q7 may be "passive" axes such that central wrist joint 314 may be rotated relative to proximal wrist joint 312 without any applied impedance from system 200, and distal wrist joint 316 may be rotated relative to central wrist joint 314 without any applied impedance from system 200. The distal end of distal wrist joint 316 may include a surgical instrument coupler interface 400 for removably coupling to a surgical instrument, as shown in Figures 4A and 4B, for example, via a coupler body 500, which may be removably coupled to a surgical instrument and coupler interface 400, as described in further detail below. Alternatively, wrist portion 11 may include a passive ball joint at the attachment point with a surgical instrument, as described in U.S. Pat. No. 10,582,977, the entire disclosure of which is incorporated herein by reference.

[0112] 3A , the robot arm 300 may further include a number of encoders, e.g., encoders E1-E7, disposed on at least some of the joints of the robot arm 300. For example, encoder E1 for measuring the angle formed between the base portion 302 and the shoulder portion 304 may be disposed on or adjacent to the base joint 303 in the base, encoder E2 for measuring the angle formed between the shoulder portion 304 and the proximal shoulder link 306 may be disposed on or adjacent to the shoulder joint 318 in the base, encoder E3 for measuring the angle formed between the proximal shoulder link 306 and the distal shoulder link 308 may be disposed on or adjacent to the joint 320, and encoder E4 for measuring the angle formed between the distal shoulder link 308 and the elbow link 310 may be disposed on or adjacent to the elbow joint 322. 3, an encoder E4 may be disposed adjacent to motor M3 operably coupled to elbow joint 322 in the base, an encoder E5 may be disposed on or adjacent wrist joint 324 for measuring the angle formed between elbow joint 310 and proximal wrist joint 312, an encoder E6 may be disposed on or adjacent joint 326 for measuring the angle formed between proximal wrist joint 312 and central wrist joint 314, and an encoder E7 may be disposed on or adjacent joint 328 for measuring the angle formed between central wrist joint 314 and distal wrist joint 316. Alternatively, encoder E4 may be disposed on or adjacent elbow joint 322. The encoders may be absolute encoders or other position / angle sensors configured to generate data for precisely determining the position and / or angle formed of the corresponding joints at the individual joints and / or the precise position of a surgical instrument coupled to the distal end of the robotic arm 300. Thus, the precise position of each link, joint, and distal end of the robot 300 may be determined based on measurements obtained from multiple encoders.Preferably, redundant encoders are placed at each location along the robot arm 300 where encoders are installed, as described in more detail below, to provide more accurate position data and to detect fault conditions.

[0113] Prior to attachment to a surgical instrument, the robot arm 300 may be manually manipulated by a user, for example, to position the robot arm 300 in a desired position for coupling with a surgical instrument. For example, a user may manually manipulate the robot arm 300 via the wrist portion 11, the actuator 330, and / or the actuator 332. In response to actuation of the actuator 330, a user may manually rotate the distal shoulder joint 308, and in response to actuation of the actuator 332, a user may manually manipulate the proximal wrist portion 312. In response to attachment to a surgical instrument, the robot arm 300 may still be manually manipulated by a user directly applying a force, for example, one or more linear forces and / or one or more torques, to the robot arm 300. During a laparoscopic procedure, however, the operator preferably manipulates the robotic arm 300 solely via the handle of the surgical instrument, which applies force / torque to the distal end of the robotic arm 300 and thus the connections and joints of the robotic arm 300. As the operator applies forces to a surgical instrument attached to the robotic arm 300, thereby causing movement of the surgical instrument, the robotic arm 300 will move in response to the movement of the surgical instrument, providing the operator with the ability to freely move the surgical instrument relative to the patient. As described in more detail below, the robotic arm 300 may apply impedance as the operator moves the surgical instrument to account for the weight of the surgical instrument and the robotic arm 300 itself, e.g., gravity compensation, thereby making it easier for the operator to move the instrument despite gravity and / or inertial forces being imposed on the robotic arm and / or the surgical instrument. As will be understood by one of ordinary skill in the art, the robot arm 300 may include fewer or more articulation joints and a corresponding number of motors and encoders / sensors than those shown in FIG. 3A.

[0114] 4C, a close-up view of the coupling mechanism of coupler interface 400 and coupler body 500 is provided. Coupler interface 400 may be coupled to a distal end of distal wrist linkage 316 using any suitable fastener or connector, such as magnets, screws, pins, clamps, welds, adhesives, rivets, and / or any other suitable fastener or any combination of the foregoing. As shown in FIG. 4C, coupler interface 400 may be coupled to a distal end of distal wrist portion 316 using fastener 410, which may be threaded or have other features that allow coupler interface 400 to be selectively attached to distal wrist portion 316. Fastener 410 may be coupled to an insert element 408 positioned at or within the distal end of distal wrist portion 316, having an opening for receiving fastener 410 therein. In some embodiments, the fastener 410 may be a pin, or may have other features, such as a ball, a latch, or the like, that allow the fastener 410 to selectively couple with the distal wrist portion 316.

[0115] A coupler body 500, which may have an opening 514 sized and shaped to slidably and releasably receive an elongated shaft of a surgical instrument therethrough, may be removably coupled to the coupler interface 400. For example, the coupler body 500 may be removably coupled to the coupler body 500 via a magnetic connection, thereby facilitating efficient attachment and detachment between the coupler body 500 and the coupler interface 400, for example, by overcoming magnetic coupling forces between the coupler body 500 and the coupler interface 400. Thus, as shown in FIG. 4C, the coupler body 500 may have one or more magnets 506 that, in the assembled state, contact a surface of the coupler interface 400 and extend away from the surface of the coupler body 500. Alternatively, in embodiments not having a coupler interface, the magnets 506 may directly contact the distal end of the distal wrist portion 316.

[0116] Thus, the coupler interface 400 or the distal end of the distal wrist portion 316 may have a ferrous base component configured to receive and magnetically couple with the magnet 506 of the coupler body 500 such that the coupler body 500 may be removably coupled with the coupler interface 500 and / or the distal end of the distal wrist portion 316. FIG. 4D illustrates a surgical instrument coupler interface 400. As shown in FIG. 4D , the coupler interface 400 may have a recessed portion 404 that is sized and shaped to receive the complementary geometry of the coupler body 500, defined by the protuberance 402. Thus, when the complementary geometry of the coupler body 500 is received within the recessed portion 404 in the assembled state, rotational movement of the coupler body 500 relative to the coupler interface 400 may be limited or otherwise prevented.

[0117] In addition, the coupler interface 400 may have one or more recesses or recesses 406 sized and shaped to receive one or more magnets 506 therein. The coupler interface 400 may have a ferrous base component or magnet in the recess 406 to magnetically couple with the magnet 506. For example, the magnet in the recess 406 may have a south magnetic pole and the magnet 506 may have a north magnetic pole, or vice versa. Additionally, the polarity of the magnets can ensure proper coupling orientation. The recess 406 may be sized and shaped to limit or otherwise prevent movement between the coupler body 500 and the coupler interface 400 in any direction, radial or normal to the axial (e.g., longitudinal) centerline of the magnet 506 when the coupler body 500 is in an assembled state with the coupler interface 400. As will be appreciated by one of ordinary skill in the art, the coupler interface 400 may have fewer or more than two recesses 406, as the coupler body 500 will have a corresponding amount of magnets.

[0118] 5A and 5B, a coupler body 500 is provided. As shown in FIG. 5A, the coupler body 500 may have one or more magnets 506 disposed on a portion 502 having a complementary geometry to the recessed portion 404 of the coupler interface 400, as described above, to facilitate alignment between the coupler body 500 and the coupler interface 400. In addition, the coupler body 500 may have one or more grooves 504 sized and shaped to engage with a complementary ridge 402 of the coupler interface 400. The grooves 504 and the ridges 402 may interact to aid in alignment of the coupler body 500 and the coupler interface 400 by limiting or otherwise preventing movement between the coupler body 500 and the coupler interface 400 in at least two directions D1 and D2, as shown in FIG. 4C. Thus, in the assembled state, the coupler body 500 can be prevented from moving in any axial direction relative to the coupler interface 400 .

[0119] 5A and 5B, coupler body 500 may have a first portion 508 and a second portion 510. First portion 508 may be coupled or integrally formed with second portion 510 via hinge 512, which may be an integral hinge formed from the same material as first and second portions 508, 510 and / or integrally formed with first and second portions 508, 510, such that second portion 510 may be moved or rotated relative to first portion 508 to expand (increase in size) or contract (decrease in size) an opening 514 defined by first portion 508 and second portion 510. First portion 508 and second portion 510 may form a clamp that may constrict around an elongated shaft of a surgical instrument positioned within opening 514 as a screw 516, e.g., a thumbscrew, is tightened to couple instrument 112 and coupler body 141. Thus, the coupler body 500 may transition between a first, unsecured / open state or position and a second, secured / closed state or position.

[0120] The diameter of the opening 514 may be selected based on the surgical instrument to be coupled to the coupler body 500. For example, a coupler body may be selected from a plurality of coupler bodies, each having an opening sized and shaped to receive an elongated shaft of a specific surgical instrument having a predefined elongated shaft diameter, such as surgical instruments used for orthopedic and trauma surgery (OTS), laparoscopic or other surgical instruments including needle holders, clamps, scissors, etc. The coupler body 500 may be coupled with a surgical instrument at any desired axial location on the surgical instrument.

[0121] 5C, the coupler body 500 may include a recess 520 extending through the second portion 510 and a recess 522 extending through at least a portion of the first portion 508. The recess 520 is aligned with the recess 522 for receiving the locking portion 518 of the screw 516. For example, the locking portion 518 may have a male threaded surface and the recesses 520, 522 may have female threaded surfaces and engage with the locking portion 518. The screw 516 may be loosened by hand to open or expand the opening 514 so that a surgical instrument may be removed, repositioned, rotated, and / or slid, etc. Once the coupler body 500 is coupled to a surgical instrument, for example via screws 516, the coupler body 500 and the surgical instrument coupled to the coupler body 500 may be removably coupled to the coupler interface 400 via magnets 506.

[0122] The opening 514 may be defined by a first semicircular cutout in the first portion 508 of the coupler body 500 and a second semicircular cutout in the second portion 510, thereby engaging a circular outer surface of an elongated shaft of a surgical instrument. The opening 514 may include, for example, a rubber pad, sheet, bump, O-ring, protrusion, or other component or feature configured to contact and grip the outer surface of the elongated shaft of a surgical instrument. For example, the rubber material may be silicone rubber or any other suitable type of rubber. Thus, once the coupler body 500 is coupled with a surgical instrument, for example, by securing the threads 516, the surgical instrument may be inhibited or otherwise prevented from moving at least axially, for example, in a direction along the longitudinal axis of the surgical instrument, or in some embodiments, from moving axially and rotatably relative to the coupler body 500 in the secured state. Preferably, a surgical instrument coupled to the coupler body 500 may be freely rotated by an operator relative to the coupler body 500, while axial movement of the surgical instrument relative to the coupler body 500 is blocked or otherwise prevented in the secured condition. For example, the frictional force between the outer surface of the elongated shaft of the surgical instrument and the inner surface of the coupler body 500 defining the opening 514 may be selected such that, in the secured condition, rotation of the surgical instrument relative to the coupler body 500 requires less force than axial movement of the surgical instrument relative to the coupler body 500. Thus, the coupler 500 may be configured to account for diametric and surface variations of the surgical instruments (including variations in the coefficient of friction of the surfaces).

[0123] In some embodiments, a surgical instrument may be moved axially relative to the coupler body 500 in response to application of at least a threshold force on the surgical instrument against the coupler body 500, or in response to actuation of a release or change in state of the coupler body 500. For example, such actuation may be accomplished by, for example, depressing a button, loosening a locking screw such as the locking screw 516 or other connector, moving a dial, or otherwise changing the coupler body 500 and / or the coupler interface 400 from a second locked state to a first unlocked state. Thus, a surgical instrument may be repositioned axially relative to the coupler body 500 by loosening the screw 516 or other manual fastener or fastening mechanism such as a clamp in the coupler body 500, repositioning the surgical instrument to a desired axial position, and retightening the screw 516 or other manual fastener or fastening mechanism. The coupler body 500 may be disposable, or alternatively, sterilizable such that it may be sterilized between surgical procedures.

[0124] As explained above, the diameter of the coupler body opening may be selected based on the surgical instrument to be coupled to the coupler body. Most commonly used laparoscopic surgical instruments have a predefined and known elongated shaft diameter, and thus, multiple coupler bodies may be provided, each having an opening sized and shaped to receive and engage a specific surgical instrument. For example, FIG. 6A illustrates a coupler body 600 having an opening 614 sized and shaped to receive a 5 mm diameter surgical instrument, e.g., retractor device 12. The coupler body 600 may be constructed similarly to the coupler body 500. For example, the coupler body 600 may include a first portion 608 coupled to a second portion 610 via a hinge portion 612, and recesses 620, 622 for fixedly receiving a locking portion 618 of a screw 616. As shown in FIG 6B, the coupler body 600 may receive the elongated shaft 12a of the retractor 12, for example, from the working end of the retractor 12, through an opening 614, such that the coupler body 600 may be slid over the elongated shaft 12a until the coupler body 600 engages the proximal portion 12b of the retractor 12, as shown in FIG 6C. Preferably, the coupler body 600 is coupled to the retractor 12 because when the coupler body 600 contacts the proximal portion 12b, this point along the retractor 12 is fixed, thereby providing a consistent reference point for calculating force measurements, as described in more detail below. Thus, when the coupler body 600 is in a desired location along the elongated shaft of the retractor 12, for example adjacent the proximal portion 12b, a screw 616 may be coupled to the coupler body 600 to secure the coupler body 600 to the retractor 12. As described above, the coupler body 600 is secured to the retractor 12 such that rotational movement of the retractor 12 relative to the coupler body 600 is permitted while axial movement of the retractor 12 relative to the coupler body 600 is constrained, e.g., the force required to move the retractor 12 relative to the coupler body 600 is much higher than the force required to rotate the retractor 12 relative to the coupler body 600.

[0125] 7A illustrates a coupler body 700 having an opening 714 sized and shaped to receive a 10 mm diameter surgical instrument, e.g., a laparoscopic device 10. The coupler body 700 may be constructed similarly to the coupler body 600. For example, the coupler body 700 may include a first portion 708 coupled to a second portion 710 via a hinge portion 712, and recesses 720, 722 for securely receiving a locking portion 718 of a screw 716. As shown in FIG. 7B, the coupler body 700 may receive the extension shaft 10a of the laparoscopic device 10, e.g., from the working end of the laparoscope 10, through the opening 714, such that the coupler body 700 may be slid over the extension shaft 10a until the coupler body 700 engages the proximal portion 10b of the laparoscope 10, as shown in FIG. 7C. Preferably, the coupler body 700 is coupled to the laparoscope 10 so that when the coupler body 700 contacts the proximal portion 10b, this point along the laparoscope 10 is fixed, thereby providing a consistent reference point for calculating force measurements, as described in more detail below. Thus, when the coupler body 700 is at a desired location along the elongated shaft of the laparoscope 10, e.g., adjacent the proximal portion 10b, a screw 716 may be coupled to the coupler body 700 to secure the coupler body 700 to the laparoscope 10. As described above, the coupler body 700 is secured to the laparoscope 10 such that rotational movement of the laparoscope 10 relative to the coupler body 700 is permitted, while axial movement of the laparoscope 10 relative to the coupler body 700 is constrained, e.g., the force required to move the laparoscope 10 relative to the coupler body 700 is much higher than the force required to rotate the laparoscope 10 relative to the coupler body 700.

[0126] Once an appropriately sized coupler body has been coupled to a selected surgical instrument, the coupler body may be removably coupled to the coupler interface 400 of the robotic arm 300. The coupler body 500 and coupler interface 400 may be configured for one-handed coupling such that an operator may use one hand to couple the coupler body 500, and thus the surgical instrument coupled thereto, to the coupler interface 400 of the robotic arm 300. Preferably, a surgical drape may be sandwiched or crimped between the coupler body and the coupler interface 400 and draped over the robotic arm 300 to maintain the sterility of the surgical space and prevent contact with non-sterile components of the robotic arm 300. Thus, the sterile drape may pass continuously (e.g., without holes, slits, or any other type of opening) between the coupler body and the coupler interface such that the coupler body is on a first side of the sterile drape and the coupler interface, robot arm 300, and / or other components of system 200 are on the other side of the sterile drape. In some embodiments, the coupler body may be integrated with the surgical drape. Additionally or alternatively, the surgical drape may include an adapter integrated therewith such that the coupler body 500 may be coupled to the coupler interface 400 via the adapter, e.g., the adapter may be positioned between the coupler body 500 and the coupler interface 400.

[0127] 8A and 8B, the robotic arm 300 may be positioned in a surgical drape ready configuration. As shown in FIG. 8A, the robotic arm 300 may be extended such that the wrist portion 311, elbow joint 310, and shoulder joint 305 extend away from the base shoulder portion 304 to allow a surgical / sterile drape to be draped over each component of the robotic arm 300. 8B, when there are two robot arms, e.g., robot arm 300a and robot arm 300b, robot arms 300a and 300b may be angled away from each other, e.g., by rotating shoulder portion 304a relative to base portion 302a of robot arm 300a and by rotating shoulder portion 304b relative to base portion 302b of robot arm 300b, such that wrist portion 311a, elbow connection 310a, and shoulder connection 305a extend away from wrist portion 311b, elbow connection 310b, and shoulder connection 305b. This configuration allows for efficient and accessible draping of the individual robot arms with surgical / sterile drapes. Additionally, in the extended position, the robotic arm may be outside the virtual haptic boundary such that the robotic arm is in haptic mode and a high level of impedance is applied to the robotic arm, thereby making the movement of the robotic arm more viscoelastic and making it easier for an operator to drape the robotic arm but provide movement thereto as needed. For example, FIG. 9A illustrates a single robotic arm 300 draped with a sterile drape 800, and FIG. 9B illustrates robotic arms 300a, 300b draped with sterile drapes 800a, 800b, respectively.

[0128] The sterile drape 800 may be completely closed at its end portions. In some embodiments, the sterile drape 800 may have openings (optionally with sterile seals or interfaces) in its distal portion through which the robotic arm 300, coupler interface 400, coupler body 500, and / or a portion of the surgical instrument may pass. A drape having a sealed end portion without any openings and not sealed along its length may provide a better sterile barrier for the system 200. Thus, all of the robotic arm 300 may be located inside and / or completely enclosed within the sterile drape 800, except for an opening at the proximal end of the sterile drape 800, e.g., near the base of the robotic arm 300. In some embodiments, the coupler body 500 and coupler interface 400 may have electrical connectors to produce an electronic connection between the robotic arm 300 and the surgical instrument. Thus, electrical signals may be transmitted through the sterile drape 800. Alternatively, the sterile drape 800 may include openings such that electrical wires or other components may pass through the openings to provide wired communication channels to electrical components, which may include, for example, memory chips for calibration, radio frequency probes for ablation, cameras, and other electronic components. The surgical instruments and coupler bodies may instead be passive or non-electronic, such that electrical wires do not need to pass through the sterile drape 800.

[0129] 10A-10D, rotation of the distal shoulder link 308 relative to the proximal shoulder link 306 of the shoulder link 305 is provided. As explained above, axis Q3 may be a "set" axis such that the distal shoulder link 308 may be rotated relative to the proximal shoulder link 306 in response to actuation of the actuator 330 during a set up phase of the robotic arm 300, for example, prior to operation of the robotic arm 300 in a surgical procedure. As shown in FIG. 10A, the shoulder portion 304 may optionally first be rotated relative to the base portion 302 to a desired position, thereby causing rotation of all links distal to the proximal shoulder link 306 that are coupled to the shoulder portion 304, rotated relative to the base portion 302, and providing sufficient space for rotation of the robotic arm 300 about joint 320. Additionally, as shown in FIG. 10, the wrist portion 311 may be at least partially extended away from the base portion 302 to avoid colliding with any components of the robot arm 300 upon rotation of the robot arm 300 about the joint 320. As shown in FIG. 10B, the actuator 330 must be actuated to allow rotation of the distal shoulder link 308 relative to the proximal shoulder link 306 at the joint 320. FIG. 10C illustrates the robot arm 300 in a desired location for a specific laparoscopic procedure upon rotation of the distal shoulder link 308 relative to the proximal shoulder link 306. FIG. 10D illustrates the robot arm 300a in a desired location upon rotation of the distal shoulder link 308a relative to the proximal shoulder link 306a compared to the robot arm 300b.

[0130] 11A and 11B, an exemplary collaborative robotic surgical system having an optical scanner is provided. As shown in FIG. 11A, the system may be constructed similarly to the system 200 of FIG. 2, and has multiple robotic arms, e.g., robotic arm 300a and robotic arm 300b. As described above, only two robotic arms are shown in FIG. 11A, but fewer or more robotic arms may be used in conjunction with the optical scanner 1100. In addition, the system may include the optical scanner 1100, e.g., a LiDAR scanner or other suitable optical scanning device, such as an RGBD camera or sensor, an RGB camera with machine learning, a time-of-flight depth camera, structured light, multiple projection cameras, a stereoscopic camera, an ultrasonic sensor, a laser scanner, other types of coordinate measurement area scanners, or any combination of the foregoing. For example, the LiDAR camera / scanner may be capable of recording both color (RGB) and depth (D) of the surgical field and may include, for example, the Intel RealSense LiDAR Camera L515 or Intel RealSense Depth Camera D435i (commercially available from Intel, Santa Clara, California) or other LiDAR or depth camera having similar or suitable specifications, including, but not limited to, any of the following specifications: (i) range: 25 cm to 500 cm; depth accuracy: 5 mm or approximately 5 mm; depth field of view: 70×55 or approximately 70×55 (degrees); depth output resolution: 1,024×768 pixels or approximately 1,024×768 pixels; depth / RGB frame rate: 30 frames per second; RGB frame resolution: 1,920×1,080; and / or RGB field of view: 70×43 degrees or approximately 70×43 degrees. The LiDAR scanner or optical scanner may further include both 1 / 4-20 UNC threaded or 2×M3 threaded mounting points. As would be understood by one of ordinary skill in the art, the optical scanner 1100 may be used in other collaborative robotic surgical systems described herein, such as system 200 or any variations thereof.

[0131] As shown in FIG. 11A, the platform supporting the robotic arms 300a, 300b may support the optical scanner 1100 and any other electronics, wiring, or other components of the system such that the optical scanner 1100 is mounted in a fixed location relative to other objects in the surgical space, and the position and orientation of the optical scanner 1100 may be known or determined relative to the global coordinate system of the system, and thus the robotic arms. This allows all data streams to be converted to a single coordinate system for development purposes. For example, the optical scanner 1100 may be supported on a rod or shaft, such as rod 1102, which may have an adjustable height or otherwise be adjustable in any direction, such as up / down, left / right, toward / away from the patient, allowing the optical scanner 1100 to obtain an optimal view or position relative to other components of the system, such as the robotic arms 300a, 300b, the surgical instruments attached thereto, the surgeon, and / or the surgical assistant. Additionally, telemetry data captured by the optical scanner 1100, for example, showing movements of the surgeon's hands, other body parts, and other components of the system, may be recorded to provide a rich, detailed data set describing the precise movements and forces applied by the surgeon throughout the procedure.

[0132] For example, the acquired data may be used to optimize procedures performed by the system, including, for example, automatic servo control (i.e., movement) of one or more portions of the robotic arm 300. By tracking the surgeon's tendency to keep the tool within a particular region of interest and / or the surgeon's tendency to avoid moving the tool into a particular region of interest, the system may optimize the automatic servo control algorithms to provide more stability within the particular region of interest. In addition, the acquired data may be used to optimize procedures performed by the system, including, for example, automatic re-centering of the field of view of the optical scanning device of the system. For example, if the system detects that the surgeon has moved out of the field of view or predicts that the surgeon may move out of the field of view, the system may cause the optical scanning device, e.g., the robotic arm supporting the laparoscope, to automatically adjust the laparoscope to track the desired location of the image as the surgeon performs the desired procedure. This behavior may be surgeon specific and may require understanding of the particular surgeon's preferences for the region of interest of operation. Thus, the system may control the robotic arm according to the specific operational requirements and / or preferences of the particular surgeon.

[0133] 11C, another exemplary cooperative robotic surgical system having multiple optical sensors is provided. As shown in FIG. 11C, the system 200 has multiple robotic arms, e.g., robotic arm 300a and robotic arm 300b, supported by the platform 100 having multiple wheels for providing mobility to the platform 100. As described above, each of the multiple wheels may include a braking mechanism that may be engaged and actuated to prevent movement of the platform 100. For example, the braking mechanism may be operably coupled to a controller of the system 200. Additionally, the system 200 may include multiple optical sensors, e.g., optical scanners 1100a, 1100b, and 1100c, disposed on the platform 100. For example, the optical scanner 1100a may be disposed on the top of the platform 100, and the optical scanners 1100b and 1100c may be disposed on the sides of the platform 100, as described above with respect to the optical scanner 1100 of FIG. 11A. Additionally or alternatively, one or more optical scanners may be positioned beneath the platform 100. The optical scanners 1100a, 1100b, and 1100c are configured to capture depth data. For example, the optical scanners 1100a, 1100b, and 1100c may be, for example, depth cameras, stereoscopic RGB cameras, LiDAR devices, and / or electromagnetic, capacitive, or infrared proximity sensors, etc.

[0134] The depth data generated by the optical sensors may be used by the controller of the system 200 to generate, in real time, a virtual map, e.g., a "bird's-eye view," of an area surrounding the platform 100, e.g., in an operating room. For example, the virtual map may illustrate the operating room from a top view. Additionally, as shown in FIG. 11C, the virtual map may include a graphical representation of the platform 100 (including the robotic arms 300a, 300b), as well as one or more objects, e.g., a patient table PT, and / or one or more persons, e.g., an operator O, a person P1, and a person P2, in the area surrounding the platform 100. In particular, the virtual map may graphically illustrate the proximity between the platform 100 and one or more objects / persons, e.g., as the platform 100 is moved through the operating room by the operator O. The controller may cause the display 110 to display the virtual map such that the operator O may view the virtual map on the display 110 in real time as the operator O moves the platform 100 through the operating room. Thus, the operator O may see objects and / or people in the area surrounding the platform 100 that they would not otherwise be able to see with their own eyes, for example, due to the platform 100 and / or the robotic arms 300a, 300b obstructing the operator O's view, and avoid collisions between the platform 100 and / or the robotic arms 300a, 300b and objects / people in the operating room. Additionally, the controller may cause the display 110 to display an alert, for example, a visual or audible alert, when the virtual map indicates that the platform 100 and / or the robotic arms 300a, 300b are approaching or within a predetermined distance of one or more objects / people in the operating room.

[0135] In some embodiments, the controller may cause the display 110 to only display the virtual map while the platform 100 is being moved within the operating room. For example, the platform 100 may include one or more actuators, e.g., buttons, levers, or handlebars, that may be operatively coupled to braking mechanisms of the wheels of the platform 100 such that, upon actuation of the actuator, the braking mechanisms are disengaged, thus allowing mobility of the platform 100. Thus, when the actuator is not actuated, the braking mechanisms are engaged such that mobility of the platform 100 is prevented. Thus, upon actuation of the actuator, the controller may automatically cause the display 110 to display the virtual map such that the operator O may view the area surrounding the platform 100 before, during, or after the movement of the platform 100 while the braking mechanisms are disengaged. Once the actuator is released such that the braking mechanisms are re-engaged, the display 110 may stop displaying the virtual map. In some embodiments, when the virtual map indicates that the platform 100 and / or the robotic arms 300a, 300b are approaching or within a predetermined distance of one (person) or more objects / persons in the operating room, the controller may override actuation of the actuators by the operator and re-engage the braking mechanism, thereby preventing further movement of the platform 100. Thus, the actuators may need to be released and re-actuated by the operator to disengage the braking mechanism and allow further movement of the platform 100.

[0136] FIG. 12 shows the system with the optical scanner 1100 operating during a laparoscopic procedure. As shown in FIG. 12, an optional additional optical scanner, e.g., a camera 1200, may be utilized to provide additional viewpoints, e.g., redundant measurements of the movement of the instruments held by the robotic arms, for monitoring and analysis, and / or to provide a video stream of the surgical scene, e.g., via streaming. As shown in FIG. 12, the system may include two robotic arms, e.g., robotic arms 300a, 300b, such that the robotic arm 300a holds the laparoscope 10 in a fixed position relative to the patient while the surgeon operates and manipulates the retractor 12, which is coupled to the distal end of the robotic arm 300b. Additionally, during the surgical procedure, the robotic arms 300a, 300b may be draped with sterile drapes 800a, 800b, respectively. As described above, the surgeon can freely manipulate the retractor 12 while the retractor 12 is coupled to the robot arm 300b, thereby causing the movement of the robot arm 300b due to the movement of the retractor 12 by the surgeon, and while the robot arm 300b takes into account the weight of the retractor 12 and the robot arm 300b. During a surgical procedure, the optical scanner 1100 may be used to monitor the identification, position, orientation, and / or movement of a surgical instrument, such as the laparoscope 10, coupled to the robot arm 300a, the identification, position, orientation, and / or movement of a surgical instrument, such as the retractor 12, coupled to the robot arm 300b, and whether any of the surgical instruments have been removed from the respective robot arms, either intentionally or unintentionally. Additionally, the optical scanner 1100 may be used to monitor the identification, position, orientation, and / or movement / displacement of any of the trocars Tr to ensure proper alignment of the robot arm and / or the surgical instrument with respect to the respective trocars. The system may be used in surgical procedures having one, two, three, four or more trocars, depending on the surgical procedure intended to be performed by the system.

[0137] 13A and 13B illustrate example data produced by the optical scanner 1100. For example, FIG. 13A illustrates image data captured by the optical scanner 1100, and FIG. 13B illustrates a depth map of at least some objects in the surgical space generated from the data captured by the optical scanner 1100. In particular, the optical scanner 1100 may create a depth map, e.g., a point cloud, where the value of each pixel is related to a distance from the optical scanner 1100. For example, the difference between the pixels for a first object (such as a first surgical instrument) and a second object (such as a trocar) would allow the system to calculate the distance between the surgical instrument and the trocar. Furthermore, the difference between the pixels for the first object (such as a first surgical instrument) at a first time point and the first object at a second time point would allow the system to calculate whether the first object has moved, the trajectory of the movement, the speed of the movement, and / or other parameters associated with the changed position of the first object.

[0138] As shown in Figures 13A and 13B, the surgeon S manipulates surgical tools and / or the draped and undraped robotic arms (DA and UA) positioned relative to the insufflated abdomen (A). As described above, data streams from the robotic arms, camera feeds from the laparoscope, data obtained from the optical scanner 1100, and optionally data captured from one or more imaging devices located on structures adjacent to the robotic arms, walls, ceilings, or other structures within the operating room may be recorded, stored, and used individually or in combination to understand and control the surgical system and the procedure of the surgical system. The aforementioned components, devices, and combinations thereof are collectively referred to herein as optical scanners or optical scanning devices.

[0139] For example, the system may measure and record any of the following within the coordinate space of the system: movement of handheld surgical instruments (attached to a robotic arm or separate therefrom) operated by the surgeon, the presence or absence of other surgical staff (e.g., instrument nurses, circulating nurses, anesthesiologists, etc.), the height and angular orientation of the surgical table, the patient position and volume on the surgical table, the presence or absence of drapes on the patient, the presence or absence of trocar ports and their position and orientation if present, gestures made by the surgical staff, the tasks being performed by the surgical staff, interactions between the surgical staff and the system, surgical instrument identification, attachment or detachment "actions" of surgical instruments and the system, position and orientation tracking of specific features of the surgical instrument relative to the system (e.g., camera head, couplers, fiducial markers, etc.), measurements of motion profiles or specific features within the scene that allow phases of the surgery to be identified, and the position, orientation, identification, and / or movement of any other instruments, features, and / or components of the system or being used by the surgical team.

[0140] The system may combine the measurements and / or other data described above with any other telemetry data from the system and / or video data from the laparoscope to provide a comprehensive data set that may be used to improve the overall usability, functionality, and safety of the co-operative robotic-assisted surgical system described herein. For example, as the system is set to begin a procedure, the optical scanner 1100 may detect the height and orientation of the surgical table. This information may enable the system to automatically configure the degrees of freedom of the platform 200 supporting the robotic arm 300 to a desired or correct position relative to the surgical table. Specifically, the optical scanner 1100 may be used to ensure that the height of the platform 100 is optimally positioned to ensure that the robotic arm 300 overlaps with the intended surgical workspace. Additionally, based on data acquired by the optical scanner 1100, the system may alert surgical staff to potential collisions between the system and other pieces of capital equipment in the operating room, such as a surgical table, laparoscope tower, camera boom, etc., and members of the surgical staff, such as an inadvertent bump by a staff member (either during set-up or during surgery). The system may use this information to recommend repositioning the platform 100 and / or other components of the system, the surgical table, and / or the patient, and / or to prevent the robot arm from switching into a collaborative operation mode as a result of a force applied to the robot arm due to a collision with a staff member, even if the force exceeds a predetermined force threshold of the robot arm.

[0141] Additionally, data acquired from the optical scanner 1100 may be used to monitor the progress of the setup for a surgical procedure and may be combined with the known state of the system to inform remote hospital staff (e.g., a surgeon) of the overall readiness to begin the procedure. Such progress steps may include confirmation of (i) patient on table, (ii) patient draped, (iii) sterile instruments available, (iv) robotic arms draped, (v) trocar ports to be inserted, and (vi) instruments (e.g., laparoscope and retractor) attached to the robotic arms of the system. For example, data acquired from the optical scanner 1100 may include detected gestures indicating system status (e.g., system draped), readiness to begin the procedure, etc., and may be used to further prepare the system for attachment or detachment of surgical instruments.

[0142] In addition, the optical scanner 1100 may identify the specific surgeon performing the procedure so that the system may use the surgeon's identity to load into the system a system profile associated with the particular surgeon. The system profile may include information related to the surgeon's operating parameters and / or preferences, the surgeon's patient list with per-patient parameters, desired or required algorithm sensitivities for the surgeon, positioning freedom of the support platform, etc. Examples of algorithm sensitivities that may be surgeon specific include tailoring / adjusting the force required to transition from a passive mode to a collaborative manipulation mode (e.g., low to high force), tailoring / adjusting the viscoelasticity felt by the surgeon when collaboratively manipulating the robotic arms (e.g., low to high viscoelasticity), etc. Additionally, the surgeon's preferences may include the preferred alignment of the robotic arm 300 with respect to the specific surgical instrument, e.g., the positioning of the articulations and joints of the robotic arm 300 relative to the patient, e.g., the preferred alignment may differ between a laparoscope and a retractor.

[0143] In some embodiments, the surgeon's preferences may be learned based on data from sensors that collect information about past procedures and / or the current procedure, including the surgeon's current posture, the surgeon's height, the surgeon's handedness, and other similar factors. For example, the system may record when a user interacts with the system and also record what the user is doing with the system, so that a data set may allow the surgeon preferences to "learn" and be updated over time. This learning may occur either via traditional algorithmic methods (i.e., trends over time, averages, optical flow, etc.) or via machine learning approaches (classification, discrimination, neural networks, reinforcement learning, etc.). FIG. 24 illustrates a data flow 2400 for updating the system configuration based on the user's learned behavior. As shown in FIG. 24, the system may be connected to an online database that may store the surgeon profile and each of several possible data sources, which may include a database of optical sensors, encoders, and / or other sensors, and / or manually entered user inputs. Data sources may be associated with a given surgeon, his / her preferred robotic arm configuration and motion parameters, and each procedure performed with the system, which may allow recording and analysis of system configurations and how they change from procedure to procedure and within a procedure. In the case of machine learning, the collaborative capabilities of the system may be leveraged so that user actions can be used to annotate data and create training data sets.

[0144] With regard to positioning freedom, the height of a surgical table is typically adjusted in some operating rooms to accommodate the height of the surgeon. Thus, by detecting the surgeon and loading the surgeon's specific profile, the system may position the platform at a suitable height for the individual surgeon to match the preferred height of the surgical table. In addition, the horizontal translation of the robotic arm may depend on the size of the patient. Thus, by accessing a patient list, the system may adjust the position of the arm based on the patient's body mass index ("BMI"). For example, for a patient with a high BMI, the system may move the robotic arm away from the operating table, and for a patient with a low BMI, the system may move the robotic arm closer to the operating table. Thus, the system allows the surgical team to fine-tune the position of the robotic arm relative to the patient, as needed. The system may further be configured to access a hospital medical records database and access the procedure type and any other available medical data (e.g., CT scan images, X-ray images, MRI images, and / or other patient-specific information), which may be used to inform the positioning of the trocar ports and the position and orientation of the platform 100 relative to the patient.

[0145] Based on the data captured by the optical scanner 1100, the system may generate virtual models of pieces of capital equipment and / or other objects in the operating room that are within range of the robot arm's movement in the same coordinate space as the robot arm and the surgical instruments coupled thereto, such that the virtual models may be stored and monitored, e.g., to detect potential collisions. In addition, the system may track the position and orientation of objects in the virtual models as they move relative to one another, such that the system may alert a user if the proximity (i.e., the spacing between them) of either the virtual model or the object falls below a predefined threshold, e.g., within 50 mm, 75 mm, 30 mm or less to 100 mm, or more. In some embodiments, the distance threshold may be based on the Euclidean distance between the closest points on the two virtual models, the normal distance between two surfaces of the virtual models, or the like. Further, the system may stop or block (e.g., prevent) further movement of the robotic arm, e.g., freeze the robotic arm, if the proximity of either the virtual model or an object, e.g., the robotic arm, reaches or falls below a predefined threshold relative to a laparoscopic tower, or the surface of a surgical table, or other object in the surgical space. In addition, the system may freeze the robotic arm if the system detects that the proximity between an object, e.g., capital equipment or a member of the surgical staff other than the surgeon, moving towards an individual robotic arm reaches or falls below a predefined threshold, thereby preventing inadvertent movement of the robotic arm that might otherwise result from such a collision or inadvertent force, e.g., an inadvertent bump from a member of staff or another piece of capital equipment, etc.

[0146] In addition, based on the data captured by the optical scanners 1100a, 1100b, 1100c, the system may generate a virtual map with graphical representations of objects and / or people present within a predefined area surrounding the platform and robotic arm in the operating room in the same coordinate space as the platform and robotic arm such that the virtual map may be stored and displayed to the user to detect potential collisions, for example, while the user is moving the platform throughout the operating room. In addition, the system may track the position and orientation of the graphical representations in the virtual map such that the system may alert the user if the proximity between any of the objects and / or people from the platform and / or robotic arm falls within a predefined threshold, for example, within 50 mm, 75 mm, 30 mm or less to 100 mm, or more.

[0147] Additionally, based on data captured by the optical scanner 1100, the system may track the motion of a handheld surgical instrument that is not coupled to the robotic arm, but is controlled directly and independently by the surgeon. For example, the optical scanner 1100 may track a clearly defined feature of the instrument, a fiducial marker attached to the instrument or the surgeon's glove (e.g., a sterile glove), a coupling between the robotic arm and the instrument, the distal tip of the instrument, and / or any other defined location on the instrument. For example, the fiducial marker may include a Manus virtual reality glove (commercially available by Manus, The Netherlands) or other wearable and / or an OptiTrack system (commercially available by Natural Point, Corvallis, Oregon). The following are examples of uses and purposes of the motion data: (i) close the control loop between the handheld instrument and the robotic arm holding the camera, thus allowing the surgeon to servo (i.e., move) the camera by "pointing" the handheld instrument; (ii) track information that can be used independently or in combination with other data streams to identify phases of the surgical procedure; (iii) identify the surgeon's handedness; (iv) monitor measurements that are associated with the surgeon's experience; (v) identify which tools the surgeon is using and when to change them for other tools; and / or (vi) track the number, location, and orientation of the patient's skin surface and trocar ports. This data and information may also be used and calculated by the system as part of a collaborative control framework. By measuring the true location and orientation of the trocar ports, the system can be provided with an additional safety check to ensure that system level calculations are correct, for example, ensuring that the actual movement of the robotic arm or instrument matches the commanded movement of the robotic arm or instrument in the robot-assisted mode.

[0148] Based on the data captured by the optical scanner 1100, the system may further track instruments being used at individual ports, how often instruments are swapped between ports, if the system holds instruments in place while the patient or surgical table is moved (in which case the system may accommodate the movement by changing the robotic arm's operating mode to a passive mode and repositioning the robotic arm 300 and / or platform 100), manually held instruments versus ports with instruments coupled to a robotic arm, and / or other conditions or parameters of the operating room or system to monitor and determine if additional trocar ports are added. Knowledge of the location and orientation of the skin surface and trocar ports relative to the robotic arm may facilitate the implementation of a "virtual boundary", as described in more detail below.

[0149] 14, components that may be included in a collaborative robotic platform 1400 are described. The platform 1400 may include one or more processors 1402, communication circuitry 1404, a power supply 1406, a user interface 1408, and / or a memory 1410. One or more electrical components and / or circuits may perform some or all of the roles of the various components described herein. Although described separately, it should be understood that the electrical components need not be separate structural elements. For example, the platform 1400 and communication circuitry 1404 may be embodied in a single chip. Additionally, while the platform 1400 is described as having a memory 1410, the memory chip may be provided separately.

[0150] The platform 1400 may contain memory and / or be coupled via one or more buses to read or write information to the memory. The memory 1410 may include a processor cache, including a multi-level hierarchical cache, in which different levels have different capacity and access speed. The memory may also include random access memory (RAM), other volatile storage devices, or non-volatile storage devices. The memory 1410 may be RAM, ROM, flash, other volatile or non-volatile storage devices, or other known memories, or some combination thereof, and preferably includes storage in which data may be selectively stored. For example, storage devices may include, for example, hard drives, optical disks, flash memory, and Zip drives. Programmable instructions may be stored on the memory 1410 and may execute algorithms, for example, to calculate a desired force to be applied along the robot arm 300 and / or a surgical instrument coupled thereto, apply impedances at individual joints of the robot arm 300, and effect the desired force.

[0151] Platform 1400 may incorporate a processor 1402, which may be comprised of one or more processors, which may be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any suitable combination thereof designed to perform the functions described herein. Platform 1400 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0152] Platform 1400 may execute an operating system (e.g., operating system 1446), such as, for example, Windows, Mac OS, QNX, Unix, or Solaris 5.10, in conjunction with firmware / software stored in memory. Platform 1400 also executes software applications, which are stored in memory. For example, the software may be programs in any suitable programming language known to those of skill in the art, including, for example, C++, PHP, or Java.

[0153] The communications circuitry 1404 may include circuitry that allows the platform 1400 to communicate with image capture devices, such as optical scanners and / or endoscopes. The communications circuitry 1404 may be configured for wired and / or wireless communications networks, such as via the Internet, telephone networks, Bluetooth networks, and / or Wifi networks, using techniques known in the art. The communications circuitry 1404 may be communications chips known in the art, such as Bluetooth and / or Wifi chips. The communications circuitry 1404 allows the platform 1400 to transfer information, such as force measurements on the body wall at the trocar insertion point, locally and / or to a remote location, such as a server.

[0154] The power supply 1406 may provide alternating or direct current. In a direct current embodiment, the power supply may include a suitable battery, such as a replaceable or rechargeable battery, and the device may include circuitry for charging the rechargeable battery and a removable power cord. The power supply 1406 may be a port to allow the platform 1400 to be plugged into a conventional wall socket, for example, via a cord with an AC / DC power converter and / or a USB port, to power the components within the platform 1400. The power supply 1406 may be operatively coupled to an emergency switch such that upon activation of the emergency switch, power being provided to components within the platform 1400, including braking mechanisms disposed on at least some of the joints of the robot arm 300, is stopped. For example, the braking mechanism may require power to be shut off such that the braking mechanism acts to prevent movement of the robot arm 300 without power, even if power is not provided to the braking mechanism.

[0155] A user interface 1408 may be used to receive input from a user and / or provide output to a user. For example, the user interface 1408 may include a touch screen, a display, switches, dials, lights, etc. Thus, the user interface 1408 may display information such as a selected surgical instrument identification and force measurements observed during operation of the robotic arm 300. Additionally, the user interface 1408 may receive user input including adjustments to a predetermined amount of movement or a predetermined dwell period at the handle of a surgical instrument to automatically switch the robotic arm to a passive mode, a predetermined threshold of force applied at the handle of a surgical instrument to automatically switch the robotic arm to a collaborative manipulation mode, a predefined position of a force sense barrier, an identification of a surgical instrument coupled to a distal end of the robotic arm, a vertical height of the robotic arm, a horizontal position of the robotic arm, etc., such that the platform 1400 may adjust the information / parameters accordingly. In some embodiments, the user interface 1408 is not present on the platform 1400 , but instead is provided on a remote, external computing device that is communicatively connected to the platform 1400 via the communications network 1404 .

[0156] The memory 1410, which is an example of a non-transitory computer readable medium, may be used to store an operating system (OS) 1446, a surgical instrument identification module 1412, a surgical instrument calibration module 1414, an encoder interface module 1416, a robotic arm position determination module 1418, a trocar position detection module 1420, a force detection module 1422, an impedance calculation module 1424, a motor interface module 1426, an optical scanner interface module 1428, a gesture detection module 1430, a passive mode determination module 1432, a collaborative manipulation mode determination module 1434, a haptic mode determination module 1436, a robotic assistance mode determination module 1438, an obstacle detection module 1440, an indicator interface module 1442, and a fatigue detection module 1444. The modules are provided in the form of computer executable instructions / algorithms that may be executed by the processor 1402 to perform various operations according to the present disclosure.

[0157] For example, during a procedure, the system may continually run the algorithms described herein based on data collected by the system. The data may be collected and / or recorded using any of the components and methods disclosed herein, including, for example, from sensors / encoders within the robot, from an optical scanning device that communicates with other components of the robotic system, and / or from manual input by an operator of the system. Thus, the algorithms, data, and configurations of the system may enable a user to coordinate the robot arm with minimal shock and impact from the weight of the robot arm and / or the surgical instruments coupled thereto, gravity, and other forces that conventional robotic arms cannot compensate for. Some of the parameters of the algorithms described herein may control aspects of the system's behavior, including, for example, the robustness of the features detected, the sensitivity to false positives, the robot control gain, the number of features to track, the dead zone radius, etc.

[0158] A surgical instrument identification module 1412 may be executed by the processor 1402 to identify the surgical instruments coupled to each of the robotic arms and load the appropriate calibration file into the controller system. For example, a calibration file for each surgical instrument may be stored in a database accessible by the surgical instrument identification module 1412 and may include information associated with the surgical instrument, such as, for example, instrument type, weight, center of mass, length, instrument shaft diameter, etc. Thus, once the appropriate calibration file is loaded and the associated surgical instrument is coupled to the robotic arm 300, the system will automatically account for the mass of the surgical instrument based on the data in the calibration file, for example, compensating for the force of gravity on the surgical instrument when it is attached to the robotic arm 300, so that the robotic arm 300 can hold the surgical instrument in place after it is coupled to the robotic arm and the operator releases the surgical instrument. For example, the surgical instrument identification module 1412 may identify the surgical instrument based on user input via the user interface 1408, e.g., an operator may select the surgical instrument from a database of surgical instruments stored in the memory 1410.

[0159] In some embodiments, the surgical instrument identification module 1412 may automatically identify a surgical instrument coupled to the robotic arm via the coupler body and coupler interface using, for example, an RFID transmitter chip and reader or receiver (e.g., placing an RFID sticker or transmitter on the surgical instrument that can transmit information about the surgical instrument to a receiver of the system), a near-field communication ("NFC") device such as a near-field magnetic induction communication device, a barcode and scanner or other optical device, a magnet-based communication system, a reed switch, a Bluetooth transmitter, an optical scanner and instrument weight and / or data gathered from a look-up table, and / or any other feature or mechanism described herein or suitable for identification of a surgical instrument. As described above, the coupler body may be selected based on the size and shape of the lumen extending therethrough to accommodate and engage a surgical instrument having a known extended shaft diameter. Thus, the surgical instrument identification module 1412 can automatically identify a surgical instrument based on the coupler body that is coupled to the surgical instrument via the magnetic connection between the coupler body and the coupler interface.

[0160] In some embodiments, the surgical instrument identification module 1412 may identify a surgical instrument, e.g., a type of surgical instrument, based on data acquired by the optical scanner 1100 via the optical scanner interface module 1428, described in more detail below. For example, the data may include such data associated with a specific instrument such that the surgical instrument identification module 1412 may compare the measurement data with information contained in a database, identify the instrument, and load the appropriate calibration file into the controller system. Similarly, the surgical instrument identification module 1412 may detect if an instrument has been removed and return the calibration parameters to a default configuration.

[0161] The surgical instrument calibration module 1414 may be executed by the processor 1402 to calibrate a surgical instrument, for example, a surgical instrument that does not currently have a calibration file associated with it in the database stored in the memory 1410. Thus, the surgical instrument calibration module 1414 may calculate measurements and specifications of a surgical instrument based on force measurements of the robotic arm 300 applied by the surgical instrument via the force detection module 1422 when it is coupled to the robotic arm 300 and the system is in a calibration mode, as described in more detail below with respect to FIG. 16. For example, the surgical instrument calibration module 1414 may generate a calibration file for the surgical instrument that includes information such as the surgical instrument's instrument type, weight, center of mass, length, instrument shaft diameter, viscoelastic parameters, etc. At least some of the surgical instrument information in the calibration file, e.g., instrument type, may be provided by user input via the user interface 1408, or, for example, the instrument type, center of mass of the instrument, instrument length, and instrument diameter may be detected by the optical scanner interface module 1428.

[0162] If the surgical instrument calibration module 1414 determines that the recalibration results are consistently different from configurations already loaded into the system, the surgical instrument calibration module 1414 may replace the existing information or add to its list of known tools and automatically load them without any user input. The surgical instrument calibration module 1414 may determine that the calibration coefficients are not correct to compensate for gravity, for example, if the surgical instrument is coupled with a robotic arm and the robotic arm moves solely due to gravity acting on the robotic arm and / or the surgical instrument, which may occur when the surgical instrument is positioned completely outside the patient's body. Additionally, the surgical instrument calibration module 1414 may automatically update or adjust the calibration coefficients (e.g., forces applied to the joints of the robotic arm) if it determines that the calibration coefficients are not correct to compensate for gravity. Thus, the surgical instrument calibration module 1414 may update the calibration coefficients for a particular surgical instrument and store the updated calibration coefficients for the particular surgical instrument in an associated calibration file for future use.

[0163] An encoder interface module 1416 may be executed by the processor 1402 to receive and process formation angle measurement data in real time from multiple encoders of the robot arm 300, e.g., encoders E1-E7. For example, the encoder interface module 1416 may calculate the change over time in the formation angle of a joint of the robot arm 300 that is rotatably coupled to a given joint associated with an encoder. As explained above, the system may include redundant encoders at each joint of the robot arm 300, thereby ensuring safe operation of the robot arm 300. Furthermore, additional encoders may be placed on the platform 100 to measure the formation angle / position of each robot arm relative to the platform 100, e.g., the vertical and horizontal positions of the robot arm relative to the platform 100. Thus, one encoder may be placed on the platform 100 to measure the movement of the robot arm along the vertical axis of the platform 100, and another encoder may be placed on the platform 100 to measure the movement of the robot arm along the horizontal axis of the platform 100.

[0164] The robot arm positioning module 1418 may be executed by the processor 1402 in real time to determine the position of the robot arm 300 and, if applicable, a surgical instrument attached thereto in 3D space based on the angle measurement data generated by the encoder interface module 1416. For example, the robot arm positioning module 1418 may determine the positions of the various linkages and joints of the robot arm 300 and along with a surgical instrument coupled to the robot arm 300. Based on the position data of the robot arm 300 and / or the surgical instrument, the robot arm positioning module 1418 may calculate, in real time, the speed and / or acceleration of the movement of the robot arm 300 and the surgical instrument attached thereto. The robot arm position determination module 1418 may determine a resultant velocity of the distal end of the robot arm 300, for example by determining the individual velocities of the various joints of the robot arm 300, e.g., via encoders associated with each of the various joints, which may be used by the passive mode determination module 1432 to determine whether the movement of the distal end of the robot arm 300 is within a predetermined threshold for purposes of transitioning the system 200 to a passive mode, as described in more detail below.

[0165] The trocar position detection module 1420 may be executed by the processor 1402 to determine the position and / or orientation of one or more trocar ports to be inserted into the patient. The position and / or orientation of the trocar port may be derived based on data obtained from, for example, inertial measurement units and / or accelerometers, optical scanners, electromechanical tracking instruments, linear encoders, sensors and data as described above. For example, the position of the trocar port on the patient may be determined using a laser pointing system that may be mounted on one or more of the components of the system, for example, the wrist portion 311 of the robotic arm, and controlled by the system to point the trocar to an optimal or determined position on the patient's body for insertion. Furthermore, upon insertion of a surgical instrument attached to the robotic arm 300 through a trocar, a virtual line may be established continuously along the longitudinal axis of the surgical instrument, the alignment / orientation of which may be automatically determined in real time as the surgical instrument moves about the trocar point, upon attachment of the surgical instrument to the coupler interface 400 via the coupler body via a magnetic connection, as described above. Furthermore, when the surgical instrument is inserted into the trocar port, it is pointed toward the trocar point, and thus the distal wrist joint 316 is also pointed toward the trocar point, the angle of which may be measured by an encoder associated therewith. Thus, the trocar point may be calculated as the intersection of multiple virtual lines established continuously along the longitudinal axis of the surgical instrument. In this manner, the calculated trocar point will remain fixed relative to the patient as the surgical instrument is steered about the trocar port, e.g., rotated or moved in and out of the patient.

[0166] Based on the known position of the distal end of the robot arm 300 from the robot arm positioning module 1418, plus the known position and / or orientation of the trocar port, the system may maintain the position of the distal end of the robot arm 300 relative to the trocar point as the robot arm 300 moves, for example, via its vertical or horizontal adjustment by the platform 100, or as the height of the patient table is adjusted, thereby moving the height of the patient's abdomen, thereby keeping the surgical instrument within the patient's body and coupled to the robot arm 300 stationary during these external movements. To accomplish this, the known position of the distal end of the robot arm 300 from the robot arm positioning module 1418 is calculated in the global frame of the system by adding the position of the platform 100 to the kinematic calculations (e.g., the "forward kinematics" of the robot arm 300 in the context of a serial-chain robotic manipulator). Using a globally known position of the distal end of the robot arm 300, the system can hold its position stationary during external movement by applying appropriate forces to the robot arm 300 that minimize the error between its current position and the desired position.

[0167] The force detection module 1422 may be executed by the processor 1402 to detect forces, e.g., body wall forces, applied on the robot arm 300, e.g., at a joint or joint of the robot arm 300, or along a surgical instrument, and on a trocar. For example, the force detection module 1422 may receive, in real time, motor current measurements at each motor, e.g., M1, M2, M3, disposed in the base of the robot arm 300, which are operably coupled to joints, e.g., base joint 303, shoulder joint 318, elbow joint 322, wrist joint 332, respectively, of the robot arm 300. The motor current measurements indicate the amount of force applied to the associated joint. Thus, the force applied to each joint of the robot arm 300 and the surgical instrument attached thereto may be calculated based on the motor current measurements and position data generated by the robot arm positioning module 1418 and / or the trocar position detection module 1420.

[0168] Due to the passive axis at the distal end of the robotic arm 300, the force applied by an instrument coupled with the robotic arm on the trocar can generally remain consistent throughout the robotic arm's workspace. The force on the trocar can be affected by the interaction of the distal tip of the instrument with tissue within the body. For example, when a tissue retractor advanced through the trocar engages (e.g., grasps) body tissue or another object inside the body, the force imparted on the end of the instrument from the body tissue or other object can cause a change in the force applied to the trocar. In some aspects, the force on the trocar can be a function of the amount of weight being lifted by the instrument being used.

[0169] The impedance calculation module 1424 may be executed by the processor 1402 to determine the amount of impedance / torque needed to be applied to individual joints of the robot arm 300 to achieve a desired effect, e.g., holding the robot arm 300 in a static position in a passive mode, allowing the robot arm 300 to move freely while compensating for the gravity of the robot arm and a surgical instrument attached thereto in a collaborative manipulation mode, applying increased impedance to the robot arm 300 in a haptic mode when the robot arm 300 and / or a surgical instrument attached thereto is within a pre-defined virtual haptic barrier, etc.

[0170] For example, the impedance calculation module 1424 may determine an amount of force required by the robot arm 300 to achieve a desired effect based on the position data of the robot arm 300 generated by the robot arm position determination module 1418 and the trocar position data generated by the trocar position detection module 1420. For example, by determining the position of the distal end of the robot arm 300 and the entry point of the surgical instrument into the patient, e.g., the trocar position, and with knowledge of one or more instrument parameters, e.g., the mass and center of mass of the surgical instrument stored by the surgical instrument calibration module 1414, the impedance calculation module 1424 may calculate an amount of force required to compensate for the gravity of the surgical instrument (compensation force), as described in more detail below with respect to FIG. 18A. Thus, the amount of compensation force required to compensate for the gravity of the surgical instrument may be converted into a torque to be applied at a joint of the robot arm 300 by a motor operably coupled to the joint of the robot arm 300, e.g., as indicated by motor current measurements.

[0171] Further, by determining the position of the distal end of the robot arm 300 and thus the change in position over time due to, for example, external forces applied to the distal end of the robot arm 300, for example, by tissue held by the working end of the surgical instrument, and with knowledge of one or more instrument parameters, for example, the mass, center of mass, and length of the surgical instrument stored by the surgical instrument calibration module 1414, the impedance calculation module 1424 may calculate the amount of force required to maintain the surgical instrument in a static position (the holding force), as described in further detail below with respect to FIG. 18B. Thus, the amount of holding force required to resist changes in the position of the distal end of the robot arm 300, plus an amount of compensation force required to compensate for the gravity of the surgical instrument, may be converted into a torque to be applied at the joints of the robot arm 300 to maintain the robot arm 300 in a static position, for example, as indicated by motor current measurements, by motors operably coupled to the joints of the robot arm 300. Additionally, the impedance calculation module 1424 and / or the force detection module 1422 may calculate the amount of force applied by the surgical instrument to the patient at the entry point, e.g., the trocar, and the amount of force applied to the working end of the surgical instrument, e.g., the grasper end of the surgical instrument, based on one or more parameters of the surgical instrument, such as the compensation force, the holding force, the mass, center of mass, and length of the surgical instrument, and the distance from the center of mass to the entry point.

[0172] Additionally or alternatively, by determining the force applied on the robot arm 300 via the force detection module 1422, as well as the position / velocity / acceleration of the distal end of the robot arm 300 in 3D space via the robot arm positioning module 1418, a desired force / impedance to be applied to the robot arm 300 to compensate for the applied force may be calculated, for example for gravity compensation or to hold the robot arm 300 in a static position in a passive mode. Thus, the desired force may be converted into a torque to be applied at the joints of the robot arm 300, for example by a motor operably coupled to the joints of the robot arm 300. For example, a robot Jacobian matrix may be used for this purpose. The Jacobian matrix is ​​a matrix calculated at each given support of the robot arm, relating the velocity at the joint to the velocity at the distal end of the robot arm 300. [ka]

[0173] where V is the velocity vector at the distal end of the robot arm 300, J is its Jacobian matrix, and q dot is the joint velocity, expressed in vector form. Using energy principles, assuming negligible mass on the joints of the robot arm 300 and negligible friction / damping, the power of the system can be determined by multiplying its forces and velocities: [ka]

[0174] where F is the generalized force vector at the distal end of the robot 300. Furthermore, the vector manipulation yields: [ka]

[0175] where t denotes the transpose of a matrix so that the forces at the distal end of the robot arm 300 can be converted to torques to be applied at the joints using the Jacobian matrix.

[0176] The motor interface module 1426 may be executed by the processor 1402 to receive motor current readings at each motor, e.g., M1, M2, M3, located within the base of the robot arm 300, and to actuate the individual motors, e.g., by applying a predetermined impedance, to achieve a desired result as described herein and / or to move a joint operatively coupled to the individual motor, such as in a robotic-assisted mode.

[0177] The optical scanner interface module 1428 may be executed by the processor 1402 to receive depth data acquired by the optical scanner 1100 and process the depth data, e.g., to detect predefined conditions therein. Additionally, the optical scanner interface module 1428 may generate a depth map indicative of the received depth data, which may be displayed to an operator, e.g., via a monitor. For example, the optical scanner interface module 1428 may map the location of the trocar port in 3D space, such that the mapping of the trocar port may be communicated to an operator, e.g., via a display or user interface 1408. The optical scanner interface module 1428 may further receive image data from additional optical scanning devices, as defined herein, including, e.g., an endoscope, operably coupled to the system. Additionally, the optical scanner interface module 1428 may receive depth data acquired by the optical scanners 1100a, 1100b, 1100c coupled to the platform 100, as described above with respect to FIGURE 11C, process the depth data, and generate a virtual map of the area surrounding the platform 100, which may be displayed to the operator via a monitor, e.g., the display 110. For example, the optical scanner interface module 1428 may generate a graphical representation of the system 200, including the platform 100 and the robotic arms 300a, 300b, as well as any objects and / or people in the area surrounding the platform 100, for display in the virtual map.

[0178] The gesture detection module 1430 may be executed by the processor 1402 to detect predefined gesture patterns as user inputs and perform actions associated with the user inputs. The predefined gesture patterns may include, for example, movement of a surgical instrument (whether attached to the robotic arm 300 or not), movement of the robotic arm 300 or other components of the system, such as foot pedals, buttons, etc., and / or movement of an operator in a predefined pattern. For example, reciprocating movement of a surgical instrument in a first direction (e.g., left / right, up / down, forward / backward, circular) may be associated with a first user input that requests a first action by the system, and / or reciprocating movement in a second direction different from the first direction (e.g., left / right, up / down, forward / backward, circular) may be associated with a second user input that requests a second action by the system. Similarly, pressing a foot pedal or button operatively coupled to the system in a predefined manner may be associated with a third user input requesting a third action by the system, and repetitive back and forth or up and down movements of the operator's head may be associated with a fourth user input requesting a fourth action by the system. Various predefined gesture patterns associated with different components or operators of the system may be redundant such that the associated user inputs may be identical for different gesture patterns. The predefined gesture patterns may be detected, for example, by an optical scanning device such as a laparoscope or optical scanner 1100 via the optical scanner interface module 1428, or directly by forces applied to the robot arm 300 via the force detection module 1422 or other components of the system.

[0179] Actions responsive to user inputs associated with predefined gesture patterns may include, for example, enabling tool tracking, servoing (i.e., moving) the laparoscope based on the motion of the handheld tool, applying a braking force on the robotic arm (e.g., preventing its further movement), applying a software lock on the robotic arm, dynamically changing the amount of time it takes for the robotic arm to transition between states from a default setting, and / or, if applicable, identifying a member of the surgical staff touching the robotic arm. This information may be used to ensure that the system does not move the robotic arm if the surgeon is not touching it, for example, to avoid scenarios where an external force acts on the robotic arm (e.g., an optical cable or other wire is pulled across the robotic arm), and that the system perceives forces that are intentional from the surgeon. The same information may be used to detect the surgeon's gaze direction, for example, whether the surgeon is looking at a video feed or elsewhere in the room, so that the system may freeze the robotic arm if the surgeon's gaze is not in the direction it should be. Additionally, the system may reposition the camera's field of view based on, for example, data from the optical scanner 1100, based on the direction the surgeon is facing, or based on the object the surgeon is believed to be looking at.

[0180] In some embodiments, an operator may actively switch the system, for example via user interface 1408, to a command mode in which certain movements or gestures of the robotic arm, surgical instrument, operator, or other as described herein are monitored by gesture detection module 1430 to determine whether they are consistent with predefined gesture patterns associated with predefined user inputs.

[0181] A passive mode decision module 1432 may be executed by the processor 1402 to analyze the motion characteristics of the robot arm 300 and determine whether to switch the motion of the robot arm 300 to a passive mode in which the system applies impedance to the joints of the robot arm 300 via the motor interface module 1426 in an amount sufficient to maintain the robot arm 300, and therefore, if applicable, a surgical instrument attached thereto, in a static position, thereby compensating for the mass of the robot arm 300 and the surgical instrument as well as any other external forces acting on the robot arm 300 and / or the surgical instrument. If the robot arm 300 is moved slightly while in the passive mode but not with sufficient force to switch out of the passive mode, the system may adjust the amount of impedance applied to the robot arm 300 to maintain the static position and continue this process until the robot arm 300 is held in a static position. For example, the passive mode determination module 1432 may determine to switch the operating mode of the robot arm 300 to the passive mode if the movement of the robot arm due to movement at the handle of the surgical instrument, as determined by the force detection module 1422, is less than a predetermined amount, e.g., 1-5 mm or less, for at least a predetermined dwell period associated with the robot arm 300. The predetermined dwell period refers to the length of time that the robot arm 300 and / or, if applicable, a surgical instrument attached thereto, is held in a static position. For example, the predetermined dwell time may range or exceed, e.g., 0.1-3 seconds, and may be adjusted by the operator. FIG. 19 illustrates a table or example values ​​of threshold dwell times for a range of sample instrument types.

[0182] In some embodiments, the passive mode determination module 1432 may determine to switch the operating mode of the robot arm 300 to a passive mode if the movement of the distal end of the robot arm due to movement at the handle of the surgical instrument, as determined by the force detection module 1422, has a speed that is less than a predetermined dwell speed / velocity. For example, if the passive mode determination module 1432 determines that the distal end of the robot arm 300 and / or, if applicable, a surgical instrument attached thereto is moving at a speed slower than a predetermined dwell speed for the entirety of a predetermined dwell period, the passive mode determination module 1432 may switch the operating mode of the robot arm 300 to a passive mode. Figure 19 illustrates a table or example values ​​of threshold dwell speeds for a range of sample instrument types. For example, for surgical instruments such as specula and tissue manipulation devices, the threshold dwell speed may be, for example, 3-5 mm / sec, and for surgical instruments such as suturing instruments, needle holders, high force instruments, staplers, and clip appliers, the threshold dwell speed may be, for example, 1-2 mm / sec. In some embodiments, the passive mode determination module 1432 may determine to switch the operating mode of the robotic arm 300 to the passive mode in response to the attachment of the surgical instrument to the robotic arm 300 and / or the corresponding detachment of the surgical instrument from the robotic arm 300 based on the identification of the surgical instrument.

[0183] A collaborative operation mode determination module 1434 may be executed by the processor 1402 to analyze the motion characteristics of the robot arm 300 and determine whether to switch the motion mode of the robot arm 300 to a collaborative operation mode in which the robot arm 300 is allowed to move freely in response to movements at the handles of surgical instruments to perform laparoscopic surgery using surgical instruments, while the system applies impedance to the robot arm 300 via the motor interface module 1426 in an amount sufficient to account for the mass of the surgical instruments and the robot arm 300. Additionally, the impedance applied to the robot arm 300 may provide a predetermined level of viscoelasticity perceptible by an operator. FIG. 19 illustrates a table or example values ​​of viscoelasticity levels for a range of sample instrument types. In some embodiments, the viscoelasticity level may be a function of the speed at which the surgical instrument is being moved and the distance from the trocar point to the tip of the instrument. For example, the collaborative manipulation mode determination module 1434 may determine to switch the operating mode of the robot arm 300 to the collaborative manipulation mode if a force applied at the robot arm 300 due to a force applied at the handle of a surgical instrument exceeds a predefined threshold (e.g., a "breakaway force") associated with the robot arm 300. The predefined force threshold may be, for example, at least 7 Newtons, approximately 7 Newtons, at least 7 Newtons, between 4 and 15 Newtons, between 4 and 10 Newtons. The predefined force threshold may depend on the type of surgical instrument being used and / or whether there is an external force being applied to the surgical instrument.

[0184] FIG. 19 illustrates a table or example values ​​of predefined force thresholds for a range of sample instrument types. As shown in FIG. 19, the predefined force thresholds may reflect typical external tissue forces that may be imparted on a surgical instrument. In some embodiments, the predefined force thresholds may be increased if the force is imparted on the surgical instrument by the tissue or organ or otherwise, depending on the direction of the breakaway force. For example, if the breakaway force is in the same direction as the force imparted on the surgical instrument from the tissue or organ, the predefined force threshold may be increased by an amount equal to or corresponding to the force imparted on the surgical instrument from the tissue or organ. In some embodiments, the predefined force thresholds for individual robotic arms are adjusted based on the patient's body mass index ("BMI"). For example, a patient with a higher BMI may have a heavier liver that would likely impart a larger force on the instrument. Thus, the predefined force thresholds may be selected to be higher for patients with a higher BMI. Thus, an operator may activate, for example, via user interface 1408, a "high force mode," in which the predefined force threshold is increased to accommodate working with heavier tissue or organs. For example, the predefined force threshold may be selectively increased by 20-100% or more.

[0185] Additionally, the force applied by the user on the surgical instrument and any external tissue force applied to the surgical instrument may be direction dependent. For example, if the force applied by the user on the surgical instrument is in the same direction as the external tissue force applied to the surgical instrument, the two forces may be additive such that the amount of force applied by the user on the surgical instrument required to overcome a predefined force threshold may be reduced by the magnitude of the external tissue force such that a force lower than the predefined force threshold would be required to exit the passive mode and enter the collaborative operation mode. On the other hand, if the force applied by the user on the surgical instrument is in the opposite direction to the external tissue force applied to the surgical instrument, the necessary amount of force applied by the user on the surgical instrument required to overcome a predefined force threshold may be increased by the magnitude of the external tissue force such that a force higher than the predefined force threshold would be required to exit the passive mode and enter the collaborative operation mode.

[0186] In addition, if the force applied by the user on the surgical instrument is in a direction perpendicular to the external tissue force applied to the surgical instrument, the required amount of force applied by the user on the surgical instrument required to overcome the predefined force threshold may not be affected by the magnitude of the external tissue force, such that the required force applied by the user on the surgical instrument required to exit the passive mode and enter the collaborative manipulation mode will be equal to the predefined force threshold. For other directions, the force vector of the applied force may be added to or offset from the force vector of the external tissue force to overcome the predefined force threshold for the particular surgical instrument coupled with the system or robotic arm, depending on the external tissue force, and the direction of the force applied by the user, if applicable. In some embodiments, the collaborative manipulation mode determination module 1434 may determine to switch the operating mode of the robotic arm 300 to the collaborative manipulation mode based on the identity of the surgical instrument.

[0187] The haptic mode determination module 1436 may be executed by the processor 1402 to analyze the motion characteristics of the robot arm 300 and determine whether to switch the motion mode of the robot arm 300 to a haptic mode in which the system applies impedance to the robot arm 300 via the motor interface module 1426 in an amount higher than that applied in the collaborative manipulation mode, making the movement of the robot arm 300 more viscoelastic than in the collaborative manipulation mode, in response to movements at the handle of the surgical instrument. For example, the haptic mode determination module 1436 may determine to switch the motion mode of the robot arm 300 to the haptic mode if at least a portion of the robot arm 300 and / or a surgical instrument attached thereto is within a predefined virtual haptic boundary. In particular, a virtual haptic boundary may be established by the system such that the robot arm or a surgical instrument coupled thereto should not violate the boundary. For example, a virtual boundary may be established at the surface of the patient to prevent any portion of the robot arm or an instrument supported by the robot arm from contacting the patient except through one or more trocars. Similarly, the virtual haptic boundary may include a haptic funnel to help guide the instrument into the patient as the operator inserts the instrument into the trocar port. Thus, for example, based on position data of the robot arm 300 and / or the surgical instrument coupled thereto received by the robot arm position determination module 1418 and / or the trocar position detection module 1420, the haptic mode determination module 1436 may determine whether the robot arm 300 and / or the surgical instrument are within a predefined virtual haptic boundary and therefore transition the robot arm 300 into a haptic mode in which the processor 1402 may instruct the associated motors to apply an effective amount of impedance to the joints of the robot arm 300 that is perceptible by the operator and communicate the virtual haptic boundary to the operator. Thus, the viscoelasticity of the robot arm 300 observed by the operator will be much higher than in the cooperative manipulation mode.In some embodiments, the haptic mode determination module 1436 may determine to switch the operational mode of the robotic arm 300 to the haptic mode based on the identification of the surgical instrument.

[0188] The robot-assisted mode determination module 1438 may be executed by the processor 1402 to analyze the motion characteristics of the robot arm 300 and determine whether to switch the motion mode of the robot arm 300 to a robot-assisted mode in which the processor 1402 may command associated motors, via the motor interface module 1426, to move corresponding links and joints of the robot arm 300 to achieve a desired result. For example, the robot-assisted mode determination module 1438 may determine to switch the motion mode of the robot arm 300 to the robot-assisted mode if a predefined condition exists, e.g., based on data obtained from the optical scanner interface module 1428.

[0189] For example, the robot-assisted mode decision module 1438 may determine, based on image data acquired from the laparoscope or optical scanner 1100 via the optical scanner interface module 1428, that a condition exists, e.g., the field of view of the robotic arm 300 or the laparoscope coupled to the optical scanner 1100 is not optimal for a given surgical procedure, e.g., due to obstruction by a surgeon or assistant or another component of the system, such that the robotic arm coupled to the laparoscope or optical scanner 1100 should be repositioned or zoomed in or out to optimize the view of the surgical site for the operator. Thus, in the robot-assisted mode, the processor 1402 may instruct the robotic arm 300 to move, reposition, and / or zoom in or out the laparoscope or increase the resolution of the image, or otherwise, either automatically / semi-automatically or in response to user input by the operator. For example, user input by the operator may be determined by the gesture detection module 1430, as described above, such that in a first direction, movement of the robotic arm or surgical instrument in a predefined gesture pattern causes the endoscope to increase resolution or magnification, in a second direction, causes the endoscope to decrease resolution or magnification, and movement in another predefined gesture pattern causes the robotic arm to hold the laparoscope and move it back away from the patient's body.

[0190] Additionally, the robot-assisted mode determination module 1438 may determine that a condition exists, e.g., one or more trocars are not in an optimal position, e.g., due to patient movement, such that the robot arm 300 should be repositioned to maintain the trocars in an optimal position, e.g., the approximate center of the robot arm 300's range of motion, thereby minimizing the risk of reaching the robot arm's joint limits during the procedure. Thus, in the robot-assisted mode, the processor 1402 may instruct the system to reposition the robot arm 300, e.g., via vertical / horizontal adjustments by the platform 100 or via the joints and connections of the robot arm 300, to better align the surgical instrument workspace.

[0191] The robot-assisted mode determination module 1438 may determine that a condition exists, e.g., the distance between an object and the robot arm 300 is within a predetermined threshold, based on image data acquired from the laparoscope or optical scanner 1100 via the optical scanner interface module 1428, such that the robot arm should be frozen to avoid collision with the object. Thus, in the robot-assisted mode, the processor 1402 may instruct the robot arm 300 to apply a braking force to slow the robot arm or inhibit or prevent movement within a predetermined distance from another object.

[0192] An obstacle detection module 1440 may be executed by the processor 1402 to analyze data indicative of operational characteristics of the system, such as position data generated by the robot arm positioning module 1418 and / or the trocar position detection module 1420 and / or force measurements calculated by the force detection module 1422, to detect whether an obstacle condition exists. For example, the obstacle detection module 1440 may detect an obstacle condition of the system and determine whether the obstacle condition is a "minor obstacle," a "major obstacle," or a "catastrophic obstacle," and each category of obstacle condition may be resolved in a different predefined manner.

[0193] For example, the obstacle detection module 1440 may detect minor obstacle conditions, such as the robot arm 300 being moved at a speed exceeding a predetermined speed threshold, which may be resolved, for example, by slowing down the movement of the robot arm 300. In some embodiments, the system may automatically apply additional impedance to the robot arm 300 when the robot arm 300 is moving too fast, thereby forcing the operator to slow down the movement of the robot arm 300. Additionally, the obstacle detection module 1440 may detect major obstacle conditions, such as an inadvertent bump of the robot arm 300, as indicated by a large force applied to the robot arm 300 by someone other than the operator. In response to detecting a major obstacle condition, the obstacle detection module 1440 may activate braking mechanisms associated with each motorized joint of the robot arm 300 (or at least the joints associated with the major obstacle condition), thereby freezing the robot arm 300 and preventing further movement of the robot arm 300. Such critical fault conditions may be cleared by the operator activating a "Clear" option displayed on the user interface 1408. The fault detection module 1440 may detect a critical fault condition, such as redundant encoders associated with a given joint of the robot arm 300 generating different formation angle measurements with a delta exceeding a predetermined threshold. In response to detecting a critical fault condition, the fault detection module 1440 may activate braking mechanisms associated with each motorized joint of the robot arm 300, thereby freezing the robot arm 300 and preventing further movement of the robot arm 300. Such a critical fault condition may be cleared by the operator restarting the system. If, upon restarting the system, the critical fault condition is still detected by the fault detection module 1440, the robot arm 300 will remain frozen until the critical fault condition is cleared.

[0194] The indicator interface module 1442 may be executed by the processor 1402 to cause the indicator 334 to communicate the status of the system, e.g., the operational mode of the robotic arm 300, to an operator or other user based on, for example, decisions made by the passive mode decision module 1432, the collaborative mode decision module 1434, the haptic mode decision module 1436, and / or the robot-assisted mode decision module 1438. For example, the indicator interface module 1442 may cause the indicator 334 to illuminate in a particular color of light associated with a particular state of the system. For example, the indicator interface module 1442 may cause the indicator 334 to illuminate in a first color (e.g., yellow) to indicate that a surgical instrument is not attached to the robot arm and the robot arm may be freely moved so that the system compensates for the mass of the robot arm, in a second color (e.g., purple) to indicate that a surgical tool is attached to the robot arm and the robot arm may be freely moved so that the system compensates for the mass of the robot arm and the mass of a surgical instrument coupled to the robot arm, in a third color (e.g., blue) to indicate that a surgical instrument is attached to the robot arm and the robot arm is in a passive mode as determined by the passive mode determination module 1432, in a fourth color (e.g., pulsed orange) to indicate that at least a portion of the robot arm and / or a surgical instrument attached thereto is within a virtual force sense boundary, e.g., 1.4 m or more above the ground surface, and in a fifth color (e.g., pulsed red) to indicate that an obstacle has been detected by the system by the obstacle detection module 1440. As will be appreciated by one of ordinary skill in the art, different colors and patterns may be communicated by indicator 334 to indicate the status of the system as described above.

[0195] Additionally, the indicators 334 may be illuminated in other distinct colors and / or patterns to communicate additional maneuvers by the robotic arm 300, for example, when the robotic arm 300 retracts the surgical arm in the robotic-assisted mode or performs another robotically-assisted maneuver in the robotic-assisted mode. As described above, the indicators 334 may further include devices for emitting other alerts, such as audible or text alerts. Thus, the indicator interface module 1442 may cause the indicators 334 to communicate the status of the system to the operator using audio or text and light, or in lieu of light.

[0196] The fatigue detection module 1444 may be executed by the processor 1402 to detect user fatigue, which may occur during operation of the robotic arm 300 in a surgical procedure, as described in further detail below with respect to FIG. 25. For example, based on data from, for example, the robotic arm positioning module 1418, the force detection module 1422, and the impedance calculation module 1424, the fatigue detection module 1444 may determine a level of fatigue of an operator using a surgical instrument coupled to the robotic arm 300 and compare the level of fatigue to a predetermined fatigue threshold. For example, the fatigue detection module 1444 may assess an overall score for a given procedure and determine a level of fatigue based on, for example, the operator's hand tremor, the distance / minimum path traveled by the instrument tip, the time to accomplish a procedure step, and / or the time to complete the procedure. Based on the data generated by the fatigue detection module 1444, the impedance calculation module 1422 may determine the amount of impedance required to apply to the robotic arm 300, for example to reduce operator tremors, such that the motor interface module 1426 may cause an associated motor to apply the required impedance to the robotic arm 300. Additionally, based on the data generated by the fatigue detection module 1444, the motor interface module 1426 may cause an associated motor to move a linkage of the robotic arm 300 to guide the operator's manipulation of a surgical instrument attached thereto.

[0197] The collaborative surgical robotic system described herein may include additional modules in the memory 1410 of the platform 200 to perform additional tasks based on the acquired data. For example, the system may determine that a surgical instrument is attached to the robot arm 300 by detecting, via the force detection module 1422, a rapid or sudden change in force (a "snap movement") applied to the robot due to, for example, the attractive force of the magnetic connection between the coupler body and the coupler interface 400. For example, the attractive force of the magnets on the coupler body and the coupler interface 400 may cause a sudden movement on at least the end portion of the robot arm and / or a sudden rotation of the last joint of the robot arm when the magnets are aligned. This sudden movement may thus be detected and trigger the surgical instrument identification module 1412 to determine that an instrument is attached to or detached from the robot arm. Similarly, the surgical instrument identification module 1412 may determine that the surgical instrument has been removed from the robotic arm 300, for example, when subsequent movement of the distal end of the robotic arm 300 involves little or no rotation at the distal-most joint of the robotic arm 300.

[0198] Additionally, the system may determine whether a surgical instrument has been removed from the robot arm 300 based on the position of the distal end of the robot arm 300 relative to the trocar point generated by the trocar position detection module 1420 and data indicating the direction of the instrument shaft and / or the orientation of the most distal linkage of the robot arm 300, e.g., the distal wrist linkage 316. For example, if the instrument is pointing directly at the trocar, there is a higher probability that the tool is attached to the robot arm. Furthermore, axis Q7 of the robot arm 300 may indicate the direction in which the instrument is pointing, and if the instrument is passing through the trocar port, the distal wrist linkage 316 will be pointing in the direction of the trocar port. Thus, if the distal wrist linkage 316 is not pointing toward the trocar port, the system may determine that the robot arm is not supporting an instrument or that an instrument is not being advanced through the trocar port. For example, if an instrument is detached from the robotic arm 300 and the robotic arm 300 is moved, the calculated direction of the instrument shaft (e.g., the direction the instrument would face if attached to the robotic arm 300) may no longer be pointing toward the trocar entry point, and likely will not be pointing toward the trocar entry point. Thus, if the system determines that a tool is no longer coupled to the robotic arm 300, it may alert the user, for example, via indicator 334.

[0199] Additionally, the system may identify when a user may be attempting to remove or uncouple a surgical instrument from the robotic arm 300 and adjust the removal force required to uncouple the surgical instrument, and thus the coupler body, from the coupler interface 400. For example, if one or more magnets are used to provide a biasing force and bias the surgical coupler body against the coupler interface, a force greater than the attractive force provided by the one or more magnets must be imparted on the surgical instrument and / or the coupler body coupled to the surgical instrument in a direction opposite to the force provided by the one or more magnets to overcome the attractive force and uncouple the coupler body and surgical instrument from the coupler interface. For example, the removal force may be 30-60 Newtons.

[0200] Additionally, the system may collect and analyze telemetry data regarding the force being applied to the robotic arm, assess or estimate whether a user is attempting to remove a tool from the robotic arm, and, if applicable, reduce the coupling force between the coupler body and the coupler interface to make it easier for the user to disengage the surgical instrument from the robotic arm. For example, the coupling / removal force may be reduced by 50-80%. Based on historical data and user feedback, and data such as whether a user replaces an instrument without adjusting the instrument's location, which may indicate inadvertent removal of the instrument, the system may estimate an optimal time to reduce the coupling force between the coupler body and the coupler interface. Additionally, the coupling force may be increased to prevent inadvertent removal of the surgical instrument from the robotic arm during operation.

[0201] Additionally, the system may determine an optimal positioning of the robot arm 300 and its joints, a surgical instrument coupled to the robot arm, or other components of the robot arm and / or the system based on data obtained from an optical scanning device used with the system, and provide guidance to an operator of the system to achieve the optimal positioning. Data indicative of the optimal positioning may further be used by the processor 1402, for example, during a setup phase or thereafter, for example, in a robot-assisted mode, to command motors to move corresponding links and joints of the robot arm 300 and automatically reposition the robot arm 300 and / or the optical scanning device to an optimal position.

[0202] Additionally, the system may collect data from sensors, e.g., position data of the robot arm 300 or a surgical instrument attached thereto, via encoders or optical scanning devices, and / or operator position data via body sensors or optical scanning devices, during a procedure, e.g., during setup or operation of the robot arm 300, so that the processor 1402 may detect deviations in the current user's movements or processes compared to the model or optimal movement patterns and communicate the deviations to the current user in real time. For example, the processor 1402 may cause a monitor to display deviations in real time to the current user, and the optimal and / or actual movement patterns. Additionally or alternatively, the indicator interface module 1440 may cause the indicator 334 to indicate deviations from the model or optimal movement patterns, e.g., by illuminating in a specific color and / or pattern. Additionally or alternatively, the motor interface module 1426 may apply impedance perceivable by the operator to the robot arm 30 as haptic feedback, including vibrations, restrictions on movement, or sensations, to indicate deviations from the model or optimal movement patterns. Thus, the system may be used as a training tool for new users, as such data can be used to optimize the position of the surgical device in real time.

[0203] The system may further analyze the depth map generated by the optical scanning device and cluster different groups of (depth) pixels into unique objects, a process referred to as object segmentation. Examples of such algorithms for segmentation may include matching acquired depth map data to known templates of objects for segmentation, using a combination of depth and RGB color images to identify and isolate relevant pixels for objects, and / or machine learning algorithms trained on real or synthetic datasets for objects for identification and segmentation. Examples of such segmentation on a depth map may include locating or determining the position of a robotic arm, identifying a patient port (e.g., a trocar port) and determining the distance from an instrument to the trocar port, identifying a surgeon and distinguishing the surgeon from other operators in the room, and / or identifying a surgeon within the field of view of a sensor. Additionally, the system may use object segmentation algorithms to uniquely identify a surgeon and track the surgeon, for example, relative to a surgical table, a patient, one or more robotic arms, etc. In addition, the system may use an object segmentation algorithm to determine if the surgeon is touching or handling any of the robotic arms, and, if applicable, identify the robotic arms being touched or handled by the surgeon. The system may further use object segmentation to locate the surgical instruments and the distal ends of the robotic arms in 3D space, such that the system may determine if a surgical instrument is attached to the distal end of the robotic arms based on, for example, the proximity between the surgical instrument and the distal end of the robotic arms.

[0204] 15, an operation 1500 of the collaborative surgical robot system described herein is provided. As shown in FIG. 15, in step 1502, an operator may couple a selected surgical instrument to the coupler interface 400 of the robot arm 300 via a coupler body, e.g., coupler body 500, 600, 700. As described above, the operator may select a coupler body that is sized and shaped to couple with the selected surgical instrument based, for example, on the extension shaft diameter of the surgical instrument. Once the surgical instrument and coupler body are ready to be coupled to the robot arm 300, the operator may load a calibration file for the selected surgical instrument, e.g., via the user interface 1408, such that information associated with the selected surgical instrument, e.g., a laparoscope or retractor, is loaded into the system. For example, the operator may select a calibration file from a database of calibration files for various surgical instruments. Calibration files may be stored from previous procedures or may be pre-loaded to include calibration files for commonly used laparoscopic instruments.

[0205] If a calibration file for the selected surgical instrument is not available in the database, the operator may use the system to self-calibrate the surgical instrument. For example, FIG. 16 illustrates a surgical instrument calibration process 1600 for calibrating the surgical instrument and determining the center of mass of the surgical instrument, which may be used, for example, in calculating accurate force measurements on the surgical instrument and the robotic arm 300 during operation. In step 1601, the operator may activate the "Start" option on the user interface 1408. In step 1602, the operator may select "Load Tool Calibration" and begin the calibration process. In step 1603, the system does not apply any impedance to the robotic arm 300 for gravity compensation of the surgical instrument. The system may apply impedance to the robotic arm 300 to account for the weight of the robotic arm 300, for example, to prevent the robotic arm 300 from falling off the ground. In step 1604, the surgical instrument is coupled to the coupler interface 400 of the robotic arm 300 via an appropriately sized coupler body, which can cause the wrist portion 411 of the robotic arm 300 to rotate about axis Q7 and engage the coupler body.

[0206] In step 1605, the system compensates for the force applied by the operator's hand and the force of gravity of the surgical instrument by measuring the force applied to the distal end of the robot arm 300 due to, for example, the mass of the surgical instrument. As explained above, the force applied to the distal end of the robot arm 300 may be measured by measuring the motor current across a motor disposed in the base of the robot arm 300. If the system overcompensates for the force of gravity of the surgical instrument, in step 1606, the robot arm 300 may "runaway", e.g., drift upwards. The runaway effect may be detected in step 1607, and in step 1608, the indicator 334 may flash to indicate the runaway to the operator. In step 1609, the system may identify the runaway as a minor disturbance and therefore apply additional impedance to the robot arm 300 and freeze the robot arm 300 as it decelerates before removing the additional impedance. Once the minor faults have been addressed, the calibration process 1600 may return to step 1603.

[0207] After step 1605, when the system compensates for the surgical instrument's gravity, if the surgical instrument is removed, either accidentally or manually by the operator in step 1611, the system detects the removal of the surgical instrument from the robotic arm 300 in step 1610. As a result, the system may stop compensating for the surgical instrument's gravity and the calibration process 1600 may return to step 1603. After step 1605, when the system has compensated for the surgical instrument's gravity, the calibration process 1600 is ready to enter a calibration mode in step 1612. For example, the operator may initiate the calibration mode via the user interface 1408 in step 1613. In step 1614, the system may indicate to the operator that it is safe to release the surgical instrument, for example, via the user interface 1408 and / or the flashing of the indicator 334, so that the operator may release the surgical instrument in step 1616. In step 1615, the system calibrates the surgical instrument.

[0208] 15, once the surgical instrument and coupler body are ready to be coupled to the robotic arm 300 and the appropriate calibration file has been loaded, the operator may simply place the coupler body adjacent to the coupler interface 400 such that the magnetic connection between the coupler body and the coupler interface 400 will automatically align and couple the surgical instrument to the robotic arm 300. The system will now accurately compensate for the gravity of the selected surgical instrument. In step 1504, the user may use the cooperatively manipulated surgical system by freely manipulating the surgical instrument coupled to the robotic arm 300 in the normal manner that an operator would do without the robotic arm 300 coupled thereto. As shown in Fig. 15, as the operator manipulates the surgical instrument and thus the robotic arm 300 coupled thereto, the system may automatically switch, for example, between a collaborative manipulation mode 1506, a passive mode 1508, a haptic mode 1510, and a robot-assisted mode 1512 (collectively referred to as "operational modes") in response to detection of predefined conditions, as described below with respect to Fig. 17. In some embodiments, the system may automatically switch only between the collaborative manipulation mode 1506, the passive mode 1508, and the haptic mode 1510. In some embodiments, the operator may select an operational mode for configuring the system at step 1504 prior to using the collaboratively manipulated surgical system.

[0209] For example, an operator may apply a particular force on the distal end of the robotic arm 300, e.g., by manipulating a surgical instrument coupled to the robotic arm 300, to indicate that the operator desires to change a particular robotic arm motion mode. Sensors and / or motor current readings may be used to detect the force applied to the distal end of the robotic arm 300 and determine whether the force matches a predefined force signature associated with a motion change, e.g., by comparing the force to one or more predefined force signatures stored within the system. If there is a match, the system may change the motion mode of the robotic arm to the particular motion mode that matches the force signature.

[0210] As described above, during operation of the collaborative surgical system, the system may continuously monitor the robot arm and the forces applied thereto, as described in method 1700 of FIG. 17, to detect a predefined condition that requires switching the operating mode of the system. As shown in FIG. 17, in step 1702, the system continuously collects data related to a first motion characteristic of the robot arm and / or a surgical instrument coupled thereto. For example, as described above, the system may measure motor currents of motors operably coupled to joints of the robot arm and angles of articulations of the robot arm based on measurements by the robot arm's encoders, and calculate in real time the position of the robot arm and the surgical instrument, and forces acting on any part of the robot arm and, if applicable, the surgical instrument. In step 1704, the system may analyze the data related to the first motion characteristic to determine whether a first condition exists. For example, based on the position and force data of the robot arm and / or the surgical instrument, the system may determine whether the movement of the robot arm due to the movement of the surgical instrument coupled thereto is within a predetermined movement threshold of the robot arm for a period of time that is longer than a predetermined dwell time of the robot arm. In response to detecting this first condition, in step 1706, the system may modify a first operating parameter of the robot arm. For example, the system may switch the operating mode of the robot arm to a passive mode in which the robot arm maintains the surgical instrument in a static position.

[0211] For example, a first robotic arm may be coupled to a laparoscope, and an operator may manipulate the laparoscope within the patient, e.g., via a monitor displaying feed images from the laparoscope, until the desired view is provided by the laparoscope. In the collaborative manipulation mode, to freely move the laparoscope coupled to the first robotic arm, the operator must apply sufficient force to the laparoscope that exceeds a predetermined force threshold. The predetermined force threshold should be low enough so as not to require too much force by the operator to freely move the laparoscope. Additionally, the predetermined force threshold may be selected to resist inadvertent movement away from the passive mode. As the operator freely moves the laparoscope in the collaborative manipulation mode, as described above, the system will apply sufficient impedance to the first robotic arm such that the mass or weight of the first robotic arm is not detectable by the operator or is otherwise significantly damped, to compensate for the effects of mass (i.e., inertia) and / or gravity of the first robotic arm and laparoscope during movement. In some embodiments, when an operator couples a laparoscope to the first robotic arm, if the laparoscope is not yet positioned within the patient's body, the system may determine that there are no external forces acting on the surgical instrument and may automatically switch the first robotic arm to a force sense mode to guide the operator in moving the laparoscope to the appropriate location through the trocar port, for example, via a virtual force sense funnel established about the trocar port.

[0212] When the laparoscope is in a desired position relative to the patient and the surgical site within the patient, the system will automatically switch from the collaborative operation mode to the passive mode in response to detecting that the movement of the first robotic arm due to the movement of the surgical instrument is within a predetermined movement threshold for a period of time that exceeds a predetermined dwell time. For example, in response to reaching the desired position, the operator will hold the laparoscope in the desired position for, for example, at least 0.25 seconds. Thus, if the predetermined dwell time is 0.25 seconds, holding the laparoscope in the desired position for a time longer than the predetermined dwell period will cause the system to automatically switch to the passive mode. Furthermore, because it may be impossible for an operator to hold the laparoscope perfectly still, at least some movement of the laparoscope is allowed for the duration of the predetermined dwell time in order to enter the passive mode. As described above, in passive mode, the first robotic arm will hold the laparoscope in a static position, for example, by the system applying sufficient impedance to the first robotic arm to compensate for all external forces acting on the laparoscope.

[0213] Similarly, a second robotic arm may be coupled to the retractor, and the operator may freely manipulate the retractor within the patient in a cooperative manipulation mode, for example, by applying sufficient force to the second robotic arm due to a force applied at the retractor exceeding a predetermined force threshold of the second robotic arm to grasp tissue within the patient and retract the tissue to provide a clear view of the surgical site through a laparoscope coupled to the first robotic arm. As the operator grasps / lifts / retracts tissue with the retractor, the system may only compensate for the gravity of the second robotic arm and / or the instrument, and not that of the tissue being grasped, so that the operator may feel any other forces acting on the retractor, including but not limited to forces acting from the tissue on the instrument. In this optional configuration, the force sensation associated with the tissue being grasped may thus be preserved.

[0214] Once the retractor has sufficiently gripped and retracted the tissue, the system may automatically transition to a passive mode in response to the operator holding the retractor in place with movement not exceeding a predetermined movement threshold of the second robotic arm for a period of time, e.g., exceeding a predetermined dwell period of the second robotic arm. Thus, when the retractor retracts tissue within the patient in the passive mode, the second robotic arm will take into account the mass of the tissue in addition to the mass of the retractor and the second robotic arm. Thus, the predetermined force threshold for switching the second robotic arm out of the passive mode must exceed the force applied to the second robotic arm due to the force applied by the tissue to the tip of the retractor such that if the force applied by the tissue to the surgical instrument exceeds a predetermined first threshold of the second robotic arm, the system will automatically switch the second robotic arm out of the passive mode, e.g., to a collaborative manipulation mode. However, the predetermined force threshold should not be so high that it is too difficult for the operator to move the retractor. As described above, the operator may adjust the predetermined force threshold, for example, via the user interface 1408.

[0215] In response to retraction of tissue through a retractor coupled to the second robotic arm, the operator may need to readjust the view of the laparoscope coupled to the first robotic arm. Thus, the operator may apply a force to the laparoscope that exceeds a predetermined force threshold of the first robotic arm such that the system automatically switches the first robotic arm from a passive mode to a collaborative operation mode. Once a new desired position of the laparoscope is achieved, the first robotic arm may automatically return to the passive mode if the predefined conditions described above are met. Alternatively, the operator may choose to uncouple the laparoscope, readjust the robotic arm and / or laparoscope, and reattach the laparoscope to the first robotic arm (or other robotic arms) to readjust the laparoscope or reposition the joint of the first robotic arm to avoid potential collisions during the laparoscopic procedure, or to switch the laparoscope to a completely different robotic arm. Upon reattachment of the laparoscope to the first robotic arm, the first robotic arm may automatically switch to the passive mode if the predefined conditions described above are met.

[0216] Furthermore, as the operator moves the retractor freely in the collaborative manipulation mode, for example prior to inserting the retractor tip through a trocar in the patient, if the operator moves the retractor tip away from the trocar port very close to the patient's skin and a virtual haptic boundary is established by the system on the patient's skin outside the trocar port, the system may automatically switch to the haptic mode. Thus, the system may apply an impedance to the second robotic arm that is much higher than that applied to the second robotic arm in the collaborative manipulation mode to indicate to the operator that they are approaching or within the virtual haptic boundary. For example, the retractor movement by the operator may feel much more viscoelastic in the haptic mode. The system may remain in the haptic mode until the operator moves the retractor out of the virtual haptic boundary. In some embodiments, in the haptic mode, the second robotic arm may reduce the effects of gravity, eliminate instrument tip tremors, apply force feedback, and avoid critical structures as defined by the virtual haptic boundary. Thus, the system does not replace the operator, but rather augments the operator's capabilities through features such as gravity compensation, tremor cancellation, force barriers, and force feedback.

[0217] In some embodiments, the system may switch the second robotic arm into a robot-assisted mode. For example, as the operator attempts to retract the tissue, if more force is required to retract the tissue than the operator is able or willing to apply to the retractor, the operator may provide user input to the system indicating that the operator desires the second robotic arm to assist in retracting the tissue. For example, as described above, the operator may perform a predefined gesture pattern, which may be detected, for example, by the optical scanner 1100, to cause the system to switch the second robotic arm into a robot-assisted mode and move the motors of the second robotic arm to the second robotic arm, and thus the retractor, to provide the additional force required to retract the tissue.

[0218] Additionally, instead of manually manipulating a laparoscope coupled to the first robotic arm as described, an operator may provide another user input to the system indicating that the operator desires the system to reposition the laparoscope. For example, when an operator actively manipulates surgical scissors, which may or may not be coupled to a robotic arm of the system, such that the tips of the surgical scissors are within the field of view of a laparoscope coupled to the first robotic arm, the operator may use the tips of the surgical scissors to perform a predefined gesture pattern, e.g., rapidly move the surgical scissors back and forth in a particular direction. A predefined gesture pattern of the surgical scissors may be captured as image data by the laparoscope, and based on the data, the system may detect the predefined gesture pattern and associate it with a predefined user input that requests the system to switch the first robotic arm from a passive mode to a robot-assisted mode, causing the first robotic arm to reposition itself, and thus the laparoscope to adjust its field of view in the direction of the pattern movement of the surgical scissors. As described above, additional gesture patterns may be performed within the field of view of the laparoscope via the surgical scissors, causing the first robotic arm to retract the laparoscope and / or zoom in or out or improve resolution on the laparoscope itself. In some embodiments, using object tracking of an additional tool within the laparoscope's field of view, such as surgical scissors that are actively operated by the operator, based on data images captured by the laparoscope, the system may automatically switch a first robotic arm coupled to the laparoscope into a robot-assisted mode, reposition itself, and adjust the field of view to ensure that the tip of the surgical scissors remains within an optimal position within the laparoscope's field of view during the procedure.

[0219] The operating mode of any one of the robotic arms may be changed independently from the operating modes of the other robotic arms of the system. In addition, the operating parameters of each robotic arm may be tailored to the specific surgical instrument coupled thereto. For example, the predetermined force threshold for a robotic arm coupled to a retractor device may be higher than the predetermined force threshold for a robotic arm coupled to a laparoscope because the retractor will withstand higher forces during the procedure. The sensors, motors, etc. of the system may be active in all modes, but may act very differently in each mode, including, for example, acting as if inactive. As will be appreciated by one of ordinary skill in the art, the system may include more than two robotic arms such that an operator may couple a third surgical instrument, e.g., a grasper device, to the third robotic arm and a fourth surgical instrument, e.g., a surgical scissors device, to the fourth robotic arm for operation during a laparoscopic procedure.

[0220] In some embodiments, the operating mode of the robot arm may be changed in response to user input provided by an operator. For example, an operator may selectively change the operating mode of the robot arm by actuating a button, dial, or switch located on the robot arm, a foot pedal or foot switch, a voice command, an input on a touch screen, or using a gesture or force signature as described above. In some embodiments, the operating mode of the robot arm may be changed solely based on the coupling of a surgical instrument to the coupler interface via the coupler body. As described above, the system may automatically identify the surgical instrument based on the coupling of the coupler body to the coupler interface. Thus, based on the identification of the surgical instrument coupled to the robot arm, the system may automatically switch the operating mode of the robot arm to a predetermined operating mode, e.g., a passive mode, if the surgical instrument is an endoscope, or if the robot arm is already in a passive mode, the system will remain in the passive mode upon coupling of the endoscope with the robot arm.

[0221] Similarly, based on the identification of the surgical instrument responsive to its attachment to the robotic arm, the system may automatically switch the operating mode of the robotic arm to a collaborative operation mode, for example, if the surgical instrument identification indicates that the surgical instrument is a tool that will be actively operated by the operator during a laparoscopic procedure. Additionally, based on the identification of the surgical instrument responsive to its attachment to the robotic arm, the system may automatically switch the operating mode of the robotic arm to a robot-assisted mode, for example, if the surgical instrument identification indicates that the surgical instrument is a tool that the operator desires to be fully robotically controlled, such as an irrigation device. Thus, in response to the attachment of an irrigation device to the robotic arm, the system will switch to a robot-assisted mode and have the robotic arm position the irrigation device at a desired location within the body.

[0222] Additionally, the system may be instructed by an operator, for example, via the user interface 1408, to operate the robot arm in fewer than the four operating modes discussed above. For example, the operator may deactivate any one of the operating modes for a given procedure. In some embodiments, the system may operate the robot arm in additional operating modes, such as a locked mode, which may be similar to the passive mode, except that the robot arm's predefined force threshold for switching out of the passive / locked mode may be high enough that the robot arm is effectively frozen to protect the robot arm from inadvertently switching out of the passive / locked mode, e.g., to avoid movement of the robot arm due to an inadvertent bump. In this locked mode, if the force from an inadvertent bump is high enough to cause even slight movement of the robot arm, the system may cause the robot arm to reposition itself to the position it was in prior to the inadvertent bump.

[0223] In addition, when a surgical instrument is not coupled to the distal end of the robot arm of the system, the system is still capable of automatically switching the operation mode of the robot arm in response to the movement of the robot arm by the operator upon detection of the predefined conditions described above. Thus, the system may remain in a static position when the robot arm is in the passive mode, and in the collaborative operation mode, if the system detects that the force applied to the robot arm by the operator exceeds the predetermined force threshold of the robot arm, it will apply impedance to the joints of the robot arm and compensate for the mass of the robot arm so that the robot arm will be allowed to be freely movable by the operator. In addition, the system will switch the robot arm to the haptic mode if the operator attempts to move any part of the robot arm within a predefined virtual haptic barrier.

[0224] In step 1514, once the laparoscopic procedure is complete, the operator may remove the surgical instruments from the individual robotic arms.

[0225] 18A-18C, force measurements during operation of the robotic arm 300 are provided. As described above, upon attachment of a surgical instrument to the coupler interface 400 via a coupler body coupled to the surgical instrument, the orientation of the surgical instrument may be automatically determined based on the magnetic connection between the coupler interface and the coupler body. Additionally, as described above, a calibration file of a surgical instrument coupled to the robotic arm 300 loaded onto the system may include information of the surgical instrument including, for example, the mass of the surgical instrument, the center of mass of the surgical instrument, and the length of the surgical instrument, such that the distance D3 between the center of mass and the instrument tip may be derived. Additionally, as described above, the position of the surgical instrument at the trocar, for example, where the surgical instrument enters the patient's body, may be calculated in real time such that the distance D2 between the center of mass of the surgical instrument and the trocar may be derived in real time. Additionally, as described above, the coupler body is preferably coupled to the surgical instrument at a fixed, known location (which may be included in a calibration file) along the surgical instrument's extension shaft, e.g., adjacent a proximal portion of the surgical instrument, so that a distance D1 between the center of mass of the surgical instrument and the coupler body, e.g., the attachment point to the distal end of the robot arm 300, may be derived. Alternatively, or in addition, an optical scanning device may be used to determine any one of D1, D2, or D3, as described above.

[0226] As shown in FIG. 18A, when a surgical instrument is positioned through a trocar Tr without any additional external forces acting on the surgical instrument other than at the trocar Tr, e.g., the surgical instrument is not lifting or retracting tissue within the patient, the force applied by the body wall to the surgical instrument at the trocar Tr (e.g., the “body wall force” or “trocar force”) may be calculated using the following equation: [ka] In the formula, Feff is the force at the distal end of the robot arm 300 (e.g., the “end effector force” of the robot arm 300), W is the weight vector of the surgical instrument (=-mgz), and F tr is the trocar force. Therefore, F eff is the desired force sent to the system, which is the sum of all forces generated in the algorithm pipeline, including, for example, gravity compensation, hold, etc.

[0227] As shown in FIG. 18B, when a surgical instrument is positioned through the trocar Tr and the retained / retracted tissue such that an external force is applied to the tip of the surgical instrument, two forces for disintegration, namely, F tr and F tt Therefore, two equations are needed to solve for the two unknown vectors, which are related to the center of mass of the surgical instrument, e.g., L cg It may be a balance of forces around the axis and also a balance of moments. [ka]

[0228] where distances D1 and D3 are known as described above, and D2 may be derived based on the known position of the distal end of the robot arm 300 and the calculated position of the trocar Tr. As shown in FIG. 18B, the center of mass L of the surgical instrument cg is behind the attachment point of the coupler body to the distal end of the robot arm 300.

[0229] As explained above, the system is adapted to measure a force, e.g., a force F applied to the tip of the instrument. tt and / or the force F applied by the instrument on the trocar in use. trIf the calculated force exceeds an individual threshold force, the operator may be alerted, and thus if the calculated force exceeds a threshold force, the system may freeze and / or automatically apply a braking or stopping force to the robotic arm 300 to slow or prevent further movement of the instrument in a direction that would increase the force applied at the instrument or trocar tip, and / or by automatically moving the robotic arm in a direction that reduces the force being applied at the instrument tip and / or trocar point at the body wall, thereby reducing the force applied at the trocar point or instrument tip at the body wall.

[0230] 20, a high level example 2000 of different combinations of data inputs for various sensors and devices of the systems disclosed herein, e.g., system 200, and features and capabilities that any implementation of the systems disclosed herein may have and produce based, at least in part, on multiple possible data inputs, is provided. As shown in FIG. 20, some implementations of the system may be configured to collect data from at least three monitoring sources 2002, including telemetry from the system (which may include force data from a robotic arm, position data from a robotic arm, etc.), video from a laparoscopic tower, and / or data from an optical scanner 1100. The data collected from the monitoring sources 2002 may undergo a data processing step 2004 using one or more processors within the system. The data processing step may include, for example, data fusion (e.g., fusion of data collected from the monitoring sources 2002) and data analysis, which may include algorithmic computation. Additionally, data from the monitoring sources 2002 may undergo processing 2004 for the development of system usability features 2006, system safety features 2008, and system performance features 2010. The system may provide features in real-time. For example, system usability features may include identifying the surgeon and adjusting the platform height based on the surgeon's profile, detecting the patient's skin surface and creating a virtual boundary around the skin surface to prevent inadvertent contact with the patient's skin surface, detecting the instrument type and automatically loading the appropriate calibration file for the particular instrument, etc. Additionally, system safety features may include, for example, displaying a virtual map of the area surrounding the platform 100 as the operator moves the platform 100 throughout the operating room, providing the operator with a view of the area surrounding the platform 100 so that the operator may avoid collisions between the platform 100 and any objects and / or persons in the area surrounding the platform 100.

[0231] Referring to FIG. 21, a schematic overview of the electrical components of the electrical system and connectivity 2100 of the system is provided. This includes the flow of energy throughout the illustrated portions of the system, ports that may be used for connectivity, and other details related to the various electronic components. For example, the system may include a non-real-time computer 2102 that may be used to obtain data from an optical scanning device and perform other functions. The non-real-time computer 2102 may also control a graphical user interface of the system for the surgeon to interact with. As explained above, the graphical user interface may include a touch screen. The non-real-time computer 2102 may be, for example, a 10th generation Intel® Core™ processor. TM i7-10700 processor, 32GB RAM (optionally 2x16GB, DDR4, 2933Mhz), standard keyboard and 512GB PCIe M.2 SSD +1TB SATA 7200 RPM hard drive, wireless and Bluetooth cards, e.g. Killer TM Wi-Fi 6 AX1650i (2x2) 802.11ax radio and Bluetooth 5.1, and / or NVIDIA GeForce RTX TM The system may include a 2060 6GB GDDR6 graphics card. The system may further include a real-time computer 2104, which may be used to operate and control the robotic arm and associated robot controller and / or other functions, such as obtaining data and information from an optical scanning device. The real-time computer 2104 may include, for example, an Intel Core i7 (8th generation) processor, 32GB RAM for memory, a 500GB SDD hard drive, and / or two or more RJ45 connectors for Ethernet connectivity.

[0232] 22, a flow chart of a process 2200 for obtaining and processing data from an optical scanning device is provided. As shown in FIG. 22, in step 2202, depth data may be obtained from one or more optical scanning devices, e.g., optical scanner 1100. In step 2204, filtering / other signal processing algorithms may be performed, e.g., median filters, Gaussian noise removal, anti-aliasing algorithms, morphological operations, ambient light adjustment, etc. In step 2206, 3D object segmentation may be performed, e.g., using template matching, machine learning, brute force matching, color+depth segmentation, 2D-3D detection, pixel value thresholding, etc. In step 2208, object coordinates may be transformed into a task space. For example, transforming object coordinates into a task space may include converting the object position and orientation from the coordinate frame of the optical scanning device to the coordinate frame of the required task (e.g., robot frame for robot control, cart frame for system setup, etc.). Additionally or alternatively, transforming object coordinates to task space may include using known optical scanning device to support platform (e.g., cart) transforms, surgical robot transforms, and / or user interface screen transforms, and generating new transforms for specific tasks such as tracking the surgeon's body (e.g., face, hands, etc.) to different elements of the system (e.g., support platform, robotic arms, screen, etc.), tracking the surgical table to the cart platform, tracking patient orientation for system setup, tracking trocar port location and orientation for setup, tracking scrub staff location for safety, etc. In step 2210, the desired task may be performed, e.g., moving the robotic arms to the vicinity of the patient / trocar ports for easy setup, tracking scrub staff to ensure the system responds only to surgeon commands, recording the surgeon's hand movements during different phases of surgery, etc.

[0233] In addition, Figure 22 illustrates a flow chart of a process 2212 for obtaining and processing data from an optical scanning device. In step 2214, depth data may be obtained from one or more optical scanning devices, such as the optical scanner 1100. In step 2216, specular noise filtering may be performed. In step 2218, patient / trocar port segmentation and identification may be performed. In step 2218, the tracked port coordinates may be transformed into the robot coordinate space. In step 2222, the robot arm may be moved to a desired proximity of the patient / trocar port.

[0234]

[00136] Referring now to Figure 23, an example data flow 2300 of the system is provided. As shown in Figure 23, a non-real-time computer 2302 may collect data from an optical scanning device, e.g., optical scanner 1100, and / or from a camera feed from a laparoscope. The non-real-time computer 2302 may also receive data from a real-time computer 2308, which has a robot controller, including telemetry information such as the position of the robot arm, the forces applied to the various motors / sensors of the robot arm, operation mode information, etc. The non-real-time computer 2302 may also receive data from a patient database 2310, which has information specific to the patient in the procedure, including, for example, CT scan data, relevant health conditions, and other information that may be desired by the surgeon.

[0235] The non-real-time computer 2302 may further provide user feedback 2312 to the user via a user interface 2314. User feedback may include, for example, collision notifications, positioning information and / or recommendations regarding various components of the system, operational modes being detected by the system, etc. The non-real-time computer 2302 may further provide commands 2318, e.g., high-level commands, to the real-time computer 2308. High-level commands may include, for example, mode changes, trajectories, force barriers, user configurations, etc. The real-time computer 2308 may include a robot controller 2320 programmed to provide robot commands 2322, e.g., motion or force commands, to one or more robot arms 2324, e.g., robot arm 300. The robot controller 2320 may receive robot feedback data 2326, e.g., motion, force, and / or touch point data, etc., from the one or more robot arms 2324.

[0236] 25, a method 2500 for estimating user fatigue during a surgical procedure using the robotic arm 300 is provided. As described above, algorithms related to gravity compensation, viscoelasticity, and / or mass effects may be used to account for user fatigue. Specifically, during a laparoscopic procedure, a surgeon may suffer fatigue as the procedure progresses and may experience hand tremors or erroneous tool motion with respect to surgical tools such as scissors, needle holders, cautery tools, graspers, etc. As shown in FIG. 25, in step 2502, the system may receive and monitor data indicative of an operator's performance, for example, from an optical scanner 1100, such as a LiDAR camera, robotic telemetry, and / or endoscope, while the operator manipulates a surgical instrument coupled to the robotic arm 300 during the surgical procedure. Learning from a large dataset of clinical procedures and / or collecting and analyzing data during a procedure or part of a procedure may enable the system to infer the level of the surgeon's performance as the procedure progresses, step 2504, and may further enable the system to adapt algorithm parameters to help the surgeon move more effectively while coordinating the surgical instruments attached to the robotic arm. For example, in step 2506, the system may adjust one or more motion parameters of the robotic arm 300 to change its behavior. If the fatigue level is above a particular threshold, in step 2608, the system may alert the surgeon. Additionally, a procedure ranking step may be used to enable the system to provide the surgeon with an overview of their performance for a given procedure and indicate their overall progress per procedure.

[0237] In some embodiments, the system may collect data during the procedure indicative of at least one of the operator's hand tremor, distance / minimum path traveled by the instrument tip, time to accomplish a procedure step, and / or time to complete the procedure, compare such data to a threshold or predefined value for each factor, and determine whether the magnitude of any one of the factors reaches a level sufficient to cause the system to warn the operator and / or adjust one or more operating parameters to reduce user fatigue. For example, the system may eliminate or reduce instrument tip tremor by applying a force on the instrument, increasing the impedance or viscoelasticity of the instrument, avoiding critical structures, and / or applying force feedback. User fatigue may be identified, for example, when procedure time increases beyond a threshold for a particular procedure, when the number of movements of the surgical instrument increases beyond a threshold for a particular procedure or otherwise indicates erroneous or uncontrolled movements, when the operator moves the instrument into a kinematic barrier a predefined number of times, when the operator applies excessive force on the trocar one or a predetermined number of times, etc. As described above, such data may be collected using sensors on the robotic arm and / or one or more optical scanning devices. When a particular level of user fatigue is identified by the system, the system may increase the viscoelasticity or impedance of the instrument and / or the robotic arm associated with the instrument, and reduce the magnitude and / or number of movements of the surgical instrument and / or the robotic arm.

[0238] In addition, the system may use an estimate of the amount of tremor involved in the movement to collect data on the speed and frequency with which the operator moves various instruments / laparoscopes and estimate the additional viscoelasticity required to reduce the tremor without impeding the movement or adding unnecessary fatigue to the operator. In some embodiments, the controller of the robotic arm 300 may iteratively adjust the viscoelasticity value for a particular instrument, collect data related to the instrument's movement, and assess whether additional adjustments are needed to the viscoelasticity applied to the instrument. Furthermore, the system may employ an iterative approach to optimize a particular motion characteristic or parameter of the robotic arm 300 using an additive algorithm that includes the steps of collecting data related to a particular motion characteristic or parameter, varying the motion characteristic or parameter, collecting additional data related to the motion characteristic or parameter, and analyzing the data to determine whether additional changes to the motion characteristic or parameter should be made, which may be based, for example, on deviations between the actual data value of the motion characteristic or parameter and a preferred or optimal value.

[0239] Referring now to FIG. 26, a data flow 2600 of a distributed network of collaborative robotic surgical systems is provided. For example, a distributed network of collaborative robotic ("cobot") surgical systems may be used in multiple hospitals, each of which may be connected to an online database. This arrangement may provide significantly more data and user information that may be used by any of the cobot systems during operation. The system may aggregate data from the distributed network of systems and identify an optimal configuration based on factors such as procedure type, surgeon experience, patient attributes, etc. Through analysis or clinician input, the cobot system may identify routine procedures versus procedures that may be more complex. This information may be used to provide advice or guidance to unskilled surgeons.

[0240] Furthermore, the centralized procedure data may enable the launch of a large amount of data analysis on a wide range of clinical procedures originating from different users. Analysis of the data may result in optimized settings for a specific procedure, including, for example, optimized system positioning, optimal port placement, optimal algorithm settings per robotic arm, and / or detection of procedure anomalies (e.g., excessive force, time, bleeding, etc.). These optimal settings or parameters may depend on the patient and tool characteristics. As explained above, a surgeon may load and use optimal settings from another surgeon or group of surgeons. In this way, optimal settings may be achieved, for example, depending on the surgeon's level of expertise. In order to track different users within a distributed network of the cobot system, it may be beneficial to identify each user. Thus, a user may log into the cobot system and access their profile online, if necessary. In this way, a user may have access to their profile at any location and be able to perform clinical procedures with their settings at different hospital locations.

[0241] An exemplary user profile may contain the user's specific settings and information, including, for example, the user's name, level of expertise, different procedures performed, and / or areas of clinical practice. In addition, clinical procedures may require the user to store specific settings, such as the clinical procedure (e.g., cholecystectomy, hernia, etc.), table orientation and height, preferred port placement, settings per auxiliary arm per algorithm, patient characteristics (e.g., BMI, age, gender), and / or surgical tool characteristics and specifications (e.g., weight, length, center of gravity, etc.). The user may be able to activate his / her own profile, and may optionally activate another user's profile, such as a colleague's profile, a profile most representative of a surgeon in the user's field of practice, a profile most representative of a surgeon with a specific level of expertise, and / or a profile recommended according to patient characteristics.

[0242] Identification of the user may be performed via password, RFID key, facial recognition, etc. Learning from multiple procedures may result in a greater level of optimization of the cobot system settings for a given procedure. This may include, for example, cart position, individual robotic arm positions, surgical table height and orientation, port placement, and / or set joint positions. These settings may be based on patient height, weight, and gender, and may even be interdependent. For example, optimal port placement may depend on patient table orientation.

[0243] Additionally, a clinical procedure may be described as a sequence of clinical procedure steps. Learning these different steps may enable the cobot system to infer the actual steps in real-time for a given procedure. For example, learning the clinical steps from a procedure may enable or enable the system to adjust algorithm settings, to provide hands-on custom reminders, to notify staff of an estimated procedure completion time, to alert staff if required equipment is not available in the room, and / or to alert staff of an emergency situation occurring.

[0244] During clinical procedures, surgeons will often recognize simple and routine surgical tasks such as grasping, retracting, cutting, etc. Learning these different tasks may allow the cobot system to infer in real-time the surgeon's preferences and habits regarding real-time procedural sequences. Several algorithms of the cobot system may be adjusted (i.e., tuned and optimized) based on this sequence recognition during the procedure to help the user better perform this simple surgical task. An example of such a task is automated retraction of the liver during a gallbladder procedure. By aggregating information across many cases, optimized force vectors may be developed.

[0245] Furthermore, several complications may arise during clinical procedures that may result in unexpected steps or surgical actions. Learning how to discern these unexpected events will help the cobot system to enable several specific safety features. In case of an emergency, the robotic arm may be stopped or motion restricted depending on the level of emergency detected by the system.

[0246] 27A-27D, a setup for a collaborative surgical system is provided. Platform 2700 may be constructed similarly to platform 100 such that platform 2700 may support one or more robotic arms, e.g., robotic arm 300a' and robotic arm 300b', and move the robotic arms relative to platform 2700. As shown in FIG. 27A, platform 2700 may be moved by a user, e.g., via wheels 104', to a desired position relative to patient table PT while robotic arms 300a', 300b' are in their individual stored configurations.

[0247] As the platform 2700 is moved towards the patient, the scene may be observed directly by a depth mapping sensor, e.g., the optical scanner 1100', which may be mounted on the platform 2700. From the depth map observed and generated by the optical scanner 1100', important features may be identified, such as, for example, the height and / or location of the patient table PT, the surface of the patient's abdomen, the position and other characteristics of the surgeon, including the surgeon's height, and the trocar ports, the bases of the robotic arms 300a', 300b', e.g., base portions 302a', 302b' and shoulder portions 304a', 304b', the robotic arms 300a', 300b', and / or one or more surgical instruments coupled thereto. Identification of such important features may be performed using standard computer vision techniques, such as template matching, feature tracking, edge detection, and the like. As each feature is registered, its position and orientation may be assigned a local coordinate system and transformed to the global coordinate system of the system using a standard transformation matrix. Once all features are transformed to a single global coordinate system, optimization algorithms, such as least squares and gradient descent, may be used to identify the most appropriate vertical and horizontal positions of the robotic arms 300a', 300b', which may be adjusted to maximize the workspace of the robotic arms relative to an insertion point on the patient via the platform 2700. The optimal workspace may depend on the surgical procedure to be performed and / or the preferred position of the surgeon.

[0248] When the platform 2700 is in its desired position relative to the patient table PT such that the wheels 104' are locked, as shown in Figure 27B, the robotic arms 300a', 300b' may be extended away from their respective stored configurations. The vertical position of the robotic arms relative to the platform 2700 may be adjusted to a desired position, as shown in Figure 27C, and the horizontal position of the robotic arms relative to the platform 2700 may be adjusted to a desired position, as shown in Figure 27D.

[0249] 28A-28D, screenshots of an exemplary graphical user interface 2800 are provided. The exemplary graphical user interface 2800 may be configurable by the user and may be integrated with the display 110. FIG. 28A illustrates an exemplary start menu. An operator may initiate operation of the co-operated system by activating the "Start" option. FIG. 28B illustrates an exemplary system setup screen. As shown in FIG. 28B, when the system includes two robotic arms, the graphical user interface 2800 may identify which robotic arm should be used with which instrument, e.g., the retractor arm 2806 and the endoscope arm 2808, and the procedure to be completed. The graphical user interface 2800 may allow a user to pre-load specific calibration files or set joint positions based on the procedure being performed and / or the surgeon performing the procedure. For example, if a user inputs that a procedure is a laparoscopic cholecystectomy, the system may preload tool types known to be associated with that procedure. Populating these preloaded settings may be accomplished by monitoring the tools that the user manually selects for a given procedure. If a given tool is consistently selected for a predetermined number of procedures, the system may automatically prepopulate that tool the next time a procedure is selected by the user.

[0250] In addition, the operator may adjust the vertical and horizontal position of each robot arm, as shown in FIGS. 27C and 27D above. As shown in FIG. 28B, to adjust the vertical and / or horizontal position of the robot arm that will be or is currently coupled to the retractor device, the operator may toggle the adjustment actuator 2802, and to adjust the vertical and / or horizontal position of the robot arm that will be or is currently coupled to the endoscope device, the operator may toggle the adjustment actuator 2804. In some embodiments, the user may adjust the horizontal and vertical position of the robot arm by using the robot arm as a force-sensing input device. For example, the robot arm may be configured to sense the user's intention by measuring the force applied by the user on the robot arm. If the user applies a force in the positive horizontal direction, the platform may move the robot arm in that direction until the user no longer applies force. Similar approaches are performed for other directions, e.g., negative horizontal, positive vertical, and negative vertical. As shown in FIG. 28B, the graphical user interface 2800 may indicate, via error notification 2810, whether an error, eg, a fault condition, is detected by the system during setup or operation of the system.

[0251] As shown in FIG. 28C, the graphical user interface 2800 may display information associated with a selected surgical instrument, as described above. For example, the graphical user interface 2800 may display, for each instrument to be coupled to each robotic arm, the instrument type, overall length, distance between the coupler body and the instrument tip, distance between the center of mass and the instrument tip, mass, and a preset unlock force required to unlock the instrument. As shown in FIG. 28C, the graphical user interface 2800 may allow an operator to select between a high and low unlock force for a surgical instrument. In addition, the graphical user interface 2800 may allow an operator to initiate a surgical instrument calibration, for example, for a new surgical instrument that does not yet have an associated calibration file stored within the system. FIG. 28D illustrates an example screen during operation of the system, for example, during a surgical procedure. As shown in FIG. 28D, the graphical user interface 2800 may display the trocar force, for example, the force being applied to the tip of the surgical instrument by tissue within the patient's body.

[0252] 29, an alternative cooperative surgical robot system is provided. System 2900 may be constructed similarly to system 200 of FIG. 2. For example, platform 1400′, base portion 302′, shoulder portion 304′, encoders E1′, E2′, E3′, E5′, E6′, E7′, motor M1′, shoulder joint 318′, shoulder linkage 305′, elbow joint 322′, elbow linkage 310′, wrist portion 311′, and coupler interface 400′ for coupling a surgical instrument SI to the robot arm may be constructed similarly to platform 1400, base portion 302, shoulder portion 304, encoders E1, E2, E3, E5, E6, E7, motor M1, shoulder joint 318, shoulder linkage 305, elbow joint 322, elbow linkage 310, wrist portion 311, and coupler interface 400, respectively. System 2900 differs from system 200 in that system 2900 includes motors disposed at the joints of the robotic arm. For example, system 2900 may include motor M2' disposed at elbow joint 318' and motor M3' disposed at elbow joint 322' configured to rotate associated links and manipulate the robotic arm. Additionally, encoder E4' may be positioned on or adjacent to elbow joint 322'.

[0253] Some implementations of the systems described herein may be configured to be controlled or operated remotely, for example, via a joystick or other suitable remote control device, computer vision algorithms, force measurement algorithms, and / or by other means. However, in preferred embodiments, the systems described herein operate without any telemetry, e.g., the robotic arm is not remotely operated via a remote surgeon console separate from the robotic arm, instead the robotic arm moves in response to movements applied to a surgical instrument coupled thereto. For example, in a robotically assisted mode, any robotically assisted movements applied to the surgical instrument by the system are not responsive to user input received at the remote surgeon console.

[0254] FIG. 30A illustrates a top view of a coupler 3000 for coupling a surgical instrument SI to a robotic arm, showing a coupler body 3002 (also referred to herein as a body) coupled with a coupler interface 3001 (also referred to herein as an interface). FIG. 2B illustrates a top view of the coupler 3000 of FIG. 30A, showing the coupler body 3002 decoupled from the coupler interface 3001. As shown in FIGS. 30A and 30B, the coupler 3000 may have a coupler body 3002 and a coupler interface 3001. The coupler interface 3001 may be coupled with the robotic arm 300 and may be configured such that the coupler body 3002 may be removably coupled with the coupler interface 3001. The coupler body 150 may be coupled with the surgical instrument SI at any desired axial location on the surgical instrument SI. Once the coupler body 3002 is coupled with a surgical instrument SI, the coupler body 3002 and the surgical instrument SI coupled with the coupler body 3002 may be coupled with the coupler interface 3001. The coupler body 3002 may be configured such that once the coupler body 3002 is coupled with a surgical instrument SI, the surgical instrument SI may be at least blocked (e.g., prevented) from moving axially relative to the coupler body 3002, or in some embodiments, from moving axially and rotationally. The coupler 3000 may be configured such that the coupler body 3002 is at least blocked (e.g., prevented) from moving in any axial direction relative to the coupler interface 3001. In some embodiments, the coupler 3000 may be configured such that the coupler body 3002 is freely rotatable relative to the coupler interface 3001. In this configuration, the surgical instrument SI coupled to the coupler body 3002 may rotate freely relative to the coupler interface 3001 to which the coupler body 3002 is coupled, and may at least be hindered (e.g., prevented) from any axial movement relative to the coupler interface 3001 to which the coupler body 3002 is coupled.

[0255] In other embodiments, the coupler 300 may be configured such that the surgical instrument SI may be moved axially relative to the coupler body 3002 in response to application of at least a threshold force on the surgical instrument SI against the coupler body 3002, or in response to actuation of the release or state change of the coupler body 3002. Such actuation may be accomplished in some embodiments, for example, by depressing a button, loosening a set screw or other connector, moving a dial, or otherwise changing the coupler 3000, coupler body 3002, and / or coupler interface 3001 from a second secured state to a first unsecured state. For example, in some embodiments, the surgical instrument SI may be repositioned axially relative to the coupler 3000 by loosening one or more thumbscrews 3010 or other manual fasteners or fastening mechanisms, such as clamps, in the coupler body 3002, repositioning the surgical instrument SI to a desired axial position, and retightening the thumbscrews 3010 or other manual fasteners or fastening mechanisms.

[0256] As shown in FIG. 30B, the coupler interface 3001 may have a recess 3003 sized and shaped to receive the coupler body 3002. The recess 3003 may block (e.g., prevent) axial, or in some embodiments, axial and rotational, movement of the coupler body 3002 relative to the coupler interface 3001, while allowing free rotational movement of the coupler body 3002 relative to the coupler interface 3001. The coupler 3000 may be configured such that the surgical instrument SI may at least be blocked (e.g., prevented) from rotational movement relative to the coupler 3000. This may be accomplished by at least blocking (e.g., preventing) rotational movement between the surgical instrument SI and the coupler 3000 or between the coupler body 3002 and the coupler interface 3001. In some embodiments, a surgical drape may be sandwiched or crimped between the coupler body 3002 and the coupler interface 3001.

[0257] FIG. 30C illustrates an end view of the coupler body 3002 and surgical instrument SI showing the coupler body 3002 in a first unsecured or open state in which the surgical instrument SI may be removed and replaced or repositioned relative to the coupler body 3002. FIG. 30D illustrates an end view of the coupler body 3002 of FIG. 30C showing the coupler body 3002 in a second secured or closed state in which the surgical instrument SI may be blocked (e.g., prevented) from at least axial movement, or in some embodiments, axial and rotational movement, relative to the coupler body 3002. In some embodiments, the coupler body 3002 may have a first portion 3004 and a second portion 3006. In some embodiments, the first portion 3004 may be rigidly coupled to the second portion 3006 via a hinge 3005 or shaft, or otherwise. In some embodiments, the first and second portions 3004, 3006 may have a semicircular cutout or recess 3008 therein that is sized and shaped to receive a surgical instrument SI therein. A fastener 3010 may be used to couple the first portion 3004 and the second portion 3006, such as when the surgical instrument SI is positioned within the recess 3008, as shown in FIG. 30D. As described above, the coupler body 3002 may be configured to at least substantially impede (e.g., prevent) axial, or in some embodiments, axial and rotational, movement of the surgical instrument SI relative to the coupler body 3002. Rubber pads, sheets, bumps, O-rings, protrusions, or other components or features configured to grip the exterior of the surgical instrument SI may be used in conjunction with any of the coupler embodiments disclosed herein. For example, the rubber interface may be positioned within a recess or recesses in the coupler body, such as recess 3008 in the first portion 3004 and / or second portion 3006 of the coupler body 3002, and may be bonded to the coupler body 3002. The rubber may be silicone rubber or any other suitable type of rubber.

[0258] 31A-31D illustrate another embodiment of a coupler 3100 that may be used with any robotic system embodiment disclosed herein to couple an instrument to an end portion of a robotic arm. The coupler 3100 may include a coupler body 3101 and a coupler interface 3120 that may have a recess or depression 3190 configured to receive the coupler body 3101 therein. The coupler interface 3120 may be coupled to the end portion of the robotic arm 300. The coupler 3100 may have a coupler body 3101 that removably or non-removably couples directly to the end portion of the robotic arm 300.

[0259] 31A, the coupler body 3101 may have a cylindrical body portion 3102 having an annular flange 3104 that projects away from a surface of the cylindrical body portion 3102. The body portion 3102 may have an opening 3106 that extends axially through the body portion 3102. The opening 3106 may be sized and shaped to receive a surgical instrument SI therein. The opening 3106 may be slightly larger than the diameter or outer size of the surgical instrument SI. The coupler body 3101 may have one or more deflectable tabs 3108 (two shown) or four or more deflectable tabs 3108 that may be configured to deflect radially inward such that when the tabs 3108 are deflected radially inward, the tabs 3108 impart a force on an outer surface of the surgical instrument SI. The coupler 3100 may be configured such that when the coupler body 3101 is positioned within the recess 3109 of the coupler interface 3120 and the coupler interface 3120 is in the second closed or secured state, the coupler interface 3120 may grip the surgical instrument SI and apply a force to or otherwise bias it radially inwardly to inhibit (e.g., prevent) at least axial or axial and rotational movement of the surgical instrument SI relative to the coupler body 3101. For example, the tabs 3108 may have a greater thickness near the distal end 3110 of the tabs 3108 such that in the relaxed or first open state, the distal end 3110 of the tabs 3108 may protrude or bulge away from an outer surface of the body portion 3102 of the coupler body 3101. In this configuration, when the coupler body 3101 is positioned within the recess 3109 of the coupler interface 3120, moving the coupler interface 3120 to the second closed state can cause a force to be applied to the distal end portion 3110 of the tab 3108, thereby deflecting the tab 3108 inwardly against the outer surface of the surgical instrument SI.

[0260] In some embodiments, the recess 3109 may have an enlarged portion 3111 sized and shaped to receive the annular flange 3104 therein and allow rotational movement of the flange 3104 while also limiting or at least inhibiting (e.g., preventing) axial movement of the coupler body 3101 by providing an axial limit to the movement of the annular flange 3104. In this arrangement, the surgical instrument SI may be advanced axially through the opening 3106 of the coupler body 3101 to any desired location. The surgical instrument SI may then be positioned within the recess 3109 of the coupler interface 3120 with the coupler body 3101 coupled thereto. The coupler interface 3120 may be removably or non-removably coupled to an end portion of the robotic arm 300 of any of the cooperatively manipulated surgical systems disclosed herein.

[0261] 31C, rubber pads, sheets, bumps, O-rings, protrusions, or other gripping features 3112 (O-rings shown) configured to grip the exterior of the surgical instrument SI may be positioned within the coupler body 3101 to increase frictional forces between the surgical instrument SI and the coupler body 3101. In some embodiments, the one or more tabs 3108 may be configured to impart a force on the gripping features 3112 when the one or more tabs 3108 are deflected inwardly.

[0262] As shown in FIG. 31D, the coupler interface 3120 may have a first portion 3105 that may be coupled to a second portion 3103. In some embodiments, the first and second portions 3105, 3103 may be rigid and may be coupled to one another via a mechanical hinge 3107. Alternatively, a living hinge, a shaft, one or more fasteners, or other components or features may be used to couple the first and second portions 3105, 3103 together. In some embodiments, the second portion 3103 may be flexible, such as an elastically stretchable or elastically rigid strap, and may be configured to extend over the surgical instrument SI and / or coupler body 3101 supported within the recess 3109. Additional fasteners, clamps, clasps, or other components or features may be used in conjunction with or in place of the hinge 3107 to fixedly couple the first and second portions 3105, 3103 together once the coupler body 3101 is received within the recess 3109 of the coupler interface 3120 to fixedly couple the surgical instrument SI and the coupler 3100.

[0263] In some embodiments, the coupler may include a coupler body and a coupler interface having a recess configured to receive the coupler body. The coupler body may have an opening extending axially therethrough configured to receive an instrument and an annular flange extending around its outer surface. The recess in the coupler interface may have an enlarged portion configured to receive the annular flange and allow rotational movement of the flange while at least inhibiting (e.g., preventing) axial movement of the coupler body by providing an axial limit to movement of the annular flange. The coupler interface may be configured to mate with an end portion of a robotic arm.

[0264] 32A and 32B illustrate a coupler body 3200 that may be used with any of the robotic system embodiments disclosed herein to couple an instrument to an end portion of a robot arm. The coupler body 3200 may have any of the components, features, and / or other details of any of the other embodiments of the coupler body disclosed herein in any combination with any of the components, features, and / or other details of the embodiment of the coupler body 3200 shown in FIGs. 32A and 32B. Any of the other embodiments of the coupler body disclosed herein may have any of the components, features, and / or other details of the coupler body 3200 in any combination with any of the components, features, and / or other details of the other coupler body embodiments disclosed herein.

[0265] The coupler body 3200 may have an opening 3202 axially therethrough that is sized and shaped to receive a surgical instrument therein, and a clamping mechanism 3204 configured to reduce an inner diameter of the opening 3202 as the clamping mechanism 3204 is actuated to move the coupler body 3200 from a first unsecured or open state, as shown in FIG. 32A, to a second secured or closed state, as shown in FIG. 32B. In this arrangement, the coupler body 3200 may be positioned around an outer surface of a surgical instrument while the coupler body 3200 is in the first open or unsecured state. The clamping mechanism 3204 may then be actuated to clamp the coupler body 3200 itself to the outer surface of the surgical instrument. The coupler body 3200 may then be coupled with a coupler interface that is sized and configured to receive and support the coupler body 3200.

[0266] 33A and 33B illustrate a coupler body 3300 that may be used with any of the robotic system embodiments disclosed herein to couple an instrument to an end portion of a robot arm. The coupler body 3300 may have any of the components, features, and / or other details of any of the other embodiments of the coupler body disclosed herein in any combination with any of the components, features, and / or other details of the embodiments of the coupler body 3300. Any of the other embodiments of the coupler body disclosed herein may have any of the components, features, and / or other details of the coupler body 3300 in any combination with any of the components, features, and / or other details of the other coupler body embodiments disclosed herein.

[0267] The coupler body 3300 may have an opening 3302 axially therethrough that is sized and shaped to receive a surgical instrument therethrough, and a clamping mechanism 3304 having first and second handle members or tabs configured to reduce an inner diameter of the opening 3302 as the clamping mechanism 3304 is actuated to move the coupler body 3300 from a first unsecured or open state as shown in FIG. 33A to a second secured or closed state as shown in FIG. 33B. In this arrangement, the coupler body 3300 may be positioned around an outer surface of a surgical instrument while the coupler body 3300 is in the first open or unsecured state. The coupler body 3300 may be moved to the first open or unsecured state by squeezing or moving the handles of the clamping mechanism 3204 together as shown in FIG. 33A. The clamping mechanism 3204 may then be released to secure the coupler body 3300 itself to the exterior surface of the surgical instrument. The coupler body 3300 may then be coupled with a coupler interface that is sized and configured to receive and support the coupler body 3300.

[0268] 34A-34C illustrate a coupler 3400 that may be used with any of the robotic system embodiments disclosed herein to couple an instrument to an end portion of a robotic arm. The coupler 3400 may have any of the components, features, and / or other details of any of the other coupler embodiments disclosed herein in any combination with any of the components, features, and / or other details of the embodiments of the coupler 3400. Any of the other coupler embodiments disclosed herein may have any of the components, features, and / or other details of the coupler 3400 in any combination with any of the components, features, and / or other details of the other coupler embodiments disclosed herein.

[0269] The coupler 3400 may have one or more coupler bodies 3402 (two shown) coupled with a coupler interface 3404. The coupler body 3402 may be slidably received within an opening 3406 in the coupler interface 3404. The coupler interface 3404 may have a recess 3408 extending along its length that may be configured to receive the outer surface of the surgical instrument SI therein and may have a semicircular or other cross-sectional shape that matches the shape of the outer surface of the surgical instrument. The coupler body 3402 may have a curved end portion 3410 that is sized and shaped, at least in part, to be routed or curved around the outer surface of the surgical instrument SI. In this configuration, the coupler body 3402 may be used to selectively secure a surgical instrument SI within the recess 3408 or otherwise secure the surgical instrument SI to the coupler interface 3404 when in the second secured or closed position as shown in FIG. 34A. A spring or other biasing mechanism 3412 may be used to bias the coupler body 3402 to the second closed or secured position as shown in FIG. 34A. A user may push the coupler body 3402 in the axial direction indicated by arrow A1 to move the coupler body 3402 from the second closed or secured position to the first open or unsecured position. The force exerted on the coupler body 3402 should exceed the spring or biasing force from the spring or biasing mechanism 3412 coupled with each of the coupler bodies 3402.

[0270] 34B , the coupler body 3402 may have a sloped end surface 3414. The sloped end surface 3414 may be configured such that as a surgical instrument SI is advanced laterally toward the recess 3408 in the coupler interface 3404, an outer surface of the surgical instrument SI may contact the coupler body end surface 3414 and the space between the coupler end surface 3414 and an adjacent surface of the coupler interface 3404 is greater at locations of the coupler end surface 3414 further away from the recess such that the slope of the end surface 3414 of the coupler body 3400 will move the coupler body 3400 from the second closed or locked state toward the first open or unlocked state, allowing the surgical instrument SI to be received within the recess 3408. The coupler body 3400 may have a spring or other biasing mechanism 3416 configured to bias the coupler body 3400 into the second closed or secured state or position.

[0271] As shown in FIG. 34C , the sloped end surface 3414 of any embodiment of the coupler body 3402 may be sloped such that as the surgical instrument SI is advanced in a downward direction relative to the end surface 3418 of the coupler body 3400, such interaction between the outer surface of the surgical instrument SI and the sloped surface 3418 of the coupler body 3402 may rotate the coupler body 3400 about the pivot point 3420 away from the recess 3408, allowing the surgical instrument SI to be received within the recess 3408.

[0272] 35A-35D illustrate a coupler 3500 that may be used with any of the robotic system embodiments disclosed herein to couple an instrument to an end portion of a robotic arm. The coupler 3500 may have any of the components, features, and / or other details of any of the other coupler embodiments disclosed herein in any combination with any of the components, features, and / or other details of the embodiments of the coupler 3500. Any of the other coupler embodiments disclosed herein may have any of the components, features, and / or other details of the coupler 3500 in any combination with any of the components, features, and / or other details of the other coupler embodiments disclosed herein.

[0273] The coupler 3500 may have a coupler body 3502 that may be coupled or engaged with the coupler interface 3504. For example, the coupler body 3502 may be slidably received within a recess 3506 formed in the coupler interface 3504. The coupler body 3502 may also have a recess 3505, which may have a semicircular cross-sectional shape or other cross-sectional shape that matches the shape of the outer surface of the surgical instrument, extending along the length of the coupler body 3502, which may be configured to receive and at least partially surround, or in some embodiments, completely surround, the outer surface of the surgical instrument SI when the coupler 3500 is in a second state as shown in FIG.

[0274] The coupler body 3502 may be made from a flexible material such as rubber, including neoprene. The coupler body 3502 may have a width greater than the width of the recess and may be biased toward a flat or generally flat shape, as shown in FIG. 35A. The coupler body 3502 may be sufficiently flexible such that when the coupler body 3502 is urged toward the distal surface 3506a of the recess 3506, the coupler body 3502 bends or folds about a central portion or other portion adjacent the recess 3505. Once the coupler body 3502 is fully advanced into the recess 3506 of the coupler interface 3504, the coupler 3500 may be configured to bias the coupler body 3502 to remain in a second anchoring position within the recess 3506. In this configuration, to secure the surgical instrument SI within the coupler 3500, an operator can advance the surgical instrument SI into the recess 3505 of the coupler body 3502 and continue to advance the surgical instrument SI and / or the coupler body 3502 toward the distal surface 3506a. Some embodiments of the coupler 3500 may be configured such that once the coupler body 3502 and the surgical instrument SI are advanced into the recess 3506 of the coupler interface 3504, the surgical instrument SI will be axially and / or rotatably secured to the coupler 3500. The coupler 3500 may then be coupled to an end portion of the robotic arm 300 such that the robotic arm 300 may be coupled with the surgical instrument SI. In any embodiment, the recess may have a sloped, curved, or otherwise tapered leading edge surface 3507 that leads into the recess to facilitate advancement into the recess 3506 of the coupler interface 3504 of the coupler body 3502.

[0275] As shown in FIG. 35E, a surgical drape 800 may be positioned between the surgical instrument SI and the coupler body 3302. In other embodiments, the surgical drape 800 may be integrated into the coupler body 3052 such that the coupler body 3502 may form part of the surgical drape, as shown in FIG. 35F. The coupler body 3502 may be sufficiently flexible to return to its original shape once the coupler body 3502 is removed from the recess 3506. In any embodiment disclosed herein, the coupler body or other components or features of the coupler can be configured to radially constrain the instrument.

[0276] 35C, the coupler 3500 may be configured to have a protrusion 3503 configured to extend into the recess 3506 of the coupler interface 3504 even when the coupler body 3502 is in a first open or unlocked state as shown in FIG. 35C. The protrusion 3503 may help bias the coupler body 3502 to remain engaged with the recess 3506 of the coupler interface 3504 even when the coupler body 3502 is in a first open or unlocked state. As shown in FIG. 35D, the coupler body 3502 may also have a protrusion, flange, handle, tab, or other protrusion 3509 at its proximal end portion configured to facilitate grasping and removal of the coupler body 3502 from the recess 3506.

[0277] In some embodiments, the coupler may include a coupler body made of a flexible material and a coupler interface having a recess configured to receive the coupler body. The coupler body may have a recess having a curved profile along the length of its first major surface configured to receive an instrument therein. The coupler body may be sufficiently flexible such that when the coupler body is urged toward a distal surface of the recess, the coupler body will fold about a portion of it adjacent the recess, thereby constraining the instrument at least axially and radially. The coupler body may be sufficiently flexible to return to the original shape of the coupler body 3502 once the coupler body is removed from the recess.

[0278] 36 illustrates a coupler 3600 that may be used with any of the robotic system embodiments disclosed herein to couple an instrument to an end portion of a robotic arm. The coupler 3600 may have any of the components, features, and / or other details of any of the other coupler embodiments disclosed herein in any combination with any of the components, features, and / or other details of the embodiments of the coupler 3600. Any of the other coupler embodiments disclosed herein may have any of the components, features, and / or other details of the coupler 3600 in any combination with any of the components, features, and / or other details of the other coupler embodiments disclosed herein.

[0279] The coupler 3600 may have a coupler body 3602 that may be coupled or engaged with the coupler interface 3604. For example, the coupler body 3602 may be received within a recess 3606 formed in the coupler interface 3604. The coupler body 3602 may also have a recess 3615, which may have a semicircular cross-sectional shape or other cross-sectional shape that matches the shape of the outer surface of the surgical instrument, extending along the length of the coupler body 3602, which may be configured to receive and at least partially surround, or in some embodiments, completely surround, the outer surface of the surgical instrument SI when the coupler 3600 is in the second state.

[0280] The coupler body 3202 may be made from a flexible material such as rubber, including neoprene. Other embodiments of the coupler body 3202 may be made from multiple materials including a first layer 3608 made from a flexible material, such as rubber, which may have increased grip, and a second layer 3610, which may be a substrate or support layer for the first layer 3608, which may be made from a more rigid material, such as plastic, metal, or other. A recess 3615 may be formed in the first layer 3608. The recess 3615 may be formed in a central portion of the first layer 3608. Some embodiments of the second layer 3610 may have a hinge 3612 at or attached to its central portion. In some embodiments, the hinge 3612 may run generally parallel to the recess 3615 formed in the first layer 3608 and the recess 3606 formed in the coupler interface 3604. In some embodiments, the coupler body 3602 may be folded or hinged about the surgical instrument SI between a first open state and a second closed or secured state by folding or hinged about the hinge 3612.

[0281] The coupler body 3600 may have a width that exceeds the width of the recess 3606. The coupler body 3602 may be configured such that when the coupler body 3602 is urged toward the distal surface 3606a of the recess 3606, the coupler body 3602 will bend or fold about the hinge 3612 to collapse or close about a surgical instrument SI positioned within the recess 3615 of the coupler body 3602 to secure the surgical instrument SI within the coupler body 3602 and the coupler interface 3604.

[0282] Some embodiments of the coupler interface 3604 may have one or more rollers 3614 (two shown) at a proximal end 3606b of the recess 3606 formed in the coupler interface 3604. The one or more rollers 3614 may facilitate movement of the coupler body 3602 into the recess 3606 by allowing the coupler body 3602 to roll on the rollers as the coupler body 3602 is advanced into the recess 3606. Some embodiments of the coupler interface 3604 may have additional rollers 3616 along a sidewall surface 3606c of the recess 3606 to continue to facilitate advancement of the coupler body 3602 into the recess 3606. In some embodiments, the recess 3606 may have a generally rectangular shape. In other embodiments, the recess 3606 may have a tapered or narrowed profile.

[0283] Once the coupler body 3602 is fully advanced into the recess 3606 of the coupler interface 3604, some embodiments of the coupler 3600 may be configured to bias the coupler body 3602 to remain in a second anchoring position within the recess 3606. In this configuration, to anchor the surgical instrument SI within the coupler 3600, an operator may advance the surgical instrument SI into the recess 3615 of the coupler body 3602 and continue to advance the surgical instrument SI and / or the coupler body 3602 toward the distal surface 3606a of the recess 3606. Some embodiments of the coupler 3600 may be configured such that once the coupler body 3602 and the surgical instrument SI are advanced into the recess 3606 of the coupler interface 3604, the surgical instrument SI will be axially and / or rotationally anchored to the coupler 3600. The coupler 3600 may then be coupled to an end portion of the robotic arm 300 such that the robotic arm 300 may be coupled to a surgical instrument SI.

[0284] 37 illustrates a coupler 3700 that may be used with any of the robotic system embodiments disclosed herein to couple an instrument to an end portion of a robotic arm. The coupler 3700 may have any of the components, features, and / or other details of any of the other coupler embodiments disclosed herein in any combination with any of the components, features, and / or other details of the embodiments of the coupler 3700. Any of the other coupler embodiments disclosed herein may have any of the components, features, and / or other details of the coupler 3700 in any combination with any of the components, features, and / or other details of the other coupler embodiments disclosed herein.

[0285] The coupler 3700 may have a coupler body 3702 that may be coupled or engaged with a coupler interface 3704. The coupler body 3702 may be received within a recess 3796 formed in the coupler interface 3704. The coupler body 3702 may also have a recess 3705, which may have a semicircular cross-sectional shape or other cross-sectional shape that matches the shape of the outer surface of the surgical instrument, extending along the length of the coupler body 3702, which may be configured to receive and at least partially surround, or in some embodiments, completely surround, the outer surface of the surgical instrument SI when at least the coupler 3704 is in a second state as shown in FIG.

[0286] The coupler body 3702 may be made from multiple materials, including a first layer 3710 made from a flexible material, such as rubber, which may have increased grip, and a second layer 3712, which may be a substrate or support layer for the first layer 3710, which may be made from a more rigid material, such as plastic, metal, or other. A recess 3705 may be formed in the first layer 3710. In some embodiments, the recess 3705 may be formed in a central portion of the first layer 3710. Some embodiments of the second layer 3712 may have a hinge 3714 at or attached to its central portion. In some embodiments, the hinge 3714 may run generally parallel to the recess 3705 formed in the first layer 3710 and the recess 3706 formed in the coupler interface 3704. In some embodiments, the coupler body 3702 may be folded or hinged about the surgical instrument SI between a first open state and a second closed or secured state by folding or hinged about the hinge 3714.

[0287] The coupler body 3702 may have a width that exceeds the width of the recess 3706. The coupler body 3702 may be configured such that when the coupler body 3702 is urged toward the distal surface 3706a of the recess 3706, the coupler body 3702 will bend or fold about the hinge 3714 to collapse or close about a surgical instrument SI positioned within the recess 3705 of the coupler body 3702 to secure the surgical instrument SI within the coupler body 3702 and the coupler interface 3704. In some embodiments, the second layer 3712 may have wings or tabs 3716 that may be used to facilitate removal of the coupler body 3702 from the recess 3706. The tab 3716 may be formed such that when the coupler body 3702 is in the second position as shown in FIG. 37, the tab 3716 may be spaced apart from the first surface 3704a (which may be the upper surface when the coupler interface 3704 is positioned as shown in FIG. 37) such that a gap or space 3720 exists between the tab 3716 and the upper surface 3704a of the coupler interface 3704. The space 3720 may be large enough to allow the tab 3716 to move towards the first surface 3704a when a force is applied to the tab 3716 in the direction of the first surface 3704a. As the tab 3716 is deflected toward the first surface 3704a, such movement of the tab 3716 may urge the remainder of the coupler body 3702 to move away from the distal surface 3706a of the recess 3706, thereby enabling the coupler body 3704 to be removed from the recess 3706.

[0288] Some embodiments of the coupler interface 3704 may have one or more rollers 3717 (two shown) at a proximal end 3706b of the recess 3706 formed in the coupler interface 3704. The one or more rollers 3717 may facilitate movement of the coupler body 3702 into the recess 3706 by allowing the coupler body 3702 to roll on the rollers as the coupler body 3702 is advanced into the recess 3706. Some embodiments of the coupler interface 3704 may have additional rollers 3718 along a sidewall surface 3706c of the recess 3706 to continue to facilitate advancement of the coupler body 3702 into the recess 3706.

[0289] Once the coupler body 3702 is fully advanced into the recess 3706 of the coupler interface 3704, some embodiments of the coupler 3700 may be configured to bias the coupler body 3702 to remain in a second anchoring position within the recess 3706. In this configuration, to anchor the surgical instrument SI within the coupler 3700, an operator may advance the surgical instrument SI into the recess 3705 of the coupler body 3703 and continue to advance the surgical instrument SI and / or the coupler body 3702 toward the distal surface 3706a of the recess 3706. Some embodiments of the coupler 3700 may be configured such that once the coupler body 3702 and the surgical instrument SI are advanced into the recess 3706 of the coupler interface 3704, the surgical instrument SI will be axially and / or rotationally anchored to the coupler 3700. The coupler 3700 may then be coupled to an end portion of the robotic arm 300 such that the robotic arm 300 may be coupled to a surgical instrument SI.

[0290] 38A and 38B illustrate a coupler 3800 that may be used with any of the robotic system embodiments disclosed herein to couple an instrument to an end portion of a robotic arm. The coupler 3800 may have any of the components, features, and / or other details of any of the other coupler embodiments disclosed herein in any combination with any of the components, features, and / or other details of the embodiments of the coupler 3800. Any of the other coupler embodiments disclosed herein may have any of the components, features, and / or other details of the coupler 3800 in any combination with any of the components, features, and / or other details of the other coupler embodiments disclosed herein.

[0291] The coupler 3800 may have a coupler body 3802 that may be coupled or engaged with a coupler interface (not shown) or may be coupled or engaged with a robot arm without the presence of a coupler interface (e.g., the coupler body of any embodiment disclosed herein may directly engage or interface with an end portion of the robot arm 300). The coupler body 3802 may have a first portion 3804 and a second portion 3806 that is coupled with the first portion 3804. In some embodiments, the first portion 3804 may be hinged or rotatably coupled with the second portion 3806. For example, the coupler body 3802 may have a hinge or joint 3810 that may couple the first and second portions 3804, 3806 together.

[0292] In some embodiments, the first portion 3804 of the coupler body 3802 may have a proximal portion 3804a and a distal portion 3804b that is integrally formed with or coupled to the proximal portion 3804a. The first portion 3804 of the coupler body 3802 may have a recess 3812 and the second portion 3806 of the coupler body 3820 may have a recess 3814, each of which may have a semicircular cross-sectional shape or other cross-sectional shape that matches the shape of the outer surface of a surgical instrument extending along the length of the coupler body 3802, which may be configured to receive and at least partially surround, or in some embodiments, completely surround, the outer surface of the surgical instrumen...

Claims

Claim 1 A cooperative surgical system for assisting laparoscopic surgery performed using a surgical instrument, the surgical instrument having a handle, an operating end, and an elongate shaft therebetween, the cooperative surgical system comprising: A robotic arm, the robotic arm comprising a proximal end, a distal end configured to be removably coupled to the surgical instrument, a plurality of linkages, and a plurality of joints between the proximal end and the distal end; A base operably coupled to the proximal end of the robotic arm; A plurality of motors disposed within the base, the plurality of motors being operably coupled to at least some of the plurality of joints; A controller operably coupled to the plurality of motors and configured to enable the robotic arm to move freely relative to the base in response to movement at the handle of the surgical instrument for performing laparoscopic surgery using the surgical instrument, the controller being configured to: Measure the position of the distal end of the robotic arm; Determine an entry point of the surgical instrument into a patient by determining intersections of a plurality of virtual lines parallel to a longitudinal axis of the surgical instrument as the position of the distal end of the robotic arm moves relative to an entry point; Calculate a compensation force required to compensate for the weight of the surgical instrument based on the position of the distal end of the robotic arm, the entry point, and one or more instrument parameters stored in a memory of the controller; During operation of the cooperative surgical system, apply a torque based on the compensation force to at least some of the plurality of joints of the robotic arm via the plurality of motors to compensate for the weight of the surgical instrument; A controller programmed to perform the above; A cooperative surgical system comprising the above. Claim 2 The controller is programmed to cause the robotic arm to maintain a static position in the passive mode in response to determining that movement of the robotic arm due to movement of the handle of the surgical instrument is less than a predetermined amount over at least a predetermined dwell period. The cooperative surgical system according to claim 1.

3. The controller is configured to measure a change in position of the distal end of the robotic arm in response to an external force, and calculate a holding force required to maintain the static position of the robotic arm based on the position of the distal end of the robotic arm, the entry point, and the one or more instrument parameters, and apply a torque to at least some of the plurality of joints of the robotic arm based on the compensation force and the holding force via the plurality of motors to maintain the static position of the robotic arm in the passive mode The cooperative surgical system according to claim 2, which is programmed to perform.

4. The controller is programmed to calculate a force applied to the patient by the surgical instrument at the entry point based on the compensation force, the holding force, the one or more instrument parameters, and the entry point. The cooperative surgical system according to claim 3.

5. The controller is programmed to calculate a force applied to the operating end of the surgical instrument based on the compensation force, the holding force, the one or more instrument parameters, and the entry point. The cooperative surgical system according to claim 3.

6. The controller is programmed to switch from the passive mode to the cooperative mode in response to determining that the holding force exceeds a predetermined threshold. The controller enables the robotic arm to move freely in the cooperative mode in response to movement of the handle of the surgical instrument to perform laparoscopic surgery using the surgical instrument while compensating for the gravity of the surgical instrument. The cooperative surgical system according to claim 3.

7. The cooperative surgical system according to claim 6, further comprising a graphical user interface operably coupled to the controller.

8. The cooperative surgical system according to claim 7, wherein the graphical user interface is configured to display at least one of a force applied to the patient by the surgical instrument at the entry point, an external force applied to the operating end of the surgical instrument, the predetermined threshold value, or the one or more instrument parameters.

9. The cooperative surgical system according to claim 7, wherein the graphical user interface is configured to enable a user to adjust the predetermined threshold value.

10. The cooperative surgical system according to claim 1, wherein the one or more instrument parameters comprise at least one of an instrument type, a mass, a center of mass, a length, or an instrument shaft diameter.

11. The cooperative surgical system according to claim 10, wherein the controller is programmed to calculate the mass of the surgical instrument by executing a calibration routine.

12. The cooperative surgical system according to claim 1, wherein the controller is configured to move the base in at least one degree of freedom.

13. The cooperative surgical system according to claim 12, further comprising a platform coupled to the base, wherein the controller is configured to cause vertical and horizontal movement of the base with respect to the platform.

14. One or more sensors operably coupled to one or more of the plurality of joints of the robotic arm, the one or more sensors configured to measure the position of the one or more joints and generate data indicative of the position of the one or more joints further comprising The cooperative surgical system according to claim 1, wherein the controller is programmed to measure the position of the distal end of the robotic arm based on the data.

15. The one or more sensors include one or more encoders disposed on one or more of the plurality of joints of the robotic arm, and the plurality of encoders are configured to measure the formed angles of the corresponding connecting portions of the plurality of connecting portions at at least some of the joints. The controller is programmed to measure the position of the distal end of the robotic arm in 3D space based on the formed angle measurements by the plurality of encoders, the cooperative surgical system according to claim 14. **Claim 16** A method for assisting in laparoscopic surgery using a robotic arm, the robotic arm comprising a proximal end, a distal end configured to be removably coupled to a surgical instrument, a plurality of connecting portions, and a plurality of joints between the proximal end and the distal end, the method comprising: Measuring the position of the distal end of the robotic arm via a controller operably coupled to a plurality of motors operably coupled to at least some of the plurality of joints; Determining an entry point of the surgical instrument into a patient by determining an intersection of a plurality of virtual lines parallel to a longitudinal axis of the surgical instrument as the position of the distal end of the robotic arm moves relative to the entry point; Calculating a compensation force required to compensate for the gravity of the surgical instrument based on the position of the distal end of the robotic arm, the entry point, and one or more instrument parameters stored in a memory of the controller; During the laparoscopic surgery, applying, via the plurality of motors, a torque to at least some of the plurality of joints of the robotic arm based on the compensation force to compensate for the gravity of the surgical instrument. Including The controller is configured to enable the robotic arm to move freely relative to a base operably coupled to the proximal end of the robotic arm in response to movement at a handle of the surgical instrument while compensating for the gravity of the surgical instrument. **Claim 17** In response to determining that movement of the robotic arm due to movement of the handle of the surgical instrument is less than a predetermined amount over at least a predetermined dwell period, further comprising maintaining the robotic arm in a passive mode at a static position, the method of claim 16.

18. Measuring a change in position of the distal end of the robotic arm in response to an external force; Calculating a holding force required to maintain the static position of the robotic arm based on the position of the distal end of the robotic arm, the entry point, and the one or more instrument parameters; Applying torque to at least some of the plurality of joints of the robotic arm based on the compensation force and the holding force via the plurality of motors to maintain the static position of the robotic arm in the passive mode The method of claim 17, further comprising.

19. The method of claim 18, further comprising calculating a force applied to the patient by the surgical instrument at the entry point based on the compensation force, the holding force, the one or more instrument parameters, and the entry point.

20. The method of claim 18, further comprising calculating a force applied to the operating end of the surgical instrument based on the compensation force, the holding force, the one or more instrument parameters, and the entry point.

21. In response to determining that the holding force exceeds a predetermined threshold, further comprising switching from the passive mode to a cooperative operation mode, wherein the robotic arm compensates for the gravity of the surgical instrument and uses the surgical instrument to perform laparoscopic surgery. In response to movement of the handle of the surgical instrument, the robotic arm is freely movable in the cooperative operation mode, the method of claim 18.

22. A cooperative surgical system for assisting in a laparoscopic surgery performed using a surgical instrument, the surgical instrument having a handle, an operating end, and an elongating shaft therebetween, the cooperative surgical system comprising: A robotic arm, the robotic arm comprising a proximal end, a distal end configured to be removably coupled to the surgical instrument, a plurality of links, and a plurality of joints. A platform configured to support the robotic arm, the platform comprising a plurality of wheels configured to enable mobility of the platform A plurality of optical sensors coupled to the platform and configured to collect depth data A display operably coupled to the platform A controller configured to enable the robotic arm to be freely movable in response to movement at the handle of the surgical instrument for performing laparoscopic surgery using the surgical instrument, the controller Receiving the depth data collected by the plurality of optical sensors Generating a map of an area surrounding the platform based on the depth data Displaying the map on the display A controller programmed to perform the above A cooperative surgical system comprising the above

23. The cooperative surgical system according to claim 22, wherein the plurality of optical sensors comprises at least one of a depth camera, a stereo RGB camera, a LiDAR device, or an electromagnetic, capacitive, or infrared proximity sensor

24. The cooperative surgical system according to claim 22, wherein the map generated by the controller comprises a graphical representation of the platform relative to at least one of one or more objects or one or more persons within the area surrounding the platform

25. The cooperative surgical system according to claim 24, wherein the controller is configured to generate an alert when the map indicates that the platform is within a predetermined distance of at least one of the one or more objects or one or more persons within the area surrounding the platform

26. A braking mechanism configured to be engaged to prevent mobility of the platform An actuator operably coupled to the braking mechanism, the actuator being configured to actuate to engage and disengage the braking mechanism to enable mobility of the platform Further comprising the above The cooperative operation type surgical system according to claim 22, wherein when the braking mechanism is engaged or disengaged, the controller automatically causes the map to be displayed on the display.

27. The cooperative operation type surgical system according to claim 22, wherein the controller is configured to move the robotic arm relative to the platform in at least one degree of freedom.

28. A method for assisting laparoscopic surgery using a robotic arm, the robotic arm comprising a plurality of connecting parts, a plurality of joints, a proximal end supported by the platform configured to enable mobility of the platform, and a distal end configured to be removably coupled to a surgical instrument, the method comprising: collecting depth data from a plurality of optical scanners coupled to the platform; generating a map of an area surrounding the platform based on the depth data, the map comprising a graphical representation of the platform relative to at least one of one or more objects or one or more persons within the area surrounding the platform; displaying the map on a display while the platform is moving to guide movement of the platform within the operating room; and including a method.

29. A cooperative operation type robotic surgical system for performing a surgical procedure, the cooperative operation type robotic surgical system comprising: a first surgical robot, the first surgical robot comprising: a base; an arm coupled to the base; a motor coupled to the arm and configured to move the arm relative to the base; a first surgical robot; a controller configured to control the arm; an optical scanner configured to collect depth data; and a cooperative operation type robotic surgical system.

30. The cooperative operation type robotic surgical system according to claim 29, wherein the optical scanner is configured to collect depth data related to the position and orientation of an instrument relative to the cooperative operation type robotic surgical system.

31. The collaborative surgical robotic system according to claim 30, wherein the controller is programmed to determine whether the instrument is coupled to the first surgical robot based on the depth data. **Claim 32** The collaborative surgical robotic system according to claim 29 or 30, wherein the controller is programmed to identify the type of the instrument based at least in part on the depth data. **Claim 33** The collaborative surgical robotic system according to claim 29 or 30, wherein the optical scanner is configured to collect depth data related to the position and movement of the instrument, and the instrument is freely held by a surgeon and not coupled to the surgical robot. **Claim 34** The collaborative surgical robotic system according to claim 29 or 30, wherein the optical scanner is configured to collect depth data related to a trocar inserted into a patient. **Claim 35** The collaborative surgical robotic system according to claim 34, wherein the system is configured to move the arm and / or the base of the first surgical robot when the position of the trocar changes by more than a threshold amount. **Claim 36** Further comprising a second surgical robot, wherein the second surgical robot comprises a second base, a second arm coupled to the second base, and a second motor coupled to the second arm and configured to move the second arm relative to the second base. The collaborative surgical robotic system according to claim 29 or 30. **Claim 37** The collaborative surgical robotic system according to claim 29 or 30, wherein the optical scanner has an accuracy of at least 5 mm in a range of 10 meters. **Claim 38** The collaborative surgical robotic system according to claim 29 or 30, wherein the optical scanner is configured to collect depth data related to the hand of a surgeon during a surgical procedure. **Claim 39** The controller is configured to control the arm of the first surgical robot according to at least one of the following operation modes, namely, a passive assistance mode, a cooperative operation assistance mode, a robot assistance mode, a force sensing mode In the passive assistance mode, the arm is static. In the passive assistance mode, the arm is static. In the coordinated operation assistance mode, while the arm can be freely moved by the operator, the motor at least partially and simultaneously moves the arm to improve the position and / or orientation of the instrument coupled to the end portion of the arm and / or to compensate for at least the gravity acting on the arm and the instrument coupled to the end portion of the arm. In the robot assistance mode, the motor moves the arm and relocates the instrument coupled to the end portion of the arm. In the force sensing mode, while the arm can be moved by the operator, the motor compensates for at least the gravity acting on the arm and / or the instrument coupled to the end portion of the arm, guides at least the instrument along a pre-defined trajectory, prevents unwanted movement of the arm and / or the instrument coupled to the end portion of the arm, prevents movement of the arm outside a specific space and / or prevents movement of the arm into a specific space. The cooperative surgical robot system according to claim 29 or claim 30.

40. The optical scanner uses the depth data to identify potential inadvertent collisions between the arm of the first surgical robot and at least one of the patient, the support platform supporting at least the first surgical robot, another surgical robot, and / or another object in the operating room, warns the user of the potential inadvertent collision and / or blocks the movement of the arm of the first surgical robot and / or causes the movement of the arm of the first surgical robot to avoid such collisions. The cooperative surgical robot system according to claim 29 or claim 30.

41. The first surgical robot is supported by a support platform, and the cooperative surgical robot system is configured to move the first surgical robot relative to the support platform based on the depth data collected by the optical scanner to optimize the position of the first surgical robot on the support platform. The cooperative surgical robot system according to claim 29 or claim 30.

42. The cooperative surgical robot system according to claim 29 or claim 30, wherein the optical scanner is configured to collect depth data used to record the movement of a surgeon's hand during a surgical procedure.

43. The cooperative surgical robot system according to claim 29 or claim 30, wherein the first surgical robot is supported by the support platform comprising a plurality of wheels configured to enable mobility of the support platform, and the plurality of wheels comprise a braking mechanism configured to be engaged to prevent mobility of the support platform.

44. The cooperative surgical robot system according to claim 43, further comprising one or more additional optical scanners disposed on the support platform and configured to collect depth data.

45. The cooperative surgical robot system according to claim 44, wherein at least one of the optical scanner or the one or more additional optical scanners comprises at least one of a depth camera, a stereo RGB camera, a LiDAR device, or an electromagnetic, capacitive, or infrared proximity sensor.

46. Further comprising a display operably coupled to the controller, The cooperative surgical robot system according to claim 44, wherein the controller is configured to generate a map of an area surrounding the support platform based on the depth data collected from at least one of the optical scanner or the one or more optical scanners, and to display the map on the display.

47. The cooperative surgical robot system according to claim 46, wherein the map generated by the controller comprises a graphical representation of the support platform with respect to one or more objects and / or one or more persons within the area surrounding the support platform.

48. The cooperative surgical robot system according to claim 47, wherein the controller is configured to generate an alert when the map indicates that the support platform is within a predetermined distance of the one or more objects and / or one or more persons within the area surrounding the support platform.

49. An actuator operably coupled to the braking mechanism, the actuator being configured to operate to engage and disengage the braking mechanism and enable mobility of the support platform further comprising The collaborative operation type surgical robot system according to claim 46, wherein when the braking mechanism is engaged and disengaged, the controller automatically causes the map to be displayed on the display

50. A collaborative operation type surgical robot system for performing a surgical procedure, the collaborative operation type surgical robot system comprising a surgical robot, comprising a base, an arm coupled to the base, a motor coupled to the arm and a surgical robot; an optical scanner configured to track movement of one or more objects around a patient; a controller configured to collect data regarding movement of one or more objects from an optical sensor and move the arm of the surgical robot in response to movement of the one or more objects and a collaborative operation type surgical system.

51. The collaborative operation type surgical system according to claim 50, wherein the controller is configured to move the base in at least one degree of freedom

Images