Systems and methods for controlling and enhancing movement of a surgical robotic unit during surgery - Patents.com

Abstract

Disclosed herein are surgical robotic systems and methods for controlling the location of one or more robotic arms within a constrained space. In some embodiments, the robotic system includes a robotic unit having a robotic arm. The system further includes a camera assembly for generating a patient's anatomy. The system further includes a controller configured or programmed to define a retraction region that defines a safe movement of the robotic arm. The method includes defining a safe movement of the robotic arm relative to the retraction region.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 323218, filed March 24, 2022, and U.S. Provisional Patent Application No. 63 / 339179, filed May 6, 2022, the entire contents of which are incorporated by reference herein.

[0002] The present disclosure is directed to minimally invasive surgical devices and associated control methods, and more particularly to the control of robotic surgical systems inserted into a patient during a surgical procedure. [Background technology]

[0003] The field of minimally invasive surgery has expanded rapidly since its introduction in the early 1990s. Although minimally invasive surgery significantly improves patient outcomes, this improvement comes at the expense of the surgeon's ability to operate with precision and ease. During laparoscopic surgery, surgeons must insert laparoscopic instruments through small incisions in the patient's abdominal wall.

[0004] Existing robotic surgical devices have attempted to solve these problems. Some existing robotic surgical devices replicate non-robotic laparoscopic surgery with additional degrees of freedom at the end of the instrument. However, even with many costly modifications to the surgical procedure, existing robotic surgical devices have failed to provide improved patient outcomes in the majority of procedures in which they are used. Additionally, existing robotic devices increase the separation created between the surgeon and the surgical end effector. This increased separation causes injuries due to the surgeon's misinterpretation of the motions and forces applied by the robotic device. Because the multiple degrees of freedom of many existing robotic devices are unfamiliar to human operators, such as surgeons, surgeons typically undergo extensive training on a robotic simulator before operating on a patient to minimize the possibility of inadvertently causing injury to the patient. Summary of the Invention [Problem to be solved by the invention]

[0005] To control existing robotic devices, the surgeon sits at a surgeon console or station and controls manipulators with his hands and feet. In addition, the robotic camera remains in a semi-fixed position and is moved by combined foot and hand movements from the surgeon. These semi-fixed cameras with limited field of view make visualization of the surgical field difficult. [Means for solving the problem]

[0006] The present disclosure is directed to systems and methods for controlling movement of a robotic unit during surgery. According to some embodiments, the system includes a controller configured or programmed to receive tissue contact constraint data and execute instructions retained in a memory to control a robotic unit having a robotic arm in a manner that reduces potential damage to tissue in an area identified by the tissue contact constraint data. The system may further include a camera assembly for generating a view of the patient's anatomy and a display unit configured to display the view of the anatomy.

[0007] In some embodiments, the present disclosure is directed to a method of controlling a position of one or more robotic arms within a constrained space, the method including receiving tissue contact constraint data and controlling the one or more robotic arms to reduce potential damage to tissue in an area defined by the tissue contact constraint data.

[0008] According to some embodiments, the present disclosure is directed to a system including a robotic arm assembly having a robotic arm, a camera assembly that generates image data of an internal region of a patient, and a controller configured or programmed to detect one or more markers in the image data, control movement of the robotic arm based on the one or more markers in the image data, and store the image data.

[0009] These and other features and advantages of the present disclosure will be more fully understood by reference to the following detailed description in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout the various views, the drawings illustrate the principles of the present disclosure and, although not to scale, show relative dimensions. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a schematic of an exemplary surgical robotic system, according to some embodiments. [Figure 2A] FIG. 2A is an exemplary perspective view of a patient cart including a robotic support system coupled to a robotic subsystem of a surgical robotic system, according to some embodiments. [Figure 2B] FIG. 2B is an exemplary perspective view of an exemplary operator console of the surgical robotic system of the present disclosure, according to some embodiments. [Figure 3A] FIG. 3A illustrates generally an exemplary side view of a surgical robotic system for performing a surgical procedure within an internal cavity of a subject, according to some embodiments. [Figure 3B] FIG. 3B illustrates, in accordance with some embodiments, an exemplary top view of a surgical robotic system for performing a surgical procedure within the internal cavity of the object of FIG. 3A. [Figure 4A] FIG. 4A is an exemplary perspective view of a robotic arm subsystem, according to some embodiments. [Figure 4B] FIG. 4B is an exemplary side perspective view of a single robotic arm of the single robotic arm subsystem of FIG. 4A, according to some embodiments. [Diagram 5] FIG. 5 is a front perspective view of the camera assembly and the robotic arm assembly, according to some embodiments. [Figure 6] FIG. 6 is a schematic diagram of a controller of the present disclosure for providing control of movement of a robotic unit within a patient in accordance with the teachings of the present disclosure. [Figure 7]7A-7D are exemplary representations of the types of markers that may be applied to a patient during a surgical procedure. [Figure 8] 8A-8B are example representations of a robotic arm that automatically moves (eg, snaps) to a particular marker when placed within a threshold distance thereof in accordance with the teachings of the present disclosure. [Figure 9A] FIG. 9A is a representation showing constrained movement of a robot arm relative to a selected plane in accordance with the teachings of the present disclosure. [Figure 9B] FIG. 9B is a representation showing constrained movement of a robotic arm when positioned within a selected volume during a surgical procedure in accordance with the teachings of the present disclosure. [Figure 9C] FIG. 9C is a representation showing a system for preventing placement of a robotic arm in one or more selected volumes during a surgical procedure in accordance with the teachings of the present disclosure. [Figure 9D] FIG. 9D is a representation showing a system for restricting movement of a robotic arm to a selected volume during a surgical procedure in accordance with the teachings of the present disclosure. [Figure 10] FIG. 10 illustrates a retraction surface for protecting patient tissue during surgery in accordance with the teachings of the present disclosure. [Figure 11] FIG. 11 illustrates a schematic of a motion control system for a surgical robotic system in accordance with the teachings of the present disclosure. [Figure 12] FIG. 12 is a representation of a tissue region identified by contact with a robotic arm in accordance with the teachings of the present disclosure. [Figure 13] FIG. 13 is a flow chart depicting a process for identifying tissue regions. [Figure 14] FIG. 14 is a flow chart depicting a process for identifying tissue regions. [Figure 15] FIG. 15 is a representation of the defined depth tolerance below the visceral bed in accordance with the teachings of the present disclosure. [Figure 16A] FIG. 16A is a representation of the movement of a robotic arm at a first side of a visceral bed in accordance with the teachings of the present disclosure. [Figure 16B]FIG. 16B is a representation of the movement of the robotic arm within the visceral bed depth tolerance in accordance with the teachings of the present disclosure. [Figure 16C] FIG. 16C is a representation of the movement of the robotic arm within the depth tolerance and approximately the midpoint of the visceral bed in accordance with the teachings of the present disclosure. [Figure 16D] FIG. 16D is a representation of the operation of a robotic arm with depth tolerance in accordance with the teachings of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The robotic system of the present disclosure assists the surgeon in controlling the movement of the robotic unit during surgery where the robotic unit is operable within the patient and minimizes the risk of accidental injury to the patient during surgery. The surgeon defines an operable area with respect to tissue at the surgical site, and the system implements one or more constraints on the arms of the robotic unit to prevent or impede progression of the arms outside of the constraints. The operable area or constraints may be defined with markers or by visual identification of portions of tissue.

[0012] In the following description, numerous specific details are described with respect to the disclosed system and method, and the environment in which the system and method may operate, to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter may be practiced without such specific details, and that certain features well known in the art have not been described in detail to avoid complexity and enhance clarity of the disclosed subject matter. Furthermore, it will be understood that any examples provided below are merely illustrative and should not be construed in a limiting manner, and that other systems, apparatuses, and / or methods may be used to implement or accomplish the teachings of the present invention and are considered to be within the scope of the present invention.

[0013] Despite advances in the field of robotic surgery, the possibility of accidentally injuring a patient when a surgical robotic unit is first deployed on a patient or during a surgical procedure remains an unaddressed technical problem. During surgery, the surgeon can articulate the robot to access the entire internal abdominal region. Due to the extensive movement of the robotic unit, injuries can occur during insertion of the robotic unit or "off camera," where the surgical robotic unit accidentally injures tissue, organs, or blood vessels outside the surgeon's field of view. For example, the surgical robotic unit may tear or pinch tissue within the surgical site, such as the visceral bed. Thus, this type of injury may go undetected, which is of great concern to the patient.

[0014] Described herein are systems and methods for solving technical problems that accidentally injure a patient. The system may define a region corresponding to tissue surrounding a surgical site, and may include user input to identify the tissue. The system may then prevent movement or slow movement of the robotic arm beyond the identified region or beyond a depth tolerance beyond the identified region to prevent tissue damage. Additionally or alternatively, the system may provide an indication to the user to inform the user of the location of the robotic arm relative to the identified region.

[0015] Although the exemplary embodiment is described as using multiple units to perform the exemplary process, it is understood that the exemplary process may also be performed by one or more modules or units. Furthermore, it is understood that terms such as controller, control unit, computing unit, etc. refer to hardware devices that include at least a memory and a processor and are specifically programmed to perform the processes described herein. The memory is configured to store modules, and the processor is specifically configured to execute the functions and operations associated with the modules to perform one or more processes further described herein.

[0016] Furthermore, the control logic of the present disclosure may be embodied as a non-transitory computer readable medium on a computer readable medium containing executable program instructions executed by a processor, a controller / control unit, or the like. Examples of computer readable media include, but are not limited to, ROM, RAM, compact disk (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer readable recording medium may also be distributed in a network-coupled computer system such that the computer readable medium is stored and executed in a distributed manner, for example, by a telematics server or a controller area network (CAN). The control logic may also be implemented using application software stored in suitable storage and memory and processed using known processing devices. The control or computing unit described herein may be implemented using any selected computer hardware with a processor, storage, and memory.

[0017] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0018] Unless specifically stated or clear from the context, the term "about" as used herein is understood to be within the normal tolerance in the art, for example, within 2 standard deviations of the mean. "About" can be understood to be within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise clear from the context, all numerical values ​​provided herein are modified by the term "about".

[0019] As used herein, the term "constriction area" is defined as a three-dimensional volume or a two-dimensional plane. A three-dimensional volume may be defined as a cube, a cone, a cylinder, or other three-dimensional shape or combination of shapes.

[0020] While the systems and methods of the present disclosure may be designed for use with one or more surgical robotic systems, the robotic systems of the present disclosure may be used in connection with any type of surgical system, including, for example, robotic surgical systems, linear stick surgical systems, virtual reality surgical systems, and laparoscopic systems. Additionally, the systems of the present disclosure may be used in other non-surgical systems where a user requires access to a myriad of information while controlling a device or instrument.

[0021] The robotic system of the present disclosure assists the surgeon in controlling the movement of the robotic unit during surgery when the robotic unit is operable within the patient's body. Thus, the control features of the present disclosure enable the surgeon to minimize the risk of accidental injury to the patient during surgery.

[0022] Similar numerical identifiers are used throughout the figures to refer to the same elements.

[0023] 1 is a schematic diagram of a surgical robotic system 10 in which aspects of the present disclosure may be employed according to some embodiments of the present disclosure. The surgical robotic system 10 includes an operator console 11 and a robotic subsystem 20 according to some embodiments.

[0024] The surgical robotic system 10 of the present disclosure employs a robotic subsystem 20 including a robotic unit 50 that can be inserted into a patient via a trocar through a single incision point or site. The robotic unit 50 is small enough to be deployed in vivo at a surgical site and is sufficiently maneuverable when inserted into the patient's body to be movable within the body to perform various surgical procedures at multiple different points or sites. The robotic unit 50 includes multiple separate robotic arms 42 that are deployable within the patient along different or separate axes. Additionally, a surgical camera assembly 44 can also be deployed along a separate axis and form part of the robotic unit 50. Thus, the robotic unit 50 employs multiple different components, such as a pair of robotic arms and a surgical or robotic camera assembly, each of which can be deployed along a different axis and can be separately steerable, steerable, and movable. Notably, the robotic unit 50 is not limited to the robotic arms and camera assemblies described herein, and additional components may be included in the robotic unit. A robotic arm and camera assembly that is disposable along a separate steerable axis is referred to herein as a split-arm (SA) architecture. The SA architecture is designed to simplify and increase efficiency of the insertion of robotic surgical instruments through a single trocar at a single insertion site while assisting in the deployment of the surgical instruments to a surgery-ready state and the subsequent removal of the surgical instruments through the trocar. As an example, surgical instruments can be inserted through the trocar to access the abdominal cavity of a patient and perform surgery in vivo. In some embodiments, a variety of surgical instruments may be utilized, including but not limited to robotic surgical instruments and other surgical instruments known in the art.

[0025] The systems and methods disclosed herein may be incorporated into and utilized with, for example, the robotic surgical devices and associated systems disclosed in U.S. Patent No. 10,285,765 and PCT Patent Application No. PCT / US20 / 39203, and / or the camera assemblies and systems disclosed in U.S. Patent Application Publication No. 2019 / 0076199, the entire contents and teachings of which are incorporated herein by reference. The robotic unit 50 may form part of a robotic subsystem 20, which in turn forms part of a surgical robotic system 10 including suitable sensors and display units, as well as a surgeon or user workstation, including a robotic support system (RSS) or patient cart, for interacting with and supporting the robotic unit of the present disclosure. The robotic subsystem 20 may, in one embodiment, include portions of the RSS, such as, for example, drive units and associated mechanical linkages, and the surgical robotic unit 50 may include one or more robotic arms and one or more camera assemblies. The surgical robotic unit 50 provides multiple degrees of freedom so that the robotic unit can be maneuvered to a single position or multiple different positions within the patient. In one embodiment, the robotic support system can be mounted directly to the operating table or to the floor or ceiling within the operating room. In another embodiment, mounting is accomplished by various fastening means including, but not limited to, clamps, screws, or combinations thereof. In yet other embodiments, the structure may be freestanding, transportable, or mobile. The robotic support system can mount a motor assembly that is coupled to the surgical robotic unit and can include gears, motors, drive trains, electronics, etc. for powering the components of the surgical robotic unit.

[0026] The robotic arm and camera assembly are capable of multiple degrees of freedom of movement (e.g., at least seven degrees of freedom). According to one practice, when the robotic arm and camera assembly are inserted into a patient through a trocar, they are capable of moving in at least axial, yaw, pitch, and roll directions. The robotic arm assembly is designed to incorporate and utilize a multi-degree of freedom of movement robotic arm having an end effector region attached to its distal end that corresponds to a user's wrist and hand region or joint. In other embodiments, the working end (e.g., end effector end) of the robotic arm is designed to incorporate and utilize other robotic surgical instruments, such as those described in U.S. Patent Application Publication No. 10,799,308, the contents of which are incorporated herein by reference.

[0027] The operator console 11 includes a display 12, an image computation module 14 (which may be a three-dimensional (3D) computation module), a hand controller 17 having a sensing and tracking module 16, and a computation module 18. Additionally, the operator console 11 may include a foot pedal array 19 including a plurality of pedals. The image computation module 14 may include a graphical user interface 39. The graphical user interface 39, the controller 26, or the image rendering device 30, or both, may render one or more images or one or more graphical user interface elements on the graphical user interface 39. For example, pillar boxes associated with a mode of operating the surgical robotic system 10, or any of the various components of the surgical robotic system 10, may be rendered on the graphical user interface 39. Live video footage captured by the camera assembly 44 may also be rendered on the graphical user interface 39 by the controller 26 or the image rendering device 30.

[0028] The operator console 11 may include a visualization system 9 including a display 12, which may be any selected type of display for displaying information, images, or videos generated by the image computation module 14, the computation module 18, and / or the robotic subsystem 20. The display 12 may include or form part of, for example, a head mounted display (HMD), an augmented reality (AR) display (e.g., an AR display, or AR glasses in combination with a screen or display), a screen or display, a two-dimensional (2D) screen or display, a three-dimensional (3D) screen or display, or the like. The display 12 may also include an optional sensing and tracking module 16A. In some embodiments, the display 12 may include an image display for outputting images from a camera assembly 44 of the robotic subsystem 20.

[0029] The hand controller 17 is configured to sense the movement of the operator's hand and / or arm to operate the surgical robotic system 10. The hand controller 17 may include a sensing and tracking module 16, circuitry, and / or other hardware. The sensing and tracking module 16 may include one or more sensors or detectors that sense the movement of the operator's hand. In some embodiments, the one or more sensors or detectors that sense the movement of the operator's hand are disposed in the hand controller 17 that is grasped or engaged by the operator's hand. In some embodiments, the one or more sensors or detectors that sense the movement of the operator's hand are coupled to the operator's hand and / or arm. For example, the sensors of the sensing and tracking module 16 may be coupled to regions of the hand and / or arm, such as the fingers, wrist region, elbow region, and / or shoulder region. In some embodiments, additional sensors may also be coupled to the head and / or neck region of the operator. In some embodiments, the sensing and tracking module 16 may be external and coupled to the hand controller 17 via electrical components and / or mounting hardware. In some embodiments, the optional sensor and tracking module 16A may sense and track movement of one or more of the operator's head, the operator's eyes, or at least a portion of the operator's neck based at least in part on imaging of the operator, in addition to or instead of sensors attached to the operator's body.

[0030] In some embodiments, the sensing and tracking module 16 may use sensors coupled to the operator's torso or any other body part. In some embodiments, the sensing and tracking module 16 may use, in addition to the sensors, an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of a magnetometer may reduce sensor drift around the vertical axis. In some embodiments, the sensing and tracking module 16 also includes sensors placed within surgical materials such as gloves, surgical scrubs, or surgical gowns. The sensors may be reusable or disposable. In some embodiments, the sensors may be disposed external to the operator, such as in a fixed location in a room such as an operating room. The external sensors 37 may generate external data 36 that may be processed by the computation module 18 and thus used by the surgical robotic system 10.

[0031] The sensors generate position and / or orientation data indicative of the position and / or orientation of the operator's hands and / or arms. The sensing and tracking modules 16 and / or 16A may be utilized to control the movement (e.g., change in position and / or orientation) of the camera assembly 44 and the robotic arm 42 of the robotic subsystem 20. The tracking and position data 34 generated by the sensing and tracking module 16 may be communicated to the computation module 18 for processing by at least one processor 22.

[0032] The computation module 18 may determine or calculate from the tracking and position data 34 and 34A the position and / or orientation of the operator's hands or arms, and in some embodiments the operator's head as well, and communicate the tracking and position data 34 and 34A to the robotic subsystem 20. The tracking and position data 34, 34A may be processed by the processor 22 and stored, for example, in the storage 24. The tracking and position data 34 and 34A may also be used by the controller 26, which in response may generate control signals for controlling the movement of the robotic arm 42 and / or the camera assembly 44. For example, the controller 26 may change the position and / or orientation of at least a portion of the camera assembly 44, at least a portion of the robotic arm 42, or both. In some embodiments, the controller 26 may also adjust the pan and tilt of the camera assembly 44 to follow the movement of the operator's head.

[0033] The robotic subsystem 20 may include a robotic support system (RSS) 46 having a motor 40 and a trocar 50 or trocar mount, a robotic arm 42, and a camera assembly 44. The robotic arm 42 and camera assembly 44 may form part of a single support axis robotic unit as disclosed and described in U.S. Pat. No. 10,285,765, or may form part of a split-arm (SA) architecture robotic system as disclosed and described in PCT Patent Application No. PCT / US2020 / 039203, both of which are incorporated herein by reference in their entireties.

[0034] The robotic subsystem 20 may employ multiple different robotic arms deployable along different or separate axes. In some embodiments, the camera assembly 44, which may employ multiple different camera elements, may also be deployed along a common separate axis. Thus, the surgical robotic system 10 may employ multiple different components, such as a pair of separate robotic arms and camera assemblies 44 deployable along different axes. In some embodiments, the robotic arm assembly 42 and the camera assembly 44 are separately operable, steerable, and movable. The robotic subsystem 20, including the robotic arm 42 and the camera assembly 44, is disposable along separate operable axes, and is referred to herein as an SA architecture. The SA architecture is designed to simplify and increase the efficiency of the insertion of robotic surgical instruments through a single trocar at a single insertion point or site, while also aiding in the deployment of the surgical instruments into a surgery-ready state, as well as the subsequent removal of the surgical instruments through the trocar 50, as further described below.

[0035] The RSS 46 may include a motor 40 and a trocar 50 or trocar mount. The RSS 46 may further include a support member supporting the motor 40 coupled to its distal end. The motor 40 may be coupled to each of the camera assembly 44 and the robotic arm assembly 42. The support member may be configured and controlled to move one or more components of the robotic subsystem 20 linearly or in any other selected direction or orientation. In some embodiments, the RSS 46 may be upright. In some embodiments, the RSS 46 may include a motor 40 coupled at one end to the robotic subsystem 20 and coupled at an opposite end to an adjustable support member or element.

[0036] The motors 40 can receive control signals generated by the controller 26. The motors 40 can include gears, one or more motors, drive trains, electronics, etc., for powering and driving the robotic arm 42 and the camera assembly 44 individually or together. The motors 40 can also provide mechanical power, electrical power, mechanical communications, and electrical communications to the robotic arm 42, the camera assembly 44, and / or the RSS 46 and other components of the robotic subsystem 20. The motors 40 can be controlled by the computation module 18. Thus, the motors 40 can generate signals to control one or more motors that can control and drive the robotic arm 42, including, for example, the position and orientation of each articulating joint of each arm, as well as the camera assembly 44. The motors 40 can further provide translational or linear degrees of freedom that are primarily utilized to insert and remove each component of the robotic subsystem 20 through the trocar 50. The motors 40 can also be used to adjust the insertion depth of each robotic arm 42 as it is inserted into the patient 100 through the trocar 50.

[0037] The trocar 50 is a medical device that may consist of a claw (which may be a metal or plastic sharp or bladeless tip), a cannula (essentially a hollow tube), and a seal in some embodiments. The trocar 50 may be used to place at least a portion of the robotic subsystem 20 within an internal cavity of a subject (e.g., a patient) and may withdraw gas and / or fluid from the body cavity. The robotic subsystem 20 may be inserted through the trocar 50 to access the patient's body cavity and perform surgery in vivo. In some embodiments, the robotic subsystem 20 of the present invention may be supported, at least in part, by the trocar 50 or a trocar mount with multiple degrees of freedom such that the robotic arm 42 and camera assembly 44 may be maneuvered within the patient to a single location or multiple different locations. In some embodiments, the robotic arm 42 and camera assembly 44 may be supported by the trocar 50 or a trocar mount with multiple degrees of freedom such that the robotic arm 42 and camera assembly 44 may be maneuvered within the patient to a single location or multiple different locations.

[0038] In some embodiments, the RSS 46 may further include an optional controller for processing input data from one or more of the system components (e.g., the display 12, the sensing and tracking module 16, the robotic arm 42, the camera assembly 44, etc.) and for generating control signals in response thereto. The motor 40 may also include a memory element for storing data in some embodiments.

[0039] The robotic arm 42, in some embodiments and in some modes of operation, may be controlled to follow scaled-down movements or motions of an operator's arm and / or hand as sensed by associated sensors. The robotic arm 42 includes a first robotic arm including a first end effector having an instrument tip disposed at a distal end of the first robotic arm, and a second robotic arm including a second end effector having an instrument tip disposed at a distal end of the second robotic arm. In some embodiments, the robotic arm 42 may have portions or regions that may be associated with shoulder, elbow, and wrist joints as well as movements that may be associated with the fingers of the operator. For example, a robotic elbow joint may follow the position and orientation of a human elbow, and a robotic wrist joint may follow the position and orientation of a human wrist. The robotic arm 42 may also have a terminal region associated therewith, which in some embodiments may terminate in an end effector that follows the movement of one or more fingers of the operator, such as the index finger, when the user pinches the index finger and thumb together, for example. In some embodiments, the robotic arm 42 may follow the movement of the operator's arms in some control modes, while the virtual chest of the robotic arm assembly may remain stationary (e.g., in an instrument control mode). In some embodiments, the position and orientation of the operator's torso is subtracted from the position and orientation of the operator's arms and / or hands. This subtraction allows the operator to move the torso without the robotic arms moving. Further disclosure control of the movement of the individual arms of the robotic arm assembly is provided in International Patent Application Publication Nos. WO 2022 / 094000 A1 and WO 2021 / 231402 A1, each of which is incorporated herein by reference in its entirety.

[0040] The camera assembly 44 is configured to provide the operator with image data 48, such as, for example, a live video feed of the operation or surgical site, as well as to allow the operator to operate and control the cameras forming part of the camera assembly 44. In some embodiments, the camera assembly 44 may include one or more cameras (e.g., a pair of cameras) whose optical axes are axially spaced apart by a selected distance, known as the inter-camera distance, to provide a stereoscopic view or image of the surgical site. In some embodiments, the operator may control the movement of the camera via hand movement, via a sensor coupled to the operator's hand, or via a hand controller 17 grasped or held by the operator's hand, thus allowing the operator to obtain a desired view of the surgical site in an intuitive and natural manner. In some embodiments, the operator may additionally control the movement of the camera via movement of the operator's head. The camera assembly 44 is movable in multiple directions, including, for example, a yaw direction, a pitch direction, and a roll direction, with respect to the viewing direction. In some embodiments, the stereoscopic camera components may be configured to provide a natural and comfortable user experience. In some embodiments, the axial distance between the cameras may be modified to adjust the depth of the surgical site as perceived by the operator.

[0041] Image or video data 48 generated by camera assembly 44 may be displayed on display 12. In an embodiment, if display 12 includes an HMD, the display may include an embedded sensing and tracking module 16A that obtains raw orientation data in yaw, pitch, and roll directions of the HMD, as well as position data in Cartesian space (x,y,z) of the HMD. In some embodiments, position and orientation data for the operator's head may be provided via a separate head tracking module. In some embodiments, sensing and tracking module 16A may be used to provide supplemental position and orientation tracking data for the display instead of, or in addition to, the HMD's embedded tracking system. In some embodiments, operator head tracking is not used or employed. In some embodiments, an image of the operator may be used by sensing and tracking module 16A to track at least a portion of the operator's head.

[0042] 2A illustrates an exemplary robotic arm assembly 20 (also referred to herein as a robotic subsystem) of a surgical robotic system 10 integrated into or mounted on a mobile patient cart, according to some embodiments. In some embodiments, the robotic arm assembly 20 includes an RSS 46, which in turn includes a motor 40, a robotic arm 42 having an end effector 45, and a camera assembly 44 having one or more cameras 47, which may also include a trocar 50 or a trocar mount.

[0043] 2B shows an example of an operator console 11 of the presently disclosed surgical robotic system 10, according to some embodiments. The operator console 11 includes a display 12, a hand controller 17, and also includes one or more additional controllers, such as a foot pedal array 19 for control of a robotic arm 42, a camera assembly 44, and other aspects of the system.

[0044] 2B also illustrates a left hand controller subsystem 23A and a right hand controller subsystem 23B of the operator console. The left hand controller subsystem 23A includes and supports the left hand controller 17A, and the right hand controller subsystem 23B includes and supports the right hand controller 17B. In some embodiments, the left hand controller subsystem 23A may be removably connected to or engaged with the left hand controller 17A, and the right hand controller subsystem 23B may be removably connected to or engaged with the right hand controller 17A. In some embodiments, the connections may be both physical and electronic such that the left hand controller subsystem 23A and the right hand controller subsystem 23B may receive signals from the left hand controller 17A and the right hand controller 17B, respectively, including signals conveying inputs received from user selections on buttons or touch input devices of the left hand controller 17A or the right hand controller 17B.

[0045] Each of the left and right hand controller subsystems 23A and 23B may include components that enable a range of motion for the respective left and right hand controllers 17A and 17B, such that the left and right hand controllers 17A and 17B may translate or displace in three dimensions and may additionally move in roll, pitch, and yaw directions. Additionally, each of the left and right hand controller subsystems 23A and 23B may register the motion of the respective left and right hand controllers 17A and 17B in each of the aforementioned directions and may transmit signals providing such movement information to a processor (not shown) of the surgical robotic system.

[0046] In some embodiments, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may be configured to receive and connect or engage a different hand controller (not shown). For example, hand controllers having different configurations of buttons and touch input devices may be provided. Additionally, hand controllers having different shapes may be provided. The hand controllers may be selected to match with a particular surgical robotic system or a particular surgical robotic procedure, or may be selected based on operator preferences for buttons and input devices or with respect to the shape of the hand controller to provide greater comfort and ease for the operator.

[0047] FIG. 3A illustrates, in accordance with some embodiments and for some surgical procedures, a side view of the surgical robotic system 10 performing surgery within an internal cavity 104 of a subject 100. FIG. 3B illustrates, in accordance with some embodiments and for some surgical procedures, a perspective top view of the surgical robotic system 10 performing surgery within an internal cavity 104 of a subject 100. The subject 100 (e.g., a patient) is positioned on a surgical table 102 (e.g., a surgical table 102). In some embodiments and for some surgical procedures, an incision is made in the patient 100 to gain access to the internal cavity 104. A trocar 50 is then inserted into the patient 100 at a selected location to provide access to the internal cavity 104 or surgical site. The RSS 46 can then be manipulated into position on the patient 100 and the trocar 50. In some embodiments, the RSS 46 includes a trocar mount that couples to the trocar 50. The robotic arm assembly 20 may be coupled to the motor 40, and at least a portion of the robotic arm assembly may be inserted into the trocar 50, and thus into the internal cavity 104 of the patient 100. For example, the camera assembly 44 and the robotic arm assembly 42 may be individually and sequentially inserted into the patient 100 through the trocar 50. Although the camera assembly and the robotic arm assembly may include some portions that remain outside the subject's body during use, references to inserting the robotic arm assembly 42 and / or the camera assembly 44 into the internal cavity of the subject, and disposing the robotic arm assembly 42 and / or the camera assembly 44 into the internal cavity of the subject refer to the portions of the robotic arm assembly 42 and the camera assembly 44 that are intended to be within the internal cavity of the subject during use. The sequential insertion method has the advantage of supporting smaller trocars, and therefore smaller incisions may be made in the patient 100, thus reducing the trauma experienced by the patient 100. In some embodiments, the camera assembly 44 and the robotic arm assembly 42 may be inserted in any order or in a specific order. In some embodiments, the camera assembly 44 may be followed by a first robotic arm of the robotic arm assembly 42, which may be followed by a second robotic arm of the robotic arm assembly 42, all of which may be inserted into the trocar 50 and thus into the internal cavity 104.Once inserted into the patient 100 , the RSS 46 can move the robotic arm assembly 42 and camera assembly 44 to the surgical site manually or automatically controlled by the operator console 11 .

[0048] Disclosure controls relating to managing the movement of individual arms of a robotic arm assembly are provided in International Patent Application Publication Nos. 2022 / 094000(A1) and 2021 / 231402(A1), each of which is incorporated by reference in its entirety into this specification.

[0049] 4A is a perspective view of the robot arm subassembly 21, according to some embodiments. The robot arm subassembly 21 includes a robot arm 42A, an end effector 45 having an instrument tip 120 (e.g., monopolar scissors, needle driver / holder, bipolar grasper, or any other suitable tool), and a shaft 122 supporting the robot arm 42A. A distal end of the shaft 122 is coupled to the robot arm 42A, and a proximal end of the shaft 122 is coupled to a housing 124 of the motor 40 (as shown in FIGS. 1 and 2A). At least a portion of the shaft 122 can be outside the internal cavity 104 (as shown in FIGS. 3A and 3B). At least a portion of the shaft 122 can be inserted into the internal cavity 104 (as shown in FIGS. 3A and 3B).

[0050] 4B is a side view of the robotic arm assembly 42, which in some implementations includes a virtual shoulder 126, a virtual elbow 128 with a capacitive proximity sensor 132, a virtual wrist 130, and an end effector 45. In some implementations, the virtual shoulder 126, virtual elbow 128, and virtual wrist 130 can include a series of hinges and revolute joints to provide seven positionable degrees of freedom for each arm, along with an additional grasping degree of freedom for the end effector 45.

[0051] 5 shows a perspective front view of an internal portion of the robot arm subsystem 20. The robot arm assembly 20 includes a robot arm 42A and a robot arm 42B. The two robot arms 42A and 42B may define a virtual chest 140 of the robot arm assembly 20 in some embodiments. In some embodiments, the virtual chest 140 may be defined by a chest plane extending between a first pivot point 142A of the most proximal joint of the robot arm 42A (e.g., shoulder joint 126), a second pivot point 142B of the most proximal joint of the robot arm 42B, and a camera imaging center point 144 of the camera 47. A pivot center 146 of the virtual chest 140 is at the center of the virtual chest.

[0052] In some embodiments, sensors on one or both of the robotic arm 42A and the robotic arm 42B can be used by the system to determine a change in position in three-dimensional space of at least a portion of the robotic arms. In some embodiments, sensors on one or both of the first robotic arm and the second robotic arm can be used by the system to determine a position in three-dimensional space of at least a portion of one robotic arm relative to a position in three-dimensional space of at least a portion of the other robotic arm.

[0053] In some embodiments, the camera assembly 44 is configured to acquire images that allow the system to determine a relative position in three-dimensional space. For example, the camera assembly may include multiple cameras, at least two of which are laterally displaced from one another relative to the imaging axis, and the system may be configured to determine a distance to a feature in an internal body cavity. Further disclosure regarding a surgical robotic system including a camera assembly and an associated system for determining a distance to a feature may be found in International Patent Application Publication No. WO2021 / 159409, entitled "System and Method for Determining Depth Perception In Vivo in a Surgical Robotic System," published on August 12, 2021, and incorporated herein by reference in its entirety. Information regarding the distance to a feature and information regarding the optical properties of the camera may be used by the system to determine a relative position in three-dimensional space.

[0054] The hand controllers for the surgical robotic systems described herein can be used with any of the surgical robotic systems described above, or any other suitable surgical robotic system. Additionally, some embodiments of the hand controllers described herein may be used with semi-robotic endoscopic surgical systems that are only partially robotic.

[0055] As mentioned above, it may be desirable for a controller for a surgical robotic system to provide sufficient inputs to provide control of the system, ergonomics, and a "natural" feel during use.

[0056] Some embodiments described herein refer to a left hand controller and a corresponding left robot arm (which may be a first robot arm), and a right hand controller and a corresponding right robot arm (which may be a second robot arm). In some embodiments, the robot arm considered to be the left robot arm and the robot arm considered to be the right robot arm may vary due to the configuration of the robot arms, and the camera assembly is adjusted so that the second robot arm corresponds to the left robot arm with respect to the view provided by the camera assembly, and the first robot arm corresponds to the right robot arm with respect to the view provided by the camera assembly. In some embodiments, the surgical robotic system, during use, identifies a robot arm as corresponding to the left hand controller and identifies a robot arm as corresponding to the right hand controller. In some embodiments, at least one hand controller includes one or more operator input devices for providing one or more inputs for additional control of the robotic assembly. In some embodiments, the one or more operator input devices receive one or more operator inputs for at least one of engaging a scan mode, resetting an orientation and position of a camera assembly to align the camera assembly's field of view with the instrument tip and the chest, displaying a menu, traversing a menu or highlighting an option or item for selection, selecting an item or option, selecting and adjusting an elbow position, and engaging a clutch associated with a respective hand controller.

[0057] In some embodiments, additional functionality may be accessed via a menu, such as, for example, selecting a level of gripping force (e.g., high / low), selecting insertion, extraction, or exchange mode, adjusting focus, lighting, or gain, camera cleaning, motion scaling, rotating the camera to allow for looking down, etc.

[0058] As described herein, the robotic unit 50 may be inserted into the patient through a trocar. The robotic unit 50 may be used by the surgeon to place one or more markers in the patient according to known techniques. For example, the markers may be applied using a biocompatible ink pen, or the markers may be passive or active objects, such as a QR code. The surgical robotic system 10 may then detect or track the markers in the image data using the detection unit 60. The markers may also be configured to emit RF or electromagnetic signals that are detected by the detection unit 60. The detection unit 60 may be configured to identify specific structures, such as different marker types, and may be configured to utilize one or more known image detection techniques, such as by using sensors or detectors that form part of a computer vision system, or by using image parallax techniques using the camera assembly 44. According to one embodiment, the detection unit 60 may be configured to identify the markers in the captured image data 44, thus enabling the system 10 to detect and track the markers. By identifying and tracking the markers, the system enables the surgeon to accurately identify and navigate the robotic unit through the unpredictable anatomy of the patient.

[0059] Markers may also be used to mark or identify the location where a selected surgical procedure or task, such as, for example, a suturing procedure, will be performed. For example, one or more of the robotic arms 42 may be used by the surgeon to place a marker at a selected location, such as at or around the incision 72. For example, as shown in FIG. 7A, the surgeon may control the robotic arms 42 to draw or place a marker 70, for example, using a biocompatible pen having a fluorescent dye or other imaging agent, near or around the incision 72 in the patient's abdomen to mark the area to be sutured. The robotic arms 42 may also be employed to place different types of markers, such as a series of "X" type markings, as illustrated in FIG. 7B. According to another practice, the surgeon may employ active or passive type markers. For example, as shown in FIG. 7C, the robotic arms 42 of the robotic unit 50 may be employed to place a QR code type marking 90 around the incision 72.

[0060] Once the markers have been placed at the selected surgical locations, the robotic unit 50 can be used to perform the selected surgical task. For example, as shown in FIG. 7D, the robotic arm 42 can be controlled by the surgeon to place one or more sutures in the incision 72. The surgeon can place the sutures in one or more of the markers, such as, for example, the X-shaped markings 80, using, for example, a suitable biocompatible thread 94.

[0061] Alternatively, the controller 18 can automatically control the movement of the robotic arm based on the image data 48 and the output signals generated by the detection unit 60 to perform a surgical task, such as making an incision 72 or closing an incision with sutures.

[0062] As shown in FIG. 6, the controller 18 may further include a detection unit 60 for detecting markers present in the image data 48 generated by the camera assembly 44. The controller 18 may also include a prediction unit 62 for analyzing the image data 48 to identify and / or predict selected types of images in the image data 48 by applying one or more known or custom artificial intelligence or machine learning (AI / ML) models or techniques to the image data. The prediction unit 62 may identify selected markers or anatomical structures in the image data based on the image data and generate insights and predictions therefrom. According to one practice, the AI / ML techniques employed by the prediction unit 62 may be supervised learning techniques (e.g., regression or classification techniques), unsupervised learning techniques (e.g., mining techniques, clustering techniques, and recommendation system techniques), semi-supervised techniques, self-learning techniques, or reinforcement learning techniques. Examples of suitable machine learning techniques include random forests, neural networks, clustering, XGBoost, bootstrap XGBoost, deep learning neural nets, decision trees, regression trees, and the like. The machine learning algorithms may also extend from the use of a single algorithm to the use of a combination of algorithms (e.g., ensemble techniques), and may use some of the existing methods of boosting algorithmic learning, bagging results to enhance learning, methods incorporating probabilistic and deterministic approaches, etc., to ensure that the machine learning is comprehensive and complete. The prediction unit 62 may be used to iteratively label portions of the image data generated by the camera assembly 44 that correspond to regions of interest, such as markers or specific anatomical structures, such as tissues, veins, organs, etc. Additionally, the prediction unit 62 may be trained on a set of training data to identify markers used by the surgeon or selected anatomical structures of the patient. The illustrated controller 18 may also include an image data storage unit 66 for storing image data 48 generated by the camera assembly 44, or image data 64 provided by a separate external data source.The external image data may include Magnetic Resonance Imaging (MRI) data, X-ray data, etc. The image data storage unit 66 may form part of the storage unit 24 or may be separate therefrom.

[0063] The controller 18 may also be configured to include a motion controller 68 for controlling the movement of the robotic unit, for example, by controlling or adjusting the movement of one or more robotic arms. The motion control unit may be configured to adjust the movement of the robotic unit based on markers detected in the image data and / or selected anatomical structures identified in the image data. The markers may be detected by the detection unit 60 and the anatomical structures may be identified by the prediction unit 62. As contemplated herein, the motion control unit may be configured to adjust the movement of the robotic unit by changing or modifying the movement speed of one or more of the robotic arms, such as by increasing or decreasing the movement speed. The motion control unit may also be configured to adjust the movement of the robotic unit by changing or modifying the torque of one or more of the robotic arms. The motion control unit may also be configured to limit, restrict, stop or prevent the movement of one or more robotic arms to one or more selected planes or one or more selected volumes.

[0064] The surgical robotic system may also be configured to perform selected surgical tasks manually (e.g., under full control of the surgeon), semi-autonomously (e.g., under partial manual control of the surgeon), or autonomously (e.g., fully automated surgical procedures). According to one practice of the present disclosure, as shown in FIG. 7D, markers may be placed within the patient's body, and then the markers may be utilized to guide the robotic unit to a selected surgical location under the control of the surgeon to perform a surgical task, such as tying one or more sutures at the identified location. According to another practice, the system 10 may be configured to provide semi-autonomous control, where the system 10 allows the surgeon to perform a manual surgical task with subsequent automated assistance or control. For example, FIG. 8A shows 8B, markers 80 may be applied around the incision 72, after which the surgeon may manipulate the robotic arm 42 to approach one of the markers 80A. The system 10 may be configured to store a preselected or predetermined threshold distance 98 around the marker 80 such that when the end effector region of the robot arm enters or comes within a threshold distance (e.g., less than the threshold distance), the system automatically generates a control signal 46 to operate the robot arm 42 and then the system automatically places or "snaps" the end effector region directly to the location of the marker. Specifically, image data 48 acquired by the camera assembly is processed by the detection unit 62 to identify the marker 80 and detect the location or proximity of the robot arm relative to the marker 80. The detected markings in the image data 48 may then be processed by the calculations 18 and compared to the threshold distance 98. When the robot arm 42 is within the threshold distance from the marker 80, such as marker 80A, the controller 18, via the motion controller 68, may generate a control signal 46 that is processed by the robotic unit 50 to coordinate the movement of the robotic unit.According to one practice, the motion controller 68 via the control signals 46 adjusts the movement of the robotic units such as by increasing the speed of movement of either or both of the robotic arms such that the robotic arms snap into a position such that the end effector region of the robotic arm 42 is located immediately adjacent to the marker 80A. Automatically positioning the robotic arm directly or immediately adjacent to the marker 80A ensures that the robotic arm is accurately positioned by the system 10 every time. The surgeon can then manually tie the stitch or suture. The threshold distance can be stored in any suitable location within the system 10, and is preferably stored within the controller 18, such as the memory unit 24 or the motion controller 68.

[0065] Alternatively, the surgical robotic system 10 may be operated in a fully automated mode in which the surgeon places markers at selected locations within the patient using the robotic unit. Once the markers are in place, the system may be configured to perform a predetermined surgical task. In this mode, image data 48 acquired by the camera assembly 48 may be processed by the detection unit 60 to detect the markers. Once the markers are detected, the motion controller 68, or alternatively, the controller, may be configured to generate control signals 46 for controlling or coordinating the movement of the robotic unit 50 and for automatically performing the selected surgical task with the robotic unit. This process allows the surgeon to plan the surgical procedure in advance and increase the probability that the robot will accurately follow the surgical plan in either the autonomous, semi-autonomous, and manual control modes. The various operating modes of the system 10 effectively allow the surgeon to maintain control of the robotic unit (i.e., decision-making and procedure planning) while simultaneously maximizing the benefits of the robotic unit's automated movement.

[0066] The present disclosure also contemplates the surgeon utilizing the robotic arm 42 to touch or contact selected points within the patient, and information associated with each contact location may be stored as a virtual marker that may be used by the controller 18 when controlling the movement of the robotic unit 50.

[0067] The surgical robotic system 10 of the present disclosure may also be configured to control or limit the movement or movement of the robotic unit 50 when positioned within a patient. The benefit of limiting or restricting the movement or movement of the robotic unit is to minimize the risk of accidental injury to the patient, for example, when manipulating the robotic unit during insertion into a cavity or while moving the robotic unit within the patient's abdomen, and while exchanging tools used by the robotic arm. To protect the patient from accidental and undetected off-camera injury, the system 10 may be configured to define a series of surgical zones, spaces, or volumes within the operating room. The predefined zones may be used to constrain or limit the movement of the robotic unit and may also be used to modify certain types of movements of the robotic unit, such as speed, torque, resolution of motion, volumetric limitations, etc., as needed.

[0068] The present disclosure is directed to systems and methods whereby a surgical robotic system can assist a surgeon in performing surgical tasks. The surgeon needs to be able to adapt to deformations in the patient's anatomy throughout the procedure. The anatomical deformations can make it difficult for the system to adapt and perform autonomous operations. A prediction unit 62 can be used to enable the surgeon to address the anatomical deformations in the patient. The prediction unit can identify selected anatomical structures from the image data. Data associated with the identified anatomical structures can be used by a controller to control the movement of the robotic unit.

[0069] As further shown in FIG. 6, the illustrated controller 18 can employ image data associated with the patient, as well as an image storage data unit 66 for storing the associated image data. The image data can include image data acquired by the camera assembly 44, as well as image data associated with other patients and acquired by other types of data acquisition techniques. For example, the patient image data acquisition techniques can include pre-stored image data associated with the patient, three-dimensional (3D) map information associated with the patient and the surgical environment or operating room, as well as MRI data, X-ray data, computed tomography (CT) data, and the like. The 3D map can be generated from a variety of different data generation techniques known in the art, including Light Detection and Ranging (LIDAR) techniques, stereoscopy techniques, image parallax techniques, computer vision techniques, and pre-operative or simultaneous 3D image techniques. The prediction unit 62 can be used to process and analyze the image data 48, and optionally process image data stored in the image data storage unit 66, to automatically identify selected types of anatomical structures, such as organs, veins, tissues, etc., that need to be protected from the robotic unit 50 during use. Thus, the prediction unit 62 may be configured or programmed (e.g., trained) to automatically identify anatomical structures and, in combination with the motion controller 68, define the types of motion controls to implement during the procedure. The present disclosure also contemplates the surgeon pre-operatively defining the types of motion restrictions to be implemented during the surgical procedure.

[0070] As described herein, during surgery, surgeons often need to adapt to changes in the patient's anatomy. Patient anatomical deformations often make it difficult for the system 10 to function properly in semi-autonomous and autonomous operating modes, and also to prevent accidental injury to the patient when operating the robotic unit in manual mode. Thus, to improve the overall effectiveness of the surgical procedure and to ensure that the system 10 operates reliably, the system may be configured to identify selected anatomical structures and then control or limit the movement of the robotic unit during the surgical procedure based on the identified structures. The camera assembly 44 may be used to capture image data of the interior of the patient's abdomen to identify selected anatomical structures.

[0071] According to the present disclosure, the motion controller 68 can generate and implement multiple different types of motion control. According to one embodiment, the motion controller 68 can confine the motion of the robotic unit to within a selected plane in a selected volume or space, while also selectively limiting one or more motion parameters of the robotic unit based on a selected patient volume or space, proximity to selected anatomical structures, and the like. The motion parameters may include range of motion, speed of movement in a selected direction, torque, and the like. The motion controller 68 can also exclude the robotic unit from entering a certain volume or space. The motion limits can be predefined and pre-established, or can be generated and changed in real-time based on image data acquired by the camera assembly during the surgical procedure.

[0072] According to one embodiment, the controller 18 can define a retraction plane based on the image data to restrict movement of the robotic unit to within the defined plane. For example, as shown in FIG. 9A, the controller 18, or the motion controller 68, can be used to define a selected retraction plane 110 within the patient's interior volume based on the image data 44. The robotic unit 50, and in particular the robotic arm 42, can be limited to movement within the retraction plane 110. Thus, even if the surgeon attempts to accidentally or intentionally move the robotic arm 42 into an area outside the retraction plane 110, the motion controller 68 prevents this type of movement from occurring.

[0073] The motion controller 68 may also be configured to define a constraint volume based on the image data and based on the output of the prediction unit 62 to constrain or limit the movement of the robotic unit when positioned within the specified volume. The prediction unit 62 may be configured to receive and process the image data 48, and optionally the external image data 64, to identify or predict a selected type of anatomical structure, such an organ. The predicted or identified data associated with the anatomical structure may then be processed by the motion controller 68 to define a selected constraint volume around the anatomical structure or the selected surgical site. According to one embodiment, for example, as shown in FIG. 9B, the prediction unit 62 identifies an organ 116, and the motion controller 68 defines or generates a constraint volume 114 around the organ 116. If the robotic unit is positioned outside or external to the constraint volume, the motion controller 68 does not impose a motion restriction on the robotic arm. However, as shown, when the robotic arm 42 enters the constraint volume 114, the motion controller 68 limits the selected type of movement of the robotic arm. According to one embodiment, the speed of movement of the robotic arm 42 is reduced by a selected predetermined amount. The slowing down of the robotic arm provides an indication to the surgeon that an organ is being approached. One skilled in the art will readily recognize that other types of motion limitations may also be used.

[0074] According to other embodiments, the controller 18 or motion controller 68 may be configured to eliminate or significantly reduce the movement capabilities of the robotic unit when the robotic unit cannot enter a defined space or is within a defined space or volume. The prediction unit 62 may be configured to receive and process the image data 48, and optionally the external image data 64, to identify or predict a selected type of anatomical structure, such as an organ or tissue. The predicted or identified anatomical structure data, such as data related to the organ 116, may be processed by the motion controller 68 to define a selected exclusion volume 120C around the organ 116. For example, in FIG. 9C, the motion controller 68 may also define additional exclusion zones or volumes, including exclusion volumes 120A and 120B, which may define other regions of the patient volume that are or need to be protected. The robotic arm 42 may be manipulated by the surgeon to perform a selected surgical task at the illustrated surgical site 118. The surgical site 118 may represent, by way of example only, a tear that needs to be surgically closed. According to one embodiment, the exclusion volumes 120A-120C may correspond to volumes that the robotic unit is prohibited from entering or penetrating, thus actively limiting the range of motion of the robotic unit and protecting the contents of the volume. According to one practice of the present disclosure, the motion controller 68 may be pre-configured to define one or more specific exclusion zones or volumes to protect vital organs or tissues that should not be contacted.

[0075] According to some embodiments, the motion controller 68 may be configured to limit the extent or range of motion of the robotic unit within a specified volume or zone. In some applications, instead of defining multiple exclusion volumes or zones, the surgeon may instead define an enclosed volume within which the robotic unit 50 may move freely. In the enclosed zone, the outer or external surroundings or perimeters cannot be penetrated by the robotic unit. The prediction unit 62 may be configured to receive and process the image data 48, and in some embodiments, the external image data 64, to identify or predict a selected type of anatomical structure, such as the organs 116A and 116B illustrated in FIG. 9D. The predicted or identified anatomical structure data, such as data related to the organs 116A, 116B, may be processed by the motion controller 68 to define a selected enclosed volume 130. The enclosed volume 130 may include, for example, the surgical site 118. The enclosed volume 130 may be configured to encompass the surgical site 118 while simultaneously avoiding or excluding the organs 116A and 116B, thus protecting the organs. The robotic arm 42 can be controlled by the surgeon to perform a selected surgical task at the illustrated surgical site 118 within the enclosed volume 130. While within the enclosed volume 130, the motion controller 68 is not configured to limit or constrain the movement of the robotic unit, such that the surgeon is free to control the robotic unit within the enclosed volume 130 without artificially limiting its speed and range of motion.

[0076] FIG. 11 illustrates a schematic of an exemplary motion control system for a surgical robotic system according to the teachings of the present disclosure. In some embodiments, control comes from a user interacting with a position control input, such as a hand controller, to provide workspace position commands to the motion control processing unit 302. The motion control processing unit 302 is configured or programmed to generate individual joint position commands to achieve workspace end effector positions. The motion control processing unit 302 may include a combination of circuitry and software to process the inputs and provide the described outputs. In systems with more than six degrees of freedom supporting end effector position control, some joint positions are not discrete values, but rather a range of possible values ​​across the range of remaining degrees of freedom.

[0077] The motion control processing unit 302 also provides the logic for selecting the optimal solution for all joints within the remaining degrees of freedom. Once optimized, the joint commands are executed by the motion control processing unit 302. The joint position feedback, after passing through forward kinematics processing, is returned to the motion control processing unit 302 to determine the end effector position error in the workspace.

[0078] A separate workspace mapping unit 310 is shown to illustrate the behavior of capturing the constraint surface. End effector workspace coordinates 312 are provided to the mapping unit for creation and storage of the workspace constraint surface or region. A marker position estimation engine 314 is included to accommodate changes to marker positions driven by changes in system configuration (e.g., trocar de-gassing), changes to visual marker positions (e.g., as a result of patient movement), or any other type of supported marker position change. A surface map management unit 316 supports user interaction with the mapping function to verify updated constraint points or resolved surfaces.

[0079] In response to the creation or modification of workspace constraints or operational violations during changes in operational performance, the video processing unit 318 overlays relevant information on the live image stream that may ultimately be rendered as a video before being presented to the user on the display unit 12. Task space constraints may include tissue surfaces (e.g., visceral beds) and / or depth tolerances, both of which are discussed in further detail below.

[0080] If any part of the system operating within the abdominal cavity exceeds the upper limit of the lower depth tolerance, for example, the visceral bed level, movement of that particular part of the system may not be permitted. This may essentially prevent all movement. Once this occurs, an override feature is described to ensure that the user is not restrained.

[0081] In some embodiments, the system is used in the patient's abdominal cavity, for example in the area near the intestine. While surface adhesions to other tissues of the intestine can be visualized, manipulated, and then surgically addressed as part of the treatment workflow, tissues deeper within the viscera have both normal connective tissue and / or potentially unexpected adhesions that cannot be easily visualized. If attachments provide a counterforce to forcibly displace resisting tissue, this can quickly lead to trauma. If components of the system 10 operate without visualization deeper below the visceral bed, there is increased concern of lateral tissue displacement causing trauma.

[0082] In abdominal surgery, insufflation provides intraperitoneal space above the viscera, thus creating a visceral bed. Aside from the benefit of allowing more space for visualization and access, the visceral bed is a somewhat arbitrary surface of interest. In ventral hernia repair, there is often a hernia sac located outside the abdominal cavity that protrudes through the hernia itself. Prior to reducing the hernia contents, a column of intestine and adipose tissue rises from the visceral bed to the hernia in the abdominal wall. In that scenario, it is useful to establish a circumferential surface that encapsulates the tissue column, protecting it from inadvertent and / or unvisualized contact.

[0083] Using the controller 18, the system 10 can define regions or zones of movement of the robotic unit and, conversely, regions or zones where movement is restricted. According to some embodiments, as shown, for example, in Fig. 10, the controller 18 can be configured to define tissue contact constraint data, e.g., a two-dimensional model such as a contraction surface 140, that corresponds to the location of one or more selected anatomical structures of the patient, e.g., tissue to be protected. The contraction surface 140 can be defined with a curvature.

[0084] In some embodiments, the controller may define a three-dimensional region or volume rather than a plane. The volume may be shaped as a cube, cone, cylinder, or other useful three-dimensional shape.

[0085] In some embodiments, the tissue constraint data includes a predetermined three-dimensional or two-dimensional shape associated with the surgical field, e.g., an insufflated abdominal or thoracic cavity. In this manner, the robotic system may have a predetermined retraction field in which to begin work and may be updated to reflect the patient's particular anatomy. In some embodiments, the tissue constraint data is calculated using either virtual or physical markers or by identifying portions of tissue as discussed herein relative to the retraction field or plane. In some embodiments, the predetermined tissue constraint data may be updated based on image data of the patient's surgical field or tissue identified within the surgical field.

[0086] The contraction plane 140 may be located directly on the physical tissue. For example, the contraction plane 140 may correspond to a defined floor, such as a visceral floor. In some embodiments, the plane 140 may be a specified distance above or below the tissue.

[0087] In some embodiments, surfaces of objects may be segmented according to their sensitivity to touch. For example, liver tissue present within an internal organ may be separately identified. Liver tissue is soft and fragile, and if damaged during surgery, it becomes particularly sensitive to touch, creating a potential safety risk.

[0088] A user may identify the retraction plane 140 with a first robotic arm prior to insertion of a subsequent robotic arm. Insertion of the second robotic arm may be monitored by camera assembly 44 while the first robotic arm remains off-screen. Because the retraction plane 140 is already defined, a user may be alerted if the off-screen robotic arm dips into plane 140.

[0089] The controller 18 may control the robotic arm 42 to reduce potential damage to tissue in areas identified by the tissue contact constraint data. The controller 18 is configured or programmed to determine areas or zones relative to the retraction surface 140 that are safe for manipulating the robotic arm 42 of the robotic unit. For example, the controller 18 may be configured or programmed to enable or permit movement of the robotic arm 42 on a first side 140A of the retraction surface 140 and inhibit or retract movement of the robotic arm on a second, opposing side 140B of the plane. The second side 140B corresponds to an area of ​​patient tissue of concern.

[0090] In some embodiments, the controller 18 is configured or programmed to determine a depth tolerance at which the robotic arm 42 can safely operate relative to the retraction surface 140. Depth tolerance is discussed in more detail in connection with FIG. 12 below.

[0091] 10, the contraction surface 140 may include a point having a vector 141 that is normal to the intended plane of Cartesian coordinates. The normal vector 141 may be configured to point to a side of the contraction surface 140 along which the elbow region 54B of the robot arm may move. Once the placement of the elbow region is calculated, it is adjusted to move away from the prohibited side of the contraction surface 140.

[0092] The elbow region 54B of the robotic arm 42 may be moved in a circular motion, as shown by the motion circle 144, according to one movement sequence. Once the virtual contraction plane 140 is defined by the controller 18 and corresponds to the anatomical structures that need to be protected, the elbow region 54B of the robotic arm 42 may be allowed to move, if desired, along a first arc portion 144A of the motion circle 144 located on a first side 140A of the contraction plane 140. This first arc portion 144A may be referred to as a safe direction of movement. For example, the controller calculates the safe direction to be "up" or away from gravity. In some embodiments, the elbow region 54B is prohibited from moving along a second arc portion 144B of the motion circle 44 located on a second side 140B of the contraction plane 140 to avoid contact with tissue. By preventing movement of the robotic arm, such as the elbow region 54B on the second side of the contraction plane 140, the patient's tissue is protected from accidental injury. In particular, multiple shrink surfaces may be combined to approximate more complex shapes.

[0093] In some embodiments, the user may redefine the retraction plane 140 after each robotic arm insertion. However, immediately after insertion is complete, the user may need to establish the viscera bed plane and depth tolerance before being presented with an indication that they are free to operate the system 10 or proceed at their own risk.

[0094] To prevent unacceptable unvisualized tissue contact by system 10 components, the user may define a boundary in the space where the acceptability of incidental tissue contact begins to change to unacceptable contact. In abdominally insufflated hernia repair procedures, there is no specific point, line, or plane that defines this boundary, but a continuous plane that approximates the visceral bed provides a useful model for controlling risk.

[0095] FIG. 12 is a representation of a tissue region identified by contact with a robotic arm, according to the teachings of the present disclosure. FIG. 13 is a flow chart depicting a process 200 for identifying a tissue region. In step S202, the system 10 may prompt the user to identify a portion of the tissue 148, for example, by contacting the end effector 52 or other distal end of the robotic arm 42 with the portion 148. For example, the user may touch the highest point within the abdomen to define a surface, for example, the visceral bed. In some embodiments, the user need not physically contact the tissue, but may point with the distal end of the robotic arm 42 and the portion 148. In some embodiments, the robotic arm 42 includes one or more tissue contact sensors at the distal end of the arm. The tissue contact sensors may be shaped to reduce damage to the tissue when contacting the tissue. Force sensors may also be included in the robotic arm 42 to measure unintended forces acting on the arm by contacting tissue.

[0096] Step S202 may be repeated one, two or multiple times to identify multiple portions 148 of tissue.

[0097] In an alternative embodiment, the tissue area may be identified using a single point laser range finder to define the horizontal plane. As another alternative, the tissue region may be identified using a single visual marker and calibrated optics to use a range of focal positions to find the point that creates the horizontal plane. As another alternative, the tissue region may be identified by manual angle definition around and relative to the gravity vector. An alternative embodiment involves using focal positions for range finding of multiple visual targets using calibrated targets and optics. An alternative embodiment involves the use of an integrated tissue contact sensor built into the instrument to define one or more points, as previously described.

[0098] In some embodiments, two portions 148 of tissue are identified by the robot arm 42. Both points may be placed on a defined contraction plane that allows for the inclusion of an angle. The rotation of the contraction plane about the line formed between the two portions 148 is further constrained by the gravity vector. The rotation of the contraction plane about the line used to define the plane is controlled by a secondary plane formed by the two portions on the line and the gravity vector. The contraction plane and the secondary plane must be perpendicular.

[0099] An alternative embodiment involves the use of a single visual marker of known shape and dimensions to estimate position and orientation based on an image of the marker by a single imager camera system with known optical properties. An example is an equilateral triangle cut from a thin but rigid material. Placing the rigid shape on top of the tissue aligns the shape with the tissue plane. Imaging the shape from a known position introduces some size variation and distortion. Given an optical system with known distortion properties, the image data can be processed to estimate the distance and orientation of the visual marker. This same approach can be used with a dual imager system and improved by taking advantage of visual differences.

[0100] The method continues in step S204 when the user prompts the system 10, for example, by pressing a button on the hand controller 17 or foot pedal assembly 19, manipulating a grasper of the robotic arm 42, or giving an audio cue to store the location of the identified portion of tissue 148 for purposes of defining a tissue region. The location may be stored in a memory of the controller 18. In some embodiments, the user identifies multiple portions of tissue 148, for example with the robotic arm 42, before the tissue region is defined. The user may prompt the system 10 after identifying each portion 148, or may prompt the system after identifying multiple portions in succession.

[0101] In step S206, the controller defines a contraction region based on the one or more identified portions of the tissue 148. As described above, the contraction region may be a three-dimensional volume or a two-dimensional plane. For example, the controller may define a plane that represents a visceral bed. In some embodiments, the controller defines tissue contact constraint data based on the one or more identified portions of the tissue 148. The tissue contact constraint data may include a contraction region or a plane, or may include a predetermined volume associated with the tissue site.

[0102] 14 is a flow chart depicting a process 300 for identifying a tissue region. In step S302, the system 10 may prompt the user to identify a portion of the tissue 148, for example, by identifying a marker disposed on the portion 148. The marker may be any marker described herein. Step S302 may be repeated once, twice, or multiple times to identify multiple portions 148 of the tissue. Alternatively, multiple portions 148 of the tissue may be marked with a marker and multiple markers may be identified at one time.

[0103] The method continues at step S304 when the user prompts the system 10, for example, by pressing a button on the hand controller 17 or foot pedal assembly 19, manipulating a grasper of the robotic arm 42, or giving an audio cue to store the location of the identified portion(s) of tissue 148 for purposes of defining a tissue region. The location may be stored in a memory of the controller 18.

[0104] In step S306, the controller defines a contraction region based on the one or more identified portions of the tissue 148. As described above, the contraction region may be a three-dimensional volume or a two-dimensional plane. For example, the controller may define a plane that represents the visceral bed.

[0105] In some embodiments, the system 10 projects an image of the shrink area onto an existing video feed provided to the user for evaluation or verification purposes.

[0106] The following examples use a defined visceral bed, but other anatomical elements are equally suitable if system 10 defines a plane or region and it is desirable to define a depth tolerance beyond the defined plane or region. In some embodiments, the depth tolerance error defines a tissue depth relative to the bed, where one or more constraints may be applied to control the robotic arm between the bed and the depth tolerance. FIG. 15 is a representation of a defined depth tolerance below the visceral bed, according to the teachings of the present disclosure. The likelihood of risky unvisualized tissue contact increases with depth below the surface approximating the visceral bed. Visceral tissue tends to be approximately self-level under the influence of gravity, but not completely, allowing for mounding, tilting, or capping. When referring to the visceral bed surface, the term "below" refers to a direction normal to the visceral bed surface, referring to the viscera, regardless of the orientation of the patient or system 10. The particular sensitivity to the degree of unvisualized tissue contact from system 10 components moving below the visceral bed is unique to each particular patient and is informed by the medical expertise and training of the user.

[0107] In some embodiments, the user control of the controller 18 of the arm 42 is reduced as the arm 42 moves through a defined visceral bed, contraction region, contraction surface, or other defined constraint. For example, the controller 18 may increase the constraint on the rate of movement of the arm 42 as the arm 42 moves through more of the defined visceral bed, contraction region, contraction surface, or other defined constraint. Additionally or alternatively, the controller 18 may increasingly decrease the torque of the arm 42 as the arm 42 moves through more of the defined visceral bed, contraction region, contraction surface, or other defined constraint.

[0108] The system 10 may also provide sensory feedback to the user when one or more of the arms 42 reach or cross a defined visceral bed. The sensory feedback may include a visual indicator on a display, an audio cue or alarm (i.e., a ring, bell, alarm, or spoken cue), and / or haptic, tactile feedback to the hand controller. Similar or different sensory feedback may be provided when one of the arms 42 reaches or crosses a defined depth tolerance.

[0109] In some embodiments, the controller 18 may be configured or programmed with a predefined depth tolerance at a retracted area or plane 140, e.g., a specified distance below a defined visceral bed. In some embodiments, the memory holds executable depth tolerance instructions to define the depth tolerance for the retracted area. In some embodiments, the user may determine an appropriate depth tolerance 146 below the defined visceral bed. In some embodiments, setting the depth tolerance 146 involves pressing a slider switch or one or more up or down buttons to navigate through a list of preprogrammed depth increments. Based on the patient's habits, the user may decide to adjust the depth tolerance 146 from its default value. For example, a patient with a higher BMI may have a thicker layer of adipose tissue on top of the viscera, so the user may increase the depth tolerance 146 to account for additional padding between the top surface and the more delicate structures.

[0110] The controller 18 may be configured or programmed with a default upper limit for the travel depth tolerance to eliminate the possibility of misuse where an unreasonable travel depth tolerance can be selected. For example, allowing a depth tolerance of 1 meter is not acceptable and serves to override the protection provided. In some embodiments, the upper limit for the travel depth tolerance is set at 2 centimeters to ensure a reasonable maximum travel limit below the visceral bed level such that accidental contact does not pose an unacceptable risk of harm to the patient. The upper limit may be 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 centimeters, or any distance in between. In some embodiments, the depth tolerance may be a negative value such that the depth tolerance is "above" the retracted region 140. For example, the upper limit may be -1, -2, -3, -4, -5, -6, -7, -8, -9, or -10 centimeters, or any distance in between.

[0111] In some embodiments, the user selects the depth tolerance by, for example, engaging a slider control input on a hand controller. In an alternative embodiment, the user may move the end effector away from a defined constriction area or surface by a distance that is used as the depth tolerance. In another alternative embodiment, the user may select from a set of existing standard depth tolerances based on the location of the surface constraint, the patient's orientation, the area of ​​the abdomen where the procedure is focused, or any such similar anatomically driven pre-definition.

[0112] Additionally or alternatively, to inhibit or constrict movement of the robot arm 42 on the second opposing side 140B of the plane, the system may provide one or more warnings to the user that the robot arm 42 is approaching or has entered the plane 140. For example, as shown in FIG. 16A, if the robot arm 42 is greater than a predetermined distance from the plane 140, the system 10 may provide a safety indication 150A to the user, for example, on a display, that the arm 42 is in a "safe" area relative to the plane 140. The safety indication 150A may include, for example, a green light. In some embodiments, the system 10 does not provide an indication to the user when the robot arm 42 is in the safe area.

[0113] 16B, the system 10 may provide a warning indication 150B to the user that the arm 42 is below the plane 140. The warning indication 150B may include, for example, a yellow light. The warning indication 150B may be triggered when the robot arm 42 is below the plane 140 but above the midpoint between the plane 140 and the depth tolerance 146, for example. In some embodiments, the system may reduce the speed of movement of the robot arm 42 when the robot arm 42 is below the plane 140 but above the midpoint between the plane 140 and the depth tolerance 146.

[0114] 16C, an additional warning indication is provided to the user when the arm 42 operates at or within approximately the midpoint of the depth tolerance 146 from the plane 140. Additional warning indication 150C. Additional warning indication 150C may include, for example, an orange light. In some embodiments, the system may further reduce the speed of movement of the robot arm 42 when the arm 42 operates at or within approximately the midpoint of the depth tolerance 146 from the plane 140.

[0115] As shown in FIG. 16D, the system 10 may provide a danger indication 150D to the user that the arm 42 is at or immediately adjacent to the depth tolerance 146. The danger indication 150D may include, for example, a red light. As another example, the plane 140 may be a defined visceral bed, as described above. The system 10 may provide the danger indication 150D when the robot arm 42 is at or immediately adjacent to the depth tolerance 146 defined by the user or the system 10. In some embodiments, movement of the robot arm 42 is prevented or stopped below the depth tolerance 146 when the danger indication 150D is provided.

[0116] In situations where the user determines the need to override the depth tolerance due to an acute problem requiring intervention, the user may engage a manual override. During the override, existing status indications may not be disabled, but may be modified to indicate that the system 10 is in an override state. When the override is no longer needed, the user may not need to manually release the override. For example, if the user overrides the limits of motion below the depth tolerance error and then returns the arm to within the previously established depth tolerance limit, the override may be automatically canceled.

[0117] The above-mentioned indications may be provided in the form of haptic feedback. For example, one or more hand controllers 17 may vibrate when one of the robotic arms 42 contacts, passes through, or comes within a predetermined threshold of the contraction surface 140. The vibrations may increase in intensity as one of the arms 42 retracts closer to the contraction surface 140.

[0118] The surgical robotic system 10 of the present disclosure may also be configured to control or limit the movement or travel of the robot unit 50 relative to the retraction region 140 or depth tolerance 146. For example, the system 10 may prevent or stop the robot arm from moving through the retraction region 140. In some embodiments, the system 10 allows movement of the arm 42 along the virtual retraction region 140, especially when the region 140 is located a distance from the tissue. The motion control processing unit may assign an increased cost to a joint position because that particular joint operates near the depth tolerance. This provides a preventative adjustment to reduce the utilized depth tolerance 146.

[0119] The user may redefine an already established virtual contraction plane 140. For example, during operation, the user may change the virtual center location (i.e., the "out-gassing" of the trocar forming the patient port), which requires an adjustment to the relative position of the user-defined visceral bed surface. The relative positions of the surfaces must be adjusted to account for the corresponding movement of the instruments and camera relative to the visceral bed. To do so, the user prompts the system 10, for example, by pressing a button or giving an audio cue, to define a new virtual contraction plane 140. The user may then proceed to define a new plane using the markers or end effectors described above. The plane 140 may need to be redefined if the patient moves or is moved, or if the robotic arm is located in a new direction or in a new area. In some embodiments, the system 10 may automatically recalculate the plane 140 when the robotic arm is located in a new direction or in a new area.

[0120] In an alternative embodiment, the system 10 employs complex surface definitions utilizing DICOM format CT or MRI images to define surface boundaries based on differences in tissue density and type. This type of information would likely need to be obtained from intraoperative imaging due to differences in insufflated abdomens. As another alternative, the system 10 may utilize the geometry of the instrument arm itself, positioned and selected by the user, to define a set of raylines that are raised together to define the interface. As another alternative, the system 10 uses visual parallax to generate or map a 3D point cloud onto the surface of the existing tissue. The use of simultaneous localization and mapping (SLAM) algorithms to accomplish this mapping is a well-known technique. As another alternative, the system 10 uses point or array LIDAR data accumulated over time to build a surface map from range data relative to a system coordinate frame. As another alternative, the system 10 uses multiple visual markers of known shape and size placed at various locations on the tissue surface to determine the distance, location, and orientation of points along that surface. This embodiment uses the same camera system characterization as the single visual marker embodiment for a single plane definition.

[0121] In an alternative embodiment, the system 10 employs customization of the surface constraints at specific locations, employing a user interface to select localized regions of the constraining surface to define a smaller depth tolerance than the remainder of the constraining surface. As another alternative, the system 10 uses fluorescent dyes and / or imaging to define areas of high perfusion where the depth tolerance is reduced.

[0122] In an alternative embodiment, the system 10 provides dead-reckoning sensing relative to the constraining surface plane using visual markers. Monitoring the location of the dead-reckoning visual markers determines if the constraining surface has moved. As another alternative, the system 10 monitors the insufflation pressure to determine when the internal organs are likely to have moved. As another alternative, the system 10 uses a specific location estimation sensor placed over the patient's anatomy where the contraction area is defined. As this location estimation sensor moves, the contraction area also moves. Location estimation can be accomplished in a number of ways, including electromagnetic pulse detection.

[0123] In an alternative embodiment, the system 10 employs sensor fusion of internal robot control feedback (current monitoring, proximity sensor fusion, etc.) in proximity to the contraction region. Feedback from the system can be used to modify the interpretation of the motion relative to the contraction region.

[0124] In some embodiments, the controller 18 limits lateral (i.e., parallel to the contraction plane) movement in proportion to how far the robot or camera has penetrated through the contraction region toward the depth tolerance. In another alternative embodiment, the controller 18 utilizes a workspace cost function to minimize the depth tolerance utilized by any given joint.

[0125] The many features and advantages of the present disclosure will be apparent from the detailed specification, and it is, therefore, intended by the appended claims to cover all such features and advantages of the present disclosure that fall within the true spirit and scope of the present disclosure. Moreover, because numerous modifications and variations will readily occur to those skilled in the art, it is not desired to limit the disclosure to the exact construction and operation illustrated and described, and therefore, all suitable modifications and equivalents may be resorted to within the scope of the present disclosure.

Claims

1. 1. A surgical robotic system, comprising: a robot unit having a robot arm; a camera assembly for generating an anatomy view of the patient; a memory containing executable instructions for controlling the robotic arm; A controller configured or programmed to execute instructions retained in the memory, the memory comprising: receiving tissue contact constraint data; and a controller that controls the robotic arm to reduce potential damage to tissue in areas identified by the tissue contact constraint data; a display unit configured to display a view of the anatomical structure.

2. 10. The surgical robotic system of claim 1, wherein the controller executes instructions retained in memory to define a floor relative to the tissue based on the contraction constraint data, the floor defining an area or region in which the robotic arm can operate with minimal risk of damage to the tissue.

3. 3. The surgical robotic system of claim 2, wherein the controller executes instructions retained in memory to define a depth tolerance relative to the bed based on the tissue shrinkage constraint data, the depth tolerance defining a tissue depth relative to the bed at which one or more constraints may be applied to control the robot arm when the robot arm is between the floor and the depth tolerance.

4. the memory holds executable instructions for stopping movement of one of the robotic arms when the robotic arms pass the depth tolerance; 4. The surgical robotic system of claim 3, wherein the controller executes the instructions to stop movement of the robotic arms beyond the depth tolerance when the one of the robotic arms passes the depth tolerance.

5. 10. The surgical robotic system of claim 1, wherein the controller is configured or programmed to define a contraction region based on the tissue contact constraint data and a portion of an anatomical structure identified at a distal end of at least one of the robotic arms.

6. 6. The surgical robot system of claim 5, wherein the display unit is configured to display an indicator when one of the robotic arms enters a space beyond the retraction area.

7. 6. The surgical robot system of claim 5, wherein the memory holds executable depth tolerance instructions to define a depth tolerance for the constriction region.

8. 8. The surgical robotic system of claim 7, further comprising a display unit, the display unit configured to display an indicator when the one of the robotic arms enters a region between the retraction region and the depth tolerance.

9. 8. The surgical robotic system of claim 7, wherein the display unit is configured to display an indicator when the one of the robotic arms reaches the depth tolerance.

10. 6. The surgical robotic system of claim 5, wherein the controller is configured or programmed to execute instructions retained in the memory to restrict movement of the robotic arm into the retraction region.

11. 8. The surgical robotic system of claim 7, wherein the controller is configured or programmed to execute instructions retained in the memory to stop movement of the robotic arm beyond a depth tolerance defined for the retraction region.

12. 6. The surgical robotic system of claim 5, wherein the contraction area is represented by a plane and the controller is configured to be programmed to stop movement of the robotic arm on one side of the plane.

13. 1. A method for controlling a position of one or more robotic arms in a constrained space, comprising: receiving tissue contact constraint data; defining a region identified by said tissue contact constraint data; and controlling the one or more robotic arms to reduce potential damage to tissue within the area defined by the tissue contact constraint data.

14. Identifying a portion of an anatomical structure at a distal end of one of the robotic arms; and defining a contraction region based on the tissue contact constraint data and the identified portion of the anatomical structure.

15. The method of claim 14 , further comprising displaying an indicator on a display unit when one of the robotic arms enters a space beyond the retracted area.

16. 14. The method of claim 13, further comprising defining a floor for the tissue within the region identified by the tissue contact constraint data, the floor defining an area or region within which the one or more robotic arms can operate with minimal risk of damage to the tissue.

17. 17. The method of claim 16, further comprising defining a depth tolerance relative to the floor based on the tissue shrinkage constraint data, the depth tolerance defining a tissue depth relative to the floor at which one or more constraints may be applied to control the one or more robotic arms when the robotic arms are between the floor and the depth tolerance.

18. The method of claim 17 , further comprising stopping movement of the one or more robotic arms when the one or more robotic arms extend beyond the depth tolerance.

19. The method of claim 17 , further comprising displaying an indicator on a display unit when one of the robotic arms reaches the depth tolerance.

20. The method of claim 17 , further comprising slowing movement of the robot arm when the robot arm is between the depth tolerance and the retracted region.

21. The method of claim 17 , further comprising stopping movement of the robotic arm when the robotic arm extends beyond the depth tolerance.

22. 14. The method of claim 13, wherein the contraction region is represented by a plane, and controlling the one or more robotic arms to reduce potential damage to tissue within the region defined by the tissue contact constraint data includes stopping movement of the one or more robotic arms on one side of the plane.

23. 1. A surgical robotic system, comprising: a robotic arm assembly having a robotic arm; a camera assembly that generates image data of an internal region of a patient; and A controller, Detecting one or more markers in the image data; Controlling movement of the robotic arm based on the one or more markers in the image data; The system of claim 1 , comprising: a controller configured or programmed to store the image data.

24. 24. The surgical robotic system of claim 23, wherein the controller is configured or programmed to control the robotic arm to place the one or more markers.

25. 24. The surgical robotic system of claim 23, wherein the marker comprises an X-shape, a quick response (QR) code marking, a reflective tape, a reflective film, a sticker, a fabric, a staple, a tack, an LED object, or an emitter.

26. 24. The surgical robotic system of claim 23, wherein the controller is programmed to define a threshold distance for the one or more markers or to define a threshold distance for the one or more markers and to alter the rate of movement of at least one of the robotic arms when one of the robotic arms is positioned relative to the one or more markers at a distance less than the threshold distance, such that one of the robotic arms is automatically positioned adjacent to the one or more markers.

Images