Operating room systems, devices, and methods of use

Abstract

Described here are systems, devices, and methods useful for minimally invasive surgical procedures such as using a robotic surgical system. Methods of monitoring a surgical procedure may comprise generating one or more of support arm data, end effector data, and patient data using one or more sensors of a robotic surgical system, identifying a surgical procedure event based on one or more of the support arm data, the end effector data, and the patient data, and estimating a surgical procedure chronology based on the identified surgical procedure event and surgeon data.

Description

OPERATING ROOM SYSTEMS, DEVICES, AND METHODS OF USECROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 664,570, filed on June 26, 2025, which is incorporated by reference herein in its entirety for all purposes.TECHNICAL FIELD

[0002] Devices, systems, and methods herein relate to monitoring and managing minimally invasive procedures using a robotic surgical system.BACKGROUND

[0003] Many surgical procedures are shifting toward the use of minimally invasive approaches in order to minimize the number and size of incisions that are made in a patient. Minimally invasive procedures such as endoscopic, laparoscopic, and thoracoscopic procedures may be associated with lower pain, quicker post-surgical recovery, shortened hospitalization, and reduced complications when compared to open surgical procedures. However, robotic surgical systems may be capital intensive and impractical for smaller medical facilities. For example, conventional minimally invasive robotic surgery is performed by two skilled surgeons (e.g., operators), which may limit the number of surgeries performed per day. Accordingly, it may be desirable to provide a robotic surgical system and method of monitoring and / or managing a robotic surgery system to improve patient outcomes with increased operating efficiency.SUMMARY

[0004] Described here are systems, devices, and methods useful for minimally invasive surgical procedures. For example, described herein are methods of monitoring a surgical procedure to estimate a surgical procedure chronology. In some variations, a method of monitoring a surgical procedure may comprise generating one or more of support arm data, end effector data, and patient data using one or more sensors of a robotic surgical system, identifying a surgical procedure event based on one or more of the support arm data, the end effector data, and the patient data, andestimating a surgical procedure chronology based on the identified surgical procedure event and surgeon data. In some variations, the surgical procedure chronology may comprise a sequence of surgical procedure events. In some variations, the surgical procedure event may comprise one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish.

[0005] In some variations, the support arm data may comprise one or more support arm movement data, support arm position data, support arm orientation data, and support arm configuration data. In some variations, the end effector data may comprise one or more of end effector movement data, end effector position data, end effector orientation data, and end effector configuration data. In some variations, the patient data may comprise one or more of health data and patient procedure data. In some variations, the surgeon data may comprise one or more of surgeon position data and historic surgical procedure data of one or more surgeons. In some variations, the one or more sensors may comprise one or more of a motion sensor, pressure sensor, optical sensor, audio sensor, patient sensor, magnetic sensor, and proximity sensor.

[0006] In some variations, the surgical procedure chronology may be compared to the historic surgical procedure data of one or more surgeons.

[0007] In some variations, a communication channel may be established between the one or more sensors and a controller of the robotic surgical system. In some variations, the controller may be in a same room or a different room as the patient. In some variations, the controller may be configured to control one or more of end effector movement and end effector orientation.

[0008] In some variations, a notification may be generated corresponding to the identified surgical procedure event. In some variations, the surgical procedure event may be identified based on the surgeon data comprising audio data.

[0009] Also described herein are methods of managing a surgical procedure. In some variations, a method of managing a surgical procedure may comprise generating data for a first surgical procedure using one or more sensors of a robotic surgical system, estimating a first surgical procedure chronology based on the first surgical procedure data, data for a first patient, and data fora surgeon data, and scheduling a second surgical procedure using a robotic surgical system based on the estimated first surgical procedure chronology, data for a second patient, and the surgeon data. In some variations, the first surgical procedure may be performed in a first operating room and the second surgical procedure may be performed in a second operating room. In some variations, the first surgical procedure may be started before the second surgical procedure. In some variations, one or more surgical procedure events of the second surgical procedure may be performed prior to completion of the first surgical procedure. In some variations, the data for the first surgical procedure may comprise one or more of movement data, position data, orientation data, and configuration data of one or more of a support arm and an end effector of the robotic surgical system. In some variations, the data for one or both of the first and second patients may comprise health data and patient procedure data. In some variations, the surgeon data may comprise one or more of surgeon position data and historic surgical procedure data of one or more surgeons.

[0010] In some variations, the first surgical procedure chronology may comprise a sequence of surgical procedure events. In some variations, the sequence of surgical procedure events may comprise one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish.

[0011] In some variations, a notification may be generated corresponding to one or more surgical procedure events of the sequence of surgical procedure events. In some variations, a notification may be generated corresponding to the scheduled second surgical procedure.

[0012] Also described herein is a robotic surgical system. In some variations, a robotic surgical system may comprise a support arm sensor configured to generate support arm data corresponding to one or more support arm characteristics, an end effector sensor configured to generate end effector data corresponding to one more end effector characteristics, a patient sensor configured to generate patient data corresponding to one or more patient characteristics, a controller coupled to each of the sensors, the controller comprising a processor and memory, the controller configured to identify a surgical procedure event based on one or more of the support arm data, end effector data, and patient data, and estimate a surgical procedure chronology based on the identified surgical procedure event and surgeon data, the surgical procedure chronology comprises a sequence ofsurgical procedure events. In some variations, the surgical procedure event may comprise one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish. In some variations, the support arm characteristics may comprise one or more of support arm movement data, support arm position data, support arm orientation data, and support arm configuration data. In some variations, the end effector characteristics may comprise one or more of end effector movement data, end effector position data, end effector orientation data, and end effector configuration data. In some variations, the patient characteristics may comprise one or more of health data and patient procedure data. In some variations, the surgeon data may comprise one or more of surgeon position data and historic surgical procedure data of one or more surgeons. In some variations, the patient data may comprise imaging data. In some variations, the end effector sensor may be configured to measure one or more fiducials of the end effector. In some variations, the controller may be in either a same or a different room as the patient.

[0013] In some variations, the end effector may comprise one or more magnetic portions. In some variations, the one or more magnetic portions may comprise an external magnet configured to be placed over or on skin of a patient. In some variations, the external magnet may be coupled to a support arm.

[0014] In some variations, the processor may be configured to establish a communication channel between one or more of the support arm sensor, the end effector sensor, and the patient sensor. In some variations, the communication channel may be a wired or wireless communication channel.

[0015] Also described here are methods of operating a robotic surgery system. In some variations, a method of operating a robotic surgery system may comprise estimating one or more parameters of the robotic surgery system. A real-time image of a field of view an operator may be generated using one or more of an optical sensor coupled to the operator and a visualization device. An image of a field of view of an operator may be provided via a display coupled to an input device. The image may comprise at least the robotic surgery system comprising one or more of a support arm, an end effector, and an external magnet, and at least one reference marker overlaid onto or adjacent to therobotic surgery system. The reference marker may comprise one or more of the estimated parameters of the robotic surgery system.

[0016] In some variations, a gaze of the operator may be measured using the input device. One or more of the support arm, the end effector, and the external magnet may be controlled based on the measured gaze of the operator.

[0017] In some variations, the reference marker may comprise one or more of a symbol, image, or text. In some variations, one or more parameters of the robotic surgery system may comprise one or more support arm data, end effector data, and external magnet data. In some variations, the support arm parameters may comprise one or more of kinematics, workspace, dynamics, and a state machine. In some variations, the end effector parameters may comprise one or more of kinematics, workspace, dynamics, and state machine. In some variations, the external magnet parameters may comprise one or more of magnitude and direction of magnetic force.

[0018] In some variations, estimating one or more parameters of a patient may comprise one or more of patient data, a surgical procedure event, a surgical procedure chronology, and a surgical parameter, and wherein the at least one reference marker comprises the one or more parameters of the patient. In some variations, the surgical procedure chronology may comprise a sequence of surgical procedure events. In some variations, the surgical procedure event may comprise one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish. In some variations, the patient data may comprise one or more of health data and patient procedure data.

[0019] In some variations, the support arm data may comprise one or more of support arm movement data, support arm position data, support arm orientation data, and support arm configuration data. In some variations, the end effector data may comprise one or more of end effector movement data, end effector position data, end effector orientation data, and end effector configuration data.

[0020] In some variations, providing the image may comprise tracking one or more of the support arm, the end effector, and the external magnet. The reference marker may be overlayed onto or adjacent one or more of the support arm, the end effector, and the external magnet in real-time.

[0021] In some variations, a plurality of virtual actuators may be provided within the image, each configured to respond to the gaze of the operator to generate one or more control signals. In some variations, the display and the input device may be each configured to be worn over an eye of an of the operator. In some variations, the input device may comprise one or more of a headset, goggles, glasses, and a contact lens. In some variations, the end effector may comprise one or more of a visualization device, a grasper, a retractor, a magnetic positioning device, a sensor, an intracavity device, a delivery device, a retrieval device, a stapler, a clip applier, and an electrocautery hook.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG. 1 is a block diagram of an illustrative variation of a robotic surgical system.

[0023] FIG. 2 depicts a flowchart representation of an illustrative variation of a method of monitoring a surgical procedure.

[0024] FIG. 3 depicts a flowchart representation of an illustrative variation of a method of managing a surgical procedure.

[0025] FIG. 4 depicts a flowchart representation of an illustrative variation of a method of managing a plurality of surgical procedures.

[0026] FIG. 5A is a schematic side view of an illustrative variation of a robotic surgical system in a storage configuration. FIG. 5B is a schematic side view of an illustrative variation of a robotic surgical system in a deployed configuration.

[0027] FIG. 6 is a schematic side view of an illustrative variation of a robotic surgical system.

[0028] FIGS. 7A and 7B depict schematic views of an illustrative variation of end effector visualization.

[0029] FIGS. 8A and 8B depict additional illustrative variations of end effector visualization.

[0030] FIG. 9 is a schematic cross-sectional view of yet another illustrative variation of end effector visualization.

[0031] FIG. 10A and 10B provide schematic views of an illustrative variation of robotic surgical system communication.

[0032] FIGS. 11 A and 1 IB are graphical user interfaces of an illustrative variation of a method of managing a plurality of surgical procedures.

[0033] FIGS. 12A-12C are graphical user interfaces of an illustrative variation of a method of monitoring a surgical procedure.DETAILED DESCRIPTION

[0034] Described here are systems, devices, and methods for use in minimally invasive surgical procedures such as reduced-port laparoscopic surgical procedures. Conventional laparoscopic surgical procedures may include a first operator performing one or more surgical steps using one or more end effectors and a second operator controlling visualization using an endoscope. In some variations, the robotic surgical systems described herein may assist the operator in performing a minimally invasive surgical procedure. For example, the robotic surgical system may comprise one or more support arms configured to control a corresponding end effector under the direction of the operator. Furthermore, the robotic surgical system may provide visualization without the conventional need for an additional assistant. In some variations, the systems described herein may be a magnetic robotic surgical system. For example, the system may comprise an external magnet placed over the patient and configured to control a detachable end effector (e.g., grasper) that enables grasper movement and control without the constraints of a fixed-position pivot point.

[0035] As described in detail herein, the systems, devices, and methods described herein may improve minimally invasive robotic surgery by: facilitating visualization, tracking, and identification during a robotic surgical procedure; estimating a surgical chronology (e.g., set of milestone events in a surgical procedure) in real-time for resource management (e.g., scheduling toincrease surgical procedure throughput); operating and / or managing a robotic surgical system with a minimum number of operators (e.g., surgeons) and support personnel; facilitating communication and collaboration among operators and / or robotic surgical systems; monitoring the performance of the operator and / or support personnel; enhancing visualization (e.g., virtual reality, augmented reality, extended reality) of a robotic surgical procedure; and ensuring patient safety.

[0036] In some variations, one or more sensors of the system may generate sensor data, support arm data, and end effector data corresponding to movement, position, orientation, and configuration to identify a surgical procedure event and / or track the system. Sensor data may include imaging data (e.g., generated by an endoscope) provided to an operator to facilitate visualization of the surgical procedure by the surgeon directly beside the patient without additional personnel. Thus, the systems, devices, and methods described herein may improve visualization, surgeon control of instruments, and operating room efficiency.

[0037] In some variations, the real-time progress of a surgical procedure may be monitored and used to estimate (e.g., extrapolate) a chronological timeline of the remaining steps of the surgical procedure. For example, an estimated duration of a particular event (e.g., step) in a surgical procedure may be based on a set of previous performances by the surgeon (and / or set of surgeons) performing the procedure.

[0038] In some variations, an estimated surgical procedure chronology for a robotic surgery procedure may in part be the basis for scheduling one or more additional surgical procedures using the robotic surgical system. For example, a second patient may enter an operating room and undergo one or more pre-surgery preparation steps when a first patient reaches a predetermined step (e.g., before the surgical procedure is completed). The surgeon may thus be able to transition from the first patient to the second patient with a minimum of downtime. Accordingly, the robotic surgical system and operator may be efficiently allocated to increase the number of patients treated in a given time period. For example, the systems and methods described herein may increase the rate of ambulatory or same-day discharge abdominal surgeries performed at a surgical facility.

[0039] In some variations, safety monitoring may include tracking the system relative to the patient using the sensors. In some variations, the robotic surgical system may ensure patient safetyby locating and monitoring a position and orientation of an end effector relative to a trocar, and monitoring one or more of a position of an external end effector relative to the patient and an end effector attachment status. In this manner, the robotic surgical system, the patient, and the procedure may be monitored.

[0040] Furthermore, single operator control of one or more robotic arms and end effectors may be facilitated by freeing the hands and visual attention of the operator to be elsewhere. For example, an input device configured to receive operator input to control one or more elements of the robotic surgery systems herein. For example, the input device (e.g., a gaze-actuated input / output device) may be communicably coupled to one or more robotic arms and / or one or more end effectors to control, for example, movement of the robotic arm(s) and / or end effector(s). In some variations, the input device may be configured to receive input from one or more body parts of an operator (e.g., hand, arm, foot, leg, head, eyes, etc.). In some variations, the input device (e.g., controller) may be configured to be worn over an eye of an operator such that both hands of the operator may remain free for other purposes.

[0041] Moreover, the systems herein may include augmented reality (AR) or virtual reality (VR) devices(s) that can aid an operator during a surgical procedure by providing enhanced views of one or more of a surgical space (including within a body cavity of a patient) including labels and / or information about the space, procedure status, patient status (e.g., organ parameters), support arm status, and end effector status (e.g., magnetic forces), thereby improving accessibility and efficiency.

[0042] Some of the surgery systems described herein may be used to perform surgical procedures such as a cholecystectomy, appendectomy, colectomy, hernia repair, sleeve gastrectomy or other bariatric procedures, nephrectomy, hysterectomy, oophorectomy, and lobectomy.I. Systems and Devices

[0043] Generally, the robotic surgical systems described herein may be operated by a single operator. A block diagram of an exemplary robotic surgical system 100 is depicted in FIG. 1. The system 100 may comprise one or more of a support arm 112, a patient platform 114, a supportstructure 116, an end effector 118, a sensor 120, an input device 122, a processor 124, a memory 126, a communication device 128, an output device 130, and an optional external magnet 132, each of which are described in more detail herein. In some variations, the support arm 112 may be configured to moveably suspend and hold an end effector 118 relative to a patient (e.g., patient on a patient platform) based on control of a single operator (e.g., control inputs, manual manipulation). In some variations, a patient may be disposed on the patient platform 114. The support arm(s) 112 and support structure 116 may be disposed around the patient platform 114 such that an operator (e.g., surgeon) may operate directly beside the patient on the patient platform 114. For example, the operator may optionally manually manipulate an end effector 118 disposed within the patient on the patient platform 114 while standing on a side of the patient platform 114.

[0044] In some variations, the support structure 116 may be configured to support (e.g., hold) one or more of the support arm(s) 112, end effector 118, sensor 120, input device, processor 124, memory 126, communication device 128, output device 130, and external magnet 132. In some variations, the support structure 116 may be moveable (e.g., mobile) relative to the patient platform 114, which may allow the support arms 112 to be positioned as desired by an operator relative to the patient platform 114.

[0045] In some variations, the end effector 118 may comprise one or more end effectors used in a surgical procedure. Optionally, the end effector 118 may comprise an external magnet 132 configured to be placed over or on a skin of a patient.

[0046] In some variations, the sensor 120 may comprise one or more sensors configured to measure one or more characteristics corresponding to one or more of the patient, the operator, and surgery system 100 including, but not limited to, the support arm 112, the patient platform 114, support structure 116, and the end effector 118.

[0047] In some variations, the input device 122 may be configured to generate an input signal based on operator input. The input signal may be configured to control one or more components of the robotic surgical system 100 such as, for example, the support arm 112, end effector 118, and communication device 128. In some variations, the processor 124 and memory 126 may be configured to control the surgery system 100. In some variations, the communication device 128may be configured to communicate with one or more components of the system 100 as well as with networks and other computer systems. In some variations, the output device 130 may be configured to output data corresponding to a surgery system 100.A. Support arm

[0048] The surgery systems described herein may comprise one or more support arms. Generally, one or more end effectors may be releasably coupled to a support arm where the support arm may be configured to moveably suspend the end effector so as to move and hold the end effector at a desired location. With the end effector suspended or held at a desired location by the support arm, an operator and / or controller may move at least a portion of the end effector externally of a patient. The support arm may be, for example, an articulated robotic arm, SCARA robotic arm, and / or linear robotic arm. The support arm may comprise one or more segments coupled together by a joint (e.g., shoulder, elbow, wrist). In some variations, the support arm may have six or more degrees of freedom. The set of Cartesian degrees of freedom may be represented by three translational (position) variables (e.g., surge, heave, sway) and by the three rotational (orientation) variables (e.g., roll, pitch, yaw). In some variations, the support arm may have less than six degrees of freedom. The support arm may be configured to move over all areas of a patient body in up to three dimensions and may also maintain the end effector at an orientation perpendicular to a surface of the patient. The support arm may comprise one or more motors configured to translate and / or rotate the joints and move the support arm to a desired location and orientation. In some variations, the position of the support arm may be temporarily locked to fix the position of the end effector. The support arm may be mounted to any suitable object, such as a medical cart, furniture (e.g., a bed rail), a wall, a ceiling, or may be self-standing. Additionally or alternatively, the support arm may be configured to be moved manually by, for example, a single operator without the assistance of a second person. The support arm may be configured to carry a payload comprising the support arm, end effector, and any tissue coupled to the end effector (e.g., a gallbladder held by a grasper). In some variations, the support arm may be mounted to any suitable object, such as a support structure, medical cart, furniture (e.g., a bed rail), a wall, a ceiling, or may be self-standing.B. Patient platform

[0049] Generally, a robotic surgical system may comprise a patient platform configured to support a patient disposed thereon. For example, the patient may lay on their back on top of the patient platform. In some variations, a patient platform may provide a plurality of degrees of freedom to move a patient on a patient platform to a desired position and orientation. In some variations, a patient platform may be configured to be adjustable with a plurality of degrees of freedom to position a patient and facilitate patient treatment. For example, the patient platform may comprise one or more of a yaw drive system, an axial drive system, and a vertical drive system. The yaw drive system may be configured to yaw the patient platform about a pivot point. The vertical drive system may be configured to control a height and / or pitch of the patient platform. The axial drive system may be configured to translate the patient platform.

[0050] In some variations, an operator may be located on a first side of a patient platform during a surgical procedure. In some variations, a support structure 116 (e.g., base) coupled to a support arm 112 may be located on the ground and along a second side of a patient platform adjacent the first side of the patient platform in order to maximize a range of the support arm.C. Support structure

[0051] Generally, a robotic surgical system may comprise a support structure configured to support (e.g., couple to, hold) a support arm. For example, the support structure 116 may be configured to position a support arm 112 (and corresponding end effectors 118) relative to a patient platform 114, patient, and operator. Therefore, the support structure 116 may function as a base for a set of robotic arms used in a surgical procedure. In some variations, the support structure 116 may comprise a housing configured to enclose one or more of a sensor 120, input device 122, processor 124, memory 126, communication device 128, and output device 130. Additionally or alternatively, the support structure 116 may comprise medical equipment such as a medical cart. In some variations, the support structure 116 may comprise a power source and / or power connector configured to power one or more of the support arm 112, end effector 118, sensor 120, input device 122, processor 124, memory 126, communication device 128, output device 130, and external magnet 132.

[0052] In some variations, the support structure 116 may comprise existing structures such as furniture (e.g., table, desk, chair), building structures (e.g., doors, walls, ceilings), and the like. For example, the support structure 116 may be configured to releasably detach from the patient platform 114 and may be portable. For example, the support structure 116 may be configured to move within an operating room (e.g., relative to the patient platform 114 and / or operator). Additionally or alternatively, the support structure may be configured to be portable to operate in a plurality of operating rooms. In some variations, the support structure 116 may comprise a set of wheels. In some variations, the support structure 116 may comprise a handle to facilitate mobility and portability. Additionally or alternatively, the support structure may comprise one or more wall- mounted rails. In some variations, the support structure may flexibly transition between different configurations as described in more detail with respect to FIGS. 5A and 5B.

[0053] FIG. 6 is a schematic side view of a support structure 600 comprising a set of wheels 602 and coupled to a support arm 610. In some variations, the support structure 600 may be locked to a particular location using one or more suitable locking features such as a brake (not shown). The support structure 600 may be coupled to an input device 620 (e.g., camera) and an output device 630 (e.g., speaker). The support arm 610 may be coupled to another input device 622 (e.g., microphone), and controller 640 (e.g., processor, memory). An operator 650 may interact with one or more of the input devices 620, 622 and output device 630. For example, the operator 650 may communicate with a remote surgeon (not shown) using the microphone 622 and speaker 630. The surgeon 650 and / or remote surgeon may view images of a surgical procedure generated by camera 620.D. End effector

[0054] Generally, the end effectors described herein are not particularly limited and may comprise one or more of a visualization device, a grasper, a retractor, a magnetic positioning device, a sensor, an intracavity device, a delivery device, a stapler, a clip applier, an electrocautery hook, and other surgical instruments that may be advanced in a minimally invasive manner through an access site.

[0055] In some variations, an end effector may be configured to be introduced into a body cavity or lumen through an access site such as a trocar or other suitable port, or through a natural orifice.The end effectors advanced into the body cavity or lumen through an access site may be advanced such that the end effector does not block the introduction and / or retrieval of other end effectors using the access site. Thus, a plurality of end effectors may be disposed and actuated within a patient body cavity or lumen.

[0056] The end effectors may be configured to be attracted to one or more magnets positioned externally of the body to move, reposition, and / or hold the end effector (which may in turn provide traction for tissue held by or otherwise in contact with the end effector). Accordingly, at least a portion of the end effectors described herein may be formed from or otherwise include one or more metallic or magnetic materials which may be attracted to a magnetic field. The materials may include one or more magnetic or ferromagnetic materials, such as, for example, stainless steel, iron, cobalt, nickel, neodymium iron boron, samarium cobalt, alnico, ceramic ferrite, alloys thereof and / or combinations thereof. The magnetic portion of the end effector may thus be attracted to a magnetic field produced by an external magnetic positioning device. Furthermore, in some variations, the magnetic portion of the end effector may allow coupling to a delivery device, as described in more detail herein.

[0057] The end effectors may be used within any suitable body cavity or lumen such as but not limited to the abdominal cavity, thoracic cavity, stomach, or intestines. The end effectors advanced into a body cavity or lumen may perform a number of functions.

[0058] In some variations, an end effector may comprise a visualization device (e.g., endoscope) configured to visualize a desired field of view during a minimally invasive procedure. In some variations, an end effector may comprise a grasper used to grasp, retract or otherwise provide remote manipulation and / or traction to tissue. In particular, magnetically controlled graspers may be advanced into a patient and releasably engage tissue. Graspers suitable for use in the surgery systems here are described in U.S. Patent Application No. 14 / 019,370, filed September 5, 2013, and titled “Grasper with Magnetically-Controlled Positioning,” U.S. Patent Application No. 15 / 195,898, filed June 28, 2016, and titled “Laparoscopic Graspers and Systems Therefor,” U.S. Patent Application No. 13 / 132,185, filed August 17, 2011, and titled “Remote Traction and Guidance Systems for Mini-Invasive Surgery,” and International Patent Application No.PCT / US2016 / 027390, filed April 13, 2016, and titled “Grasper with Magnetically-Controlled Positioning,” each of which is hereby incorporated by reference in its entirety.

[0059] In some variations, an end effector may comprise a retractor used to retract or otherwise support and / or move internal organs of a patient. In particular, magnetically controlled retractors may be advanced into a patient and retract tissue to displace it from a surgical site inside the patient and / or otherwise engage tissue to increase surgical access to that tissue. Furthermore, the retractors may be configured to be maintained in position without requiring a handle or grasper. For example, in some variations, a retractor may be configured to form a sling to retract tissue. The terminal ends may comprise a magnetic material or have magnetic masses disposed on them, such that they are configured to be attracted to a magnetic field. When a portion of the retractor is looped underneath a portion of tissue, at least a portion of the tissue may be suspended by the retractor and moved towards the patient wall. In some variations, the retractor may be configured to transition between a substantially linear configuration and the curvilinear configuration.

[0060] Other retractors suitable for use in the surgery systems here are described in International Patent Application No. PCT / US2016 / 027385, filed April 13, 2016, and titled “Retractor Systems, Devices, and Methods for Use,” which is hereby incorporated by reference in its entirety. Other suitable retractors may include, for example, one or more of a coiled retractor, cradle retractor, lever retractor, platform retractor, and J-hook.Delivery device

[0061] The delivery devices described herein are generally configured to releasably carry one or more end effectors. A delivery device may be used to deliver one or more end effectors into a body cavity or lumen. Because the delivery devices may be releasably coupled to the end effectors, the delivery devices may be removed from the body cavity after delivery of the end effector, which may keep the access site (e.g. trocar or natural orifice) free for the delivery of other end effectors or other tools. In some instances, the delivery device may be configured to re-couple to the end effector to reposition or remove the end effector from a body cavity or lumen. In other instances, the system may comprise a separate retrieval device configured to reposition or remove the end effector from abody cavity or lumen. In some variations, the delivery device or retrieval device may be further configured to actuate an end effector.

[0062] When the end effector is a grasper, the delivery devices described here may be configured to releasably carry a grasper, and may be further configured to actuate the grasper to selectively connect the grasper to tissue or release the grasper from tissue. The delivery devices may be typically further configured to release the grasper from the delivery device (e.g., after the grasper has been connected to tissue). In some instances, the delivery device may be configured to re-couple to the grasper to reposition or remove the grasper from a body cavity or lumen. In other instances the system may comprise a separate retrieval device configured to reposition or remove the grasper from a body cavity or lumen. In some instances, the delivery device or retrieval device may be used with the grasper to remove tissue from the body. For example, the grasper may be connected to a tissue such as a gall bladder, the tissue may be severed from the body (e.g., using one or more surgical tools), and the grasper may be retrieved using the delivery device or another retrieval device to remove the grasper and tissue from the body.

[0063] Delivery devices suitable for use in the surgery systems here are described in U.S. Patent Application No. 14 / 019,370, filed September 5, 2013, and titled “Grasper with Magnetically- Controlled Positioning,” which was previously incorporated by reference in its entirety.

[0064] It should be appreciated that while delivery devices are described herein primarily with reference to use with a grasper, the delivery devices described herein may also be used to reversibly couple to another end effector to deliver, position and reposition, and / or remove another end effector. For example, in some instances the delivery devices may be used to deliver, position and reposition, and / or remove a visualization device, such as a camera and / or light source.Magnetic positioning device

[0065] The surgery systems described herein may comprise one or more external magnetic positioning devices comprising an external magnet 132, support arm 112, and / or sensors 120. The external magnets may generate a magnetic field configured to attract one or more end effectors. By controlling the position and / or strength of the external magnets and thereby the position and / orstrength of the magnetic fields, the external magnets may control the position of the end effectors disposed within a body cavity or lumen of a patient. This may free space at an access site (e.g., port) of the patient to allow additional end effectors to be advanced into the patient and reduce, if not eliminate, the need for a second operator such as a skilled surgeon.

[0066] The external magnets may be configured to generate a magnetic field, such that when the external magnet is positioned near a patient, a magnetic field may be generated inside the patient. This magnetic field may apply a force to and manipulate an end effector. In some variations, the external magnet may comprise one or more permanent magnets, one or more electromagnets, and / or one or more electropermanent magnets. Permanent magnets may be formed from suitable magnetic and ferromagnetic materials such as, but not limited to, rare-earth magnets (e.g., samarium-cobalt magnets, neodymium magnets), cobalt, gadolinium, iron, nickel, alnico alloys, ferrites, alloys thereof, combinations thereof, and the like. The external magnets may comprise any number of individual magnets, which in some instances may be formed in an array. The external magnets may have any suitable size and shape, such as cylindrical shape having a circular, oval, or semi-circle cross-section, a bar magnet having a rectangular or triangular cross section, a spherical magnet, or the like. In some variations, the external magnets may comprise permanent magnets, while in other variations, the external magnets may comprise electromagnets or electropermanent magnets. When the external magnets comprise electromagnets or electropermanent magnets, the current may be manipulated to change the strength of the external magnets and / or to turn them on / off. For example, an increase in the magnetic field generated by the external magnet may bring an end effector in contact with a body cavity wall of a patient while a decrease in the magnetic field may reposition the end effector away from the body cavity wall. Additionally, a stronger magnetic field may be needed to magnetically couple the end effector with the external magnet through a thick body cavity wall (e.g., a thick abdominal wall), whereas a weaker magnetic field may be desirable to reduce the attractive force between the end effector and the external magnet through a thin body cavity wall (e.g., a thin abdominal wall).

[0067] When an external magnetic positioning device is magnetically coupled to an end effector, movement of the external magnet via movement of the support arm may in turn move the end effector disposed within a body cavity or lumen of the patient. For example, coronal movement ofthe external magnet relative to the patient may result in a corresponding coronal movement of the end effector. As another example, moving the external magnet closer to the end effector using the support arm may increase the attraction between the external magnet and the end effector so as to bring the end effector in contact with a patient cavity wall, while moving the external magnet further away from the end effector may reduce the magnetic attraction and reposition the end effector away from the body cavity wall. Thus, by controlling the strength of the external magnet and position of the external magnet using the support arm, and thereby the strength and position of the magnetic field, the magnetic positioning device may control the position of the end effectors disposed within a body cavity or lumen of a patient. In some variations, strength and / or position of the external magnet may be used to control a force of a magnetically coupled end effector against a body cavity wall or lumen wall using the sensors described in detail hereinE. Sensor

[0068] The surgery systems described herein may comprise one or more sensors to determine a location of a portion of one or more support arms, external magnetic positioning devices (e.g., external magnets), patient body surfaces (e.g., abdomen, internal cavity wall, breasts), surgery system components (e.g., end effector, intracavity devices, trocar, support structure, patient platform, input device, output device), and operator. For example, an end effector (e.g., endoscope) may comprise one or more sensors (e.g., optical sensors) configured to detect a location of a patient body surface and another end effector to calculate a proximity of the end effector relative to the patient such that the system ensures that the support arm and / or the end effector does not contact the patient. For example, each segment of a support arm may comprise an inductive proximity sensor to calculate a distance between the support arms. As another example, an infrared, radar, or ultrasonic range finder mounted on the support arm may be configured to calculate a distance to the patient. A controller (e.g., processor and memory) may be configured to maintain a predetermined distance between the end effector and a patient body surface such as a distance of about 1 mm, about 5 mm, or about 10 mm. Thus, a controller may limit a range of motion of the support arm.

[0069] As another example, a position of an end effector may be controlled based on a force sensor such as for an end effector in contact with an internal body cavity wall. A contact force ofthe end effector with the body cavity wall may be reduced if a force sensor detects that the force exceeds a predetermined threshold. The sensors may comprise one or more of a force sensor (e.g., Hall sensor, load cell, springs), proximity sensor, optical sensor, motion sensor, accelerometer, gyroscope, laser rangefinder, radar, and LIDAR.F. Input device

[0070] Generally, an input device 122 of a robotic surgical system 100 may serve as a communication interface between an operator and the surgery system 100. The input device 122 may be configured to receive input data and output data to one or more of the support arm 112, sensor 120, end effector 118, and output device 130. For example, operator control of an input device 122 (e.g., foot controller joystick, keyboard, touch screen, microphone, eye gaze tracker) may be processed by processor 124 and memory 126 for input device 122 to output a control signal to one or more end effectors 118, support arms 120, and patient platform 114. As another example, images generated by an end effector 118 comprising a visualization device (e.g., endoscope) may be received by input device 122, processed by processor 124 and memory 126, and displayed by the output device 130 (e.g., monitor display). Sensor data from one or more sensors 120 may be received by input device 122 and output visually, audibly, and / or through haptic feedback by one or more output devices 130. In some variations, a single operator may control one or more components of a surgery system 100 using one or more input devices 122. Some variations of an input device may comprise at least one switch configured to generate a control signal. The input device may be coupled to a support arm and / or disposed on a patient platform or support structure adjacent to the patient and / or operator. However, the input device may be mounted to any suitable object, such as furniture (e.g., a bed rail), a wall, a ceiling, or may be self-standing. The control signal may include, for example, a movement signal, device switch signal, activation signal, magnetic field strength signal, and other signals. In some variations, the input device may comprise a wired and / or wireless transmitter configured to transmit a control signal to a wired and / or wireless receiver of a controller. A movement signal (e.g., for the control of movement, position, and orientation) may control movement in at least four degrees of freedom of motion, and may include yaw and / or pitch rotation. For example, an input device comprising a touch surface may be configured to detect contact and movement on the touch surface using any of a plurality of touch sensitivity technologies includingcapacitive, resistive, infrared, optical imaging, dispersive signal, acoustic pulse recognition, and surface acoustic wave technologies.

[0071] In other variations, a plurality of input devices may be configured to control a single component of the surgery system (e.g., end effector) to enhance operator flexibility. For example, an operator may choose to control a support arm using combinations of a joystick, directional pad, soft keys, voice commands, eye gaze, and the like.

[0072] In still other variations, each input device of a surgery system may be associated with a corresponding component of the surgery system. Some non-limiting examples include: a joystick may be configured to control movement of a support arm; a touch screen may be configured to pan, tilt, and / or zoom a visualization device; a jog dial may be configured to control the jaw positions of a grasper; and a step switch may be configured to release a delivery device from an end effector.

[0073] In variations of the input device comprising one or more buttons, button presses of varying duration may execute different functions. For example, a lumen output level of a light source may be configured to increase with a longer button press. Conversely, a shorter duration button press may correspond to a different function such as deactivating the light source.

[0074] In some variations, a surgery system may comprise a plurality of input devices provided in separate housings, where for example a first input device may be handheld and / or portable while a second input device may be stationary. In some variations, a first input device may comprise a tablet including a touch screen display and a second input device may comprise a step switch or foot pedal. The step switch may in some variations be a safety switch that must be engaged at the same time as contact with the touch screen before a control signal is transmitted to the surgery system. Output of a control signal upon simultaneous engagement of a first input device and second input device may confirm that operator input to the first input device is intentional.

[0075] The input device 122 may be configured to receive a control signal from an operator. Nonlimiting examples of the control signal may include an applied force, a measurement of an operator parameter (e.g., operator gaze), a movement signal, a device switch signal, an activation signal, and / or a magnetic field strength signal. In some variations, the control signal may includeone or more control signals, such as at least two control signals, or a plurality of control signals. The input device 122 may be configured to transmit and / or receive signals to and / or from other components of the robotic surgery system 100 via a wired and / or wireless connection. For example, the input device may comprise a wired and / or wireless transmitter configured to transmit a control signal to a wired and / or wireless receiver of a controller (e.g., via the communication device 128). A movement control signal (e.g., for the control of movement, position, and / or orientation of a support arm or end effector) may control movement in one, two, three, four, five, or six degrees of freedom (i.e., up / down, forward / back, left / right, pitch, roll, and / or yaw).

[0076] In some variations, the input device 122 may include at least one switch or actuator (e.g., a virtual actuator) configured to generate a control signal. In some variations, the systems herein may comprise a plurality of input devices, such as two or more or at least two input devices. In some variations, a plurality of input devices may comprise one or more types of input devices. For example, as described below, the surgery systems herein may comprise a gaze-actuated input device and a user interface. In some variations, an input device may include an AR or VR device (e.g., portable or wearable augmented reality devices) configured to enhance visualization of a surgical procedure, and may include a headset, goggles, glasses, or contact lens(es) configured to be worn by an operator.

[0077] In some variations, the input devices herein may comprise a device for providing a user interface configured to receive information related to a patient, surgical procedure, and / or components of the robotic surgery system. The device may be, for example, a display (e.g., standalone display, wearable display). In some variations, the user interface may be configured to provide one or more (e.g., a series of) prompts for an operator to provide one or more inputs for a system.Gaze-Actuated Input Device

[0078] In some variations, the input device may comprise a gaze-actuated input device (e.g., gaze controller) configured to receive input from one or both eyes of an operator. In some variations, a gaze-actuated input device may be configured to operate one or more support arms, patient platform, support structure, sensor, communication device, output device, external magnet, and endeffectors of a robotic surgery system described herein. Here, a control signal may be generated by measuring a gaze of the operator. For example, the gaze-actuated input device may comprise one or more (e.g., a plurality of) actuators each configured to respond to a gaze of the operator to generate a corresponding control signal. In some variations, the plurality of actuators may be virtual actuators provided with an AR or VR device configured to be worn over one or more eyes of an operator, such as a headset, goggles, glasses, contacts, and / or the like. The operator may actuate each virtual actuator by directing a gaze toward the actuator for a time period. Put another way, once a length of the gaze of the operator is determined to be about equal to or greater than a time period, the actuator at which the gaze is directed may transmit corresponding control signal to one or more components of the robotic surgical system 100. In some variations, the time period may be about 0.1 seconds (s) to about 15 s, such as about 0.5 s to about 12.5 s, about 1 s to about 10 s, about 1.25 s to about 7.5 s, about 1.5 s to about 5 s, or about 1.75 s to about 2.5 s (including all ranges and subranges therebetween). In some variations, the time period may be adjusted (e.g., individually for each operator).

[0079] The gaze-actuated input device may comprise any suitable number of actuators (e.g., virtual actuators) to control movement of one or more of a support arm, external magnet, support structure, or end effector in one or more degrees of freedom, such as in one, two, three, four, five, or six degrees of freedom. For example, a gaze-actuated input device may comprise one or more of: a first actuator configured to translate a support arm (or external magnet, support structure, end effector) along a first axis, a second actuator configured to translate the support arm (or external magnet, support structure, end effector) in along a second, different axis, and a third actuator configured to translate the support arm (or external magnet, support structure, end effector) along a third, different axis. Each of the first, second, and third axes may be one of the X, Y, or Z axes. In some variations, the gaze-actuated input device may comprise two separate actuators for translating the support arm (or external magnet, support structure, end effector) in opposite directions along a same axis. Additionally, or alternatively, the gaze-actuated input device may comprise one or more of: a first actuator configured to rotate the support arm (or external magnet, support structure, end effector) around a first axis, a second actuator configured to rotate the support arm (or external magnet, support structure, end effector) around a second, different axis, and a third actuatorconfigured to translate the support arm (or external magnet, support structure, end effector) around a third, different axis. Accordingly, each one of these actuators may control pitch, yaw, or roll of the support arm (or external magnet, support structure, end effector). In some variations, the gaze- actuated input device may comprise two separate actuators for rotating the support arm (or external magnet, support structure, end effector) in opposite directions around a same axis.

[0080] In some variations, the gaze-actuated input device may be coupled to an output device, such as a display (as described below) in order to provide actuators. For example, the gaze-actuated input device may comprise a combination input / output device configured to provide an augmented or virtual experience for an operator throughout a surgical procedure.

[0081] In some variations, a robotic surgery control system may include a plurality of input devices. For example, such a system may include a foot-actuated input device, a display that provides a user interface, and an AR or VR device (e.g., a headset). One or both of the display and AR or VR device may also function as an output device. For example, the display may be configured to provide feedback or medical information for an operator via the user interface, and the AR or VR device may be configured to provide a mixed-reality image as an aid for the operator during a procedure. Any of the support arm, external magnet, support structure, and end effector may be actuated via one or more of the input devices.E. Processor

[0082] A surgery system 100, as depicted in FIG. 1, may comprise a processor 124 and a machine-readable memory 126 (e.g., controller) in communication with one or more support arms 112 and / or end effectors 118. The processor 124 may be connected to the support arms 112, patient platform 114, end effectors 118, and / or sensors 120 by wired or wireless communication channels. The processor 124 may be located in the same or different room as the patient. In some variations, the processor 124 may be coupled to a patient platform or disposed on a support structure adjacent to the patient and / or operator. The processor 124 may be configured to control one or more components of the system 100, such as an end effector 118 that may visualize a body cavity or lumen, grasp tissue, retract tissue, hold and / or drive a needle, and the like. In some variations, theprocessor 124 may be configured to coordinate movement and orientation of end effectors 118 within a body cavity or lumen through corresponding movement and control of the support arm 112.

[0083] The processor 124 may be implemented consistent with numerous general purpose or special purpose computing systems or configurations. Various exemplary computing systems, environments, and / or configurations that may be suitable for use with the systems and devices disclosed herein may include, but are not limited to software or other components within or embodied on personal computing devices, network appliances, servers or server computing devices such as routing / connectivity components, portable (e.g., hand-held) or laptop devices, multiprocessor systems, microprocessor-based systems, and distributed computing networks.

[0084] Examples of portable computing devices include smartphones, personal digital assistants (PDAs), cell phones, tablet PCs, phablets (personal computing devices that are larger than a smartphone, but smaller than a tablet), wearable computers taking the form of smartwatches, portable music devices, and the like, and portable or wearable augmented reality devices that interface with an operator’s environment through sensors and may use head-mounted displays for visualization, eye gaze tracking, and user input.

[0085] The processor 124 may incorporate data received from memory 126 and operator input to control one or more support arms 112 and end effectors 118. The memory 114 may further store instructions to cause the processor 124 to execute modules, processes, and / or functions associated with the system 100. The processor 124 may be any suitable processing device configured to run and / or execute a set of instructions or code and may comprise one or more data processors, image processors, graphics processing units, physics processing units, digital signal processors, and / or central processing units. The processor 124 may be, for example, a general purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), configured to execute application processes and / or other modules, processes, and / or functions associated with the system and / or a network associated therewith. The underlying device technologies may be provided in a variety of component types such as metal-oxide semiconductor field-effect transistor (MOSFET) technologies like complementary metal-oxide semiconductor (CMOS), bipolar technologies like emitter-coupled logic (ECL), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, combinations thereof, and the like.G. Memory

[0086] Some variations of memory 126 described herein relate to a computer storage product with a non-transitory computer-readable medium (also may be referred to as a non-transitory processor- readable medium) having instructions or computer code thereon for performing various computer- implemented operations. The computer-readable medium (or processor-readable medium) is non- transitory in the sense that it does not include transitory propagating signals per se (e.g., a propagating electromagnetic wave carrying information on a transmission medium such as air or a cable). The media and computer code (also may be referred to as code or algorithm) may be those designed and constructed for a specific purpose or purposes. Examples of non-transitory computer- readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as Compact Disc / Digital Video Discs (CD / DVDs), Compact Disc-Read Only Memories (CD-ROMs), and holographic devices; magnetooptical storage media such as optical discs; solid state storage devices such as a solid state drive (SSD) and a solid state hybrid drive (SSHD); carrier wave signal processing modules; and hardware devices that are specially configured to store and execute program code such as Application- Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), Read-Only Memory (ROM), and Random-Access Memory (RAM) devices. Other variations described herein relate to a computer program product, which may include, for example, the instructions and / or computer code disclosed herein.

[0087] The systems, devices, and / or methods described herein may be performed by software (executed on hardware), hardware, or a combination thereof. Software modules (executed on hardware) may be expressed in a variety of software languages (e.g., computer code), including C, C++, Java®, Python, Ruby, Visual Basic®, and / or other object-oriented, procedural, or other programming language and development tools. Examples of computer code include, but are not limited to, micro-code or micro-instructions, machine instructions, such as produced by a compiler, code used to produce a web service, and files containing higher-level instructions that are executedby a computer using an interpreter. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code.H. Communication device

[0088] In some variations, surgery systems 100 described herein may communicate with networks and computer systems through a communication device 128. In some variations, the surgery system 100 may be in communication with other devices via one or more wired and / or wireless networks. A wireless network may refer to any type of digital network that is not connected by cables of any kind. Examples of wireless communication in a wireless network include, but are not limited to cellular, radio, satellite, and microwave communication. However, a wireless network may connect to a wired network in order to interface with the Internet, other carrier voice and data networks, business networks, and personal networks. A wired network is typically carried over copper twisted pair, coaxial cable and / or fiber optic cables. There are many different types of wired networks including wide area networks (WAN), metropolitan area networks (MAN), local area networks (LAN), Internet area networks (IAN), campus area networks (CAN), global area networks (GAN), like the Internet, and virtual private networks (VPN). Hereinafter, network refers to any combination of wireless, wired, public and private data networks that are typically interconnected through the Internet, to provide a unified networking and information access system.

[0089] Cellular communication may encompass technologies such as GSM, PCS, CDMA or GPRS, W-CDMA, EDGE or CDMA2000, LTE, WiMAX, and 5G networking standards. Some wireless network deployments combine networks from multiple cellular networks or use a mix of cellular, Wi-Fi, and satellite communication. In some variations, the network interface 116 may comprise a radiofrequency receiver, transmitter, and / or optical (e.g., infrared) receiver and transmitter. The communication device 128 may communicate by wires and / or wirelessly with one or more of the support arm 112, end effector 118, sensor 120, input device 122, output device 130, network, database, server, combinations thereof, and the like.I. Output device

[0090] An output device 130 of a surgery system 100 may be configured to output data corresponding to a surgery system, and may comprise one or more of a display device, audio device, and haptic device. The output device may be coupled to a patient platform and / or disposed on a support structure adjacent to the patient and / or operator. In other variations, the output device may be mounted to any suitable object, such as furniture (e.g., a bed rail), a wall, a ceiling, and may be self-standing.

[0091] A display device may allow an operator to view images of one or more end effectors, support arms, body cavities, and tissue. For example, an end effector comprising a visualization device (e.g., camera, optical sensor) located in a body cavity or lumen of a patient may be configured to image an internal view of the body cavity or lumen and / or end effectors. An external visualization device may be configured to image an external view of the patient and one or more external magnetic positioning devices. Accordingly, the display device may output one or both of internal and external images of the patient and system components.

[0092] In some variations, an output device 130 may be configured to output data corresponding to a surgery system or surgical procedure, and may comprise one or more of a display device, audio device, and haptic device. The output device 130 may be coupled to a patient platform and / or disposed on a medical cart adjacent to the patient and / or operator. In other variations, the output device 130 may be mounted to any suitable object, such as furniture (e.g., a bed rail), a wall, a ceiling, and may be self-standing.

[0093] An output or display device may allow an operator to view images of one or more end effectors, support arms, external magnets, support structures, body cavities, and tissue. For example, an end effector comprising a visualization device (e.g., camera, optical sensor) located in a body cavity or lumen of a patient may be configured to image an internal view of the body cavity or lumen and / or intracavity devices. An external visualization device may be configured to image an external view of the patient and one or more external magnetic positioning devices. Accordingly, the display device may output one or both of internal and external images of the patient and system components.

[0094] In some variations, the output device 130 may be configured to provide an AR or VR experience for an operator by generating one or more images having enhanced and / or virtual features. These features may comprise graphical overlays onto an image, reference markers associated with objects being imaged (e.g., surgical tools, external magnets, support structure 116, support arm, drapes, end effectors, operator hands, etc.), and / or the like to conveniently provide information to an operator during a procedure. A mixed reality experience provided by the output device 130 may help to guide a procedure by providing easily-interpretable, real-time feedback to the operator.

[0095] In some variations, the output device 130 may be a combined input / output device, such as a display configured to receive user input and output, for operator interpretation, information related to a patient, surgical procedure, and / or components of the robotic surgery system. For example, the user interface may be configured to provide (e.g., graphically) an indication of one or more dimensions of an end effector that were determined by the processor 124. In some variations, the user interface may be configured to provide one or more (e.g., a series of) prompts for an operator to determine one or more parameters for the surgical procedure.

[0096] In some variations, the combined input / output device may be configured to receive a first image (e.g., a real time image captured by an end effector) and simultaneously capture a second image (e.g., an image of a real time field of view of an operator detected by a sensor). The input / output device may be configured to provide the first and second images in combination (e.g., oriented adjacent to each other or overlayed with each other) via a display thereof so that an operator may access both images throughout a surgical procedure. In some variations, the input / output device may comprise an AR or VR device, such as one or more of an AR or VR headset, goggles, glasses, and contact lens. The AR or VR device may be configured to provide, via a display, one or more augmented or virtual images received and / or detected by the device. An augmented image may comprise, for example, a reference marker overlayed onto or adjacent to a desired system component (e.g., a portion of an end effector or surgical instrument) such that the system component may be tracked within the image via the reference marker. In some variations, the reference marker may be generated using an RFID tag located on the end effector or surgical instrument. In some variations, a reference marker may be overlayed onto one or more hands of anoperator. The reference marker may comprise one or more of a symbol, image, or text. In some variations, the reference marker may comprise a digital reproduction or digital outline of an end effector or surgical instrument.

[0097] In some variations, an output device may comprise a display device including at least one of a light emitting diode (LED), liquid crystal display (LCD), electroluminescent display (ELD), plasma display panel (PDP), thin film transistor (TFT), organic light emitting diodes (OLED), electronic paper / e-ink display, laser display, and / or holographic display.

[0098] An audio device may audibly output patient data, sensor data, system data, alarms and / or warnings. For example, the audio device may output an audible warning when monitored patient data (e.g., blood pressure) falls outside a predetermined range or when a malfunction in a support arm is detected. As another example, audio may be output when operator input is overridden by the surgery system to prevent potential harm to the patient and / or surgery system (e.g., collision of support arms with each other, excessive force of the end effector against a patient cavity wall). In some variations, an audio device may comprise at least one of a speaker, piezoelectric audio device, magnetostrictive speaker, and / or digital speaker. In some variations, an operator may communicate to other users using the audio device and a communication channel. For example, the operator may form an audio communication channel (e.g., VoIP call) with a remote operator and / or observer.

[0099] A haptic device may be incorporated into one or more of the input and output devices to provide additional sensory output (e.g., force feedback) to the operator. For example, a haptic device may generate a tactile response (e.g., vibration) to confirm operator input to an input device (e.g., touch surface). Haptic feedback may in some variations simulate a resistance encountered by an end effector within a body cavity or lumen (e.g., magnetic field and tissue resistance).Additionally or alternatively, haptic feedback may notify that an operator input is overridden by the surgery system to prevent potential harm to the patient and / or system (e.g., collision of support arms with each other). Operator interaction with a user interface utilizing an input and output device is discussed in more detail herein.II. Methods

[0100] Also described here are methods for monitoring and / or managing one or more surgical procedures using the robotic surgical systems described herein. A robotic surgical system is generally a high-cost expenditure that can treat a limited number of patients per day. Operating room time may also be expensive and it is therefore desirable to efficiently schedule patients, surgeons, hardware, and operating rooms by reducing downtime. The methods as described herein may facilitate parallel treatment of patients in different operating rooms, thereby increasing usage efficiency of operating room and robotic surgical system resources and increasing patient throughput.

[0101] Workflow may be improved by performing surgical procedures in parallel, thus treating a plurality of patients in the workflow simultaneously (at different stages of treatment). This may have numerous benefits, such as reducing the cost of surgery and increased access to services for patients. For example, pre-surgery preparation for a first patient may be performed in a first room and be followed by surgery performed in a second room (e.g., first operating room). The first patient may then be moved into a third room post-surgery. A second patient may enter a fourth room (e.g., second operating room) as soon as the robot surgery system and operator complete surgery of the first patient in the first operating room. Accordingly, patients need not occupy an operating room until they are ready to be treated by the operator using the robotic surgical system.

[0102] Generally, the methods described herein comprise monitoring a surgical procedure based on data generated by the system to estimate a chronological timeline of the surgical procedure, which may facilitate resource management and improve operating efficiency. In some variations, the methods described herein comprise managing a surgical procedure by scheduling a surgical procedure of a second patient based on a surgical procedure chronology of a first patient.A. Monitoring a surgical procedure

[0103] Described herein are methods of monitoring a surgical procedure to estimate a chronological timeline of a surgical procedure. FIG. 2 is a flowchart that generally describes a method of monitoring a surgical procedure 200 of at least one patient using any of the systems anddevices described herein. A patient may be disposed on a patient platform adjacent a support arm and operator. The operator may control a robotic surgical system to treat the patient. During a surgical procedure, the method 200 may comprise generating one or more of support arm data, end effector data, and patient data 202. The support arm data, end effector data, and patient data may be generated by one or more sensors of the robotic surgical system. For example, the sensor may comprise one or more of a motion sensor, pressure sensor, optical sensor, audio sensor, patient sensor, magnetic sensor, and proximity sensor. The generated data may comprise a combination of sensor data. Additionally or alternatively, patient data and surgeon data may be input data and / or received from memory (e.g., database).

[0104] In some variations, the support arm data may comprise one or more of movement data, position data, orientation data, and configuration data corresponding to the support arm. In some variations, the end effector data may comprise one or more of movement position data, orientation data, and configuration data of the end effector. In some variations, the patient data may comprise one or more of position data and historic surgical procedure data of one or more surgeons. For example, the position of the patient may correspond to imaging data generated by an optical sensor. Patient data may comprise vital signs, demographic data, patient history (e.g., health data), patient procedure data, and the like.

[0105] In some variations, a surgical procedure event may be identified based on one or more of the support arm data, the end effector data, and the patient data 204. A surgical procedure chronology may comprise a sequence of surgical procedure events. The surgical procedure event may comprise one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish. In some variations, a patient start event may correspond to patient admission to a surgical facility. In some variations, surgery preparation may include a set of pre-operation steps prior to surgery. For example, the patient may complete forms, change into a surgical gown, and move onto a patient platform. In some variations, first incision may correspond to the first surgical cut made through the skin. In some variations, surgery start may correspond to the first step of the principal procedure. For example, surgery start may comprise insufflation and / or a visualization device (e.g., endoscope) entering a trocar. In some variations, treatment preparation may correspond to preparing a treatmentsite. For example, one or more organs (e.g., portion of a liver) may be grasped and / or retracted to a predetermined location. In some variations, treatment delivery may correspond to performing treatment on a desired organ. For example, treatment may comprise stapling a stomach while the liver remains retracted. In some variations, treatment finish may correspond to a final step of a treatment. For example, a final ligation may be performed and / or the retracted liver may be released. In some variations, surgery finish may correspond to a final step of the surgery. For example, the visualization device may be retracted (e.g., withdrawn) from the trocar. In some variations, patient finish may correspond to an end of the surgical procedure. For example, the surgical procedure may be completed upon closure of all incisions in the patient, retraction of the support arms into a retracted configuration, and / or patient discharge. Different surgical procedures may comprise different criteria for each of the surgical procedure events.

[0106] In some variations, a surgical procedure event may be identified at least in part on one or more of support arm data, end effector data, and patient data. For example, FIGS. 5A and 5B illustrate a robotic surgical system 500 comprising a support structure 510 (e.g., cart) comprising a set of wheels 520, where the support structure 510 is coupled to a support arm 530. In FIG. 5 A, a support arm 530 is disposed in a first configuration 532 (e.g., retracted, stowed, idle) where the support arm 530 is not in motion, positioned (e.g., located) away from a patient platform, and oriented away from a patient platform. One or more sensors of the support arm 530 may be configured to generate support arm data comprising one or more of movement data (e.g., zero acceleration), position data (e.g., GPS coordinates), orientation data (e.g., facing away from patient platform coordinates), and configuration data corresponding to the support arm. In some variations, support arm data corresponding to the first configuration may identify the surgical procedure event as one of patient start and patient finish. In FIG. 5A, end effector data may comprise an unattached configuration where the end effector is not coupled to the support arm 530. In some variations, an optical sensor may be configured to generate imaging data corresponding to support arm data and end effector data. For example, the support structure 510 may comprise an optical sensor (not shown) configured to generate imaging data of the surgical procedure.

[0107] FIG. 5B depicts the transition of the support arm 530 from the first configuration 532 to a second configuration 534 (e.g., extended, ready) where the support arm 530 is positioned apredetermined distance from a patient platform 540 (e.g., within reach of the support arm), and oriented towards the patient platform 540. For example, the support arm 530 in the first configuration 532 may exceed a predetermined distance from a patient platform 540 (e.g., located in a room different from the operating room) while the support arm 530 in the second configuration 534 may within a predetermined distance from the patient platform 540 (e.g., located within about 2 meters of the patient platform 540). In some variations, support arm data corresponding to the second configuration may identify the surgical procedure event as one of surgery preparation and surgery finish. In FIG. 5B, end effector data may comprise an attached configuration where the end effector 536 is coupled to the support arm 530. In some variations, the support arm and end effector configurations may be based on the imaging data generated by the optical sensor using imaging data processing techniques (e.g., computer vision) that identify and characterize objects within the image.

[0108] In some variations, a surgical procedure event may be identified based at least in part on imaging data generated by one or more of an end effector and optical sensor. For example, the initial advancement of an end effector (e.g., endoscope) through a trocar and visual confirmation of an end effector (e.g., tissue grasper) within a patient body cavity may correspond to a surgery start event of a surgical procedure. Likewise, the final retraction of the endoscope from the trocar and visual confirmation of treatment delivery (e.g., stapled stomach) may correspond to a surgery finish event of the surgical procedure.

[0109] FIG. 7A illustrates an end effector 710 comprising an optical sensor 720 advanced through a trocar 730. As the end effector 710 is advanced, the optical sensor 720 may generate at least a first image 712 and a subsequent second image 714. Likewise, as the end effector 710 is withdrawn, the optical sensor 720 may generate a third image 716 and a subsequent fourth image 718. The dark circular portion of the images 712, 714, 716, 718 corresponds to an opening in the trocar 730. The diameter of the dark circular portion will increase as the end effector 710 is advanced through the trocar 730 and will decrease as the end effector 710 is withdrawn from the trocar 730.

[0110] In some variations, the images 712, 714, 716, 718 (and other images) generated by the optical sensor 720 may be input to a machine-learning model (e.g., surgical procedure event model)that outputs a corresponding surgical procedure event (e.g., surgery start, surgery finish). In some variations, a surgical procedure event machine-leaning model may be trained using a training set of images corresponding to endoscope images. For example, the training set may include a plurality of images of graspers disposed within an abdominal cavity, stapled stomachs, and the like. In some variations, imaging data may comprise one or more of an external portion of a patient, operator, support arm, support structure, patient platform, operating room, and the like.

[0111] FIGS. 8A and 8B illustrate additional examples of surgical procedure event identification based on imaging data. For example, FIGS. 8A and 8B are schematic diagrams of images 800, 802 generated by an endoscope disposed within a body cavity of the patient. The images 800, 802 may comprise an end effector 850 (e.g., grasper) configured to grasp tissue for retraction (e.g., liver retraction) where first image 800 corresponds to a start of a treatment preparation surgical procedure event and second image 802 corresponds to completion of the treatment preparation event as described in more detail herein. In some variations, the end effector 850 may comprise one or more fiducials configured to facilitate image processing and analysis of the end effector 850 in imaging data. For example, an end effector sensor (e.g., optical sensor) may be configured to measure one or more fiducials of the end effector 850 to facilitate identification of the grasper.

[0112] For the sake of explanation, first image 800 may be divided into four quadrants including first quadrant 810, second quadrant 812, third quadrant 814, and fourth quadrant 816, although the first image 800 may be divided into a plurality of portions where each portion may have any predetermined shape. The location of the grasper 850 within the third quadrant 814 and a generally horizontal orientation of the grasper 850 may correspond to a non-retracted organ state. That is, first image 800 corresponds to the grasper 850 in the process of grasping organ 840 (organ not shown for clarity).

[0113] Next, FIG. 8B depicts the second image 802 where the grasper is located in the second quadrant 812 and in a generally vertical position which corresponds to the grasper 850 retracting tissue upward to an abdominal wall of the patient. In some variations, the end effector 850 may comprise one or more magnetic portions. An external magnet may be placed on an opposite side of the abdominal wall to magnetically attract the grasper 850 to the generally vertical position.Accordingly, the second image 802 corresponds to the completion of the treatment preparation event when an external end effector (not shown) such as an external magnet or magnet arm is located in a predetermined position (e.g., on or over the abdominal cavity of the patient).

[0114] In some variations, an exterior image of a patient may be divided into a plurality of portions and analyzed to identify a surgical procedure event. For example, image analysis of an exterior portion of the patient may be used to identify surgical procedure events including but not limited to surgery preparation, first incision, and patient finish.

[0115] In some variations, a surgical procedure chronology may comprise an end effector status. For example, an end effector attached to a support arm and located within a body cavity of the patient may be data used to identify a surgical procedure event (e.g., treatment delivery) of a surgical procedure. In FIG. 9, a first end effector 910 may be advanced through a trocar 930 into a body cavity and configured to image a second end effector 920 (e.g., cutter, stapler, cauterizer, etc.) within the body cavity. An end effector status of “instrument in use” may be identified based on image analysis of the second end effector 920 within the body cavity. In some variations, endoscope imaging of end effectors during a surgical procedure may facilitate end effector counting (e.g., instrument use tally) for disposable tracking and ensure patient safety.

[0116] In some variations, a surgical procedure event may be identified based at least in part on surgeon data such as surgeon position data. FIG. 10A illustrates a robotic surgical system 1020, operator 1030, and computing device 1040 (e.g., operator mobile device) each located in a first operating room 1010. The robotic surgical system 1020 and computing device 1040 may be configured to establish a communication channel with each other. For example, the robotic surgical system 1020 may be configured to estimate a proximity of the computing device 1040 to the robotic surgical system 1020 in order to indirectly estimate the proximity of the operator 1030 to the robotic surgical system 1020. For example, a Bluetooth connection established between the robotic surgical system 1020 and the computing device 1040 may correspond to operator 1030 presence in the first operating room 1010. In some variations, one or more surgical procedure events (e.g., first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, etc.) may comprise the location of the operator 1030 to be within a predetermined proximity of the roboticsurgical system 1020. By contrast, as shown in FIG. 10B, the absence of a Bluetooth connection between the robotic surgical system 1022 and the computing device 1040 may correspond to operator 1030 absence from a second operating room 1012.

[0117] In some variations, the surgical procedure event may be identified based on the surgeon data comprising one or more of audio data, imaging data, and input data. For example, the surgeon may audibly identify the start and / or end of a surgical procedure event (e.g., “first incision”, “scope entering trocar”). In some variations, the surgeon may perform a gesture (e.g., hand gesture) towards an optical sensor that may be analyzed to identify the surgical procedure event. In some variations, the surgeon may manually input the surgical procedure event using a keyboard and / or mouse.

[0118] In some variations, a communication channel may optionally be established 206. For example, a communication channel may be established between the surgeon using the robotic surgical system and another operator (e.g., another surgeon, technician) at a remote location. In some variations, the communication channel may be established using the robotic surgical system in response to identifying a predetermined surgical procedure event. For example, at the start of a treatment preparation event, a communication channel may be established with a remote operator comprising an audio and video feed (e.g., video conference) of the procedure generated by the robotic surgical system. This may allow another operator to participate in the surgical procedure (e.g., remotely consult) at predetermined events of the surgical procedure (e.g., during a treatment delivery event). As another example, a communication channel (e.g., audio channel) may be established between a first operating room and a second operating room when a second patient is at a surgery preparation step such that respective operators in each operating room can communicate their status to each other. Additionally or alternatively, a communication channel may be established by a predetermined set of operators on an ad hoc basis.

[0119] In some variations, a notification may optionally be generated based on predetermined criteria 208. For example, a notification (e.g., audible alert, visual message) may be generated corresponding to an identified surgical procedure event. The notification may include additional data including but not limited to time (e.g., elapsed time), sensor measurements, and patient vitalsigns. In some variations, the patient data may comprise one or more of the notifications and may be added as part of a patient’s health record. Additionally or alternatively, notifications may be generated when predetermined criteria are not met. For example, an alert notification may be transmitted to (and / or a communication channel may be established with) a remote operator if the time elapsed between a treatment finish event and a surgery finish event exceeds a predetermined threshold. In this manner, another operator may be made available to assist in case of possible delays and / or complications in a surgical procedure.

[0120] In some variations, a notification may comprise a graphical user interface (GUI) such as shown in FIGS. 11 A, 11B, and 12A-12C. In particular, FIGS. 11A and 11B depict a first GUI 1100 and a second GUI 1102, respectively. The GUIs 1100, 1102 may represent a surgical dashboard displayed for an operator (e.g., surgeon, technician, administrator, health care professional) that may provide the status of one or more surgical procedures. For example, the GUIs 1100, 1102 may comprise the status of a plurality of operating rooms 1114, 118, 1122 including the name of the surgeon and procedure being performed. Furthermore, the GUIs may comprise a color-coded status 1112 of the surgical procedure event (e.g., surgery preparation 1130, surgery 1132, post-surgery 1134). For example, a first operating room 1114 may be in surgery preparation 1116 while each of a second operating room 1118 and third operating room 1112 may be in surgery 1120, 1124. FIG.1 IB shows that the status of each operating room is independent of each other as the third operating room 1112 transitions to a post-surgery status 1125. In some variations, an operator may select a specific operating room to view more detailed information. For example, FIGS. 12A-12C depict GUIs 1200, 1202, 1204 comprising surgery details 1220, 1222, 1224, real-time images 1230, 1232, 1234, and a corresponding surgical procedure chronology 1240, 1242, 1244 with an indication of the current surgical procedure event. The real-time images may be generated by, for example, an endoscope.

[0121] In some variations, a surgical procedure chronology may be estimated based on the identified surgical procedure event and surgeon data 210. The estimate may be generated in realtime throughout a surgical procedure. The estimated surgical procedure chronology may be updated continuously or at periodic intervals. In some variations, surgeon data may comprise one or more of position data and historical surgical procedure data of one or more surgeons. Historical surgicalprocedure data may include an amount of time taken by the surgeon to perform one or more of the surgical procedure events. For example, an estimate of when a treatment finish event will occur may be based on the historical duration of a treatment delivery event for a predetermined surgical procedure performed by the surgeon. In this manner, the estimate of the surgical procedure chronology may be improved. If historical data for the surgeon is incomplete, then an average of historical data for a set of surgeons may be used to estimate the surgical procedure chronology.

[0122] In some variations, the surgical procedure chronology may optionally be compared to historic surgical procedure data 212. For example, the surgical procedure chronology may be compared to historic surgical procedure data of one or more surgeons 212 to compare the performance of the surgical procedure.B. Managing a surgical procedure

[0123] Described herein are methods of managing a surgical procedure by scheduling a surgical procedure of a second patient based on a surgical procedure chronology of a first patient. Efficient scheduling of a set of surgical procedures may increase patient throughput of a surgical facility. FIG. 3 is a flowchart that generally describes a method of managing a surgical procedure 300 of at least one patient using any of the systems and devices described herein. A first patient may be disposed on a patient platform adjacent a support arm and operator. The operator may control a robotic surgical system to treat the patient. During a first surgical procedure, the method 300 may comprise generating first surgical procedure data of a first surgical procedure using one or more sensors of a robotic surgical system 302. In some variations, the first surgical procedure may be performed in a first operating room. In some variations, the first surgical procedure data may comprise one or more of movement data, position data, orientation data, and configuration data of one or more of a support arm and an end effector. In some variations, the first surgical procedure data may comprise first patient data comprising first health data and first patient procedure data. In some variations, the first surgical procedure data may comprise surgeon data comprising one or more of position data and historic surgical procedure data of one or more surgeons.

[0124] In some variations, a first surgical procedure chronology may be estimated based on the first surgical procedure data, first patient data, and surgeon data 304. For example, the first surgicalprocedure chronology may comprise a sequence of surgical procedure events. In some variations, the surgical procedure event may comprise one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish.

[0125] In some variations, a second surgical procedure may be scheduled based on the estimated first surgical procedure chronology 306. In some variations, the first surgical procedure may be started before the second surgical procedure. The second surgical procedure may be performed in a second operating room. In some variations, one or more events of a second surgical procedure may be performed prior to the completion of a first surgical procedure. That is, the first and second surgical procedures may be performed in parallel. In some variations, an operator may be in a first operating room for predetermined portions of a surgical procedure. For example, a first surgeon may perform first incision, surgery start, and treatment preparation. A second surgeon may perform treatment delivery, treatment finish, surgery finish, and patient finish. Accordingly, after a first surgeon finishes their portion of a first surgical procedure for a first patient, the second surgeon may begin their portion of the surgical procedure chronology for the first patient. This allows the first surgeon to simultaneously perform their portion of a second surgical procedure for a second patient, thereby increasing operating room efficiency.

[0126] In some variations, a notification may optionally be generated 308. In some variations, the surgical procedure event notification may correspond to one or more surgical procedure events. For example, notification may comprise one or more of the GUIs shown in FIGS. 11 A, 1 IB, and 12A- 12C as described herein.

[0127] In some variations, a second surgical procedure notification may optionally be generated 310. For example, the second surgical procedure notification may notify a second patient of a scheduled patient start time or a change in the scheduled patient start time. For example, a second surgical procedure may be rescheduled for an earlier time if an estimated completion time of a first surgical procedure becomes sooner than an initial estimate. Conversely, a delay in the estimated first surgical procedure chronology may delay the second surgical procedure. In this manner, thescheduled second surgical procedure may be updated in real-time based on the estimated progress of the first surgical procedure.C. Example workflow

[0128] FIG. 4 depicts a flowchart representation of a patient workflow for a surgical facility comprising a first operating room 410, second operating room 420, and third operating room 430. Each operating room 410 may comprise a robotic surgical system 406 configured to perform a surgical procedure such as a cholecystectomy, appendectomy, colectomy, hernia repair, sleeve gastrectomy or other bariatric procedures, nephrectomy, hysterectomy, oophorectomy, and lobectomy. Additionally or alternatively, the robotic surgical system 406 may be moved between operating rooms.

[0129] In some variations, the robotic surgical system 406 may be configured to provide visualization of a surgical position (e.g., via endoscopic visualization) via computer vision and / or machine-learning models, position tracking of the system (e.g., support arm, support structure, end effector, surgeon), communication using an operator control interface (e.g., input device, output device), surgical procedure chronology estimation, and surgical procedure scheduling. In some variations, the robotic surgical system 406 may comprise one or more magnetic surgical end effectors 402 such as a magnetic grasper and external magnet (e.g., permanent magnet, electromagnet).

[0130] In some variations, a first patient start event 412 may comprise a first patient entering the first operating room 410. For example, the first patient may be wheeled into the first operating room 410 via a wheelchair. The first patient may then undergo surgery preparation 414. For example, the patient may be disposed on a patient platform and the support arms of the robotic surgical system 406 may transition from a first configuration to a second configuration as described with respect to FIGS. 5A and 5B. Next, the first patient may undergo a first incision 416 to create one or more ports. For example, a first port may be configured for a visualization device such as an endoscope and a second port may be configured for an end effector such as a grasper, retractor, cutter, and the like. Then, first patient surgery start 418 may begin. For example, a visualization device may be advanced through a first port of the first patient. Image processing and analysis 404 may beperformed for each of the surgical procedure events where imaging data is generated (e.g., surgery start through surgery finish). For example, imaging data may be input to a machine-learning model trained on a training set of images.

[0131] In some variations, treatment preparation 420 may comprise preparing a treatment site while under visualization. For example, a grasper may be introduced into a body cavity of the patient using the second port and used to retract a liver while being visualized by the endoscope advanced through the first port. Once the treatment site is prepared, the operator may perform treatment delivery 422. For example, a gastric sleeve procedure may comprise delivering staples to the stomach of the patient. Subsequent treatment finish 424 may comprise steps such as ligation and organ release (e.g., liver release from retraction). Surgery finish 426 may comprise removing the end effectors from the patient. For example, the visualization device may be retracted (e.g., withdrawn) from the first port. Patient finish 428 may comprise preparing the patient to leave the operating room. For example, the ports may be closed and the support arm may transition from the second configuration to a first configuration.

[0132] In some variations, the treatment preparation event 420 for the first patient may be the basis for a second patient start event 430 in the second operating room 450. The second patient may then undergo surgery preparation 432. Next, the second patient may undergo first incision 434 to create one or more ports. In some variations, treatment preparation 436 of the second patient may be the basis for a third patient start event 440 in the third operating room 460. The second patient may leave the second operating room 450 after the second patient finish event 438. The third patient may proceed through a similar workflow as the first patient and the second patient, at least a portion of which, is performed in parallel with another patient.

[0133] In some variations, an estimated surgical procedure chronology and / or scheduled surgical procedure may be updated continuously or at periodic intervals (e.g., upon completion of each surgical procedure event). In some variations, the estimate may be based on historical data (e.g., events database). For example, surgeons may perform a step of the same procedure at different rates.

[0134] In some variations, a communication channel may be established at predetermined intervals or as desired. For example, a communication channel (e.g., teleconference, videoconference) may be established between operating rooms using respective robotic surgical systems. Additionally or alternatively, a communication channel may be established between an operating room and an operator (e.g., administrator, technician, support personnel) in another room.C. Method of operating a robotic surgery system

[0135] Also described here are methods of operating a robotic surgery system. In some variations, a method of operating a robotic surgery system may comprise estimating one or more parameters of the robotic surgery system. In some variations, one or more parameters of the robotic surgery system may comprise one or more support arm data, end effector data, and external magnet data. In some variations, the support arm parameters may comprise one or more of kinematics, workspace, dynamics, and a state machine. In some variations, the end effector parameters may comprise one or more of kinematics, workspace, dynamics, and a state machine. In some variations, the external magnet parameters may comprise one or more of magnitude and direction of magnetic force. The parameters may be estimated from one or more sensors of the system. In some variations, the support arm data may comprise one or more of support arm movement data, support arm position data, support arm orientation data, and support arm configuration data. In some variations, the end effector data may comprise one or more of end effector movement data, end effector position data, end effector orientation data, and end effector configuration data.

[0136] In some variations, estimating one or more parameters of a patient may comprise one or more of patient data, a surgical procedure event, a surgical procedure chronology, and a surgical parameter. The reference marker may comprise one or more parameters of the patient. In some variations, the surgical procedure chronology may comprise a sequence of surgical procedure events. In some variations, the surgical procedure event may comprise one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish. In some variations, the patient data may comprise one or more of health data and patient procedure data. One or more of these parameters may be output onan AR or VR device. In some variations, a surgical procedure event may include patient events such as bleeding that may be sensed using an optical sensor coupled to an AR or VR device.

[0137] A real-time image of a field of view an operator may be generated using one or more of an optical sensor coupled to the operator and a visualization device. For example, the visualization device may comprise an endoscope (e.g., laparoscope camera) configured to visualize tissue within a body cavity of a patient. The optical sensor coupled to the operator may be, for example, a camera mounted on a headset worn by the operator. The real-time image may comprise a plurality of realtime images from different devices that may be selected based on the procedure and / or operator preference.

[0138] The real-time image of a field of view of an operator may be provided via a display coupled to an input device. The real-time image may comprise at least the robotic surgery system comprising one or more of a support arm, an end effector, and an external magnet, and at least one reference marker overlaid onto or adjacent to the robotic surgery system. In some variations, the reference marker may comprise one or more of a symbol, image, or text. The reference marker may comprise one or more of the estimated parameters of the robotic surgery system.

[0139] In some variations, an image of a view (i.e., perspective or field of view) of an operator may be provided by the display. The image may include an augmented end effector, such as a reproduction of an end effector (or other component of the system) including a virtual overlay (e.g., a reference marker or tag) on or adjacent to the end effector. For example, the end effector may be tracked by overlaying a reference marker onto or adjacent to the end effector in real-time. In some variations, the reference marker may comprise a digital copy and / or digital outline of the end effector.

[0140] In some variations, the reference marker displayed on a display (e.g., AR or VR headset) may correspond to support arm parameters such as kinematics, workspace, dynamics, and a state machine. For example, the reference marker may include a virtual overlay on a support arm showing an estimated workspace of the support arm. As the support arm reaches a predetermined edge of the workspace, the virtual overlay may change (e.g., change color, size) and may further include audio notifications such as verbal warnings, beeps, and the like. As another example, asupport arm (or end effector, or other component of the system) may comprise one or more fiducials configured to be measured by an optical sensor (e.g., mounted on an AR or VR headset). The position of the fiducials relative to the optical sensor (as a proxy for the operator) may be used to estimate and / or confirm one or more system parameters (e.g., support arm workspace). In some variations, the fiducials may be configured to be visible and discernable (e.g., size, color) through a sterile drape. In some variations, the reference marker may include an end effector parameter such as external load that may be overlayed on an image of the end effector. As another example, external magnet parameters including a magnitude and a direction of magnetic force (e.g., IO N downward force on an end effector) may be overlayed over an image of one or more end effectors and based on one or more of external magnet and support arm parameters.

[0141] In some variations, a patient parameter may be estimated in real-time during a procedure. For example, an organ (e.g., intestine) grasped by an end effector may be imaged by an optical sensor and used to estimate a length of the organ (e.g., length of intestine). For example, a portion of the intestine may be grasped by a pair of graspers during a bariatric procedure and measured using an optical sensor image. The length of the grasped portion of the intestine may be estimated at least in part on the known dimensions of the graspers in the image. In some variations, a length measurement of the tissue may be performed when each of the graspers are in a closed configuration. As sequential portions of the intestine are grasped, sequential length measurements may be performed to estimate a total length of the grasped intestine.

[0142] In some variations, a gaze of the operator may be measured. This may be achieved using the input / output control device. For example, the device may include one or more sensors for monitoring the gaze of the operator.

[0143] Next, the robotic surgical system may be controlled based on the measured gaze of the operator. One or more of the support arm, the end effector, and the external magnet may be controlled based on the measured gaze of the operator. For example, in some variations, controlling the system may comprise directing the gaze of the operator at a virtual actuator that is provided on the image (e.g., via the display of the input / output device) for a time period (e.g., at least 1 s, at least 2 s, at least 5 s, at least 8 s, at least 10 s, etc.), (via a support arm control signal) after the gaze isdetected for at least a threshold duration. Additionally or alternatively, a virtual actuator may be actuated based on gesture controls. For example, an operator use their hand to activate a virtual actuator provided in the image displayed to the operator. The virtual actuator may comprise one or more commands or control signals corresponding to one or more of the robotic surgery system and the real-time image.

[0144] In some variations, providing the image may comprise tracking one or more of the support arm, the end effector, and the external magnet. The reference marker may be overlayed onto or adjacent one or more of the support arm, the end effector, and the external magnet in real-time.

[0145] In some variations, a plurality of virtual actuators may be provided within the image, each configured to respond to the gaze of the operator to generate one or more control signals. In some variations, the display and the input device may be each configured to be worn over an eye of an of the operator. In some variations, the input device may comprise one or more of a headset, goggles, glasses, and a contact lens. In some variations, the end effector may comprise one or more of a visualization device, a grasper, a retractor, a magnetic positioning device, a sensor, an intracavity device, a delivery device, a retrieval device, a stapler, a clip applier, and an electrocautery hook.

[0146] Although the foregoing variations have, for the purposes of clarity and understanding, been described in some detail by illustration and example, it will be apparent that certain changes and modifications may be practiced, and are intended to fall within the scope of the appended claims. Additionally, it should be understood that the components and characteristics of the systems and devices described herein may be used in any combination. The description of certain elements or characteristics with respect to a specific figure are not intended to be limiting or nor should they be interpreted to suggest that the element cannot be used in combination with any of the other described elements. For all of the variations described herein, the steps of the methods may not be performed sequentially. Some steps are optional such that every step of the methods may not be performed.

Claims

CLAIMSWe claim:

1. A method of monitoring a surgical procedure for a patient, comprising: generating one or more of support arm data, end effector data, and patient data using one or more sensors of a robotic surgical system; identifying a surgical procedure event based on one or more of the support arm data, the end effector data, and the patient data; and estimating a surgical procedure chronology based on the identified surgical procedure event and surgeon data.

2. The method of claim 1, wherein the surgical procedure chronology comprises a sequence of surgical procedure events.

3. The method of claim 2, wherein the surgical procedure event comprises one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish.

4. The method of claim 1, wherein the support arm data comprises one or more of support arm movement data, support arm position data, support arm orientation data, and support arm configuration data.

5. The method of claim 1, wherein the end effector data comprises one or more of end effector movement data, end effector position data, end effector orientation data, and end effector configuration data.

6. The method of claim 1, wherein the patient data comprises one or more of health data and patient procedure data.

7. The method of claim 1, wherein the surgeon data comprises one or more of surgeon position data and historic surgical procedure data of one or more surgeons.

8. The method of claim 7, further comprising comparing the surgical procedure chronology to the historic surgical procedure data of one or more surgeons.

9. The method of claim 1, further comprising establishing a communication channel between the one or more sensors and a controller of the robotic surgical system in response to identifying a predetermined surgical procedure event.

10. The method of claim 9, wherein the controller is in either of a same room or a different room as the patient.

11. The method of claim 9, wherein the controller is configured to control one or more of end effector movement and end effector orientation.

12. The method of claim 1, further comprising generating a notification corresponding to the identified surgical procedure event.

13. The method of claim 1, wherein the one or more sensors comprise one or more of a motion sensor, a pressure sensor, an optical sensor, an audio sensor, a patient sensor, a magnetic sensor, and a proximity sensor.

14. The method of claim 1, wherein identifying the surgical procedure event is based on the surgeon data comprising audio data.

15. A method of managing a surgical procedure for a patient, comprising: generating data for a first surgical procedure using one or more sensors of a robotic surgical system;estimating a first surgical procedure chronology based on the first surgical procedure data, data for a first patient, and data for a surgeon data; and scheduling a second surgical procedure using a robotic surgical system based on the estimated first surgical procedure chronology, data for a second patient, and the surgeon data.

16. The method of claim 15, wherein the first surgical procedure chronology comprises a sequence of surgical procedure events.

17. The method of claim 16, wherein the sequence of surgical procedure events comprises one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish.

18. The method of claim 16, further comprising generating a notification corresponding to one or more surgical procedure events of the sequence of surgical procedure events.

19. The method of claim 15, further comprising generating a notification corresponding to the scheduled second surgical procedure.

20. The method of claim 15, wherein the first surgical procedure is performed in a first operating room and the second surgical procedure is performed in a second operating room.

21. The method of claim 15, wherein the first surgical procedure is started before the second surgical procedure.

22. The method of claim 21, wherein one or more surgical procedure events of the second surgical procedure are performed prior to completion of the first surgical procedure.

23. The method of claim 15, wherein the data for the first surgical procedure comprises one or more of movement data, position data, orientation data, and configuration data of one or more of a support arm and an end effector of the robotic surgical system.

24. The method of claim 15, wherein the data for one or both of the first and second patients comprises health data and patient procedure data.

25. The method of claim 15, wherein the surgeon data comprises one or more of surgeon position data and historic surgical procedure data of one or more surgeons.

26. A robotic surgical system, comprising: a support arm sensor configured to generate support arm data corresponding to one or more support arm characteristics; an end effector sensor configured to generate end effector data corresponding to one or more end effector characteristics; a patient sensor configured to generate patient data corresponding to one or more patient characteristics; and a controller coupled to each of the sensors, the controller comprising a processor and memory, the controller configured to: identify a surgical procedure event based on one or more of the support arm data, the end effector data, and the patient data; and estimate a surgical procedure chronology based on the identified surgical procedure event and surgeon data.

27. The system of claim 26, wherein the surgical procedure chronology comprises a sequence of surgical procedure events.

28. The system of claim 26, wherein the surgical procedure event comprises one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish.

29. The system of claim 26, wherein the support arm characteristics comprise one or more of support arm movement data, support arm position data, support arm orientation data, and support arm configuration data.

30. The system of claim 26, wherein the end effector characteristics comprise one or more of end effector movement data, end effector position data, end effector orientation data, and end effector configuration data.

31. The method of claim 26, wherein the patient characteristics comprise one or more of health data and patient procedure data.

32. The system of claim 26, wherein the surgeon data comprises one or more of surgeon position data and historic surgical procedure data of one or more surgeons.

33. The system of claim 26, wherein the patient data comprises imaging data.

34. The system of claim 26, wherein the end effector sensor is configured to measure one or more fiducials of the end effector.

35. The system of claim 26, wherein the end effector comprises one or more magnetic portions.

36. The system of claim 35, wherein the one or more magnetic portions comprises an external magnet configured to be placed over or on skin of a patient.

37. The system of claim 36, wherein the external magnet is coupled to a support arm.

38. The system of claim 26, wherein the processor is configured to establish a communication channel between one or more of the support arm sensor, the end effector sensor, and the patient sensor.

39. The system of claim 38, wherein the communication channel is a wired or wireless communication channel.

40. The system of claim 26, wherein the controller is in either a same or a different room as the patient.

41. A method of operating a robotic surgery system, comprising: estimating one or more parameters of the robotic surgery system; generating a real-time image of a field of view an operator using one or more of an optical sensor coupled to the operator and a visualization device; and providing, via a display coupled to an input device, the real-time image of the field of view of the operator, the image comprising at least the robotic surgery system comprising one or more of a support arm, an end effector, and an external magnet, and at least one reference marker overlaid onto or adjacent the robotic surgery system, wherein the at least one reference marker comprises one or more of the estimated parameters of the robotic surgery system.

42. The method of claim 41, wherein the reference marker comprises one or more of a symbol, image, or text.

43. The method of claim 41, wherein the one or more parameters of the robotic surgery system comprise one or more support arm data, end effector data, and external magnet data.

44. The method of claim 43, wherein the support arm parameters comprise one or more of kinematics, workspace, dynamics, and state machine.

45. The method of claim 43, wherein the end effector parameters comprise one or more of kinematics, workspace, dynamics, and state machine.

46. The method of claim 43, wherein the external magnet parameters comprise one or more of magnitude and direction of magnetic force.

47. The method of claim 41, further comprising estimating one or more parameters of a patient comprising one or more of patient data, a surgical procedure event, a surgical procedure chronology, and a surgical parameter, and wherein the at least one reference marker comprises the one or more parameters of the patient.

48. The method of claim 47, wherein the surgical procedure chronology comprises a sequence of surgical procedure events.

49. The method of claim 48, wherein the surgical procedure event comprises one or more of patient start, surgery preparation, first incision, surgery start, treatment preparation, treatment delivery, treatment finish, surgery finish, and patient finish.

50. The method of claim 47, wherein the patient data comprises one or more of health data and patient procedure data.

51. The method of claim 43, wherein the support arm data comprises one or more of support arm movement data, support arm position data, support arm orientation data, and support arm configuration data.

52. The method of claim 43, wherein the end effector data comprises one or more of end effector movement data, end effector position data, end effector orientation data, and end effector configuration data.

53. The method of claim 41, further comprising: measuring a gaze of the operator using the input device; and controlling one or more of the support arm, the end effector, and the external magnet based on the measured gaze of the operator.

54. The method of claim 41, wherein providing the image comprises: tracking one or more of the support arm, the end effector, and the external magnet; and overlaying the reference marker onto or adjacent one or more of the support arm, the end effector, and the external magnet in real-time.

55. The method of claim 53 further comprising providing, within the image, a plurality of virtual actuators each configured to respond to the gaze of the operator to generate one or more control signals.

56. The method of claim 41, wherein the display and the input device are each configured to be worn over an eye of an of the operator.

57. The method of claim 41, wherein the input device comprises one or more of a headset, goggles, glasses, and a contact lens.

58. The method of claim 41, wherein the end effector comprises one or more of a visualization device, a grasper, a retractor, a magnetic positioning device, a sensor, an intracavity device, a delivery device, a retrieval device, a stapler, a clip applier, and an electrocautery hook.

Images