Systems and methods for cooperation between surgeon and assistant in virtual procedure training

Abstract

A surgical robotic training system includes a plurality of augmented and / or virtual reality headsets that are worn by a plurality of trainees. At least one of trainee is operating a surgeon console and another trainee is an assistant. The system includes a simulator for generating a multi-user virtual environment, which renders a surgical robotic system that is controllable by the surgeon console. Another user, such as a surgical assistant trainee, wears one of the headsets and may navigate in the virtual environment as well as interact with the virtual surgical robotic system. The virtual environment provides for interactions between the users and the environment, allowing the surgeon trainee to instruct the assistant to perform certain manual tasks, such as moving arms to attach to access ports, retracting and inserting instruments, and the like.

Description

SYSTEMS AND METHODS FOR COOPERATION BETWEEN SURGEON AND ASSISTANT IN VIRTUAL PROCEDURE TRAININGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 612,393, filed December 20, 2023, the entire content of which is incorporated herein by reference.BACKGROUND

[0002] Surgical robotic systems are currently being used in a variety of medical procedures, including minimally invasive surgical procedures. Some surgical robotic systems include a surgeon console controlling a surgical robotic arm and a surgical instrument having an end effector (e.g., forceps or grasping instrument) coupled to and actuated by the robotic arm. In operation, the robotic arm is moved to a position over a patient and then guides the surgical instrument into a small incision via a surgical port or a natural orifice of a patient to position the end effector at a work site within the patient’s body. A variety of different types of instruments are used with surgical robotic systems that are designed to perform specific functions, such as suturing during minimally invasive robotically assisted surgical procedures.

[0003] Surgical robotic systems involve complex interactions between the surgeon at the surgeon console and one or more assistants at the bedside operating the robotic arms. Current virtual training solutions focus on only one of these roles. A unified training system would bridge the gap between role-specific trainers.SUMMARY

[0004] The present disclosure provides a surgical robotic training system including a surgeon console having a main screen and a side screen and optionally one or more robotic arms. The system also includes a first headset for use by a surgeon trainee and a second headset for use by a bedside assistant. The headsets may be augmented (AR), virtual reality (VR), or mixed reality headsets. The surgeon trainee may test run full surgical procedures, starting from the setup of the surgical system in a virtual environment, which is generated by the system. One or both of the first and second headsets are virtually disposed inside the virtual environment allowing for cooperative training of the surgeon trainee and the bedside assistant. This allows for procedure training for the surgeon trainee on the surgeon console with a full virtual operating room, i.e., virtual environment, view displayed on the first headset. The virtual environment may also be shown on the side screen of the surgeon console and / or as apassthrough on the headset. The first headset may operate in a passthrough mode when the headset is pointed at the first screen, allowing the surgeon trainee to view the first screen directly or in a virtual mode, displaying the virtual environment when the first headset is pointing away from the first screen of the surgeon console. The bedside assistant wearing the second headset may be located in any physical space, such as the same room as the surgeon trainee using the surgeon console or anywhere else where the system and the second headset are communicatively connected to each other, e.g., via an Internet connection. In further embodiment, multiple headsets, for each member of the surgical team may be used to provide for virtual training on a wider scale.

[0005] Providing a differentiated simulation training experience reduces training costs associated with training using physical robotic systems and operating rooms. In addition, the disclosed training system also improves training quality by enabling high fidelity training, i.e., high resolution AR / VR experience, without access to the physical robotic systems.

[0006] According to one embodiment of the present disclosure, a surgical robotic training system is disclosed. The surgical robotic training system includes a simulator for generating a virtual environment including a virtual patient and one or more robotic arms having a surgical instrument and / or a camera. The system may include a surgeon console having one or more surgeon handle controllers for receiving surgeon input for moving one or more robotic arms and / or their corresponding surgical instruments in the virtual environment. The console also includes a first screen for displaying a laparoscopic view of the surgical instrument inside the virtual patient in the virtual environment. The system further includes one or more assistant handle controllers for receiving assistant input for engaging with one or more robotic arms and / or the surgical instrument in the virtual environment. The system additionally includes an assistant headset for displaying an outside view of the virtual environment based on a position and an orientation of the assistant headset, where the outside view displays movement of one of the robotic arms or the surgical instruments in the virtual environment in response to the surgeon input.

[0007] Implementations of the above embodiment may include one or more of the following features. According to one aspect of the above embodiment, the assistant input may include coupling the surgical instrument to a robotic arm or decoupling the surgical instrument from a robotic arm. The assistant input may include retracting the surgical instrument from the virtual patient or inserting the surgical instrument into the virtual patient. The laparoscopic view may include movement of the surgical instrument inside the virtual patient in the virtual environment in response to the assistant input. The surgeon console may include a secondscreen for displaying graphical user interface for providing instructions to the assistant headset. The instructions may include set up instructions for one or more robotic arms in the virtual environment. The instructions may include exchanging of the surgical instrument or retracting of the surgical instrument. The headset may be a virtual reality headset, an augmented reality headset, or a mixed reality headset.

[0008] According to another embodiment of the present disclosure, a surgical robotic training system is disclosed. The surgical robotic training system includes a simulator for generating a virtual environment including a virtual patient and one or more robotic arms having a surgical instrument. The system further includes a surgeon console having one or more surgeon handle controllers for receiving surgeon input for moving one or more of the robotic arms or the corresponding surgical instruments in the virtual environment. The surgeon console also includes a first screen for displaying a laparoscopic view of the surgical instrument inside the virtual patient in the virtual environment. The system also includes a surgeon headset operable in a passthrough mode in which the surgeon headset displays the first screen of the surgeon console and a virtual mode in which the surgeon headset displays the virtual environment, where the surgeon headset automatically switches between the passthrough mode and the virtual mode based on a position and orientation of the surgeon headset relative to the first screen.

[0009] Implementations of the above embodiment may include one or more of the following features. According to one aspect of the above embodiment, the surgical robotic training system may also include one or more assistant handle controllers for receiving assistant input for engaging with one or more robotic arms and the surgical instrument in the virtual environment, and an assistant headset for displaying an outside view of the virtual environment based on a position and orientation of the assistant headset, where the outside view shows movement of the at least one robotic arm or the surgical instrument in the virtual environment in response to the surgeon input.

[0010] The surgeon headset may be a virtual reality headset, an augmented reality headset, or a mixed reality headset. The assistant input may include coupling the surgical instrument to a robotic arm or decoupling the surgical instrument from a robotic arm. The assistant input may also include retracting the surgical instrument from the virtual patient or inserting the surgical instrument into the virtual patient. The laparoscopic view may include movement of the surgical instrument inside the virtual patient in the virtual environment in response to the assistant input. The surgeon console may include a second screen for displaying a graphical user interface for providing instructions to the assistant headset. The instructions may includeset up instructions for the robotic arms in the virtual environment. The instructions may also include exchanging of the surgical instrument or retracting of the surgical instrument.

[0011] According to a further embodiment of the present disclosure, a surgical robotic training system is disclosed. The surgical robotic training system includes a simulator for generating a multi-user virtual environment and hosting a plurality of users therein, where the virtual environment includes a virtual patient and one or more robotic arms having a surgical instrument and / or a camera. The virtual environment also provides for interaction between the plurality of users. The system further includes a surgeon console usable by a first user of the plurality of users. The surgeon console also includes one or more surgeon handle controllers for receiving surgeon input for moving the robotic arms and / or the surgical instrument in the virtual environment and a first screen for displaying a laparoscopic view of the surgical instrument inside the virtual patient in the virtual environment. The system additionally includes an assistant handle controller usable by a second user of the plurality of users for receiving assistant input for engaging with the robotic arms and / or the surgical instrument in the virtual environment. The system also includes an assistant headset usable by the second user of the plurality of users for displaying an outside view of the virtual environment based on position and orientation of the assistant headset, where the outside view includes movement of the robotic arms and / or the surgical instrument in the virtual environment in response to the surgeon input.

[0012] Implementations of the above embodiment may include one or more of the following features. According to one aspect of the above embodiment, the surgeon console may also include a second screen for displaying a graphical user interface for providing instructions to the assistant headset. The simulator may also transmit instructions to the assistant headset.

[0013] According to a further embodiment of the present disclosure, a surgical robotic training system is disclosed. The system includes a robotic arm having a surgical instrument and a simulator for generating a multi-user virtual environment and hosting a plurality of users therein, wherein the virtual environment includes a virtual patient and a virtual robotic arm having a virtual surgical instrument simulating the robotic arm and the surgical instrument. The system also includes a surgeon console, which has at least one surgeon handle controller for receiving surgeon input for moving at least one of the virtual robotic arm or the virtual surgical instrument in the virtual environment and mirroring same movement on at least one of the robotic arm or the surgical instrument. The surgeon console also includes a first screen for displaying a laparoscopic view of the virtual surgical instrument inside the virtual patientin the virtual environment, wherein an exchange of the surgical instrument is reflected in the virtual environment.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Various embodiments of the present disclosure are described herein with reference to the drawings wherein:

[0015] FIG. 1 is a perspective view of a surgical robotic system including a control tower, a console, and one or more surgical robotic arms each disposed on a movable cart according to an embodiment of the present disclosure;

[0016] FIG. 2 is a perspective view of a surgical robotic arm of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0017] FIG. 3 is a perspective view of a movable cart having a setup arm with the surgical robotic arm of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0018] FIG. 4 is a schematic diagram of a computer architecture of the surgical robotic system of FIG. 1 according to an embodiment of the present disclosure;

[0019] FIG. 5 is a plan schematic view of movable carts of FIG. 1 positioned about a surgical table according to an embodiment of the present disclosure;

[0020] FIG. 6 is a perspective view of a surgical robotic training system including a first headset and a second headset according to an embodiment of the present disclosure;

[0021] FIG. 7 is a perspective view of a headset and hand controllers of the system of FIG. 6 according to an embodiment of the present disclosure;

[0022] FIG. 8 is a schematic diagram of a training simulator console of the surgical robotic training system of FIG. 6 according to an embodiment of the present disclosure;

[0023] FIG. 9 is a view of a virtual environment generated by a simulator and displayed on the second headset according to an embodiment of the present disclosure;

[0024] FIG. 10 is a view of the virtual environment generated by the simulator and displayed on the first headset according to an embodiment of the present disclosure;

[0025] FIGS. 11A-C are screenshots of setup screens displayed during training on the surgeon console according to an embodiment of the present disclosure; and

[0026] FIGS. 12A-C are views of the virtual environment generated by the simulator and displayed on the second headset according to an embodiment of the present disclosure.DETAILED DESCRIPTION

[0027] Embodiments of the presently disclosed surgical robotic system are described in detail with reference to the drawings, in which like reference numerals designate identical or corresponding elements in each of the several views.

[0028] As will be described in detail below, the present disclosure is directed to a surgical robotic system, which includes a surgeon console, a control tower, and one or more movable carts having a surgical robotic arm coupled to a setup arm. The surgeon console receives user input through one or more interface devices. The input is processed by the control tower as movement commands for moving the surgical robotic arm and an instrument and / or camera coupled thereto. Thus, the surgeon console enables teleoperation of the surgical arms and attached instruments / camera. The surgical robotic arm includes a controller, which is configured to process the movement commands and to generate a torque commands for activating one or more actuators of the robotic arm, which would, in turn, move the robotic arm in response to the movement commands.

[0029] With reference to FIG. 1, a surgical robotic system 10 includes a control tower 20, which is connected to all the components of the surgical robotic system 10 including a surgeon console 30 and one or more movable carts 60. Each of the movable carts 60 includes a robotic arm 40 having a surgical instrument 50 coupled thereto. The robotic arms 40 also couple to the movable carts 60. The robotic system 10 may include any number of movable carts 60 and / or robotic arms 40.

[0030] The surgical instrument 50 is configured for use during minimally invasive surgical procedures. In embodiments, the surgical instrument 50 may be configured for open surgical procedures. In further embodiments, the surgical instrument 50 may be an electrosurgical forceps configured to seal tissue by compressing tissue between jaws and applying electrosurgical current thereto. In yet further embodiments, the surgical instrument 50 may be a surgical stapler including a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners, e.g., staples, and cutting stapled tissue. In yet further embodiments, the surgical instrument 50 may be a surgical clip applier including a pair of jaws configured apply a surgical clip onto tissue.

[0031] One of the robotic arms 40 may include an endoscopic camera 51 configured to capture video of the surgical site. The endoscopic camera 51 may be a stereoscopic endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The endoscopic camera 51 is coupled to a video processing device 56, which may be disposed within the control tower 20. The videoprocessing device 56 may be any computing device as described below configured to receive the video feed from the endoscopic camera 51 and output the processed video stream.

[0032] The surgeon console 30 includes a first screen 32, which displays a video feed of the surgical site provided by camera 51 disposed on the robotic arm 40, and a second screen 34, which displays a user interface for controlling the surgical robotic system 10. The first screen 32 and second screen 34 may be touchscreens allowing for displaying various graphical user inputs.

[0033] The surgeon console 30 also includes a plurality of user interface devices, such as foot pedals 36 and a pair of handle controllers 38a and 38b which are used by a user to remotely control robotic arms 40. The surgeon console further includes an armrest 33 used to support clinician’s arms while operating the handle controllers 38a and 38b.

[0034] The control tower 20 includes a screen 23, which may be a touchscreen, and outputs on the graphical user interfaces (GUIs). The control tower 20 also acts as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as to move the robotic arms 40 and the corresponding surgical instrument 50, based on a set of programmable instructions and / or input commands from the surgeon console 30, in such a way that robotic arms 40 and the surgical instrument 50 execute a desired movement sequence in response to input from the foot pedals 36 and the handle controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, repositioning camera movement and electrosurgical activation / deactivation. In particular, the foot pedals 36 may be used to perform a clutching action on the hand controllers 38a and 38b. Clutching is initiated by pressing one of the foot pedals 36, which disconnects (i.e., prevents movement inputs) the hand controllers 38a and / or 38b from the robotic arm 40 and corresponding instrument 50 or camera 51 attached thereto. This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and the instrument 50 and / or camera 51. This is useful when reaching control boundaries of the surgical space.

[0035] Each of the control tower 20, the surgeon console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21, 31, 41 are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term “network,” whether plural or singular, as used herein, denotes a data network, including, but not limited to, the Internet, Intranet, a wide area network, or a local area network, and without limitation as to the full scope of the definition of communication networks as encompassed by the present disclosure. Suitable protocols include, but are notlimited to, transmission control protocol / intemet protocol (TCP / IP), datagram protocol / intemet protocol (UDP / IP), and / or datagram congestion control protocol (DCCP). Wireless communication may be achieved via one or more wireless configurations, e.g., radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances, using short length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high level communication protocols using small, low-power digital radios based on the IEEE 122.15.4-1203 standard for wireless personal area networks (WPANs)).

[0036] The computers 21, 31, 41 may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM (EEPROM), non-volatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and / or set of instructions described in the present disclosure including, but not limited to, a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted for by using any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and / or set of instructions described herein.

[0037] With reference to FIG. 2, each of the robotic arms 40 may include a plurality of links 42a, 42b, 42c, which are interconnected at joints 44a, 44b, 44c, respectively. Other configurations of links and joints may be utilized as known by those skilled in the art. The joint 44a is configured to secure the robotic arm 40 to the movable cart 60 and defines a first longitudinal axis. With reference to FIG. 3, the movable cart 60 includes a lift 67 and a setup arm 61, which provides a base for mounting of the robotic arm 40. The lift 67 allows for vertical movement of the setup arm 61. The movable cart 60 also includes a display 69 for displaying information pertaining to the robotic arm 40. In embodiments, the robotic arm 40 may include any type and / or number of joints.

[0038] The setup arm 61 includes a first link 62a, a second link 62b, and a third link 62c, which provide for lateral maneuverability of the robotic arm 40. The links 62a, 62b, 62c are interconnected at joints 63a and 63b, each of which may include an actuator (not shown) for rotating the links 62b and 62b relative to each other and the link 62c. In particular, the links 62a, 62b, 62c are movable in their corresponding lateral planes that are parallel to each other, thereby allowing for extension of the robotic arm 40 relative to the patient (e.g., surgical table).In embodiments, the robotic arm 40 may be coupled to the surgical table (not shown). The setup arm 61 includes controls 65 for adjusting movement of the links 62a, 62b, 62c as well as the lift 67. In embodiments, the setup arm 61 may include any type and / or number of joints.

[0039] The third link 62c may include a rotatable base 64 having two degrees of freedom. In particular, the rotatable base 64 includes a first actuator 64a and a second actuator 64b. The first actuator 64a is rotatable about a first stationary arm axis which is perpendicular to a plane defined by the third link 62c and the second actuator 64b is rotatable about a second stationary arm axis which is transverse to the first stationary arm axis. The first and second actuators 64a and 64b allow for full three-dimensional orientation of the robotic arm 40.

[0040] The actuator 48b of the joint 44b is coupled to the joint 44c via the belt 45a, and the joint 44c is in turn coupled to the joint 46b via the belt 45b. Joint 44c may include a transfer case coupling the belts 45a and 45b, such that the actuator 48b is configured to rotate each of the links 42b, 42c and a holder 46 relative to each other. More specifically, links 42b, 42c, and the holder 46 are passively coupled to the actuator 48b which enforces rotation about a pivot point “P” which lies at an intersection of the first axis defined by the link 42a and the second axis defined by the holder 46. In other words, the pivot point “P” is a remote center of motion (RCM) for the robotic arm 40. Thus, the actuator 48b controls the angle 0 between the first and second axes allowing for orientation of the surgical instrument 50. Due to the interlinking of the links 42a, 42b, 42c, and the holder 46 via the belts 45a and 45b, the angles between the links 42a, 42b, 42c, and the holder 46 are also adjusted to achieve the desired angle 0. In embodiments, some or all the joints 44a, 44b, 44c may include an actuator to obviate the need for mechanical linkages.

[0041] The joints 44a and 44b include an actuator 48a and 48b configured to drive the joints 44a, 44b, 44c relative to each other through a series of belts 45a and 45b or other mechanical linkages such as a drive rod, a cable, or a lever and the like. In particular, the actuator 48a is configured to rotate the robotic arm 40 about a longitudinal axis defined by the link 42a.

[0042] With reference to FIG. 2, the holder 46 defines a second longitudinal axis and configured to receive an instrument drive unit (IDU) 52 (FIG. 1). The IDU 52 is configured to couple to an actuation mechanism of the surgical instrument 50 and the camera 51 and is configured to move (e.g., rotate) and actuate the instrument 50 and / or the camera 51. IDU 52 transfers actuation forces from its actuators to the surgical instrument 50 to actuate components of an end effector 49 of the surgical instrument 50. The holder 46 includes a sliding mechanism 46a, which is configured to move the IDU 52 along the second longitudinal axis defined by the holder 46. The holder 46 also includes a joint 46b, which rotates the holder 46 relative to thelink 42c. During endoscopic procedures, the instrument 50 may be inserted through an endoscopic access port 55 (FIG. 3) held by the holder 46. The holder 46 also includes a port latch 46c for securing the access port 55 to the holder 46 (FIG. 2).

[0043] The IDU 52 is attached to the holder 46, followed by a sterile interface module (SIM) 43 being attached to a distal portion of the IDU 52. The SIM 43 is configured to secure a sterile drape (not shown) to the IDU 52. The instrument 50 is then attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46. The SIM 43 includes a plurality of drive shafts configured to transmit rotation of individual motors of the IDU 52 to the instrument 50 thereby actuating the instrument 50. In addition, the SIM 43 provides a sterile barrier between the instrument 50 and the other components of robotic arm 40, including the IDU 52.

[0044] The robotic arm 40 also includes a plurality of manual override buttons 53 (FIG. 1) disposed on the IDU 52 and the setup arm 61, which may be used in a manual mode. The user may press one or more of the buttons 53 to move the component associated with the button 53.

[0045] With reference to FIG. 4, each of the computers 21, 31, 41 of the surgical robotic system 10 may include a plurality of controllers, which may be embodied in hardware and / or software. The computer 21 of the control tower 20 includes a controller 21a and safety observer 21b. The controller 21a receives data from the computer 31 of the surgeon console 30 about the current position and / or orientation of the handle controllers 38a and 38b and the state of the foot pedals 36 and other buttons. The controller 21a processes these input positions to determine desired drive commands for each joint of the robotic arm 40 and / or the IDU 52 and communicates these to the computer 41 of the robotic arm 40. The controller 21a also receives the actual joint angles measured by encoders of the actuators 48a and 48b and uses this information to determine force feedback commands that are transmitted back to the computer 31 of the surgeon console 30 to provide haptic feedback through the handle controllers 38a and 38b. The safety observer 21b performs validity checks on the data going into and out of the controller 21a and notifies a system fault handler if errors in the data transmission are detected to place the computer 21 and / or the surgical robotic system 10 into a safe state.

[0046] The computer 41 includes a plurality of controllers, namely, a main cart controller 41a, a setup arm controller 41b, a robotic arm controller 41c, and an instrument drive unit (IDU) controller 41d. The main cart controller 41a receives and processes joint commands from the controller 21a of the computer 21 and communicates them to the setup arm controller 41b, the robotic arm controller 41c, and the IDU controller 4 Id. The main cart controller 41a also manages instrument exchanges and the overall state of the movable cart 60, the robotic arm 40,and the IDU 52. The main cart controller 41a also communicates actual joint angles back to the controller 21a.

[0047] Each of joints 63a and 63b and the rotatable base 64 of the setup arm 61 are passive joints (i.e., no actuators are present therein) allowing for manual adjustment thereof by a user. The joints 63a and 63b and the rotatable base 64 include brakes that are disengaged by the user to configure the setup arm 61. The setup arm controller 41b monitors slippage of each of joints 63a and 63b and the rotatable base 64 of the setup arm 61, when brakes are engaged or can be freely moved by the operator when brakes are disengaged, but do not impact controls of other joints. The robotic arm controller 41c controls each joint 44a and 44b of the robotic arm 40 and calculates desired motor torques required for gravity compensation, friction compensation, and closed loop position control of the robotic arm 40. The robotic arm controller 41c calculates a movement command based on the calculated torque. The calculated motor commands are then communicated to one or more of the actuators 48a and 48b in the robotic arm 40. The actual joint positions are then transmitted by the actuators 48a and 48b back to the robotic arm controller 41c.

[0048] The IDU controller 4 Id receives desired joint angles for the surgical instrument 50, such as wrist and jaw angles, and computes desired currents for the motors in the IDU 52. The IDU controller 4 Id calculates actual angles based on the motor positions and transmits the actual angles back to the main cart controller 41a.

[0049] The robotic arm 40 is controlled in response to a pose of the handle controller controlling the robotic arm 40, e.g., the handle controller 38a, which is transformed into a desired pose of the robotic arm 40 through a hand eye transform function executed by the controller 21a. The hand eye function, as well as other functions described herein, is / are embodied in software executable by the controller 2 la or any other suitable controller described herein. The pose of one of the handle controllers 38a may be embodied as a coordinate position and roll-pitch-yaw (RPY) orientation relative to a coordinate reference frame, which is fixed to the surgeon console 30. The desired pose of the instrument 50 is relative to a fixed frame on the robotic arm 40. The pose of the handle controller 38a is then scaled by a scaling function executed by the controller 21a. In embodiments, the coordinate position may be scaled down and the orientation may be scaled up by the scaling function. In addition, the controller 21a may also execute a clutching function, which disengages the handle controller 38a from the robotic arm 40. In particular, the controller 21a stops transmitting movement commands from the handle controller 38a to the robotic arm 40 if certain movement limits or other thresholdsare exceeded and acts like a virtual clutch mechanism, e.g., limits mechanical input from effecting mechanical output.

[0050] The desired pose of the robotic arm 40 is based on the pose of the handle controller 38a and is then passed by an inverse kinematics function executed by the controller 21a. The inverse kinematics function calculates angles for the joints 44a, 44b, 44c of the robotic arm 40 that achieve the scaled and adjusted pose input by the handle controller 38a. The calculated angles are then passed to the robotic arm controller 41c, which includes a joint axis controller having a proportional-derivative (PD) controller, the friction estimator module, the gravity compensator module, and a two-sided saturation block, which is configured to limit the commanded torque of the motors of the joints 44a, 44b, 44c.

[0051] With reference to FIG. 5, the surgical robotic system 10 is set up around a surgical table 90. The system 10 includes movable carts 60a-d, which may be numbered “1” through “4.” During setup, each of the carts 60a-d are positioned around the surgical table 90. Position and orientation of the carts 60a-d depends on a plurality of factors, such as placement of a plurality of access ports 55a-d, which in turn, depends on the surgery being performed. Once the port placements are determined, the access ports 55a-d are inserted into the patient, and carts 60a-d are positioned to insert instruments 50 and the endoscopic camera 51 into corresponding ports 55a-d.

[0052] During use, each of the robotic arms 40a-d is attached to one of the access ports 55a-d that is inserted into the patient by attaching the latch 46c (FIG. 2) to the access port 55 (FIG. 3). The IDU 52 is attached to the holder 46, followed by the SIM 43 being attached to a distal portion of the IDU 52. Thereafter, the instrument 50 is attached to the SIM 43. The instrument 50 is then inserted through the access port 55 by moving the IDU 52 along the holder 46.

[0053] FIG. 6 shows a surgical robotic training system 200 for use along with the surgical robotic system 10 in training. The training system 200 is a virtual reality system and includes a plurality of headsets, e.g., a first headset 202 and a second headset 203, which may be substantially identical. With reference to FIG. 7, which shows outside and interior views of each of the headsets 202 and 203, which include a head-mounted display worn by a user around their head. The headsets 202, 203 are configured to display a virtual environment, i.e., virtual system 10. Each of the headsets 202, 203 includes a controller 206, an imaging device 210 (e.g., camera), and a screen 208. The controller 206 includes a memory configured to have instructions stored thereon and a processor configured to execute the instructions.

[0054] The headsets 202, 203 may also operate with one or more handheld controllers 220 that enable the users to navigate and / or interact with the virtual system 10. In some embodiments,the headsets 202, 203 and / or the handheld controllers 220 may be modified versions of those included in any suitable virtual reality hardware system that is commercially available for applications including virtual and augmented reality environments, such as HOLOLENS® from Microsoft, of Redmond, WA, META QUEST PRO® available from Meta, of Menlo Park, CA. In embodiments where an enclosed virtual reality set is being used, front facing cameras, i.e., imaging device 210, may be disposed on the headsets 202, 203 to provide a front-facing view (e.g., external images of the patient) to be displayed on the headsets 202, 203 along with virtual or augmented projections. In embodiments, where the headsets 202, 203 is an augmented reality display having a projection window, only projections are displayed on the screen.

[0055] The headsets 202, 203 and / or the handheld controllers 220 may be modified to enable interaction by a user with a virtual robotic surgical environment (e.g., a handheld controllers 220 may be modified to operate the surgeon console 30). In some embodiments, the virtual reality system 400 may further include one or more tracking emitters as part of the external vision system that emit infrared light into the workspace. The tracking emitters may, for example, be mounted on a wall, ceiling, fixture, or other suitable mounting surface. Tracking features 209, which may be sensors, markers, or reflectors are disposed on outward-facing surfaces of the headsets 202, 203 and / or handheld controllers 220 for detecting the emitted infrared light or reflecting the light back to the external vision system. Based on the location of any sensors that detect the emitted light and when such sensors detect the emitted light after the light is emitted, the controller 21a may be configured to determine (e.g., through triangulation) the location and orientation of the headsets 202, 203 and / or handheld controllers 220 within the workspace. In other embodiments, various position tracking sensors 212, (such as inertial measurement units (IMU), accelerometers or gyroscopes, etc.) may be used to determine location and orientation of the headsets 202, 203 and handheld controllers 220.

[0056] In some embodiments, the headsets 202, 203 may include straps (e.g., with buckles, elastic, snaps, etc.) that facilitate mounting to the user’s head. For example, the headsets 202, 203 may be structured like goggles, a headband or headset, a cap, etc. The headsets 202, 203 may include two eyepiece assemblies providing a stereoscopic immersive display, though alternatively may include any suitable display.

[0057] The handheld controllers 220 may include interactive features that the user may manipulate to interact with the virtual robotic surgical environment. For example, the handheld controllers 220 may include one or more buttons, triggers, touch-sensitive features, scroll wheels, switches, and / or other suitable interactive features. Additionally, the handheldcontrollers 220 may have any of various form factors, such as a wand, pinchers, round shapes (e.g., ball or egg-shaped), etc. In some embodiments, the handheld controller may include a carried device (e.g., wand, remote device, etc.) and / or a garment worn on the user’s hand (e.g., gloves, rings, wristbands, etc.) and including sensors and / or configured to cooperate with external sensors to thereby provide tracking of the user’s hand(s), individual fmger(s), wrist(s), etc. Other suitable controllers may additionally or alternatively be used (e.g., sleeves configured to provide tracking of the user's arm(s)).

[0058] During training, the system 10 may be set up in the same manner as shown in FIG. 5 around a training dummy disposed on the table 90. This allows for the surgeon trainee to practice using the system 10, such that the surgeon console 30 controls the robotic arms 40, camera 51, and the instruments 50, etc.

[0059] In embodiments, the surgeon console 30, may be configured to simulate operation of the system 10 and may be operated in a virtual simulation (i.e., training) mode. In this mode, the surgeon console 30 is configured to simulate operation of the surgical robotic system 10 (e.g., clutching, camera control, suturing, and stapling) based on a set of programmable instructions and / or input commands from the surgical console 30 via the handle controllers 38a and 38b and the foot pedals 36 without operating actual robotic arms 40, camera 51, and the instruments 50, etc. The surgeon console 30 simulates, in response to programmable instructions and / or input commands, virtual instances of the control tower 20, one or more movable carts 60, the robotic arm 40, the surgical instrument 50, and the camera 51 disposed along with the surgical instrument 50 on the robotic arm 40.

[0060] With reference to FIG. 4, the surgeon console 30 may include one or more simulation computers, each including a plurality of controllers, namely, a primary controller 110, a simulation controller 140, and a simulator 170 operably connected to a shared memory 130. The primary controller 110 simulates the controller 21a. The shared memory 130 is configured to store session data and instrument information. The session data contains information such as, a scenario name, an initial position of an instrument, name of the instrument, and functionality of the instrument, e.g., whether instruments operate with electrosurgical generators, staple tissue, etc. The initial position of the instrument includes the pivot point “P” e.g., a tool center point (TCP) and joint 46b of holder 46, e.g., the remote center of motion (RCM). Optionally, the name of the instrument may be encoded in a vector look-up table, e.g., 256 x 1 vector, identified by a numerical number corresponding to an instrument identifier including additional instrument information and may be received from the simulator 170.

[0061] The instrument information may include a maximum joint limit, a minimum joint limit of the surgical instrument 50, an appropriate kinematic parameters of the instrument 50 (e.g., jaw offset and wrist length), an actual position of the surgical instrument 50 and camera 51, jaw opening ratios, and active instrument functions. The shared memory 130 may further include additional information, such as, state of the main cart controller 41a, active energy states, and initial exercise information.

[0062] Primary controller 110 and the simulation controller 140 may be implemented in a computer, which may be running a Unix or Linux operating system, e.g., QNX, and the simulator 170 may be implemented in another computer, which may be running WINDOWS® operating system. The primary controller 110 and the simulator 170 may be interconnected using any suitable communication network based on wired or wireless communication protocols. Each of the primary controller 110, the simulation controller 140, and the simulator 170 may be implemented in any combination of computers, interconnected to the one or more computers using any suitable communication network based on wired or wireless communication protocols. In some instances, the primary controller 110 and the simulation controller 140 may be interconnected through one or more transmission protocols, including machine-to-machine communication protocols, such as a Data Distribution Service protocol for Real-Time Systems (DDS) including Real-Time Publish Subscribe Protocol (RTPS) enabling scalable, real-time, dependable, high performance, interoperable packet, or data exchanges. In some instances, the primary controller 110 and the simulator 170 may be set up as virtual machines.

[0063] The simulator 170 of the surgeon console 30 simulates the commands and responses of the computer 41 including the main cart controller 4 la, the setup arm controller 4 lb, the robotic arm controller 41c, and the instrument drive unit (IDU) controller 4 Id to and / or from the primary controller 110.

[0064] The primary controller 110 simulates the computer 21 of the control tower 20, including the controller 21a. In particular the primary controller 110 receives session data from simulator 170 to determine desired drive commands for each joint, e.g., of the robotic arm 40 and / or the instrument drive unit 52 and communicates the desired drive commands and the instrument drive unit 52 to a virtual representation of the robotic arm 40 of the main cart controller 41a, which is simulated by the simulator 170 of the surgeon console 30. The primary controller 110 may be further configured to receive actual joint angles of the surgical instrument 50 to determine force feedback commands transmitted to the simulator 170 to provide haptic feedback through the handle controllers 38a and 38b of the surgical console 30.

[0065] With reference to FIG. 8 the simulation controller 140 includes one or more communication interfaces. The communication interfaces include a simulator interface 142a and a primary controller interface 142b. The simulator interface 142a is coupled to the simulator 170 and facilitates communication between the simulation controller 140 and the simulator 170. The primary controller interface 142b is coupled to the primary controller 110 and configured to facilitate communication between the primary controller 110 and the simulation controller 140. The simulation controller 140 further includes an exercise initializer unit 152, a kinematics algorithm unit 154, a machine state unit 156, and an instrument function handler 158 for each robotic arm 40 simulated in the surgeon console 30. As used herein below, the robotic arm 40 and the associated components, e.g., joints 44a, 44b, 44c, instrument 50, etc. are referenced by the same numerals as the physical counterparts of FIG. 4 for simplicity, however, they are simulated by the simulation controller 140.

[0066] The machine state unit 156, based on commands received from the primary controller 110, is configured to determine the appropriate action in the simulator 170 corresponding with a machine state. The machine state unit 156 may include one or more states, such as a registration state, a tele-robotic operation control state, and instrument specific states, e.g., a clip applier state, an electrosurgical state, and a stapler state. The registration state includes an “unregistered” and “registered” state. The registration state is initially set to a default state of “unregistered,” when the session is not active, and the simulated movable cart is placed in a bedside active state to prevent tele-robotic operation control. When the session is active, the registration state is changed from “unregistered” to “registered” to allow tele-robotic operation control. The instrument-specific states, may include: “disabled,” “wait clip reload,” and “reload animation” for a clip applier; “disabled,” “enabled,” “idle,” and “cutting” for electrosurgical forceps; and “disabled,” “idle,” “advancing,” “advancing paused,” and “advancing complete” for a stapler.

[0067] The tele-robotic operation control state includes a “waiting” and “ready” state. The “ready” state may further include sub-states, such as “hold,” “teleoperable,” and instrument specific states. The tele-robotic operation control state is initially set to a default state of “waiting” until the session is active. When the session is active, the tele-robotic operation control state is changed from “waiting” to “ready,” indicating to the primary controller 110 that the movable cart is ready for tele-robotic operation with a sub-state of “hold” until the movable cart receives a command from the primary controller 110 to enter tele-robotic operation. When tele-robotic operation is entered, the sub-state is changed from “hold” to “teleoperable” state. The sub-state may be changed back and forth from “hold” to “teleoperable,” based on acommand received from the primary controller 110. If the instrument 50 is a stapler and in the process of being reloaded, the sub-state may be changed from “teleoperable” to “reload animation” to disable tele-robotic operation during the reload animation.

[0068] The instrument function handler 158 maps instrument-specific commands from the primary controller 110 and the states from the machine state unit 156 to corresponding instrument functions within the surgeon console 30. The state of instrument 50 is received from the machine state unit 156. Based on the received state of instrument 50 and the specific command from the primary controller 110, the command from the primary controller 110 is mapped to the appropriate corresponding simulated instrument 50. The kinematics algorithm unit 154 is configured to perform kinematic calculations, such as inverse and forward kinematic calculations.

[0069] The exercise initializer unit 152 is configured to obtain the stored session data and instrument information from the simulator 170 to calculate an orientation and joint positions of joints 44a, 44b, and 44c of the simulated robotic arm 40 in a virtual fixed frame. The virtual fixed frame is a virtual representation of the fixed frame on the robotic arm 40, including one or more subset frames, such as, a TCP frame and an RCM frame. In some systems, the active instrument functions may be determined based on applying bit-masking to the incoming data corresponding to various functionality of the instruments, e.g., electrosurgical generators, staple tissue, etc.

[0070] To calculate the orientation of robotic arm 40, the initial instrument information, including an initial position of the instrument 50 and camera 51 is determined based on the initial TCP position relative to the RCM position. Instrument distances are calculated based on the difference between the initial TCP position and the RCM position (RCM-TCP). Based on the calculated instrument distances, x-direction (RCM-TCPx), y-direction (RCM-TCPy), and z-direction (RCM-TCPz) are calculated. Thus, the x-direction, y-direction, the z-direction, and the initial TCP position are combined to create an initial instrument pose (e.g., position and orientation). The initial instrument pose is post-multiplied by a transformation matrix to compensate for the hand eye coordination implemented in the primary controller 110, resulting in an initial position of camera 51.

[0071] To calculate the initial joint positions of joints 44a, 44b, 44c of the simulated robotic arms 40, the kinematic algorithm unit 154 calculates a subset of the joints of the simulated robotic arms 40 (e.g., joints 44a, 44b, and 44c) from the RCM-TCP distances while the remaining joints are set to zero (0). The calculated subset of the joints 44a, 44b, and 44c of the robotic arms 40 is further processed through the kinematic algorithm unit 154 to calculate theTCP in the RCM frame for each instrument 50 and camera 51. The inverse of the calculated TCP in the RCM frame provides the RCM in the TCP frame. To determine the orientation of each simulated robotic arm 40 based in the virtual fixed frame, the RCM in the TCP frame is post-multiplied by initial pose of instrument 50 and camera 51, the results may be used in the primary controller 110 to calculate the hand eye coordination, as well as further calculation in the kinematic algorithm unit 154.

[0072] The kinematic algorithm unit 154 is further configured to calculate a desired simulated instrument poses from a desired joint positions of the robotic arm 40 and an actual joint positions of the robotic arm 40 from actual poses of simulated instrument 50. The desired joint position of the robotic arm 40 is obtained from a position of the handle controllers 38a and 38b and / or foot pedals 36. The position of the handle controllers 38a and 38b and / or foot pedals 36 may include coordinate position and RPY orientation to a coordinate in the surgical console 30 relative to the robotic arm 40 in a virtual fixed frame. The kinematic algorithm unit 154 calculates the desired positions of instrument 50 utilizing the desired joint positions of the robotic arm 40 from the primary controller 110. The resulting desired poses of instrument 50 are post-multiplied with the RCM in the virtual fixed frame. In calculating the desired poses of camera 51, the desired poses of instrument 50 are further post-multiplied with the transpose of the calculated hand eye coordination in the primary controller 110. To obtain the desired joint positions of the robotic arm 40, a switch having a time threshold may be implemented to ensure that the actual joint positions of the robotic arm 40 are initialized via the primary controller 110 at the start of each exercise.

[0073] The kinematic algorithm unit 154 calculates the joint positions of the robotic arm 40 based on an average of the obtained actual positions of instrument 50 and the desired positions of instrument 50 post-multiplied with the inverse of the RCM in the virtual fixed frame. The joint positions of the robotic arm 40 are further configured to be transmitted to the primary controller 110 to determine force feedback.

[0074] The simulation controller 140 may further include a timing control feature configured to indicate the start and end of a session. In aspects, the simulation controller 140 may further include primary controller writer configured to transmit the desired and actual joint positions based on the machine state. In the event, the simulation controller 140 is in a tele-robotic operable state, the actual joint positions of the robotic arm 40 are transmitted to the primary controller 110 for force feedback calculations. Otherwise, the desired joint positions of the robotic arm 40 are transmitted to the primary controller 110 to disable force feedback. In some systems, the simulation controller 140 further includes a GUI writer to transmit information(e.g., robotic arm status, camera head state, and registration confirmed status) to a GUI subsystem of the second display device 34a. The information displayed by the second display device 34a is displayed during an active session allowing input from the user. In some instances, the simulation controller 140 may further include a primary controller reader configured to obtain the desired joint positions of the robotic arm 40 and commands from the primary controller 110.

[0075] The simulation controller 140 may further include a simulator writer configured to transmit poses of instrument 50 and / or camera 51, jaw angles, and active instrument functions to the shared memory 130 for further calculation. The surgeon console 30 may further includes additional software components found in a physical surgical robotic system, such as logging and data management, process and deployment, graphical user interface, alarms and notifications, surgeon console software subsystem, and primary control software subsystem software.

[0076] In operation, the user selects a training exercise in the surgical console 30 to initialize a virtual training session. The start of the session may be flagged by the timing control feature of the simulation controller 140. The exercise initializer unit 152 initializes the session by calculating an initial instrument and camera positions based on the initial TCP and initial RCM positions. The simulator writer may transmit the initial instrument and camera positions to the simulator 170 to initialize the session. The session data and instrument information are read from the shared memory 130 by simulation controller 140. The simulation controller 140 calculates actual joint positions of the robotic arm 40 based on the actual positions of instrument 50 from the instrument information read from the shared memory 130 by simulation controller 140. The primary controller writer may transmit and write the calculated actual joint positions of the robotic arm 40 to the primary controller 110 for force feedback if a command is received from the primary controller 110 indicating that the machine state of the simulation controller 140 is in a tele -robotic operable state. The primary controller 110 receives desired joint positions of the robotic arm 40 and commands from the user input, and the simulation controller 140 calculates desired poses of instrument 50 and camera 51 based on the desired joint positions of the robotic arm 40 and commands. The primary controller reader may obtain the desired joint positions of the robotic arm 40 and commands from the primary controller 110. The primary controller writer may transmit the desired joint positions of the robotic arm 40 calculated to the primary controller 110 to disable force feedback if commands are received from the primary controller 110 indicating that the machine state of the simulation controller 140 is in a tele-robotic non-operable state. The simulator 170 displays the simulation of therobotic arm 40, on the first screen 32 of the surgical console. The instrument function handler 158, based on the received commands from the primary controller 110, maps the corresponding command with an instrument function within the simulator 170. To map the corresponding commands with the instrument function within the simulator 170, the simulation controller 140 determines which robotic arm 40 and instrument drive unit 52 to simulate, determines the machine state of the robotic arm 40, and instrument drive unit 52.

[0077] With reference to FIGS. 6 and 7, the training system 200 may be used to train the surgeon at the surgeon console 30 while the assistant is being trained in virtual reality at the bedside. The surgeon and assistant trainees are operating in the same virtual environment 300 shown in FIG. 9. The virtual environment 300 is a multi-user environment hosting a plurality of users, i.e., trainees, simultaneously. Each of the users may interact with objects in the virtual environment 300 based on their input controls and other user parameters and such interactions have an impact on the virtual environment 300 are observable by all the other users in realtime. The simulator 170 generates and renders the virtual environment 300, which is displayed through the headsets 202, 203 as well as the simulated view displayed on the screen 32 of the surgeon console 30. The simulated view on the screen 32 includes simulated images of the instruments 50 from the virtual environment 300 as shown in FIG. 6. The simulator 170 also hosts multiple users, e.g., surgeon and assistant trainees.

[0078] In embodiments, only the assistant trainee is wearing the headset 203 while the surgeon trainee may operate the surgeon console 30 without the headset 202 and wearing specialized eyewear (e.g., stereoscopic glasses) for viewing stereoscopic images displayed on the screen 32 of the surgeon console 30, which may display a virtual laparoscopic view inside the patient provided by a virtual laparoscopic camera 51. In further embodiments, the assistant trainee may be located in any physical environment, either the same or apart from the space where surgeon console 30 is located. In remote setups, the headset 203 may be connected to the simulator 170 via any suitable remote network as defined above.

[0079] The virtual environment 300 may include the entire surgical robotic system 10 of FIG. 1 or portions thereof, e.g., robotic arms 40, rendered, virtual form. The surgeon trainee interacts with the virtual environment 300 through the surgeon console 30 and the assistant trainee interacts with the virtual environment 300 via the headset 203 and the handle controllers 220. The inputs at the surgeon console 30 are reflected in the movements and actuations of the robotic arms 40 in the virtual environment 300, which are in turn, observable by the assistant trainee through the headset 203. Conversely, the assistant trainee may interact with virtual objects, i.e., virtual robotic arms 40 and corresponding instruments 50, using gestures and / orhand controllers 220 and the changes are also reflected in the virtual environment 300. Similarly, the surgeon trainee may observe movement of the instruments 50 due to manipulation by the assistant trainee using the controllers 220, e.g., during retraction.

[0080] The headset 202 may be worn by the surgeon trainee to provide for a more immersive experience with the virtual environment 300. In embodiments where the headset 202 is a mixed reality device, the headset 202 may operate in a passthrough mode when the headset 202 is pointed toward the surgeon console 30 and in particular toward the screen 32. This mode allows the surgeon trainee to view the surgical scene rendered on the screen 32 without obstruction. In a virtual mode, the headset 202 displays the virtual environment 300 on its screen 208. FIG. 10 shows the virtual environment 300 as displayed on the headset 202 while the surgeon trainee is sitting at the surgeon console 30. The surgeon trainee ’ s hands are shown, which may be done based on tracking. In addition, a virtual surgeon console 30 is also shown as well as the same virtual laparoscopic view.

[0081] The virtual mode is enabled when the user turns their head away from the screen 32. Thus, the headset 202 switches between the passthrough and virtual modes based on the head position and orientation of the surgeon trainee. Head position and / or orientation may be determined based on head tracking and / or tracking of the headset 202 as described above.

[0082] The surgeon console 30 may be used to train in setting up the robotic system 10 in the manner shown in FIG. 5. The second screen 34 may display a GUI 35 (FIG. 6) for setting up the system 10 in the virtual environment 300 as shown in FIGS. 11A-C as a GUI for transmission to the headset 203. Each of the slides provides instructions for patient positioning, port placement, and arm cart docking, respectively. The surgeon trainee may adjust placement, e.g., by holding and dragging, various elements that are being set up, e.g., access ports 55a-d, prior to sending the instructions to the assistant trainee’s headset 203. The instructions are provided to the assistant trainee who sets up the robotic system 10 in the virtual environment 300. The instructions may be provided as 2D or 3D animations, text, pictographic, or combinations thereof and may be displayed on the screen 208 of the headset 203. As shown in FIG. 12C, a mesh model 52’ of the IDU 52 is displayed as well the virtual IDU 52 instructing the assistant trainee to shift the IDU 52 in the virtual environment 300 to the location of the mesh model 52’.

[0083] In addition to setup, the training system 200 may be also used to train the surgeon and the assistant in a variety of operational procedures, such as exchanging instruments 50 and performing other instrument manipulation, e.g., reloading stapler cartridges. The surgeon trainee may issue requests to the assistant trainee either through the surgeon console 30, e.g.,by making a selection through a GUI displayed on the second screen 34, or via a voice request, which may be played through headphones in the second headset 203. Exchanging instruments and reloading includes retracting the instrument 50 from the patient, detaching the instrument 50 from the IDU 52, then reattaching a new instrument 50 and inserting the same through the access port 55. The manipulations of the robotic arm 40 and the instrument 50 during this process may be performed by manipulating virtual objects in the virtual environment 300 as shown in FIGS. 12A-C.

[0084] The surgeon trainee, i.e., the headset 202 and hands, may be rendered as an avatar 302 in the virtual environment 300 as shown in FIG. 12A. The headset 203 worn by the assistant trainee may be also rendered in the virtual environment 300. This allows the users to track each other in virtual environment 300.

[0085] In addition to these exemplary interactions with the virtual environment 300, the training system 200 may also be used for training the surgeon in simulated procedures using the physical robotic system 10. The system 10 is controlled through the surgeon console 30 while the assistant may interact with physical arm carts 40, reloading stapler, passing suture needles, loading clips, and other general assist tasks in the physical environment. The surgeon trainee would still interact with the virtual simulation, while robotic arms 40a-d and other components of the system 10 are real are moved in response to surgeon’s input. This would allow the bedside assistant to plan their positioning relative to the robotic arms 40a-d and use the real robotic system 10. The bedside assistant trainee would also be responsible for instrument exchange and the real -world setup of the system 10, such as instrument exchange etc. One exemplary use of the robotic system 10 by the bedside trainee may be the instrument exchange, where the surgeon trainee requests another instrument and the bedside trainee would extract the instrument and perform the instrument exchange with the IDU 52 of the corresponding robotic arm 40. After the exchange, the system 10 would send the new instrument ID back to the simulation that the surgeon is interacting with. This includes updating the instrument 3D and physics models in the simulation so that the exchanged instrument is rendered in the simulation.

[0086] An exemplary workflow or method of operation for a hybrid training system, i.e., physical system 10 with the surgeon console 30 running a simulation, includes communicatively coupling each of the robotic arms 40a-d and the control tower 20 to the surgeon console 30 such that the robotic arms 40a-d communicate to the simulator 170 of the surgeon console 30. The simulator 170 receives all state information for each of the robotic arms 40a-d, instruments 50, and / or camera 51 coupled thereto. The simulator 170 renders aprocedure exercise for the surgeon using the surgeon console 30. As the surgeon trainee moves the virtual instruments in the simulation, the robotic arms 40a-d move in parallel, i.e., the movements of the robotic arms 40a-d mirrors the movements of the simulated instruments in the same manner the robotic arms 40a-d move when being controlled during teleoperation. This hybrid approach allows the bedside assistant to work with the physical robotic arms 40a- d, the instruments 50, and / or camera 51 while the surgeon trainee operates the system in a virtual environment.

[0087] It will be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be construed as limiting, but merely as exemplifications of various embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the claims appended thereto.

[0088] The following examples are illustrative of the techniques described herein.

[0089] Example 1. A surgical robotic training system comprising: a simulator for generating a virtual environment including a virtual patient and at least one robotic arm having a surgical instrument; a surgeon console including: at least one surgeon handle controller for receiving surgeon input for moving the at least one robotic arm and / or the surgical instrument in the virtual environment; and a first screen for displaying a laparoscopic view of the surgical instrument inside the virtual patient in the virtual environment; at least one assistant handle controller for receiving assistant input for engaging with the at least one robotic arm and the surgical instrument in the virtual environment; and an assistant headset for displaying an outside view of the virtual environment based on a position and an orientation of the assistant headset, wherein the outside view shows movement of the at least one robotic arm or the surgical instrument in the virtual environment in response to the surgeon input.

[0090] Example 2. The surgical robotic training system according to Example 1, wherein the assistant input includes at least one of coupling the surgical instrument to the at least one robotic arm or decoupling the surgical instrument from the at least one robotic arm.

[0091] Example 3. The surgical robotic training system according to Example 1, wherein the assistant input includes at least one of retracting the surgical instrument from the virtual patient or inserting the surgical instrument into the virtual patient.

[0092] Example 4. The surgical robotic training system according to Example 1, wherein the laparoscopic view includes movement of the surgical instrument inside the virtual patient in the virtual environment in response to the assistant input.

[0093] Example 5. The surgical robotic training system according to Example 1, wherein the surgeon console includes a second screen for displaying a graphical user interface for providing instructions to the assistant headset.

[0094] Example 6. The surgical robotic training system according to Example 5, wherein the instructions provided to the assistant headset include set up instructions for the at least one robotic arm in the virtual environment.

[0095] Example 7. The surgical robotic training system according to Example 5, wherein the instructions provided to the assistant headset include at least one of exchanging of the surgical instrument or retracting of the surgical instrument.

[0096] Example 8. The surgical robotic training system according to Example 1, wherein the assistant headset is at least one of a virtual reality headset, an augmented reality headset, or a mixed reality headset.

[0097] Example 9. A surgical robotic training system comprising: a simulator for generating a virtual environment including a virtual patient and at least one robotic arm having a surgical instrument; a surgeon console including: at least one surgeon handle controller for receiving surgeon input for moving the at least one robotic arm and / or the surgical instrument in the virtual environment; and a first screen for displaying a laparoscopic view of the surgical instrument inside the virtual patient in the virtual environment; and a surgeon headset operable in a passthrough mode in which the surgeon headset displays the first screen of the surgeon console and a virtual mode in which the surgeon headset displays the virtual environment, wherein the surgeon headset automatically switches between the passthrough mode and the virtual mode based on position and orientation of the surgeon headset relative to the first screen.

[0098] Example 10. The surgical robotic training system according to Example 9, further comprising: at least one assistant handle controller for receiving assistant input for engaging with the at least one robotic arm and the surgical instrument in the virtual environment; and an assistant headset for displaying an outside view of the virtual environment based on position and an orientation of the assistant headset, wherein the outside view shows movement of the at least one robotic arm or the surgical instrument in the virtual environment in response to the surgeon input.

[0099] Example 11. The surgical robotic training system according to Example 10, wherein the surgeon headset is at least one of a virtual reality headset, an augmented reality headset, or a mixed reality headset.

[0100] Example 12. The surgical robotic training system according to Example 10, wherein the assistant input includes at least one of coupling the surgical instrument to the at least one robotic arm or decoupling the surgical instrument from the at least one robotic arm.

[0101] Example 13. The surgical robotic training system according to Example 10, wherein the assistant input includes at least one of retracting the surgical instrument from the virtual patient or inserting the surgical instrument into the virtual patient.

[0102] Example 14. The surgical robotic training system according to Example 10, wherein the laparoscopic view includes movement of the surgical instrument inside the virtual patient in the virtual environment in response to the assistant input.

[0103] Example 15. The surgical robotic training system according to Example 10, wherein the surgeon console includes a second screen for displaying a graphical user interface for providing instructions to the assistant headset.

[0104] Example 16. The surgical robotic training system according to Example 15, wherein the instructions provided to the assistant headset include set up instructions for the at least one robotic arm in the virtual environment.

[0105] Example 17. The surgical robotic training system according to Example 15, wherein the instructions provided to the assistant headset include at least one of exchanging of the surgical instrument or retracting of the surgical instrument.

[0106] Example 18. A surgical robotic training system comprising: a simulator for generating a multi-user virtual environment and hosting a plurality of users therein, wherein the virtual environment includes a virtual patient and at least one robotic arm having a surgical instrument and provides for interaction between the plurality of users; a surgeon console usable by a first user of the plurality of users, the surgeon console including: at least one surgeon handle controller for receiving surgeon input for moving at least one robotic arm and / or the surgical instrument in the virtual environment; and a first screen for displaying a laparoscopic view of the surgical instrument inside the virtual patient in the virtual environment; at least one assistant handle controller usable by a second user of the plurality of users for receiving assistant input for engaging with the at least one robotic arm and the surgical instrument in the virtual environment; and an assistant headset usable by the second user of the plurality of users for displaying an outside view of the virtual environment based on position and orientation of the assistant headset, wherein the outside view includes movement of the at least one robotic arm and / or the surgical instrument in the virtual environment in response to the surgeon input.

[0107] Example 19. The surgical robotic training system according to Example 18, wherein the surgeon console includes a second screen for displaying a graphical user interface for providing instructions to the assistant headset.

[0108] Example 20. The surgical robotic training system according to Example 19, wherein the simulator transmits the instructions to the assistant headset.

[0109] Example 21. A surgical robotic training system comprising: a robotic arm including a surgical instrument; a simulator for generating a multi-user virtual environment and hosting a plurality of users therein, wherein the virtual environment includes a virtual patient and a virtual robotic arm having a virtual surgical instrument simulating the robotic arm and the surgical instrument; and a surgeon console including: at least one surgeon handle controller for receiving surgeon input for moving at least one of the virtual robotic arm or the virtual surgical instrument in the virtual environment and mirroring same movement on at least one of the robotic arm or the surgical instrument; and a first screen for displaying a laparoscopic view of the virtual surgical instrument inside the virtual patient in the virtual environment, wherein an exchange of the surgical instrument is reflected in the virtual environment.

Claims

WHAT IS CLAIMED IS:

1. A surgical robotic training system (200) comprising: a simulator (170) for generating a virtual environment (300) including a virtual patient and at least one robotic arm (40) having a surgical instrument (50); a surgeon console (30) including: at least one surgeon handle controller (38a, 38b) for receiving surgeon input for moving the at least one robotic arm and / or the surgical instrument in the virtual environment; and a first screen (32) for displaying a laparoscopic view of the surgical instrument inside the virtual patient in the virtual environment; at least one assistant handle controller (220) for receiving assistant input for engaging with the at least one robotic arm and the surgical instrument in the virtual environment; and an assistant headset (203) for displaying an outside view of the virtual environment based on a position and an orientation of the assistant headset, wherein the outside view shows movement of the at least one robotic arm or the surgical instrument in the virtual environment in response to the surgeon input.

2. The surgical robotic training system according to claim 1, wherein the assistant input includes at least one of coupling the surgical instrument to the at least one robotic arm or decoupling the surgical instrument from the at least one robotic arm.

3. The surgical robotic training system according to any preceding claim, wherein the assistant input includes at least one of retracting the surgical instrument from the virtual patient or inserting the surgical instrument into the virtual patient.

4. The surgical robotic training system according to any preceding claim, wherein the laparoscopic view includes movement of the surgical instrument inside the virtual patient in the virtual environment in response to the assistant input.

5. The surgical robotic training system according to any preceding claim, wherein the surgeon console includes a second screen (34) for displaying a graphical user interface for providing instructions to the assistant headset.

6. The surgical robotic training system according to claim 5, wherein the instructions provided to the assistant headset include set up instructions for the at least one robotic arm in the virtual environment.

7. The surgical robotic training system according to claim 5, wherein the instructions provided to the assistant headset include at least one of exchanging of the surgical instrument or retracting of the surgical instrument.

8. The surgical robotic training system according to any preceding claim, wherein the assistant headset is at least one of a virtual reality headset, an augmented reality headset, or a mixed reality headset.

Images