Robotic surgical system and method for conducting a customized user-training program

Abstract

A robotic surgical system and method for conducting customized user training programs are provided. In one embodiment, the robotic surgical system determines whether a user's skill at moving the robotic arm with the user input device needs improving. In response to determining that the user's skill at moving the robotic arm with the user input device needs improving, the robotic surgical system provides a customized simulation training program to improve the skill. Other embodiments are provided.

Description

ROBOTIC SURGICAL SYSTEM AND METHOD FOR CONDUCTING A CUSTOMIZED USER-TRAINING PROGRAMRELATED APPLICATION

[0001] The present patent document claims the benefit of the filing date under 35 U.S.C.§ 119(e) of Provisional U.S. Patent Application Serial No. 63 / 595,054, filed November 1, 2023, which is hereby incorporated by reference.TECHNICAL FIELD

[0002] The following embodiments generally relate to the field of robotic surgery and more specifically to a customized training program for a user of a robotic surgical system.BACKGROUND

[0003] Minimally-invasive surgery (MIS), such as laparoscopic surgery, involves techniques intended to reduce tissue damage during a surgical procedure. For example, laparoscopic procedures typically involve creating a number of small incisions in the patient (e.g., in the abdomen), and introducing one or more surgical instruments (e.g., an end effector, at least one camera, etc.) through the incisions into the patient. The surgical procedures may then be performed using the introduced surgical instruments, with the visualization aid provided by the camera.

[0004] Generally, MIS provides multiple benefits, such as reduced patient scarring, less patient pain, shorter patient recovery periods, and lower medical treatment costs associated with patient recovery. In some embodiments, MIS may be performed with robotic systems that include one or more robotic arms for manipulating surgical instruments based on commands from an operator. A robotic arm may, for example, support at its distal end various devices, such as surgical end effectors, imaging devices, cannulae for providing access to the patient’s body cavity and organs, etc.

[0005] In some embodiments, the operator may provide commands for manipulating surgical instruments while viewing an image that is provided by a camera and displayed on a display to the user.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1A depicts an example of an operating room arrangement with a robotic surgical system and a user console of an embodiment.

[0007] FIG. IB is a schematic illustration of one exemplary variation of a robotic arm manipulator, tool driver, and cannula with a surgical tool of an embodiment.

[0008] FIG. 1C is a schematic illustration of an exemplary user console of an embodiment.

[0009] FIG. 2 is a schematic illustration of an exemplary variation of a user console for a robotic surgical system of an embodiment in communication with one or more third party devices.

[0010] FIG. 3 is a schematic of a surgical robotic platform of an embodiment with a graphical user interface (GUI) module, where the surgical robotic platform is in communication with multiple medical data resources.

[0011] FIGS. 4A and 4B are perspective and longitudinal cross-sectional views, respectively, of one exemplary variation of a handheld user input device of an embodiment.

[0012] FIG. 5 is a flow chart of a method of an embodiment for conducting a customized usertraining program on a robotic surgical system.

[0013] FIG. 6 is an illustration of a graphical user interface providing a customized usertraining program of an embodiment.

[0014] FIG. 7 is an illustration of a graphical user interface providing scores from a customized user-training program of an embodiment.

[0015] FIG. 8 is a flow chart of a method of an embodiment for conducting a customized usertraining program.

[0016] FIG. 9 is a flow chart of a method of an embodiment for conducting a customized usertraining program for a surgeon.

[0017] FIG. 10 is a flow chart of a method of an embodiment for conducting a customized user-training program for a resident.

[0018] FIG. 11 is a flow chart of a method of an embodiment for conducting a customized user-training program based on collected endoscopic positioning statistics.DETAILED DESCRIPTION

[0019] Non-limiting examples of various aspects and variations of the embodiments are described herein and illustrated in the accompanying drawings.Robotic surgical system overview

[0020] FIG. 1A is an illustration of an exemplary operating room environment with a robotic surgical system. Generally, as shown in FIG. 1A, the robotic surgical system includes a user console 100 (sometimes referred to herein as the “surgeon bridge” or “bridge”), a control tower 133, and one or more robotic arms 160 located at a robotic platform (e.g., table, bed, etc.), where surgical instruments (e.g., with end effectors) are attached to the distal ends of the robotic arms 160 for executing a surgical procedure. The robotic arms 160 are shown as a table-mounted system, but in other configurations, one or more robotic arms may be mounted to a cart, ceiling or sidewall, or other suitable support surface.

[0021] As further illustration, as shown in the exemplary schematic of FIG. IB, a robotic surgical system may include at least one robotic arm 160 and a tool driver 170 generally attached to a distal end of the robotic arm 160. A cannula 180 coupled to the end of the tool driver 170 may receive and guide a surgical instrument 190 (e.g., end effector, camera, etc.). Furthermore, the robotic arm 160 may include a plurality of links that are actuated so as to position and orient the tool driver 170, which actuates the surgical instrument 190.

[0022] Generally, as shown in FIG. 1A, the user console 100 may be used to interface with the robotic surgical system 150. A user (such as a surgeon or other operator) may use the user console 100 to remotely manipulate the robotic arms 160 and / or surgical instruments (e.g., in tele-operation). The user console 100 may be located in the same operating room as the robotic system 150, as shown in FIG. 1A. In other embodiments, the user console 100 may be located in an adjacent or nearby room, or tele-operated from a remote location in adifferent building, city, or country. In one example, the user console 100 may comprise a seat 110, foot-operated controls (pedals) 120, one or more handheld user input devices 122, and at least one user display 130 configured to display, for example, a view of the surgical site inside a patient (e.g., captured with an endoscopic camera), and / or other surgical or medical information.

[0023] In the exemplary user console shown in FIG. 1C, a user located in the seat 110 and viewing the user display 130 may manipulate the foot-operated controls 120 and / or handheld user input devices 122 to remotely control the robotic arms 160 and / or surgical instruments mounted to the distal ends of the arm. The foot-operated controls 120 and / or handheld user input devices 122 may additionally or alternatively be used to control other aspects of the user console 100 or robotic system 150. For example, in variations in which the user generally controls (at any given time) a designated “left-hand” robotic arm / instrument and a designated “right-hand” robotic arm / instrument, the foot-operated controls 120 may enable a user to designate from among a larger group of available robotic arms / instruments which robotic arms / instruments comprise the “left-hand” and “right-hand” robotic arm / instruments (e.g., via toggle or rotation in selection among the available robotic arms / instruments). Other examples include adjusting or configuring the seat 110, the foot-operated controls 120, the user input devices 122, and / or the user display 130.

[0024] In some variations, a user may operate the surgical robotic system in an “over the bed” (OTB) mode, in which the user is at the patient’s side and simultaneously manipulating a robotically-driven instrument / end effector attached thereto (e.g., with a handheld user input device 122 held in one hand) and a manual laparoscopic tool. For example, the user’s left hand may be manipulating a handheld user input device 122 to control a robotic surgical component, while the user’s right hand may be manipulating a manual laparoscopic tool. Accordingly, in these variations, the user may perform both robotic -assisted MIS and manual laparoscopic surgery on a patient.

[0025] During an exemplary procedure or surgery, the patient is prepped and draped in a sterile fashion, and anesthesia may be achieved. Initial access to the surgical site may be performed manually with the robotic system 150 in a stowed configuration or withdrawn configuration to facilitate access to the surgical site. Once access is completed, initial positioning and / or preparation of the robotic system may be performed. During the surgicalprocedure, a surgeon or other user in the user console 100 may utilize the foot-operated controls 120, user input devices 122, and / or other suitable controls to manipulate various end effectors and / or imaging systems to perform the procedure. Manual assistance may be provided at the procedure table by other personnel, who may perform tasks including but not limited to retracting tissues, or performing manual repositioning or tool exchange involving one or more robotic arms 160. Other personnel may be present to assist the user at the user console 100. Medical and surgery-related information to aid other medical personnel (e.g., nurses) may be provided on additional displays such as a display 134 on a control tower 133 (e.g., control system for the robotic surgical system) and / or a display 132 located bedside proximate the patient. For example, as described in further detail herein, some or all information displayed to the user in the user console 100 may also be displayed on at least one additional display for other personnel and / or provide additional pathways for interpersonnel communication. When the procedure or surgery is completed, the robotic system 150 and / or user console 100 may be configured or set in a state to facilitate one or more post-operative procedures, including but not limited to robotic system cleaning and / or sterilization, and / or healthcare record entry or printout, whether electronic or hard copy, such as via the user console 100.

[0026] In some variations, the communication between the robotic system 150, the user console 100, and any other displays may be through the control tower 133, which may translate user commands from the user console 100 to robotic control commands and transmit them to the robotic system 150. The control tower 133 may transmit status and feedback from the robotic system 150 back to the user console 100 (and / or other displays). The connections between the robotic system 150, the user console 100, other displays, and the control tower 133 may be via wired and / or wireless connections, and may be proprietary or performed using any of a variety of data communication protocols. Any wired connections may be built into the floor and / or walls or ceiling of the operating room. The robotic surgical system may provide video output to one or more displays, including displays within the operating room as well as remote displays accessible via the Internet or other networks. The video output or feed may be encrypted to ensure privacy, and all or one or more portions of the video output may be saved to a server, an electronic healthcare record system, or other suitable storage medium.

[0027] In some variations, additional user consoles 100 may be provided, for example to control additional surgical instruments, and / or to take control of one or more surgical instruments at a primary user console. This will permit, for example, a surgeon to take over or illustrate a technique during a surgical procedure with medical students and physicians-in- training, or to assist during complex surgeries requiring multiple surgeons acting simultaneously or in a coordinated manner.

[0028] In some variations, as shown in the schematic illustration of FIG. 2, one or more third party devices 240 may be configured to communicate with the user console 210 and / or other suitable portions of the robotic surgical system. For example, as described elsewhere herein, a surgeon or other user may sit in the user console 210, which may communicate with the control tower 230 and / or robotic instruments in a robotic system 220. Medical data (e.g., endoscopic images, patient vitals, tool status, etc.) may be displayed at the user console 210, the control tower 230, and / or other displays. At least a subset of the surgical and other medical-related information may furthermore be displayed at a third party device 240, such as a remote computer display that is viewed by a surgical collaborator in the same room or outside the room. Other communication, such as teleconferencing with audio and / or visual communication, may further be provided to and from the third party device. The surgical collaborator may be, for example, a supervisor or trainer, a medical colleague (e.g., radiologist), or other third party who may, for example, view and communicate via the third party device 240 to assist with the surgical procedure.

[0029] FIG. 3 is a schematic illustration of an exemplary variation of a system 300 including a robotic surgical system and its interaction with other devices and parties. Although a particular architecture of the various connected and communicating systems is depicted in FIG. 3, it should be understood that in other variations, other suitable architectures may be used and the arrangement shown in FIG. 3 is for illustrative purposes. The system 300 may include a surgical robotic platform 302 that facilitates the integration of medical data from discrete medical data resources generated from a variety of parties. Data from the discrete medical data resources may, for example, be used to form temporally coordinated medical data. Multi-panel displays of the temporally coordinated medical data may be configured and presented, as described further herein.

[0030] The platform 302 may be, for example, a machine with one or more processors 310 connected to one or more input / output devices 312 via a bus 314. The at least one processor may, for example, include a central processing unit, a graphics processing unit, an application specific integrated circuit, a field programmable logic device or combinations thereof.

[0031] The surgical robotic platform 302 may include one or more input ports to receive medical data from discrete medical data resources. For example, a surgical robot port 329 may receive surgical robot data from a surgical robot 330. Such data may, for example, include position data or other suitable status information. An imaging port 331 may receive imaging data from an imaging device 332, such as an endoscope, that is configured to capture images (e.g., still images, video images) of a surgical site. The endoscope may, for example, be inserted through a natural orifice or through an aperture in a surgical patient. As another example, one or more medical instrumentation ports 333 may receive patient vital information from medical instrumentation 334 (e.g., a pulse oximeter, electrocardiogram device, ultrasound device and / or the like). Additionally, as another example, one or more user control data ports 335 may receive user interaction data from one or more control devices that receive user inputs from a user for controlling the system. For example, one or more handheld user input devices, one or more foot pedals, and / or other suitable devices (e.g., eye tracking, head tracking sensors) may receive user inputs.

[0032] The surgical robotic platform 302 may further include one or more output ports 337 configured for connection to one or more displays 338. For example, the displays 338 may include an open display (e.g., monitor screen) in a user console, an immersive display or head-mounted device with a display, on supplemental displays such as on a control tower display (e.g., team display), a bedside display (e.g., nurse display), an overhead “stadium”- style screen, etc. For example, the graphical user interface disclosed herein may be presented on one or more displays 338. The one or more displays 338 may present three- dimensional images. In some variations, the one or more displays 338 may include a touchscreen. The one or more displays 138 may be a single display with multiple panels, with each panel presenting different content. Alternatively, the one or more displays 138 may include a collection of individual displays, where each individual display presents at least one panel.

[0033] In some variations, a network interface 316 may also be connected to the bus 314. The network interface 316 may, for example, provide connectivity to a network 317, which may be any combination of one or more wired and / or wireless networks. The network 317 may, for example, help enable communication between the surgical robotic platform 302 and other data sources or other devices. For example, one or more third party data sources 340 may also be connected to the network 317. The third party source 340 may include a third party device (e.g., another computer operated by a third party such as another doctor or medical specialist), a repository of video surgical procedure data (e.g., which may be relevant to a procedure being performed by a surgeon), or other suitable source of additional information related to a surgical procedure. For example, the third party device data may be ported to a panel that is displayed to a surgeon before, during or after a procedure.

[0034] As another example, one or more application databases 342 may be connected to the network 317 (or alternatively, stored locally within a memory 320 within the surgical robotic platform 302). The application database 342 may include software applications (e.g., as described in further detail below) that may be of interest to a surgeon during a procedure. For example, a software application may provide access to stored medical records of a patient, provide a checklist of surgical tasks for a surgical procedure, perform machine vision techniques for assisting with a procedure, perform machine learning tasks to improve surgical tasks, etc. Any suitable number of applications may be invoked. Information associated with an application may be displayed in a multi-panel display or other suitable display during a procedure. Additionally or alternatively, information provided by one or more applications may be provided by separate resources (e.g., a machine learning resource) otherwise suitably in communication with the surgical robotic platform 302.

[0035] In some variations, one or more of the software applications may run as a separate process that uses an application program interface (API) to draw objects and / or images on the display. APIs of different complexities may be used. For example, a simple API may include a few templates with fixed widget sizes and locations, which can be used by the GUI module to customize text and / or images. As another example, a more complex API may allow a software application to create, place, and delete different widgets, such as labels, lists, buttons, and images.

[0036] Additionally or alternatively, one or more software applications may render themselves for display. This may, for example, allow for a high level of customization and complex behavior for an application. For example, this approach may be implemented by allowing an application to pass frames that are rendered by a graphical user interface (GUI) module 324, which can be computer-readable program code that is executed by the processor 310. Alternatively, an image buffer may be used as a repository to which an application renders itself.

[0037] In some variations, one or more software applications may run and render themselves independent of the GUI module 324. The GUI module may still, however, launch such applications, instruct the application or the operating system where the application is to be positioned on the display, etc.

[0038] As another approach, in some variations, one or more applications may run completely separate from the GUI rendered by the GUI module. For example, such applications may have a physical video connection and data connection to the system (e.g., through suitable input / output devices, network, etc.). The data connection may be used to configure video feed for an application to be the appropriate pixel dimensions (e.g., full screen, half screen, etc.).

[0039] As shown in FIG. 3, in some variations, a memory 320 may also be connected to the bus 314. The memory 320 may be configured to store data processed in accordance with embodiments of the methods and systems described herein.

[0040] In some variations, the memory 320 may be configured to store other kinds of data and / or software modules for execution. For example, a user console may include a memory 320 that stores a GUI module 324 with executable instructions to implement operations disclosed herein. The GUI module may, for example, combine and aggregate information from various software applications and / or other medical data resources for display. In some exemplary variations, one or more software applications may be incorporated into base code of the GUI module, such that the module draws graphics and displays text in the appropriate location on the display. For example, the module may fetch the images from a database, or the images may be pushed to the interface from an instrument (e.g., endoscopic camera) in the operating room, via a wired or wireless interface.

[0041] In some variations, medical data may be collected from discrete medical data resources (e.g., surgical robot 330, endoscope 332, medical instrumentation 334, control devices 336, third party data source 340, application database 342, etc.). Additionally, at least some of the medical data may be temporally coordinated such that, when necessary, time sensitive information from different medical data resources is aligned on a common time axis. For example, surgical robot position data may be time coordinated with endoscope data, which is coordinated with operator interaction data from control devices. Similarly, a networked resource, such as information provided by one or more software applications, may be presented at an appropriate point in time along with the other temporally coordinated data. Multi-panel displays, and / or other suitable displays, may be configured to communicate medical information (e.g., including the temporally coordinated medical data) as part of a graphical user interface (GUI).

[0042] Various exemplary aspects of a GUI for a robotic surgical system are described herein. In some variations, the GUI may be displayed in a multi-panel display at a user console that controls the robotic surgical system. Additionally or alternatively, the GUI may be displayed at one or more additional displays, such as at a control tower for the robotic surgical system, at a patient bedside, etc. Generally, the GUI may provide for more effective communication of information to a user in the user console and / or other personnel, as well as for more effective communication and collaboration among different parties involved in a surgical procedure, as further described below.Graphical user interface (GUI) interaction

[0043] In one embodiment, the GUI is displayed on a display 130 in a user console 100 that is used to control the robotic surgical system 150 (e.g., by a surgeon), and at least some of interactive graphical objects displayed on the display 130 may be controlled, selected, or otherwise interacted with via one or more user controls that are also used to control an aspect of the surgical system (e.g., surgical instrument). For example, a user may use one or more handheld user input devices 122 and / or one or more foot pedals 120 to selectively control an aspect of the robotic surgical system 150 and selectively interact with the GUI. By enabling control of both the robotic surgical system 150 and the GUI with the same user controls, the user may advantageously avoid having to switch between two different kinds of user controls. Enabling the user to use the same input devices to control the robotic system 150and the GUI streamlines the surgical procedure and increases efficiency, as well as helps the user maintain sterility throughout a surgical procedure.

[0044] As shown generally in FIGS. 4A and 4B, an exemplary variation of a handheld user input device 122 for controlling a robotic system may include a member 410, a housing 420 at least partially disposed around the member 410 and configured to be held in the hand of a user, and a tracking sensor system 440 configured to detect at least position and / or orientation of at least a portion of the device. The housing 420 may be flexible (e.g., made of silicone). In some instances, the detected position and / or orientation of the device may be correlatable to a control of the robotic system. For example, the user input device 122 may control at least a portion of a robotic arm, an end effector or tool (e.g., graspers or jaws) coupled to a distal end of the robotic arm, a GUI, or other suitable aspect or feature of the robotic surgical system 150. Additionally, in some instances, the detected position and / or orientation of the device 122 may be correlatable to a control of a GUI. Furthermore, in some variations, the user input device 122 may include one or more sensors for detecting other manipulations of the user input device 122, such as squeezing of the housing 420 (e.g., via one or more pressure sensors, one or more capacitive sensors, etc.).

[0045] Generally, a user interface for controlling a robotic surgical system may include at least one handheld user input device 122, or may include at least two handheld user input devices 122 (e.g., a first user input device to be held by a left hand of the user, and a second user input device to be held by a right hand of the user), or any suitable number. Each user input device 122 may be configured to control one or more different aspects or features of the robotic system. For example, a user input device held in the left hand of the user may be configured to control an end effector represented on a left side of a camera view provided to the user, while a user input device held in the right hand of the user may be configured to control an end effector represented on a right side of the camera view.

[0046] In some variations, the handheld user input device 122 may be a groundless user input device configured to be held in the hand and manipulated in free space. For example, the user input device 122 may be configured to be held between the fingers of a user, and moved about freely (e.g., translated, rotated, tilted, etc.) by the user as the user moves his or her arms, hands, and / or fingers. Additionally or alternatively, the handheld user input device 122 may be a body-grounded user input device, in that the user input device 122 may be coupledto a portion of the user (e.g., to fingers, hand, and / or arms of a user) directly or via any suitable mechanism such as a glove, hand strap, sleeve, etc. Such a body-grounded user input device may still enable the user to manipulate the user input device in free space.Accordingly, in variations in which the user input device 122 is groundless or body- grounded (as opposed to permanently mounted or grounded to a fixed console or the like), the user input device 122 may be ergonomic and provide dexterous control, such as by enabling the user to control the user input device with natural body movements unencumbered by the fixed nature of a grounded system.

[0047] The handheld user input device 122 may include wired connections that, for example, may provide power to the user input device 122, carry sensor signals (e.g., from the tracking sensor assembly and / or other sensors such as a capacitive sensor, optical sensor, etc. Alternatively, the user input device may be wireless as shown in FIG. 4A and communicate commands and other signals via wireless communication such as radiofrequency signals (e.g., WiFi or short-range such as 400-500mm range, etc.) or other suitable wireless communication protocol such as Bluetooth. Other wireless connections may be facilitated with optical reader sensors and / or cameras configured to detect optical markers on the user input device 122 infrared sensors, ultrasound sensors, or other suitable sensors.

[0048] The handheld user input device may include a clutch mechanism for switching between controlling a robotic arm or end effector and controlling a graphical user interface, etc., and / or between other control modes. One or more of the various user inputs described in further detail below may, in any suitable combination, function as a clutch. For example, touching a gesture touch region of the device, squeezing the housing, flicking or rotating the user input device, etc. may function to engage a clutch. As another example, a combination of squeezing and holding the user input device, and rotating the user input device, may function as a clutch. However, any suitable combination of gestures may function as a clutch. Additionally or alternatively, user input to other user input devices (e.g., foot pedal assembly) may, alone or in combination with user input to a handheld user input device, function as a clutch.

[0049] In some variations, engagement and disengagement of a clutch mechanism may enable transition between use of a handheld user input device as a control for the robotic system and use of the handheld user input device as a control for the GUI (e.g., to operate a cursordisplayed on the screen). When a clutch mechanism is engaged such that the user input devices are used to control the GUI, positions or poses of the robotic arms may be substantially locked in place to “pause” operation of the robotic system, such that subsequent movement of the user input devices while the clutch is engaged will not inadvertently cause movement of the robotic arms.

[0050] Simulator application

[0051] Another variation of an application for a GUI is a simulator application. A simulator application may, for example, be in communication with a database storing simulated surgical robotic experiences or simulated exercises, such as to teach user-specific psychomotor skills for robotic simulation (e.g., games to practice performing a roll action of a handheld user input device and / or other skills, etc.). Simulated surgical robotic experiences may include, for example, simulation and training exercises with simulated patients. The simulator application may load such simulated experiences into the GUI, including a simulated endoscopic view and other patient parameters. The simulated experiences may further include simulated events such as robotic arm collisions, patient distress, and other suitable events that may help enable a new user (e.g., a surgeon in training) to learn how to respond and resolve issues appropriately. Simulations may be generated separately from the simulator application, such as with simulation developer software, or alternatively may be generated within the simulator application itself.

[0052] In some variations, the simulator application may grade a user based on his or her performance in the simulated exercise, such as by providing a score for the user. Such scores may be tracked over time to gauge a trainee’s progress and fluency in using the robotic surgical system. In some variations, the simulator application may display a user’s progress throughout a set curriculum (e.g., indicating a user has completed three out of ten exercises), evaluate baseline skills of the user to tailor or adjust curriculum, and / or provide recommendations for particular simulation exercises based on the user’s performance.

[0053] In general, a “simulator” refers to the hardware and / or software components (e.g., a processor executing computer-readable instruction code) that allow a simulation to be executed on the surgeon bridge 100 to allow a user to practice using the user input devices of the surgeon bridge 100. In one embodiment, the hardware / software components of thesimulator are integrated into the user console / surgeon bridge 100 of the robotic surgical system 150 (e.g., in the same housing as other components of the console / surgeon bridge 100). For example, the simulator software can be stored and activated on the surgeon bridge 100 fortraining purposes. Prior simulators are separate components that are removably attached to the external surface of the user console, with data cables plugged into the user console and a power cable plugged into a power outlet. Thus, setting up this external component can take several minutes before it is ready to use, and additional time is required to tear-down the components after use. Further, the extra cables can create a tripping or other hazard. In contrast to these master-slave training stations, with the simulator integrated in the surgeon bridge 100, a user simply clicks a button on the GUI for the simulator application, and the simulator is ready for use (perhaps after a short reset time) without requiring the user to worry about plugging in data and power cables and waiting for other components to boot. When in clinical mode, manipulation of the user input devices results in movement of the robotic arms, whereas in simulation mode, the signals from the user input devices are sent just to the simulator and not to the robotic arm controller.

[0054] In addition to the simulator being integrated in the robotic surgical system, the simulator in this embodiment is different from prior simulators in that it provides a customized user-training program. Current skills training for surgeons is not individualized or responsive to a surgeon’s need, nor does it have the capability to incorporate real surgical performance data to build a customized training program tailored to individual surgeons.For example, training sessions for users are often focused on passing a variety of skills areas in order to complete a certification. It is desirable to have trainings focused on real-life performance (e.g., during surgery or other training) of a user to enhance the skills that the user lacks. The simulator in one embodiment can pick exercises for a user based on performance feedback from past surgeries and / or training sessions of the user. As will be discussed in more detail below, a customized-training program can be provided that can be based on the sequential order of the robotic training or on a customized order, such as skills evaluation, progress tracking, and specific case studies. Data logged by the robotic system can include, but is not limited to, categories, types, and examples of data for performance evaluation in both teleop and simulation situations.

[0055] Returning to the drawings, FIG. 5 is a flow chart 500 of a method of an embodiment for conducting a customized user-training program on the robotic surgical system 150. The method can be performed by one or more processors in the system 150. For example, a single processor can be used to perform all of the steps, or multiple processors can be used, with each performing one or more different steps. The phrase “a process” is used herein to refer to either alternative. Also, the processor or processors used to perform these steps can be used to perform other functions in the system 150 or can be used exclusively for simulation training.

[0056] As shown in FIG. 5, in this embodiment, a processor in the robotic surgical system 150 maintains a skill level record for a user that tracks the user’s skill level in manipulating the robotic arm 160 with the user input device 122 (act 510). A skill level record can be any suitable data structure with any suitable format and can be stored in a memory of the robotic surgical system 150 or in a memory external to the robotic surgical system 150. The skill level record for the user can be maintained by collecting data gathered during a prior real surgery performed by the user using the robotic surgical system 150 and / or by collecting data gathered from a prior simulation exercise performed by the user.

[0057] The data collected to track the user’s skill level can take any suitable form. For example, if the data is from a real surgery, the data can be positioning data of a robotic arm (e.g., carrying a tool or an endoscope), which can include a duration that the robotic arm was positioned in a given position or an adjustment frequency of the robotic arm. The data can also include one or more of the following: data regarding movement of the robotic arm, physiological data related to the user, manually-entered data, video review data, and video data. It should be noted that the phrase “one or more of the following” is intended to mean one or more of those items (so not all of the items need to be included), rather than one or more examples of each of those items (with all the items being included). Some of this data, which will be described in more detail below, can be generated just during clinical / teleop mode, just during simulation mode (e.g., missed targets, object drops, and tissue tears), or during either mode.

[0058] Data regarding movement of the robotic arm can refer to data from the robotic arm 160 or from other components of the robotic surgical system 150. For example, such data can include, but is not limited to, user interactions, UID 122 position, joint angles, instrumentand camera positions, instrument movement while out of view, an indication that the interlock is engaged, workspace size, and the smoothness level of the motion of an instrument. Physiological data related to the user can include data gathered from sensor(s) monitoring the user (e.g., number of blinks, heart rate, or other measures that correlate to cognitive load or stress), and self-reported task load index data. Manually-entered data refers to data (errors) logged by a mentor or proctor as they occur, such as tissue-injury events. Video review data refers to data coming from later case review using a common framework for evaluation, such as a Global Evaluative Assessment of Robotic Skills (GEARS) or Robotic Skills Assessment (RSA) scale. Video data refers to video that can be the source of computer-vision detected events, errors, techniques, and movements.

[0059] Referring back to FIG. 5, based on the user’s skill level record, the processor creates a customized simulation training program for the user (act 520). For example, the processor can create a customized simulation training program for the user by analyzing the collected data and then mapping / converting the data (indicating a user skill that needs improvement) with a particular simulation training program. The resulting customized simulation training program can be a sequential order of steps performed during surgery or can be a customized order of steps that is different from a sequential order of steps performed during surgery (e.g., based on a skills evaluation, progress tracking, and a specific case study). Further, the customized simulation training program can comprise one or more of the following: a basic skills training loop, a responsive training loop, a skills refresh loop, a career progression loop, an upcoming case loop, and a systems update loop. Each of these loops is intended to provide the right training at the right time, so users can be efficient in acquiring and practicing skills and avoid training that is not helpful to them.

[0060] The basic skills training loop is the initial training simulation that includes a sequential core curriculum tied to teaching basic robotic control, safety, psychomotor, and perceptual skills. Based on a user’s performance metrics, the system can suggest the user either repeat the exercise, attempt an alternate exercise to reinforce his skills, or move on to the next exercise. In this way, the system can be responsive to a user’s skills even for a prescribed curriculum of basic skills, taking them through the shortest pathway if they are meeting benchmarks. The responsive training loop responds to the metrics collected during surgery that may point out the surgeon’s weakest skill areas. The system could then recommend thesurgeon undergo additional training by using simulation exercises or lab tasks that reinforce these areas. Once the system sees an increase in that skill area during training, it can look for an increase in the same skill area during surgery.

[0061] The skills refresh loop can be used to detect skills decay when a surgeon is not a frequent robotic user. The system can detect when there are long time intervals between scheduled patient cases and ask the user to do a quick skills assessment test. If his metrics show decay in any skill areas, the system can recommend that the user do simulation exercises or other training until the metrics return to their benchmarks. The career progression loop is a loop that recognizes that, at different stages in a surgeon’s career, he may find different skills are important to reinforce. The system can look at aggregate metrics for surgeons of different career profiles, stages, and specialties and recommend what type of training could be useful for an individual surgeon. The upcoming case loop can be used when the system looks at a surgeon’s upcoming cases. If there are any upcoming surgeries that the surgeon does not perform frequently, the system can recommend refresher training. This can be targeted towards a complex step or using a different type of instrument than usual. The system updates loop can be used to notify the surgeon for training when there are new instruments, new best practice guidelines, etc.

[0062] After the customized simulation training program for the user is created, it can be provided to the user (act 530). FIG. 6 is an illustration of a graphical user interface of a simulator providing a customized user-training program of an embodiment. As shown in FIG. 6, in this example, the exercises in the training program are games that are not visually related to surgery. For example, one game requires the user to pick up and place jacks in different colored bowls, and another game requires the user to pick up rings and place them on pegs. These exercises help the user develop skills that are useful in surgery.

[0063] After the user completes the exercises, the user’s skill level record is updated based on a result of those exercises (act 540). In one embodiment, the user can be assessed on completion time, number of dropped objects, economy of motion, excessive force, instrument collisions, and missed targets, and these scores are provided to the user (see FIG. 7) and are used to update the user’s skill level record. The updated record can be used to create additional customized simulation training programs for the future.

[0064] FIGS. 8-11 provide example implementations and will be discussed below. It should be understood that these are merely examples, and other implementations can be used.

[0065] Turning first to FIG. 8, FIG. 8 is a flow chart 800 of a method of an embodiment for conducting a customized user-training program. As shown in FIG. 8, the processor of the system 150 receives a user’s surgery performance scores in various surgical skills (act 810). The processor then imports the surgery performance scores into the training simulator (act 820). Next, the processor integrates the surgery performance scores with the training scores (act 830). The processor then automatically picks the surgical skills with the lowest scores for the user when the user starts a training session using the simulator (act 840). Finally, the processor updates the user’s training scores after the training session (act 850).

[0066] FIG. 9 is a flow chart 900 of a method of an embodiment for conducting a customized user-training program of an embodiment for a surgeon. As shown in FIG. 9, in this example, the surgeon has frequent instrument collision during surgery (act 910). The system 150 identifies skills (e.g., bimanual dexterity) to improve to avoid this problem (act 920). The system 150 checks during which exercises did the user have the most instrument collisions (act 930). The user practices those exercises (e.g., pegboard, camera targeting) until the scores improve (act 940). The system 150 then observes future surgeries of the user for skill transfer (act 950).

[0067] FIG. 10 is a flow chart 1000 of a method of an embodiment for conducting a customized user-training program of an embodiment for a resident. As shown in FIG. 10, in this example, a resident gains feedback that he needs to work on positioning the endoscope (act 1010). The system 150 then identifies the skill to improve (act 1020) and, in this example, recommends the scope positioning exercise (act 1030). The resident repeats the scope positioning exercise until the score reaches the benchmark (act 1040). The system 150 gives the resident feedback during future surgeries (act 1050).

[0068] FIG. 11 is a flow chart 1100 of a method of an embodiment for conducting a customized user-training program of an embodiment based on collected endoscopic positioning statistics. As shown in FIG. 11, the system 150 collects endoscope positioning statistics (e.g., duration, adjustment frequency, etc.) of the surgeon during his robotic surgeries (act 1110). If the system 150 determines that the surgeon’s positioning skills needimproving (act 1120), the system recommends an endoscope positioning exercise during the surgeon’s training (act 1130). This can be done by selecting one or more of a predetermined number of exercises to include in the customized simulation training program based on the user’s skill that needs improving. If the surgeon passes the training exercise (act 1140), the system 150 updates the surgeon’s endoscope positioning skills training report (act 1150).

[0069] Other Illustrative Embodiments include the following. Illustrative Embodiments for one type of claim (e.g., system, method, computer program, or computer readable storage medium) may be provided in other types (e.g., system as a method). Illustrative Embodiments for one set (e.g., Illustrative Embodiments 1-9) may be used in other sets.

[0070] Illustrative Embodiment 1. A robotic surgical system comprising: a user console comprising a display device and a user input device; a robotic arm; and a processor configured to: maintain a skill level record for a user that tracks the user’s skill level in manipulating the robotic arm with the user input device; create a customized simulation training program for the user based on the user’s skill level record; provide the customized simulation training program to the user; and update the user’s skill level record based on a result of the customized simulation training program.

[0071] Illustrative Embodiment 2. The robotic surgical system of Illustrative Embodiment 1, wherein the skill level record for the user is maintained by collecting data gathered during a prior real surgery performed by the user using the robotic surgical system.

[0072] Illustrative Embodiment 3. The robotic surgical system of any of Illustrative Embodiments 1-2, wherein the skill level record for the user is maintained by collecting data gathered from a prior simulation exercise performed by the user.

[0073] Illustrative Embodiment 4. The robotic surgical system of any of Illustrative Embodiments 1-3, wherein the skill level record for the user is maintained by collecting one or more of the following: data regarding movement of the robotic arm, physiological data related to the user, manually-entered data, video review data, and video data.

[0074] Illustrative Embodiment 5. The robotic surgical system of any of Illustrative Embodiments 1-4, wherein the customized simulation training program is created by mapping a skill that needs improvement with a particular simulation training program.

[0075] Illustrative Embodiment 6. The robotic surgical system of any of Illustrative Embodiments 1-5, wherein the customized simulation training program is based on a sequential order of steps performed during surgery.

[0076] Illustrative Embodiment 7. The robotic surgical system of any of Illustrative Embodiments 1-6, wherein the customized simulation training program provides a customized order of steps that is different from a sequential order of steps performed during surgery.

[0077] Illustrative Embodiment 8. The robotic surgical system of any of Illustrative Embodiments 1-7, wherein the customized simulation training program comprises one or more of the following: a basic skills training loop, a responsive training loop, a skills refresh loop, a career progression loop, an upcoming case loop, and a systems update loop.

[0078] Illustrative Embodiment 9. A method for conducting a customized user-training program on a robotic surgical system, the method comprising: performing the following in a robotic surgical system comprising a robotic arm and a user console comprising a display device and a user input device: determining whether a user’s skill at moving the robotic arm with the user input device needs improving; and in response to determining that the user’s skill at moving the robotic arm with the user input device needs improving, providing a customized simulation training program to improve the skill.

[0079] Illustrative Embodiment 10. The method of Illustrative Embodiment 9, further comprising updating a record for the user based on a result of the customized simulation training program.

[0080] Illustrative Embodiment 11. The method of any of Illustrative Embodiments 9-10, wherein the user’s skill is determined from a user’s surgery performance score, a user’s simulation score, or both the user’s surgery performance score and the user’s simulation score.

[0081] Illustrative Embodiment 12. The method of any of Illustrative Embodiments 9-11, wherein the user’s skill is determined by collecting one or more of the following: data regarding movement of the robotic arm, physiological data related to the user, manually- entered data, video review data, and video data.

[0082] Illustrative Embodiment 13. The method of any of Illustrative Embodiments 9-12, wherein providing the customized simulation training program comprises selecting one or more of a predetermined number of exercises to include in the customized simulation training program based on the user’s skill that needs improving.

[0083] Illustrative Embodiment 14. The method of any of Illustrative Embodiments 9-13, wherein providing the customized simulation training program comprises selecting one or more of the following based on the user’s skill that needs improving: a basic skills training loop, a responsive training loop, a skills refresh loop, a career progression loop, an upcoming case loop, and a systems update loop.

[0084] Illustrative Embodiment 15. A robotic surgical system comprising: a user console comprising a display device and a user input device; a robotic arm; means for maintaining a skill level record for a user that tracks the user’s skill level in manipulating the robotic arm with the user input device; means for creating a customized simulation training program for the user based on the user’s skill level record; means for providing the customized simulation training program to the user; and means for updating the user’s skill level record based on a result of the customized simulation training program.

[0085] Illustrative Embodiment 16. The robotic surgical system of Illustrative Embodiment 15, wherein the skill level record for the user is maintained by collecting data gathered during a prior real surgery performed by the user using the robotic surgical system.

[0086] Illustrative Embodiment 17. The robotic surgical system of any of Illustrative Embodiments 15-16, wherein the skill level record for the user is maintained by collecting data gathered from a prior simulation exercise performed by the user.

[0087] Illustrative Embodiment 18. The robotic surgical system of any of Illustrative Embodiments 15-17, wherein the skill level record for the user is maintained by collecting one or more of the following: data regarding movement of the robotic arm, physiological data related to the user, manually-entered data, video review data, and video data.

[0088] Illustrative Embodiment 19. The robotic surgical system of any of Illustrative Embodiments 15-18, wherein the customized simulation training program comprises one or

Claims

more of the following: a basic skills training loop, a responsive training loop, a skills refresh loop, a career progression loop, an upcoming case loop, and a systems update loop.[0089] The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, they thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.What is claimed is:

1. A robotic surgical system comprising: a user console comprising a display device and a user input device; a robotic arm; and a processor configured to: maintain a skill level record for a user that tracks the user’s skill level in manipulating the robotic arm with the user input device; create a customized simulation training program for the user based on the user’s skill level record; provide the customized simulation training program to the user; and update the user’s skill level record based on a result of the customized simulation training program.

2. The robotic surgical system of Claim 1, wherein the skill level record for the user is maintained by collecting data gathered during a prior real surgery performed by the user using the robotic surgical system.

3. The robotic surgical system of Claim 1, wherein the skill level record for the user is maintained by collecting data gathered from a prior simulation exercise performed by the user.

4. The robotic surgical system of Claim 1, wherein the skill level record for the user is maintained by collecting one or more of the following: data regarding movement of the robotic arm, physiological data related to the user, manually-entered data, video review data, and video data.

5. The robotic surgical system of Claim 1, wherein the customized simulation training program is created by mapping a skill that needs improvement with a particular simulation training program.

6. The robotic surgical system of Claim 1, wherein the customized simulation training program is based on a sequential order of steps performed during surgery.

7. The robotic surgical system of Claim 1, wherein the customized simulation training program provides a customized order of steps that is different from a sequential order of steps performed during surgery.

8. The robotic surgical system of Claim 1, wherein the customized simulation training program comprises one or more of the following: a basic skills training loop, a responsive training loop, a skills refresh loop, a career progression loop, an upcoming case loop, and a systems update loop.

9. A method for conducting a customized user-training program on a robotic surgical system, the method comprising: performing the following in a robotic surgical system comprising a robotic arm and a user console comprising a display device and a user input device: determining whether a user’s skill at moving the robotic arm with the user input device needs improving; and in response to determining that the user’s skill at moving the robotic arm with the user input device needs improving, providing a customized simulation training program to improve the skill.

10. The method of Claim 9, further comprising updating a record for the user based on a result of the customized simulation training program.

11. The method of Claim 9, wherein the user’s skill is determined from a user’s surgery performance score, a user’s simulation score, or both the user’s surgery performance score and the user’s simulation score.

12. The method of Claim 9, wherein the user’s skill is determined by collecting one or more of the following: data regarding movement of the robotic arm, physiological data related to the user, manually-entered data, video review data, and video data.

13. The method of Claim 9, wherein providing the customized simulation training program comprises selecting one or more of a predetermined number of exercises to include in the customized simulation training program based on the user’s skill that needs improving.

14. The method of Claim 9, wherein providing the customized simulation training program comprises selecting one or more of the following based on the user’s skill that needs improving: a basic skills training loop, a responsive training loop, a skills refresh loop, a career progression loop, an upcoming case loop, and a systems update loop.

15. A robotic surgical system comprising: a user console comprising a display device and a user input device; a robotic arm; means for maintaining a skill level record for a user that tracks the user’s skill level in manipulating the robotic arm with the user input device; means for creating a customized simulation training program for the user based on the user’s skill level record; means for providing the customized simulation training program to the user; and means for updating the user’s skill level record based on a result of the customized simulation training program.

16. The robotic surgical system of Claim 15, wherein the skill level record for the user is maintained by collecting data gathered during a prior real surgery performed by the user using the robotic surgical system.

17. The robotic surgical system of Claim 15, wherein the skill level record for the user is maintained by collecting data gathered from a prior simulation exercise performed by the user.

18. The robotic surgical system of Claim 15, wherein the skill level record for the user is maintained by collecting one or more of the following: data regarding movement of the robotic arm, physiological data related to the user, manually-entered data, video review data, and video data.19 The robotic surgical system of Claim 15, wherein the customized simulation training program comprises one or more of the following: a basic skills training loop, a responsive training loop, a skills refresh loop, a career progression loop, an upcoming case loop, and a systems update loop.

Images