Surgical system with training or assistance function

Abstract

To provide systems, methods and tangibly implemented software instructions for supporting a teleoperated surgical system (TSS).SOLUTION: A surgical input that includes monitored events of the TSS is received. A current stage of the surgical procedure is autonomously determined based on the surgical input. A call for surgeon assistance is detected in response to context-relevant assistance provided to a surgeon console of the TSS, the context-relevant assistance being stage-synchronized with the surgical procedure.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] Claiming priority This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 374,670, filed August 12, 2016, and to U.S. Provisional Patent Application No. 62 / 254,556, filed November 12, 2015, each of which is incorporated by reference in its entirety.

[0002] Aspects of the invention relate to medical devices used in connection with surgery, and more particularly, to controlling a robotic assisted surgical system to collect and evaluate surgical performance data and automatically assist or advise surgical personnel. [Background technology]

[0003] Minimally invasive teleoperated surgical systems (TSS) are being developed to increase the surgeon's manual dexterity when working at an internal surgical site as well as to allow the surgeon to operate on a patient from a remote location (outside the sterile field). In a TSS, the surgeon is provided with an image of the surgical site, often in a control console. While viewing the image of the surgical site on an appropriate viewer or display, the surgeon performs a surgical procedure on the patient by manipulating master input or control devices of the control console. Each of the master input devices controls the operation of a servo-actuated / articulated surgical instrument. During a surgical procedure, the TSS can provide mechanical actuation and control of various surgical instruments or tools having end effectors that perform various functions for the surgeon, such as, for example, holding or driving a needle, grasping a blood vessel, cutting tissue, stapling tissue, etc., in response to the manipulation of the master input devices.

[0004] It is desirable to train surgeons in the use of TSS and to assist surgeons in procedures that use TSS. Practical solutions are needed to facilitate automated surgical evaluation and automated surgical assistance, where applicable. Summary of the Invention

[0005] The following summary introduces certain aspects of the inventive subject matter to provide a basic understanding. This summary is not an extensive overview of the inventive subject matter and is not intended to identify solutions or critical elements or to delineate the scope of the inventive subject matter. While this summary includes information relevant to various aspects and embodiments of the inventive subject matter, its sole purpose is to present some aspects and embodiments in a general form as a prelude to the detailed description that follows.

[0006] One aspect of the present invention is directed to a surgeon support system for a teleoperated surgical system (TSS). The TSS generally includes a surgeon input interface that receives surgical control inputs for enabling an electro-mechanical surgical system to perform a surgical procedure.

[0007] In some embodiments, the surgeon assistance system includes a virtual surgical environment engine and an assist engine, each of which may be implemented using computer hardware.

[0008] The virtual surgical environment engine includes a surgical input evaluation engine that receives surgical input including monitored events of the TSS, and a segmentation engine that determines a current stage of the surgical procedure based on the surgical input.

[0009] The assist engine includes a TSS interface communicatively coupled to a surgeon input interface of the TSS, a help request detector that detects a request for surgeon assistance, and an assistance rendering engine that transmits context-related assistance via the TSS interface in response to the surgeon's request for assistance. The context-related assistance is stage-synchronized to the surgical procedure. Stage synchronization refers to alignment with respect to surgical process flow, rather than strict time alignment. In some embodiments, stage synchronization supports stage-offsetting, such that assistance is provided to prepare the surgeon or surgical team for a future stage of the surgical procedure.

[0010] Advantageously, phase synchronization provides the surgeon with situation-relevant assistance that can be made immediately available. Examples of assistance include automatically queued expert video segments of a current (or upcoming) portion of a surgical procedure, on-demand simulations of a current or upcoming portion of a surgical procedure that can potentially be called upon during surgery, surgical technique advice for specific scenarios from an automated expertise system, and notification of expert assistance from individuals previously assessed as having expertise in a particular portion of a surgical procedure. The foregoing examples are given for illustrative purposes and should not be considered as limiting the scope of the appended claims unless expressly called out in the claims.

[0011] In some embodiments, the virtual surgical environment engine is further configured to implement a simulator engine that processes a computational model of the surgical procedure based on the surgical input of the TSS. For example, the simulator engine can include a TSS model for computationally representing the TSS in the virtual surgical environment; and a patient model for computationally representing a patient based on physical characteristics of the patient and changes to the patient affected by manipulation of the TSS model.

[0012] In some embodiments, the surgeon support system includes a surgical technique assessment engine configured to access surgical inputs and a current phase of the surgical procedure and calculate a plurality of temporal and spatial metrics of the surgical inputs corresponding to a plurality of different phases of the surgical procedure including the current phase, The surgical technique assessment engine can generate a surgical phase specific performance score representative of the quality of the surgical performance of the surgeon generating the surgical control inputs. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a plan view of a minimally invasive teleoperated surgical system (TSS). [Diagram 2] FIG. 2 is a perspective view of the surgeon's console. [Diagram 3] FIG. 2 is a perspective view of an electronics cart. [Figure 4] FIG. 1 is a schematic diagram of a TSS. [Figure 5A] FIG. 1 is an explanatory diagram of a TSS. [Figure 5B] FIG. 2 is a perspective view of a patient side cart of the surgical system. [Figure 5C] FIG. 1 is an explanatory diagram of a surgical scene. [Figure 6] FIG. [Figure 7] FIG. 2 is a perspective view of an instrument manipulator. [Figure 8] FIG. 1 is a schematic diagram of a surgical planning tool. [Figure 9] FIG. 1 illustrates an advanced surgical system featuring a TSS having additional support systems including a virtual surgical environment, a data recording system, a surgeon evaluation system, and an assistance system, in accordance with various embodiments. [Figure 10] FIG. 10 illustrates operational elements of the virtual surgical environment of the system of FIG. 9 and their interactions, according to one embodiment. [Figure 11] FIG. 11 illustrates operational elements and their interactions of the surgical input evaluator of the virtual surgical environment of FIG. 10, according to one embodiment. [Figure 12]FIG. 11 illustrates a segmenter of the virtual surgical environment of FIG. 10 including its operational elements and their interactions, according to one embodiment. [Figure 13] FIG. 11 illustrates operational elements of the simulator of the virtual surgical environment of FIG. 10 and their interactions, according to one embodiment. [Figure 14] FIG. 10 illustrates operational elements of the surgeon evaluator of the system of FIG. 9 and their interactions, according to one embodiment. [Figure 15] 10 illustrates operational elements of the data recording system of the system of FIG. 9 and their interactions, according to an exemplary embodiment. [Figure 16] FIG. 10 illustrates the operating elements of the assist engine of the system of FIG. 9 and their interactions, according to some embodiments. [Figure 17] FIG. 1 is a block diagram illustrating a computer system in the exemplary form of a general-purpose machine, which can be converted into a special-purpose machine for implementing aspects of the embodiments described herein. [Figure 18] FIG. 18 illustrates an exemplary hardware and software architecture of a computing device such as that shown in FIG. 17, showing various interfaces between the hardware and software components. [Figure 19A] 1A-1C are diagrams illustrating an exemplary surgical instrument and actuator assembly, according to some embodiments, in which the surgical instrument is shown in different exemplary operating states of the surgical instrument. [Figure 19B] 1A-1C are diagrams illustrating an exemplary surgical instrument and actuator assembly, according to some embodiments, in which the surgical instrument is shown in different exemplary operating states of the surgical instrument. [Figure 19C] 1A-1C are diagrams illustrating an exemplary surgical instrument and actuator assembly, according to some embodiments, in which the surgical instrument is shown in different exemplary operating states of the surgical instrument. [Figure 20] 10 is a process flow diagram illustrating an exemplary method of operating a TSS support system, such as the system of FIG. 9, according to some embodiments. [Figure 21A] FIG. 15 is an illustrative diagram showing the change in horizon of the surgical scene brought about by the control setting adjuster of the surgeon evaluator of FIG. 14 to assist the surgeon in achieving a viewpoint that allows the surgeon to more precisely control surgical instruments within the surgical scene of the TSS. [Figure 21B] FIG. 15 is an illustrative diagram showing the change in horizon of the surgical scene brought about by the control setting adjuster of the surgeon evaluator of FIG. 14 to assist the surgeon in achieving a viewpoint that allows the surgeon to more precisely control surgical instruments within the surgical scene of the TSS. [Figure 21C] FIG. 15 is an illustrative diagram showing the change in horizon of the surgical scene brought about by the control setting adjuster of the surgeon evaluator of FIG. 14 to assist the surgeon in achieving a viewpoint that allows the surgeon to more precisely control surgical instruments within the surgical scene of the TSS. [Figure 22A] FIG. 15 is an illustrative diagram showing the two-dimensional positional changes of the surgical scene brought about by the control setting adjuster of the surgeon evaluator of FIG. 14 to assist the surgeon in achieving a viewpoint from which he or she can better monitor relevant information within the surgical scene of the TSS. [Figure 22B] FIG. 15 is an illustrative diagram showing the two-dimensional positional changes of the surgical scene effected by the control setting adjuster of the surgeon evaluator of FIG. 14 to assist the surgeon in achieving a viewpoint that allows the surgeon to better observe relevant information within the surgical scene of the TSS. [Figure 23A] FIG. 15 is an illustrative diagram showing the zoom level change of the surgical scene effected by the control setting adjuster of the surgeon evaluator of FIG. 14 to assist the surgeon in achieving a perspective that allows him or her to better observe relevant information within the surgical scene of the TSS. [Figure 23B] FIG. 15 is an illustrative diagram showing the zoom level change of the surgical scene effected by the control setting adjuster of the surgeon evaluator of FIG. 14 to assist the surgeon in achieving a perspective that allows him or her to better observe relevant information within the surgical scene of the TSS. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] This detailed description and the accompanying drawings, which describe aspects, embodiments, implementations, or applications of the invention, should not be construed as limiting, and the claims define the invention to be protected. Various mechanical, configurational, structural, electrical, and operational changes can be made without departing from the scope of this detailed description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the invention. Like numbers in two or more figures represent the same or similar elements.

[0015] An element described in detail with respect to one embodiment, implementation, or application may be included in other embodiments, implementations, or applications not specifically shown or described, whenever practical. For example, if an element is described in detail with reference to a first embodiment, but not with reference to a second embodiment, the element is nevertheless within the scope of the invention as included in the second embodiment. Thus, to avoid unnecessary repetition in the following detailed description, one or more elements illustrated and described in connection with one embodiment, implementation, or application may be incorporated in other embodiments, implementations, or aspects, unless specifically described otherwise, unless one or more elements render the embodiment or implementation non-functional, or unless two or more elements provide conflicting functionality.

[0016] Aspects of the invention will be described primarily in terms of implementation using the da Vinci® Surgical System (specifically, the Model IS4000, commercially available as the Da Vinci Xi® Surgical System) commercialized by Intuitive Surgical, Inc. of Sunnyvale, Calif. However, knowledgeable persons will appreciate that aspects of the invention disclosed herein may be embodied and practiced in a variety of ways, including robotic and, where applicable, non-robotic embodiments and implementations. Implementations with a da Vinci® Surgical System (e.g., the da Vinci Xi® Surgical System, the da Vinci Si® Surgical System) are merely exemplary and should not be construed as limiting the scope of aspects of the invention disclosed herein to any particular model or device, nor should such limitations be construed as limiting the scope of any aspects of the invention disclosed herein to any particular model or device, nor should such limitations be construed as limiting the scope of any aspects of the invention disclosed herein to any particular model or device, nor to the extent that such limitations are expressly set forth in one or more claims.

[0017] Minimally Invasive Teleoperated Surgical System (TSS) Teleoperation refers to the operation of a machine at a distance. In a minimally invasive teleoperated medical system, a surgeon can view the surgical site inside the patient's body using an endoscope that includes a camera. In some embodiments, stereoscopic images can be captured that allow for depth perception during the surgical procedure.

[0018] Referring now to the drawings, where like reference numerals represent like parts throughout the several views, FIG. 1 is a plan view of a minimally invasive TSS 10 typically used to perform a minimally invasive diagnostic or surgical procedure on a patient 12 lying on an operating table 14. The system includes a surgeon console 16 for use by a surgeon 18 during the procedure. One or more assistants 20 may also participate in the procedure. The minimally invasive TSS 10 further includes a patient side cart(s) 22 and an electronics cart 24. The patient side cart 22 may steer at least one surgical instrument 26 through a minimally invasive incision in the body of the patient 12 while the surgeon 18 views the surgical site via the surgeon console 16. Images of the surgical site may be acquired by a camera mounted with an endoscope 28, such as a stereo endoscope, and the camera may be steered by the patient side cart 22 to position and orient the endoscope 28. Images of the surgical site may be processed using a computer processor located on the electronics cart 24 for subsequent display to the surgeon 18 via the surgeon console 16. Although separate system elements (i.e., Patient Side Cart 22, Electronics Cart 24, and Surgeon Console 16) are shown and described for illustrative purposes, in various embodiments, the elements included therein may be combined and / or separated. For example, in some embodiments, the computer processor of the Electronics Cart 24 may be incorporated into the Surgeon Console 16 and / or the Patient Side Cart 22. The number of surgical instruments 26 used at one time generally depends on the diagnostic or surgical procedure and space constraints within the surgical site, among other factors. When it is necessary to change one or more surgical instruments 26 used during a procedure, the assistant 20 may remove the surgical instrument 26 from the Patient Side Cart 22 and replace it with another surgical instrument 26 on a tray in the operating room.

[0019] FIG. 2 is a perspective view of the surgeon console 16. The surgeon console 16 has a viewer 31 including a left eye display 32 and a right eye display 34 for presenting the surgeon 18 with a coordinated stereoscopic view of the surgical site that enables depth perception. In various other embodiments, a non-stereoscopic display may be provided to the surgeon 18. The console 16 further includes one or more control inputs 36. In response to manipulation of the one or more control inputs 36 by the surgeon 18, one or more surgical instruments mounted for use on the patient side cart 22 (shown in FIG. 1) move. The control inputs 36 may provide the same or more mechanical degrees of freedom as the associated surgical instruments 26 (shown in FIG. 1) to provide telepresence to the surgeon 18 (i.e., the perception that the control inputs 36 are integral with the instruments 26), such that the surgeon has a strong sense of directly controlling the instruments 26. To this end, in some embodiments, position, force, and tactile feedback sensors (not shown) may be used to communicate position, force, and tactile sensations from the surgical instrument 26 back to the surgeon's hand via the control input 36, subject to communication delay constraints. Although a physical console 16 with a fixed viewer 31 and mechanically coupled control input 36 is shown and described for illustrative purposes, in various other embodiments, an "ungrounded" control input and / or display structure may be used. For example, in some embodiments, the viewer 31 may be a head-mounted display and / or the control input 36 may be mechanically independent of any underlying structure (e.g., wired, wireless, or gesture-based, such as Microsoft's Kinect).

[0020] The surgeon console 16 is typically located in the same room as the patient so that the surgeon can directly monitor the procedure and be physically present if necessary to talk to the patient assistant directly rather than via telephone or other communication medium, however, the surgeon may be located in a different room, an entirely different building, or other remote location away from the patient allowing for remote medical procedures.

[0021] FIG. 3 is a perspective view of the electronics cart 24. The electronics cart 24 is coupled to an endoscope 28 and includes a computer processor for processing captured images for subsequent display to a surgeon or other suitable display located locally and / or remotely. For example, if a stereoscopic endoscope is used, the computer processor on the electronics cart 24 can process the captured images to present a calibrated stereoscopic image of the surgical site to the surgeon. Such calibration can include alignment between opposing images and adjusting the stereoscopic working distance of the stereoscopic endoscope. As another example, image processing can include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. Optionally, the equipment in the electronics cart can be integrated into the surgeon console or patient side cart, or can be located in various other locations in the operating room.

[0022] FIG. 4 illustrates a schematic of a TSS 50 (such as the minimally invasive TSS of FIG. 1). A surgeon console 52 (such as the surgeon console 16 of FIG. 1) is used by a surgeon to control a patient side cart 54 (such as the patient side cart 22 of FIG. 1) during a minimally invasive procedure. The patient side cart 54 can capture images of the surgical site using an imaging device such as a stereo endoscope and output the captured images to a computer processor located on an electronics cart 56 (such as the electronics cart 24 of FIG. 1). The computer processor typically includes one or more data processing boards for executing computer readable code stored in a non-volatile memory device of the computer processor. In one aspect, the computer processor can process the captured images in various ways before subsequent display. For example, the computer processor can overlay the captured images on a virtual control interface before displaying a composite image to the surgeon via the surgeon console 52.

[0023] Additionally or alternatively, the captured images may undergo image processing by a computer processor located outside the electronics cart 56. In one embodiment, the TSS 50 includes an optional computer processor 58 (shown in dashed lines) similar to the computer processor located on the electronics cart 56, and the patient side cart 54 outputs the captured images to the computer processor 58 for image processing before display on the surgeon console 52. In another aspect, the captured images first undergo image processing by the computer processor on the electronics cart 56 and then undergo further image processing by the computer processor 58 before display on the surgeon console 52. The TSS 50 may include an optional display 60, shown in dashed lines. The display 60 is coupled to the computer processor located on the electronics cart 56 and the computer processor 58, and the captured images processed by these computer processors are displayed on the display 60 and also on the display of the surgeon console 52.

[0024] FIG. 5A is an exemplary simplified block diagram showing the arrangement of components of a teleoperated surgical system 10 that uses one or more mechanical support arms 510 to perform a surgical procedure, according to some embodiments. Aspects of the system 10 include mechanisms that operate robotically assisted and autonomously. These mechanical support arms 510 often support surgical instruments. For example, a mechanical surgical arm (e.g., a central mechanical surgical arm 510C) can be used to support an endoscope with a stereoscopic or three-dimensional surgical image capture device 101C. The mechanical surgical arm 510C can include a sterile adapter, or a clamp, clip, screw, slot / groove, or other fastening mechanism for mechanically securing an endoscope including the image capture device 101C to the mechanical arm. In various other embodiments, the image capture device 101C (or any other surgical instrument) can be incorporated into the mechanical surgical arm 510C.

[0025] A user or operator O (typically a surgeon) performs a surgical procedure on a patient P by manipulating control input devices 36, such as hand grips and foot pedals, on a master control console 16. The operator can view video frames of images of the surgical site inside the patient's body via a stereoscopic viewer 31. A computer processor 58 in the console 16 directs the movement of the teleoperated endoscopic surgical instruments 101A-101C over control lines 159 and moves the instruments using a patient-side system 24 (also referred to as a patient-side cart).

[0026] The patient-side system 24 includes one or more mechanical support arms 510. Typically, the patient-side system 24 includes at least three mechanical surgical arms 510A-C (commonly referred to as mechanical surgical support arms 510) supported by corresponding positioning setup arms 156. The central mechanical surgical arm 510C can support or include an endoscopic camera 101C suitable for capturing images within the camera's field of view. The central left and right mechanical surgical support arms 510A and 510B can support or include instruments 101A and 101B, respectively, capable of manipulating tissue.

[0027] FIG. 5B is a perspective view of a patient side cart 500 of the minimally invasive TSS 10, according to an embodiment. The patient side cart 500 includes one or more support arm assemblies 510. A surgical instrument manipulator 512 is attached to the end of each support arm assembly 510. Additionally, each support arm assembly 510 can optionally include one or more set-up joints (e.g., non-powered and / or lockable) used to position the attached surgical instrument manipulator 512 relative to the patient for surgery. As shown, the patient side cart 500 rests on the floor. In other embodiments, the working portion of the patient side cart can be attached to a wall, ceiling, a surgical table 526 that also supports the patient's body 522, or other operating room equipment. Additionally, although a single patient side cart 500 is shown as including four surgical instrument manipulators 512, multiple patient side carts 500 and / or more or fewer surgical instrument manipulators 512 on the patient side cart(s) 500 may be used.

[0028] A functional TSS generally includes a vision system portion that allows a user of the TSS to view the surgical site from outside the patient's body 522. The vision system typically includes a camera instrument 528 for capturing video images and one or more video displays for displaying the captured video images. In some surgical system configurations, the camera instrument 528 includes optics that transfer images from the tip of the camera instrument 528 to one or more imaging sensors (e.g., CCD or CMOS sensors) outside the patient's body 522. Alternatively, the imaging sensor(s) may be located at the tip of the camera instrument 528, and signals generated by the sensor(s) may be transferred along leads or wirelessly for processing and display on one or more video displays. One example of a video display is the stereoscopic display on the surgeon's console of a surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.

[0029] 5A-5B, each surgical instrument manipulator 512 is attached to a surgical instrument 520 that operates at a surgical site within a patient's body 522. Each surgical instrument manipulator 512 is provided in a variety of forms such that its associated surgical instrument is movable in one or more mechanical degrees of freedom (e.g., six total Cartesian degrees of freedom, five or fewer Cartesian degrees of freedom, etc.). Typically, mechanical or control constraints restrict each manipulator 512 to moving its associated surgical instrument about the instrument's center of motion that is stationary relative to the patient, which is typically located at the position where the instrument enters the body.

[0030] In one aspect, the surgical instruments 520 are controlled by computer-assisted teleoperation. A functional minimally invasive TSS includes a control input that receives input from a user of the TSS (e.g., a surgeon or other medical personnel). The control input communicates with one or more computer-controlled teleoperated actuators, such as one or more motors to which the surgical instruments 520 are coupled. In this manner, the surgical instruments 520 move in response to the movement of the medical personnel's control input. In one aspect, the one or more control inputs are included in a surgeon console, such as the surgeon console 16 shown in FIG. 2. The surgeon can steer the control input device 36 of the surgeon console 16 to operate the teleoperated actuators of the patient side cart 500. Forces generated by the teleoperated actuators are transferred via a drive train mechanism that transfers the force from the teleoperated actuators to the surgical instruments 520.

[0031] 5A-5B, in one aspect, the surgical instrument 520 and the cannula 524 are removably coupled to the manipulator 512 with the surgical instrument 520 inserted through the cannula 524. One or more remotely operated actuators of the manipulator 512 move the surgical instrument 512 as a whole. The manipulator 512 further includes an instrument carriage 530. The surgical instrument 520 is removably connected to the instrument carriage 530. In one aspect, the instrument carriage 530 houses one or more remotely operated actuators therein that provide multiple controller operations that translate the surgical instrument 520 as various movements of an end effector on the surgical instrument 520. Thus, the remotely operated actuators in the instrument carriage 530 move only one or more components of the surgical instrument 520, and not the entire surgical instrument. The input for controlling the entire instrument or any of its components is such that an input provided to the control input by a surgeon or other medical personnel (a "master" command) is converted into a corresponding action by the surgical instrument (a "slave" response).

[0032] According to some embodiments, the surgical system 10 can have multiple system operating states including: docked instrument type, head in, camera movement, third arm control, ergonomic adjustments, table movement adjustments, etc. During the docked system state, one or more manipulators 512 are coupled to the cannula 524. During the next system state, the surgical instruments (slaves) are tracking the control inputs (master commands). During the instrument type system state, the system places in the system a set of instruments suitable for performing a particular surgical procedure or for performing a particular surgical activity during a surgical procedure. During the head in system state, the system waits for the surgeon to indicate that he has held the "master" control input device.

[0033] In alternative embodiments, the instrument carriage 530 does not house a remotely operated actuator. The remotely operated actuators enabling various movements of the end effector of the surgical instrument 520 are housed at a location remote from the instrument carriage 530, such as, for example, anywhere on the patient side cart 500. A cable-based force transmission mechanism or the like is used to transfer the movement of each of the remotely located remotely operated actuators to a corresponding instrument interface actuator output located on the instrument carriage 530. In some embodiments, the surgical instrument 520 is mechanically coupled to a first actuator, which controls a first movement of the surgical instrument, such as longitudinal (z-axis) rotation. The surgical instrument 520 is mechanically coupled to a second actuator, which controls a second movement of the surgical instrument, such as two-dimensional (x,y) movement. The surgical instrument 520 is mechanically coupled to a third actuator, which controls a third movement of the surgical instrument, such as opening and closing of the jaw end effector.

[0034] 5C is an illustration depicting a surgical scene 550 and shows an endoscope 101C mounting a camera 528 used to record the scene according to some embodiments. The scene 550 is positioned within a patient's body cavity. The scene 550 includes an exemplary virtual spherical anatomical structure 552 including a geometric contour 554. The scene 550 surrounds a surgical instrument 556. The camera 528 mounted on the endoscope 101C captures and records the scene displayed in the viewer 31 for later playback.

[0035] FIG. 6 is a side view of a surgical instrument 520 including a distal portion 650 and a proximal control mechanism 640 coupled by an elongated tube 610 having an elongated tube centerline axis 611. The surgical instrument 520 is configured to be inserted into a patient's body and used to perform a surgical or diagnostic procedure. The distal portion 650 of the surgical instrument 520 can provide any of a variety of end effectors 654, such as forceps, a needle driver, a cautery device, a cutting tool, an imaging device (e.g., an endoscope or ultrasound probe) as shown. The surgical end effector 654 can include mechanical degrees of freedom for functions such as opening and closing jaws or translating a knife along a path. In the illustrated embodiment, the end effector 654 is coupled to the elongated tube 610 by a wrist 652 that allows the end effector to be oriented relative to the elongated tube centerline axis 611. The surgical instrument 520 may include information stored therein (e.g., on a semiconductor memory associated with the instrument), which may be permanent or which may be updated by a surgical system configured to operate the surgical instrument 520. Thus, the surgical system may provide for either one-way or two-way communication of information between the surgical instrument 520 and one or more components of the surgical system.

[0036] FIG. 7 is a perspective view of a surgical instrument manipulator 512. The instrument manipulator 512 is shown without a surgical instrument installed. The instrument manipulator 512 includes an instrument carriage 530 to which a surgical instrument (e.g., surgical instrument 520) can be removably coupled. The instrument carriage 530 houses a number of remotely operated actuators. Each remotely operated actuator includes an actuator output 705. When a surgical instrument is installed on the instrument manipulator 512, one or more instrument inputs (not shown) of the instrument's proximal control mechanism (e.g., proximal control mechanism 640 of FIG. 6) are mechanically coupled to a corresponding actuator output 705. In one aspect, this mechanical coupling is direct, with the actuator output 705 directly contacting the corresponding instrument input. In another aspect, this mechanical coupling occurs through an intermediate interface, such as a drape element configured to provide a sterile barrier between the instrument manipulator 512 and the associated surgical instrument.

[0037] In one aspect, movement of one or more instrument inputs by a corresponding remotely operated actuator results in movement of a mechanical degree of freedom of the surgical instrument. For example, in one aspect, the surgical instrument mounted on the instrument manipulator 512 is the surgical instrument 520 shown in FIG. 6. Referring to FIG. 6, in one aspect, movement of one or more instrument inputs of the proximal control mechanism 640 by a corresponding remotely operated actuator rotates the elongated tube 610 (and attached wrist 652 and end effector 654) relative to the proximal control mechanism 640 about the elongated tube centerline axis 611. In another aspect, movement of one or more instrument inputs by a corresponding remotely operated actuator results in movement of the wrist 652, which orients the end effector 654 relative to the elongated tube centerline axis 611. In another aspect, movement of one or more instrument inputs by a corresponding remotely operated actuator results in movement of one or more moveable elements (e.g., jaw members, knife members, etc.) of the end effector 654. Thus, the various mechanical degrees of freedom of a surgical instrument mounted on instrument manipulator 512 can be moved by manipulation of remotely controlled actuators on instrument carriage 530.

[0038] Surgical Planning System FIG. 8 illustrates a schematic diagram of an exemplary surgical planning system 800. In one embodiment, the surgical planning tool 800 includes a TSS 850 in data communication with an electronic medical record database 830. The TSS 850 illustrated here is similar to the TSS 50 illustrated in FIG. 4. In one embodiment, the electronic medical record database 830 includes medical records of patients treated at a particular hospital or multiple hospitals. The database 830 can be implemented on a server located on the premises of the hospital. The medical record entries contained in the database 830 can be accessed by a computer at the hospital over an intranet network. Alternatively, the database 830 can be implemented on a remote server located away from the hospital, for example, using one of many cloud data storage services. In this case, the medical record entries of the database 830 are stored on the cloud server and can be accessed by a computer system with internet access.

[0039] In one embodiment, a surgical procedure is performed on a first patient using the TSS 850. An imaging device associated with the TSS 850 captures images of the surgical site and displays the captured images as frames of video on the display of the Surgeon Console 52. In one embodiment, a medical practitioner at the Surgeon Console 52 uses an input device of the Surgeon Console 52 to highlight or annotate a particular patient's anatomical structure shown in the displayed video. An example of such an input device is the left and right hand grip control inputs 36, 38 shown in FIG. 2, respectively, which are coupled to a cursor that operates in conjunction with a graphic user interface superimposed on the displayed video. The graphic user interface may include a pointing device such as a QWERTY keyboard, a mouse and an interactive screen display, a touch screen display, or other means for data or text input, or voice annotation and / or speech to text conversion by a microphone and a processor. Thus, the medical practitioner may highlight tissue of interest in the displayed image or input text annotation.

[0040] In one embodiment, the surgical site video is further displayed on a display located on the electronics cart 56. In one embodiment, the display of the electronics cart is a touch screen user interface useful for medical personnel to highlight and annotate specific portions of the patient's anatomy shown on the image displayed for viewing on the display on the electronics cart. By the user touching a portion of the patient's anatomy displayed on the touch screen user interface, the user can highlight a portion of the displayed image. Additionally, a graphic interface including a keyboard can be overlaid on the displayed image. The user can enter text annotations using the keyboard or other input device such as a microphone and speech-to-text driver.

[0041] In one embodiment, surgical site videos captured by imaging devices associated with the TSS 850 are recorded by the TSS 850 and stored in the database 830 in addition to being displayed to the user in real-time or near real-time. Highlights and / or annotations made by the user in association with the recorded videos can also be stored in the database 830. In one embodiment, highlights made by the user are embedded in the recorded videos prior to storage in the database 830. At a later time, the recorded videos can be retrieved for viewing. In one embodiment, a person viewing the recorded videos can select whether or not highlights are displayed. Similarly, annotations associated with the recorded videos can also be stored in the database 830. In one embodiment, annotations made by the user are used to tag the recorded videos and to provide a means of identifying the subject matter contained in the recorded videos. For example, one annotation can represent the condition of a particular disease state. This annotation is used to tag the recorded videos. At a later time, a person wishing to view the recorded procedures related to this disease state can use a keyword search to find the videos.

[0042] Searching for stored videos In some cases, it is desirable for medical personnel to be able to view video recordings of past surgical procedures performed on a given patient. In one aspect, a patient who previously underwent a first surgical procedure to treat a medical condition subsequently requires a second surgical procedure to treat a recurrence of the same medical condition or to treat an anatomical structure located near the surgical site of the first surgical procedure. In one aspect, the surgical site events of the first surgical procedure were captured in the surgical site video recording, and the video recording was stored in the database 830 as part of the patient's electronic medical record. In a related embodiment, the first surgical procedure may be indexed according to the type of procedure. Prior to performing a second surgical procedure on a patient, medical personnel may perform a search of the database 830 to find video recordings of the patient's previous surgical procedures.

[0043] In some cases, it is desirable for a medical practitioner planning to perform a surgical procedure on a patient to be able to view video recordings of similar surgical procedures performed on individuals with certain characteristics similar to the patient. In one embodiment, the surgical site video recordings of the surgical procedures can be tagged with metadata information such as the patient's age, sex, body mass index (BMI), genetic information, the type of procedure the patient underwent, etc., before each video recording is stored in the database 830. In one embodiment, the metadata information used to tag the video recordings is automatically retrieved from the patient's existing medical records and then used to tag the video recordings before storing the video recordings in the database 830. Thus, before performing a medical procedure on a patient, the medical practitioner can search the database 830 for video recordings of similar procedures performed on patients who share certain characteristics in common with the patient. For example, if the medical practitioner uses the TSS 850 to plan to perform a prostatectomy on a 65-year-old male patient with a high body mass index, the medical practitioner can use the TSS 850 to search the database 830 for surgical site video recordings of prostatectomies performed on similarly high body mass index males of a similar age.

[0044] In one embodiment, the video recording of the surgical procedure is communicated by the database 830 to an optional personal computing device 820 (shown in dashed lines), such as a personal computer, tablet, smart phone, terminal, or other electronic access device, for viewing by medical personnel planning to perform the surgical procedure. Additionally or alternatively, video recordings of previous surgical procedures are communicated by the database 830 to the TSS 850 for viewing before or during surgery. In one embodiment, the video recordings are displayed by the TSS 850 on a display located on the surgeon console 52. In another embodiment, the video recording of the first surgical procedure is displayed on a display located on the electronics cart 56.

[0045] In related embodiments, as described in more detail below, video recordings of previous procedures can be synchronized or queued with a stage of a current surgical procedure to facilitate immediate location of related surgical tasks or techniques, such as those used in future stages of the current procedure. Queuing video playback to the current surgical procedure allows the surgeon to very efficiently scroll through the playback to view future or recent steps. Notably, while stage synchronization or stage queuing is not strictly time-synchronized, there is typically a similarity in time order between the previous and current surgical procedures. Rather, as described above, stage synchronization is aligned with respect to the surgical process flow. Direct visual feature matching by video matching can be enhanced by one or more of (a) metadata, (b) stage synchronization, and (c) temporal synchronization.

[0046] Alternatively, video sequences from a current surgery can be matched to video sequences from a representative surgery performed by an experienced surgeon by matching based on visual similarity (e.g., color, time, edges, etc.) or content similarity (e.g., organs, anatomical structures, tools, etc.) if the content can be identified using computer vision methods.

[0047] Cloud-Based Video Database In one aspect, the database 830 is implemented on a remote server using a cloud data storage service and is accessible by multiple healthcare providers. Referring to FIG. 8 , as shown in dashed lines, the surgical planning tool 800 optionally includes a TSS 850 (shown in dashed lines) and a personal computing device 840 (shown in dashed lines). In one aspect, the TSS 850 is similar to the TSS 850 and the personal computing device 840 is similar to the personal computing device 820, except that the TSS 850 and the personal computing device 820 are located at a first healthcare provider and the TSS 850 and the personal computing device 840 are located at a second location or with a second healthcare provider. In one aspect, a first patient requires a surgical procedure for a medical condition and undergoes the surgical procedure using the TSS 850 at the first healthcare provider. A video recording of the surgical procedure is stored in the database 830. At a later time, a second patient requires surgical treatment for the same medical condition and is scheduled to undergo surgical treatment using the TSS 850 at a second healthcare provider. Prior to performing the surgical procedure on the second patient, the medical personnel accesses the database 830 via a secure internet connection and searches the database 830 for surgical site video recordings of similar procedures. In one embodiment, the medical personnel treating the second patient can retrieve the video recordings of the first patient's surgical procedure from the database 830 without gaining knowledge of the first patient's identity. In this manner, the privacy of the first patient is maintained. In one embodiment, the video recordings of the first patient's surgical procedure include highlights and / or annotations made by the medical personnel who treated the first patient.

[0048] Computer-Based Pattern Matching and Analysis The surgical planning tool 800 may include pattern matching and analysis algorithms implemented in the form of computer executable code. In one aspect, the pattern matching and analysis algorithms are stored in a non-volatile memory device of the surgical planning tool 800 and configured to analyze the video recordings stored in the database 830. As previously described, each video recording 830 stored in the database may be tagged with and / or embedded with certain metadata information. This metadata information may include patient information such as the patient's age, gender, and other information describing the patient's health or medical history. Additionally, as previously described, the metadata information may include highlights or annotations made by a medical professional. In one aspect, these highlights and annotations are embedded in the video recordings and stored in the database 830 along with the video.

[0049] In one aspect, the pattern matching and analysis algorithm includes an image analysis component that identifies patterns of shapes and colors shared among multiple video recordings stored in database 830. The pattern matching and analysis algorithm reviews tagged metadata associated with this subset of videos to determine if words or phrases are frequently associated with videos in this subset. Metadata can be tagged, for example, by annotating text, translating speech to text. These analyses performed by the pattern matching and analysis algorithm can be used to assist medical personnel in making decisions regarding a patient's anatomy, preferred surgical approach, disease state, potential complications, etc.

[0050] Virtual Surgical Environment, Data Recording, and Surgeon Evaluator A TSS can record video of the surgical procedure being performed, together with the surgeon's input, via a control console. It is generally desirable to use the data collected in this way to train and assist the surgeon. However, system designers face many challenges in managing the vast amounts of collected data, extracting the relevant parts of the data, and providing useful, unobtrusive assistance that the surgeon will welcome.

[0051] In various embodiments, as described in more detail below, the TSS is augmented or supported by additional dedicated machines to provide assessment and support to the medical personnel operating the TSS. FIG. 9 illustrates an advanced surgical system 900 featuring a TSS 950, similar to the TSS 50 described above in connection with FIG. 4, except that the TSS 950 includes appropriate interfaces to additional support systems, including a virtual surgical environment (VSE) 904, a data recording system 906, a surgeon assessor 908, and a support system 910. Each of these additional support systems may be realized in a variety of machine configurations. For example, one or more additional systems may be provided as dedicated units or as part of a computer platform by execution of program instructions. The computer platform may be a single physical machine, or in the case of a distributed model of cloud computing, may be distributed among multiple physical machines by role, function, processing thread, etc. The computer platform may, in some embodiments, be implemented as part of the TSS 950 (e.g., in the surgeon console 52, electronics cart 58, etc.) or may be implemented using separate hardware from the TSS (e.g., a tablet, personal or laptop computer, a smartphone, or some combination thereof). The computer platform may also be implemented as a combination of local processing on a system or personal hardware and the cloud.

[0052] In various embodiments, certain operations may be performed in a virtual machine, which in turn may be performed on one or more physical machines. Those skilled in the art will appreciate that features of the embodiments may be realized by a variety of different suitable machine implementations.

[0053] In various embodiments, these components are implemented as engines, circuits, components, or modules and are referred to as engines for consistency, although these terms may be used interchangeably. An engine may be hardware, software, or firmware communicatively coupled to one or more processors to perform the operations described herein. An engine may be a hardware engine, and such an engine may be considered a tangible entity capable of performing certain operations and may be configured or arranged in a particular manner. In one example, a circuit may be configured in a particular manner (e.g., internally or to an external entity such as another circuit) as an engine. In one example, all or a portion of one or more hardware processors may be configured as an engine that operates to perform certain operations by firmware or software (e.g., instructions, application portions, or applications). In one example, the software may reside on a machine-readable medium. In one example, the software, when executed by the engine's underlying hardware, causes the hardware to perform the specified operations. The term hardware engine, therefore, should be understood to encompass a tangible entity, an entity that is physically configured, specifically configured (e.g., hardwired), or temporarily (e.g., for a very short period of time) configured (e.g., programmed) to operate in a particular manner or to perform some or all of any of the operations described herein.

[0054] Consider the example where the engines are temporarily configured, where each engine need not be instantiated at one time. For example, if the engines include general purpose hardware processor cores configured using software, the general purpose hardware processor cores may be configured as different engines at different times. Thus, the software may configure the hardware processor cores to, for example, configure a particular engine at one time and a different engine at a different time.

[0055] The TSS 950 interfaces with a surgical environment 902 that is useful to the patient and medical personnel. Generally speaking, the control inputs of the TSS 950 are provided to electromechanical systems that produce real-world effects in the surgical environment 902. In contrast, the VSE 904 computationally models one or more portions of the actual surgical environment to form a virtual surgical environment according to some embodiments. Thus, the same control inputs of the TSS 950 are provided to the VSE 904, and these control inputs produce computationally modeled effects in the virtual surgical environment.

[0056] In one type of embodiment, the virtual surgical environment includes a model of at least a portion of a patient including physiological structures, fluids, disease states, hemodynamics, etc., in addition to the effects of modeled electromechanical outputs representing the output of the TSS 950 on the patient model. In another type of embodiment, the patient is not modeled, but rather, the electromechanical outputs of the TSS 950 are represented. In another type of embodiment, the control signaling provided to the electromechanical systems of the TSS 950 is modeled by the VSE 904.

[0057] In a related embodiment, the virtual surgical environment 904 is programmed or otherwise configured to determine a stage or segment of the surgical procedure based on an evaluation of the surgical input received from the TSS 950.

[0058] In another related embodiment, the VSE 904 is programmed or otherwise configured to determine a stage or segment of a surgical procedure based on the VSE simulated effects of surgical inputs to the TSS, including simulated effects on the patient (e.g., tissue effects at the surgical site, hemodynamics, etc.), or other simulated features of the patient-surgical instrument interaction other than the surgical inputs themselves. In another related embodiment, a combination of surgical inputs and their simulated effects are used to evaluate a segment of a surgical procedure.

[0059] FIG. 10 illustrates the operational elements of the VSE 904 and their interactions according to one embodiment. The surgical input evaluator 1002 is programmed or otherwise configured to collect event data, such as control inputs, kinematic information, sensor data, system events, or state indicators, among others, from the TSS 950 and interpret their meaning in terms of surgical effect. The segmenter 1004 is programmed or otherwise configured to distinguish between stages of a surgical procedure while the surgical procedure is being performed or in a post-processing mode. The simulator 1006 is programmed or otherwise configured to model a surgical procedure in a virtual surgical environment based on the control inputs from the TSS 950. The simulator 1006 can operate in parallel with or in place of an actual surgical procedure performed in the surgical environment 902 and can be used by the surgeon to practice a particular procedure or a portion thereof. In some embodiments, the simulator 1006 is invoked during certain parts of the procedure to allow the surgeon to practice a task before actually performing the task on the patient. More generally, the simulator 1006 may be used for training and practice by surgeons, students, or other medical personnel.

[0060] 11 is a diagram illustrating the operational elements of the surgical input evaluator 1002 and their interactions, according to one embodiment. The event monitor 1102 receives event data from the TSS 950, which may include system events, kinematics, and external inputs, and records a series of events. In a related embodiment, the event monitor 1102 receives information regarding additional actions taken in the surgical environment 902 other than or by the TSS 950, such as actions by the assistant(s) or other equipment / instruments associated with the procedure. These actions are detected and communicated to the TSS 950, for example, by one or more sensors of the TSS 950, or by environmental monitoring sensors such as video capture devices, or by the other equipment / instruments themselves. As referenced above, events may further include any of the activations of instrument controls, such as button presses, instrument motion controls, instrument selections, video system controls, wrist motion, camera motion activation, master gestures / activations, and other inputs provided by the surgeon, assistant, or other detectable actions / inputs / outputs. For example, other data collected as events may include motion data, eye movements, user authentication (e.g., iris recognition), muscle activity (e.g., measurements using electromyography), sensed surgeon posture, images captured by the endoscope, system state (e.g., docking / deployment status of various instruments, engagement / disengagement status of the arms with the master controller, type of instrument installed, number of instruments installed, operator head engagement status, forces applied to controls such as the master control, forces detected by the robotic instruments, camera movements and controls, touch screen inputs, personnel positions (e.g., if sensors are provided in the surgical environment, etc.), and a history of any of these. The sequence of events may be recorded as time series data and may be recorded collectively and represent the coordinated actions of the operator(s) of the TSS 950 and any personnel within the surgical environment 902, and / or other data compiled from the surgical environment 902.

[0061] In some embodiments, the gesture assessor 1104 reads the time series event data and applies the gesture criteria 1106 to see if a combination of events constitutes any defined gesture. In the present context, a gesture is a sequence of two or more events that constitute a composite movement of the tool and control. As an illustrative example, a single stitching action associated with a sewing task can be considered a gesture. In this example, there is generally a sequence of individual movements for which control input data is collected and classified to recognize the gesture. In one embodiment, the gesture criteria 1106 includes a library of sequences of events that constitute various gestures. In a more basic environment, the gesture criteria 1106 can simply define a time window of recent events (e.g., the most recent 40 seconds) such that for any time increment, that time window of a sequence of events is considered to be a gesture. The time window or gesture in this example may overlap with other gestures.

[0062] Notably, in other embodiments, the gesture evaluator 1104 is not present. Rather, the monitored events are provided to the task evaluator 1108. In yet another type of embodiment in which the gesture evaluator 1104 is available, a combination of detected gestures and monitored events is provided to the task evaluator 1108 in parallel.

[0063] The task evaluator 1108 is programmed or otherwise configured to evaluate the current surgical task being performed. In the present context, a task is a sequence of events, gestures, or combinations thereof that together produce a defined surgical effect. A task can be a clinically common operation (e.g., an incision task, a suturing task, etc.) or a clinically relevant step(s) / segment(s) of a procedure (e.g., UV anastomosis of a prostatectomy). The task evaluation is based on a sequence of recent events and further based on data from a task criteria database 1110 that associates a defined sequence of events with a task. In a related embodiment, the task criteria database 1110 can also include a library of surgical procedures for which those tasks are commonly performed. In this type of embodiment, the identification of the surgical procedure itself can be an input to the task evaluation algorithm. A surgical procedure in this context is a high-level descriptor, for example, a prostatectomy, an umbilical hernia repair, a mastectomy, etc.

[0064] In various embodiments, the gesture evaluator 1104 and the task evaluator 1108 may independently perform one or more classification operations such as K-nearest neighbor (KNN) classification, clustering operations such as K-means clustering, association rule mining operations, support vector machine (SVM) operations, artificial neural network (ANN) operations (e.g., recurrent neural networks, convolutional neural networks, etc.), etc. based on their respective criteria 1106, 1110 used as training set data when applicable.

[0065] Each event, gesture, and task can have various parameter values ​​such as joint angles, speed, leading / trailing idling, etc. It will be appreciated that during a teleoperated surgical procedure, control inputs occur that cause a change in the operating state of the surgical system. For example, the surgeon can move his head in or out of the viewer, resulting in a change in the head-in state. The surgeon can bring a hand or foot into or out of contact with a control input, for example, causing a change in state: The combination of instruments in use can be changed, for example causing a change in the instrument type state.

[0066] Operation of the TSS 950 to assist in surgical activities in one or more surgical conditions further results in the generation of control input information within the surgical system indicative of instrument movements indicative of the actual manner in which the surgeon performed the surgical activity. For example, the surgeon may use the control inputs to move an instrument, for example, quickly or slowly. The surgeon may, for example, move a surgical instrument along one or another path toward or away from an anatomical structure. The surgeon may, for example, use the control inputs to adjust the position of different instruments before actuating a particular instrument.

[0067] 19A-19C are illustrations of an exemplary surgical instrument 1902 and actuator assembly 1903, according to some embodiments, where the surgical instrument is shown in three different exemplary surgical instrument actuation states. The exemplary instrument 1902 includes a jaw end effector 1904 that is transitionable between open and closed states, with a continuum of partially open / partially closed states in between. The exemplary instrument 1902 also includes a two-degree-of-freedom (2-dof) wrist 1906 that can move between different two-dimensional (x,y) positional states. The exemplary actuator assembly 1903 includes a first actuator 1908 that, in some embodiments, includes a jaw motor (JM) that is used to actuate the jaw end effector 1904. The exemplary actuator assembly 1903 includes a second actuator 1910 that, in some embodiments, includes a wrist motor (WM) that is used to actuate the wrist 1906. During surgery, the surgical instrument 1902 can transition through a number of instrument actuation states that correspond to different activities during a surgical procedure. Each transition results in the generation of control input information that is captured and stored by the event monitor 1102, which control input information indicates the movement of the instrument when commanded to transition from its physical position and configuration in one state (e.g., open or closed) to its physical position and configuration in the next state.

[0068] For example, as shown in FIG. 19A, a first gesture can include a first actuator 1908 (JM) placing the jaw end effector 1904 in a fully open state and a second actuator 1910 (WM) placing the wrist 1906 in a first position state (x1, y1). For example, as shown in FIG. 19B, a surgical procedure can include a second gesture in which the first actuator 1908 transitions the jaw end effector 1904 to a fully closed state and the second actuator 1910 transitions the wrist 1906 to a second position state (x2, y2). For example, as shown in FIG. 19C, a surgical procedure can include a third surgical activity in which the first actuator 1908 transitions the jaw end effector 1104 in a partially open / partially closed state and the second actuator 1910 transitions the wrist 1906 to a third position state (x3, y3).

[0069] According to one embodiment, the surgical input evaluator 1002 cooperates with the simulator 1006 to model positional states in response to control input information (e.g., events, gestures, or a combination thereof). The virtualized movements and positional states can in turn be evaluated by the task evaluator 1108 to identify applicable tasks. In this type of embodiment, task evaluation can be performed using modeled kinematic information rather than or in addition to processed control input data.

[0070] 12, a segmenter 1004 according to an exemplary embodiment is shown in detail, including its operational elements and their interactions. In the illustrated example, the segmenter 1004 is configured to selectively apply one of two classes of algorithms for segmentation assessment: a real-time segment evaluator 1200 and a post-processing based segment evaluator 1220.

[0071] The real-time segment evaluator 1200 may be applied during surgery to perform segment evaluation with negligible delay as perceived by the surgeon or medical personnel. In an exemplary embodiment, the real-time segment evaluator 1200 is configured as a recurrent neural network (RNN) using a long short-term memory (LSTM) RNN architecture for deep learning. As shown, the feature extractor 1202 applies filtering or other selection criteria to the sequence of evaluated tasks and their corresponding parameters to generate a feature vector as input to the RNN 1204. Examples of feature vectors include poses (e.g., position and rotation matrix information) along with actuator joint angles and velocities. The RNN 1204 may be a bidirectional RNN, where there are actually two RNNs, one forward-looking and the other backward-looking. Thus, predicted future values ​​and past values ​​are considered in the RNN algorithm. In an exemplary embodiment, the RNN 1204 has two layers, a bidirectional LSTM and a gated recurrent unit (GRU) layer, with hidden nodes in each layer.

[0072] The training data 1206 includes classified feature vectors that are used to train the RNN 1204 using an appropriate training algorithm, as will be appreciated by those skilled in the art. The training data may be based on surgical inputs (e.g., events including control input parameters from surgeon operated controls, or auxiliary modifiable instrument configurations, events occurring in the surgical environment, etc.). In a related embodiment, the training data may be based on simulated post-effects of the surgical inputs. In this latter type of embodiment, the simulator 1006 may be a source of training data.

[0073] The post-processing segment evaluator can be applied at any time after the operation, using all the data recorded from the operation to perform segment evaluation. In one example of a post-processing based segment evaluator 1220, clustering is performed using a hierarchical alignment clustering algorithm (HACA) according to an illustrated embodiment. The exemplary segment evaluator 1220 architecture includes a feature extractor 1222, a pre-processor 1224, a frame kernel matrix 1226, and a clustering engine 1228 according to an illustrated embodiment. The feature extractor 1222 generates a feature vector having a particular dimension and representing a particular set of parameters. The pre-processor 1224 can use a K-means clustering algorithm according to one embodiment.

[0074] The frame kernel matrix operation 1226 calculates a self-similarity (i.e., Gram) matrix K=φ(X) related to the motion segmentation. T Calculate φ(X), where each entry s ij is the two samples x i and x j This defines the similarity between

number

[0075] The HACA clustering engine is based on the paper Zhou, Feng, Fernando De la Torre, Jessica K. Hodgins, “Hierarchical aligned cluster analysis for temporal clustering of human motion,” IEEE Transactions on Run the HACA algorithm, which matches Pattern Analysis and Machine Intelligence 35, no. 3(2013): 582-596.

[0076] In a related embodiment, a post-processing based segment evaluator 1220 is used post-operatively to update the training data 1206 and re-evaluate the segment decisions of the real-time segment evaluator 1200 .

[0077] The confidence measurement engine 1230 is programmed or otherwise configured to calculate a confidence score that represents the probability of a correct segmentation. The confidence score may be calculated using cross-entropy and log-probability calculations. As described in more detail below, the calculated confidence may be used as a metric to assess the skill of the surgeon or as a weighting on other metrics, such as time, to further improve the assessment accuracy.

[0078] 13 is a diagram illustrating the operating elements of the simulator 1006 and their interactions according to one embodiment. A patient model 1302 is programmed or otherwise configured to represent a patient based on the patient's measured or assessed physical characteristics 1304 or based on measured or diagnosed medical conditions 1306. Examples of relevant physical characteristics 1304 include size, weight, sex, age, body mass index (BMI), muscle tone, organ size and location, etc., as well as imaged anatomical structures, such as obtained from a CT scan, MRI, ultrasound, or other examination. Similarly, medical conditions 1306 may include features such as tumors, cysts, hernias, torn connective tissue, fractured bones, etc. A TSS modeler 1310 is programmed or otherwise configured to represent the TSS 950, including tools, end effectors, and other components, as represented by configuration information 1312. The TSS modeler 1310 receives control input information from the TSS 950 so that the positioning and movement of various surgical tools can be reflected in the virtual surgical environment.

[0079] The TSS modeler 1310 and the patient model 1302 can exchange state information during the simulation as the virtual surgical procedure is simulated, such that one or both of these engines can maintain and update the current state of the patient as changes are made by the surgical procedure.

[0080] In one embodiment, the simulator 1006 operates with the surgical environment 902 to track the progress of the TSS 950 and the patient during the surgical procedure. In another embodiment, a given segment of the surgical procedure can begin simulation at the surgeon's request or in response to a trigger. This intra-operative simulation is invoked when the surgeon encounters a critical or difficult moment during the procedure and wishes to practice certain movements or various techniques before actually deploying these measures on the patient. In a related embodiment, the parallel running simulations may be autonomously analyzed to determine the current surgical stage using the segmenter 1004, which may be further trained with model-based situational data in a set of feature vectors.

[0081] In another embodiment, the simulator 1006 may be used for surgical training or practice in place of an actual patient. In some implementations, the user interface of the TSS 950 for surgeons and other medical personnel can be used to realistically simulate a surgical environment and provide a machine interface experience as part of the training or practice.

[0082] Surgeon's evaluation 14 is a more detailed diagram illustrating the operational elements of the surgeon evaluator 908 and their interactions, according to one embodiment. At the heart of the surgeon evaluator 908 is a surgical technique evaluator 1402 programmed or otherwise configured to analyze the control input signaling and the evaluations by the gesture evaluator 1104 (if available), the task evaluator 1108, and the segmenter 1004 to generate an evaluation or score representative of the quality of the surgeon's surgical technique in manipulating the control inputs of the TSS 950, which may be used during a surgical procedure or during a practice or training simulation.

[0083] Inputs to the surgical technique evaluator include video information V, event information E, gestures G, and evaluated tasks T along with kinematic data K. This data may be provided by the TSS 950 or the VSE 904 (e.g., from the surgical input evaluator 1002, the simulator 1006). In a related embodiment, the inputs may include segment confidence scores SC as determined by the confidence measurement engine 1230 of the segmenter 1004 based on segment determinations by the RNN 1204 or the clustering engine 1228.

[0084] A segment corresponds to a surgical procedure portion and is defined in terms of a combination of one or more features that represent one or more combinations of video, event, gesture, task, and kinematics. A segment includes an event / kinematics / video combination that can be expressed as a gesture / task or an event / kinematics / video combination that can be directly expressed without intermediate gesture / task representation.

[0085] In one embodiment, the surgical technique evaluator 1402 calculates various statistics for each of the various temporal and spatial metrics. For example, the duration of various surgical procedure segments may be evaluated. Similarly, task-level durations may be calculated and aggregated in several ways, for example, per segment. An economy-of-motion analysis may also be performed that examines the amount of gestures or other movements, the amount of events, or some combination of these amounts, associated with the completion of a particular task or surgical segment. Idle time (i.e., no movement) may also be considered. Other metrics may also be taken into account in the evaluation of the surgical technique, such as motion speed, wrist angle of the input device, idle time (e.g., stationary), master workspace range, master clutch number, applied energy, and other such parameters. Camera control metrics may also be taken into account in the evaluation of the surgical technique, such as camera movement frequency, camera movement duration, camera movement spacing, etc. In some embodiments, camera clutch movement (CMFrq) is defined as the average number of endoscopic movements made by the surgeon per time increment (e.g., per second) over the course of an entire event or segment. Camera movement duration (CMDur) is defined as the average time increment (e.g., seconds) of all endoscope movements over the course of an entire event or segment. Camera clutch interval (CMInt) is defined as the average time increment (e.g., seconds) between endoscope movements over the course of an entire event or segment. Non-limiting examples include duration of camera use, consistency of camera use, duration of energy application (cutting tissue, cauterizing tissue, increasing energy applied), frequency of master clutch use, temporal control of different arms (e.g., time between use of arm 1 and arm 3), frequency of master alignment with end effector orientation, etc.

[0086] The evaluation criteria 1406 defines one or more criteria for calculating the surgeon's performance rating. In one type of embodiment, the surgical procedure performance is given a series of scores corresponding to different identified stages of the procedure. The evaluation criteria 1406 can include filtering, masking, and weighting information applied to various metrics to adjust their relative importance. In particular, different weights / adjustments may be applied to the metrics for various surgical procedures or for different stages of a given surgical procedure. The evaluation criteria 1406 can also define algorithms or formulas to be applied in rendering the surgical performance rating. In an embodiment where the surgical technique evaluator 1402 uses machine learning algorithms such as ANNs, classifiers, clustering agents, support vector machines, etc., the evaluation criteria 1406 can include criteria for constructing feature vectors of the metrics.

[0087] The benchmark skills database 1404 includes scores and metrics achieved by surgeons or other medical personnel who are considered experts in the corresponding surgical procedure.

[0088] For example, in some embodiments, the benchmark database 1404 includes a benchmark of a surgeon's skill level based on camera control time. The inventors conducted a study to evaluate surgeons as experienced, intermediate, or novice in using a teleoperated surgical system (TSS) by measuring camera control time for each of a number of exercises representing different surgical tasks (e.g., camera targeting, point and needle, energy dissection, match board, needle targeting, peg board, pick and place, ring and rail, ring walking, scaling, suturing, sponge, thread ring and tube). The results of the exercises are used to create a benchmark of the surgeon's skill level based on camera control time. The study found that experienced surgeons generally perform camera control much faster than novice surgeons. Intermediate surgeons often perform exercises much faster than novice surgeons. However, the study found that there was no significant difference in the time to complete the exercises between intermediate and experienced surgeons.

[0089] This study revealed many surprising findings. For example, experienced surgeons had significantly higher CMFrq than novice surgeons in most practices, intermediate surgeons had significantly higher CMFrq than novice surgeons in most practices, and there was no significant difference in CMFrq between intermediate and experienced surgeons. This study found that experienced surgeons had significantly shorter CMDur than novice surgeons in most practices. Intermediate surgeons had significantly shorter CMDur than novice surgeons in most practices. Expert surgeons had significantly shorter CMDur than intermediate surgeons in most practices. Expert surgeons had significantly shorter CMDur than intermediate surgeons in most practices. Expert surgeons had significantly shorter CMInt than novice surgeons in most practices, but intermediate surgeons had significantly shorter CMInt than novice surgeons in most practices. There was no significant difference in CMInt between intermediate and experienced surgeons.

[0090] In a related embodiment, the constructed feature vector for each task or segment that may be performed in real time for a given procedure is compared to one or more benchmark scores corresponding to the task or segment of the surgical procedure. This comparison may be a simple Cartesian distance measure between the feature vector in question and the benchmark feature vector, or some other similar measure. The determined similarity may be returned as a surgical performance score corresponding to the task, segment, etc.

[0091] The various scores calculated for each surgeon for a given surgical procedure are stored in a surgeon database 1408. As surgeons gain experience and expertise in using the TSS 950 and performing particular procedures, scores over time can be analyzed to detect trends, such as rates of improvement. According to some embodiments, the surgeon database 1408 distinguishes performance scores between simulated-only performance of the TSS 950 at 1412 and performance corresponding to a procedure performed on an actual patient at 1414.

[0092] Data logging 15 is a diagram illustrating the operational elements of the data recording system 906 and their interactions according to an exemplary embodiment. An indexer 1502 is programmed or otherwise configured to associate incoming data with other relevant data to facilitate data retrieval. In the illustrated example, various data may be recorded during a surgical procedure. For example, captured video and added annotations 1504, an assessed input log 1506, an assessed task log 1508, a simulation log 1510, and a running surgical procedure assessment log 1512 may be recorded. Each item of data may be further associated with its date and time of capture, the patient, the surgeon, the TSS, the type of surgery, and other relevant data.

[0093] In one embodiment, for captured video and annotation data 1504, indexer 1502 can divide the video into segments corresponding to evaluated segments of the surgical procedure. Evaluated input log 1506 stores evaluated gestures as well as time series events and other input information. Evaluated task log 1508 includes evaluations of tasks and surgical procedure segments performed in real time. In one embodiment, evaluated input log 1506 includes a short-term memory portion and a long-term memory portion, where the short-term memory portion stores all incoming input information including events, while the long-term memory portion stores only inputs for which a task adjudication decision was made.

[0094] The simulation log 1510 contains time-series state information describing the patient model 1302 and the TSS modeler 1310 of the simulator 1006. In one embodiment, the sampling rate of the simulation log is similar to the rate at which segment evaluations are performed (e.g., 4-20 samples / sec). Sufficient data is recorded to allow detailed review and analysis of the surgical procedures performed.

[0095] The active surgical technique assessment log 1512 stores a time series sample of continuously updated surgical performance scores generated by the surgical technique evaluator 1402. The data recorded in the active surgical technique assessment log 1512 may be specific to the current surgical task (or simulation) being performed on the current patient. This data set may overlap with data in the surgeon database 1408 (FIG. 14), which stores surgeon-specific data across multiple different patients, procedures, and dates.

[0096] Assist engine 16 is a diagram illustrating the operational elements of the assist engine 910 and their interactions, according to some embodiments. As shown, the assist engine 910 includes a help-request detector 1600 that is programmed or otherwise configured to detect a request for surgeon assistance. In some embodiments, any of these engines, the help-request detector 1600, the performance sensor 1602, and the TSS user interface (UI) 1604, can initiate assist operations of the assist engine 910 in response to a respective input. The performance sensor 1602 monitors the operation of the VSE 904 and the surgeon evaluator 908. For example, the output of the segmenter 1004 overlaid with an ongoing surgical technique evaluation (e.g., recorded in the database 1512) provides information regarding the stage of the surgical procedure and the measured performance of the surgeon at that stage.

[0097] The assist criteria 1603 include various threshold conditions that, when met, trigger the execution of an assist action by one or more of the assistance rendering engines described below. In particular, different thresholds may be associated with different types of assistance. During operation, the manual performance sensor 1602 compares the currently running surgical technique assessment of the current surgical procedure and its segments against the threshold conditions corresponding to the procedure and stage. If the threshold condition comparison is met, i.e., if the currently running surgical technique assessment score falls below a given threshold, the corresponding assistance is suggested and / or provided.

[0098] In a related embodiment, the threshold conditions of the assist criteria 1603 may be surgeon-specific or patient-specific. For example, a more experienced surgeon may configure the system with a higher threshold before assistance is provided than a novice surgeon. Similarly, a lower threshold may be selected for surgery on patients with more critical conditions (e.g., severe trauma patients, elderly patients, neonatal or premature patients, etc.).

[0099] Additionally, for example, the assist criteria 1603 can be used to selectively request input from certain skilled surgeons who have expertise in certain steps of the procedure. For example, certain surgeons may be really good at performing complex dissections, and therefore they should be on-call if someone is struggling with that step but another surgeon is looking for an easier step.

[0100] The TSS UI interface 1604 exchanges information with the surgeon console of the TSS 950. The TSS UI interface 1604 receives requests for specific assistance from the surgeon, whereupon the TSS UI interface instructs the assistance rendering engine to provide the corresponding assistance. Assistance information relevant to the situation can also be displayed to the surgeon via the TSS UI interface, which sends that information to the surgeon console.

[0101] In a related embodiment, in response to an automatic request for assistance by the treatment capability sensor 1602, the TSS UI interface 1604 provides a visual or audio notification to the surgeon providing the assistance and provides a control input for the surgeon to accept, decline, or postpone the assistance.

[0102] According to the illustrated embodiment, a number of different assistance rendering engines are available to provide corresponding types of assistance to a surgeon or other medical personnel. In various other embodiments, more or fewer assistance rendering engines are provided. As shown, a video queuer 1606 is configured to retrieve video segments from a database 1608 for playback. The video segments may have a particular correspondence to a current stage of the surgical procedure, as determined by the segmenter 1004. For example, the video segments that are queued may represent a current stage or a future stage of the surgical procedure.

[0103] The notifier 1610 is configured to initiate communication via a remote console or other interface with an on-call surgeon according to expert contact 1612 who can provide direct guidance and / or assistance to the surgeon.

[0104] The technical advisor 1614 is responsive to a current or upcoming surgical stage and recommends a particular surgical approach to be used in a given scenario, taking into account the surgeon's assessed ongoing surgical technique assessment based on the surgical skill requirements database 1616. In a related embodiment, the technical advisor 1614 may provide prompts or other instructions or recommendations applicable to other medical personnel, such as one or more assistants in the surgical environment.

[0105] In a related embodiment, the technical advisor 1614 includes a predictive function that predicts upcoming stages of surgery based on step duration, step order, current segment, and step competency level (all based on the current segment and surgeon competency, as well as optionally historical data corresponding to that surgeon). This information can be used to predict future events, such as end of procedure for optimal OR turnover, future instrument changes, etc.

[0106] The simulation composer 1618 operates to autonomously set up simulation scenarios that correspond to stages of a current or upcoming surgical procedure to be performed by the simulator 1006. This type of assistance may enable a surgeon to practice certain surgical procedures in the virtual surgical environment during surgery, before returning to the actual surgical environment to perform the procedure on a patient.

[0107] The control setting adjuster 1620 operates to automatically change one or more behaviors of the TSS in response to control inputs to facilitate a surgical task. Adjustments may be made according to control adjustment rules 1622, which may be procedure-specific, task-specific, surgeon-specific, or some combination thereof. Examples of variable TSS behaviors include adjustments to control input sensitivity (e.g., slowing down the motion speed of a given input event or gesture), gesture filtering (e.g., allowing certain input movements to produce surgical effects but not others), video capture settings, etc. Other adjustments include camera horizon / zoom / position, table position, image characteristics (color, tint, overlay, etc.), loading pre-op images for the current / future step from the current patient or related patients, scaling of master tool movements, force feedback gain, mapping control input button presses differently depending on the context / segment, or some combination thereof.

[0108] 21A-21C are diagrams illustrating the change in horizon of the surgical scene made by the control setting adjuster 1620 to assist the surgeon in achieving a perspective that allows him to more precisely control the surgical instruments in the surgical scene of the teleoperated surgical system (TSS). The surgeon stereoscopically views the surgical scene including the surgical instruments via the left eye display 32 and the right eye display 34 (a stereoscopic surgeon view is depicted for illustrative purposes, but in various other embodiments, a monoscopic view may be provided). For example, assume that the TSS is fitted with three different surgical instruments 2110, 2112, 2114 and a camera (not shown) carrying an endoscope. The surgeon can control two surgical instruments and two endoscopes using left and right hand grip control inputs 36, 38. The instruments that are in the field of view of the camera are visible to the surgeon in the surgical scene view. Further, assume that, for example, the surgeon changes instruments during the transition from one task to the next (where the surgeon changes). Furthermore, the changed from instrument is displayed as the changed from instrument to the endoscope. Assume that the surgical instrument currently under the surgeon's control is positioned at a different location in the surgical scene than the left and right hand grip control inputs 36, 38. In this case, the surgeon's grip on the left and right hand grip control inputs 36, 38 does not change, but the position and orientation of the surgical instrument in the surgical scene currently under the surgeon's control is changed. The control setting adjuster 1620 can provide adjustment of the horizon of the surgical scene to align the viewpoint of the instrument currently under the surgeon's control in the surgical scene with the position of the surgeon's hands on the left and right hand grip control inputs 36, 38 to improve or optimize precision and control of the surgical instrument.

[0109] Referring to FIG. 21A, a first surgical scene 2100 is shown with a first horizon indicated by the orientation of a horizon icon 2120, including first and second surgical instruments 2110, 2112 currently under the control of the surgeon. The horizon icon 2120 is displayed as an optional overlay within the scene, and indicates the horizon of the surgical scene, which is typically the level when the endoscope is in the neutral position of the manipulators of the patient side cart. It will be understood that the horizon level is nominally defined as the neutral position in the patient side arm, and is not relative to the left and right displays, since the left and right displays never change position. Furthermore, the orientation of the surgical instruments 2110, 2112 and the horizon of the first surgical scene shown in FIG. 21A is consistent with the natural anatomical orientation of the surgeon's hands on the left and right hand grip control inputs 36, 38 when viewing the scene placed directly in front of the surgeon's eyes. Note that the 3D Cartesian position of the surgeon's hand may differ from the 3D Cartesian position of the instrument due to the master clutch capabilities of the TSS.

[0110] 21B, a second surgical scene 2102 is shown having a first horizontal line, indicated by the orientation of a horizontal line icon 2120, that includes first and third surgical instruments 2110, 2114 that are currently under the control of the surgeon. Assume that in the course of transitioning from the first scene 2100 to the second scene 2102, the surgeon has changed from using a second surgical instrument (shown in dashed lines) 2112 to using a third surgical instrument 2114. Thus, the surgeon's hands on the left and right hand grip control inputs 36, 38 control the first and third surgical instruments 2110, 2114 in the second surgical scene 2102, rather than controlling the first and second surgical instruments 2110, 2112 as they did in the first scene 2100. The second and third surgical instruments 2112, 2114 are positioned in different locations within the surgical scene. Additionally, the horizon icon 2120 indicates that the view of the second surgical scene 2102 from the perspectives of the left and right eye displays 32, 34 is the same as the perspective of the first scene 2100. Thus, the surgeon now controls a third surgical instrument 2114 in the second surgical scene 2102, which is in a different position than the position of the previously controlled second instrument 2112. The position of the surgeon's hands on the left and right hand grip control inputs 36, 38 does not change, and the horizon of the surgical scene does not change, but the position of one of the currently controlled instruments, namely the third instrument 2114, is changed. Furthermore, the orientation of the right hand grip control input 38 must be changed to match the new orientation of the third instrument 2114, which is different from the instrument 2112, in case the surgeon needs to mirror the pose of the third instrument 2114 before gaining control over that instrument. For these reasons, and potentially others, the positions of the surgical instruments 2110, 2114 and the horizon of the second surgical scene shown in Figure 21B may not match the natural anatomical positions of the surgeon's hands on the left and right hand grip control inputs 36, 38. As a result of this mismatch, the surgeon may not be able to move the third instrument 2114 with the precision and / or dexterity that he or she might otherwise have.

[0111] Referring to FIG. 21C, a third surgical scene 2104 is shown with a second horizon line indicated by the orientation of the horizon line icon 2120, including the first and third surgical instruments 2110, 2114 currently under the control of the surgeon. The third scene 2104 is identical to the second scene 2102, except that the horizon line of the surgical scene changes as indicated by the orientation of the horizon line icon. As a result of the change in scene orientation, it will be appreciated that the position / orientation of the surgical instruments 2110, 2114 and the horizon line of the third surgical scene shown in FIG. 21C more closely matches the natural anatomical position of the surgeon's hands on the left and right hand grip control inputs 36, 38 when viewing the scene located directly in front of the surgeon's eyes. As a result of the change in orientation of the horizon line of the surgical scene, the surgeon can more easily move the third instrument 2114 more precisely than with the horizon line of the surgical scene of the second scene 2102.

[0112] The control setting adjuster 1620 provides a camera orientation control setting change to the endoscope-mounted camera to change the horizon orientation from that shown in the second surgical scene 2102 to that shown in the third surgical scene 2104. The surgical technique evaluator 1402 can determine, for example, to use the horizon orientation of the third scene 2114 when an experienced surgeon controls the first and third instruments 2100, 2104 at the positions shown in the second and third scenes 2110, 2114. The control setting adjuster 1620 can automatically perform such a horizon orientation change on behalf of a novice surgeon. Thus, for example, when a novice surgeon transitions to a task that may be improved by a change in the horizon of the surgical scene according to the accuracy of the experienced instrument movement, the control setting adjuster 1620 can automatically change the horizon orientation of the surgical scene to better correlate the horizon orientation of the surgical scene and the position of the instruments in the scene with the position of the surgeon's hands on the left and right hand grip control inputs 36, 38.

[0113] 22A-22B are illustrations showing two-dimensional positional changes of a surgical scene performed by the control setting adjuster 1620 to assist the surgeon in achieving a viewpoint from which the surgeon can better observe relevant information within the surgical scene of the TSS. Referring to FIG. 22A, a fourth surgical scene 2200 is shown in which the first and second surgical instruments 2110, 2112 have their end effector portions positioned near the edge of the surgical scene 2200. When the surgical instruments 2100, 2102 are positioned near the edge of the scene as in the fourth surgical scene 220, items of interest located near the edge of the scene, such as body tissue (not shown), may be located outside the visible scene and therefore may not be visible to the surgeon. Referring to FIG. 22B, a fifth surgical scene 2202 is shown in which the first and second surgical instruments 2110, 2112 have their end effector portions positioned near the center of the surgical scene 2202. The control setting adjuster 1620 provides camera control setting changes to the endoscope-mounted camera to change from a second surgical scene 2102 displaying a first two-dimensional area with the end effector of the instrument located at the edge of the scene to a third surgical scene 2104 displaying a second two-dimensional area with the end effector of the instrument located near the center of the scene. The surgical technique evaluator 1402 can determine, for example, that the surgeon prefers to display the instrument in a particular area (e.g., the center, etc.) within the surgical scene. Thus, for example, when a novice surgeon follows a more experienced surgeon into a task involving a scene where he or she can better see the change in the area of ​​the depicted scene as the surgeon moves the instrument toward the center of the scene, the control setting adjuster 1620 can automatically change the position of the camera to change the position of the instrument and tissue structures (not shown) depicted within the scene.

[0114] 23A-23B are diagrams illustrating the zoom level changes of the surgical scene performed by the control setting adjuster 1620 to assist the surgeon in achieving a perspective from which the surgeon can better observe relevant information within the surgical scene of the TSS. Referring to FIG. 23A, a sixth surgical scene 2300 is shown in which the first and second surgical instruments 2110, 2112 are shown zoomed out so that they appear small and far away within the surgical scene 2200. It is understood that the small and far away appearance of the surgical instruments 2100, 2102 in the sixth surgical scene 2300 also brings other items of interest, such as body tissue (not shown), into which the surgical instruments 2100, 2102 interact and thus become difficult for the surgeon to see. Referring to FIG. 23B, a seventh surgical scene 2302 is shown in which the first and second surgical instruments 2110, 2112 are shown zoomed in so that they appear closer and larger and more easily seen in detail. The control setting adjuster 1620 provides a camera control setting change to the endoscope-mounted camera to change from the fifth surgical scene 2300, which displays a first zoomed-out zoom level, to the seventh surgical scene 2302, which displays a second zoomed-in zoom level. The surgical technique evaluator 1402 can determine, for example, that an experienced surgeon prefers to view instruments at a particular zoom level for a particular task. Thus, for example, the control setting adjuster 1620 can automatically change the camera zoom level when, for example, a novice surgeon follows the experienced surgeon and transitions to a task that includes a scene that can be better viewed as the zoom level changes.

[0115] Additionally, camera alignment can include an estimate of the patient's anatomical position and camera orientation to optimize the camera position / orientation for movement and interaction. For example, if a surgeon needs to move a needle in a particular direction based on anatomy, the camera can align itself to this environmental cue.

[0116] In some embodiments, assistance by any of these assist engines can be phase-synchronized to the current stage of the surgical procedure based on a current assessment of the stage. As noted above, phase synchronization refers to alignment in terms of surgical process flow, rather than strict time alignment. In some embodiments, phase synchronization supports phase offsetting, such that assistance is provided to prepare the surgeon or surgical team for a future stage of the surgical procedure.

[0117] The assistance may be further selected to match the type of evaluated surgical procedure used, as evaluated by the segmenter 1004, the simulator 1006, the surgeon evaluator 908, or a combination including two or more of these engines.

[0118] 17 is a block diagram illustrating a computer system that is an exemplary form of a general purpose machine. In certain embodiments, programming the computer system 1700 according to one or more of the specific operating architectures and algorithms described herein produces a special purpose machine that, when executed, forms a VSE 904, a data recording engine 906, a surgeon evaluator 908, an assist engine 910, or any combination of these systems. In a network deployment, the computer system may operate in the capacity of either a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0119] The exemplary computer system 1700 includes at least one processor 1702 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both, a processor core, a computation node, etc.), a main memory 1704, and a static memory 1706, which communicate with each other via a link 1708 (e.g., a bus). The computer system 1700 may further include a video display device 1710, an alphanumeric input device 1712 (e.g., a keyboard), and a user interface (UI) navigation device 1714 (e.g., a mouse). In one embodiment, the video display device 1710, the input device 1712, and the UI navigation device 1714 are incorporated into a touch screen display. The computer system 1700 may further include a memory device 1716 (e.g., a drive unit), a signal generator 1718 (e.g., a speaker), a network interface device (NID) 1720, and one or more sensors (not shown).

[0120] The storage device 1716 includes a machine-readable medium 1722 that stores one or more sets of data structures and instructions 1724 (e.g., software) embodied in or utilized by any one or more of the methods or functions described herein. The instructions 1724 may also reside, completely or at least partially, within the main memory 1704, the static memory 1706, and / or the processor 1702 during execution thereof by the computer system 1700, with the main memory 1704, the static memory 1706, and / or the processor 1702 also constituting machine-readable media.

[0121] While the machine-readable medium 1722 is shown to be a single medium in the exemplary embodiment, the term "machine-readable medium" may include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more instructions 1724. The term "machine-readable medium" is considered to include any tangible medium that can store, encode, or convey instructions for execution by a machine, cause a machine to perform any one or more methods of the present disclosure, or store, encode, or convey data structures utilized by or in connection with such instructions. Thus, the term "machine-readable medium" is considered to include, but is not limited to, solid-state memory, optical media, and magnetic media. Specific examples of machine-readable media include, by way of example, semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and non-volatile memory, including, but not limited to, CD-ROMs and DVD-ROMs.

[0122] The NID 1730 according to various embodiments may take any suitable form factor. In one such embodiment, the NID 1720 is in the form of a network interface card (NIC) that interfaces with the processor 1702 via a link 1708. In one example, the link 1708 is a PCI Express (PCIe) bus that includes a slot in which the NIC form factor may be removably engaged. In another embodiment, the NID 1720 is a network interface circuit laid out on a motherboard along with local link circuitry, processor interface circuitry, other input / output circuitry, memory circuitry, storage device, peripheral controller circuitry, and the like. In another embodiment, the NID 1720 is a peripheral device that interfaces with the link 1708 via a peripheral input / output port, such as a universal serial bus (USB) port. The NID 1720 transmits and receives data over a transmission medium 1726, which may be wired or wireless (e.g., radio frequency, infrared, or visible light spectrum, etc.), optical fiber, etc.

[0123] FIG. 18 is a diagram illustrating an exemplary hardware and software architecture of a computer device such as that shown in FIG. 17, showing various interfaces between hardware (HW) and software (SW) components. Hardware components, as indicated by HW, are displayed below the dividing line, while software components, as indicated by SW, are above the dividing line. On the hardware side, a processing unit 1802 (which may include one or more microprocessors, digital signal processors (each having one or more processor cores), etc.) interfaces with a memory management unit 1804 and a system interconnect 1806. The memory management unit 1804 provides mapping between virtual and physical memory used by the processes being executed. The memory management unit 1804 may be an integral part of a central processing unit, which also includes the processing unit 1802.

[0124] The interconnect 1806 includes interfaces with input / output devices such as PCI, USB, etc. as well as backplanes such as memory, data, and control lines. Memory 1808 (e.g., Dynamic Random Access Memory - DRAM) and non-volatile memory such as flash memory 1809 (e.g., Electrically Erasable Read Only Memory - EEPROM, NAND Flash, NOR Flash, etc.) are interfaced with the memory management unit 1804 and the interconnect 1806 via memory controllers 1810. In some embodiments, the architecture can support direct memory access (DMA) by peripherals. I / O devices such as video and audio adapters, non-volatile storage, external peripheral links such as USB, Bluetooth, etc. as well as network interface devices such as those communicating via Wi-Fi or LTE family interfaces are collectively represented as I / O devices and network 1812, which interface with the interconnect 1806 via corresponding I / O controllers 1814.

[0125] On the software side, a pre-operating system (pre-OS) environment 1816, which runs at initial system boot, is responsible for initiating the boot of an operating system. A traditional example of a pre-OS environment 1816 is the system Basic Input / Output System (BIOS). In modern systems, the Unified Extensible Firmware Interface (UEFI) is implemented. The pre-OS environment 1816 is responsible for initiating the boot of an operating system, but also provides an execution environment for embedded applications in accordance with certain aspects of the present invention.

[0126] The operating system (OS) 1818 provides a kernel that controls hardware devices, manages memory access for programs in memory, coordinates tasks, facilitates multitasking, organizes stored data, allocates memory space and other resources, loads program binary code into memory, initiates execution of application programs (which then interact with the user and the hardware devices), and detects and responds to various defined interrupts. The operating system 1818 also provides device drivers and various common services that facilitate interfacing with peripherals and networks and provide an abstraction of application programs so that the applications do not have to be responsible for handling the details of such common operations. The operating system 1818 also provides a graphical user interface (GUI) that facilitates interaction with a user through peripheral devices such as a monitor, keyboard, mouse, microphone, video camera, touch screen, etc.

[0127] Runtime system 1820 implements portions of the execution model, including operations such as placing parameters on a stack before a function call, disk input / output (I / O) operations, and operations related to parallel execution. Runtime system 1820 may also perform support services such as type checking, debugging, or code generation and optimization.

[0128] Libraries 1822 include collections of program functions that provide further abstraction of application programs. These include, for example, shared libraries, dynamic link libraries (DLLs), and the like. Libraries 1822 may be integrated into the operating system 1818, runtime system 1820, may add functionality, or may even be remotely hosted. Libraries 1822 define an application program interface (API) through which various function calls are made by application programs 1824 to invoke services provided by the operating system 1818. Application programs 1824 are programs that perform tasks useful to a user beyond the tasks performed by lower-level system programs that coordinate the basic operation of the computing device itself.

[0129] Process Flow 20 is a process flow diagram illustrating an exemplary method 2000 of operating a TSS support system, such as system 900. It should be noted that method 2000 is a well-characterized embodiment that combines a variety of different operations for purposes of illustration. While the example shown in FIG. 20 may be a practical embodiment, it will be understood that certain operations may be modified, omitted, or reordered in various other embodiments.

[0130] The following description refers to Figure 20 in conjunction with the system component diagrams and operational component diagrams of Figures 9-16. At 2002, the event monitor 1102 of the surgical input evaluator 1002 receives surgical input. The surgical input includes monitored events of the TSS. Additionally, the surgical input may include control inputs to a computational model of the TSS in the simulator 1006. The surgical input may also include modeled events from the computational model of the TSS.

[0131] In 2004, the TSS modeler 1310 models the TSS in the virtual surgical environment. This may include computationally representing tools, end effectors, and other components as specified in configuration information 1312. The TSS modeler 1310 processes control inputs and effects virtual state changes in the model in response to those control inputs. The virtual state changes may themselves constitute types of events that the event monitor 1102 may receive.

[0132] In 2006, the patient model 1302 models the patient, including the effects of the operations of the TSS modeler 1310. In 2008, the surgical input evaluator determines the current surgical task being performed. In 2010, the segmenter 1004 determines the current stage of the surgical procedure based on the surgical input and based on the optional evaluated tasks.

[0133] In 2012, the confidence measurement engine 1230 of the segmenter 1004 determines a confidence score for the stage assessment. In 2014, the surgeon evaluator 908 uses the surgical technique evaluator 1402 to calculate temporal and spatial metrics of the surgical input. Optionally, the confidence score is taken into account. In 2016, the surgical technique evaluator generates a stage-specific performance score for the surgeon or surgical staff.

[0134] At 2018, a request for surgeon assistance may be detected based on the current stage of the surgical procedure, the performance score, or some combination of these items. The request for assistance may be provided by the surgeon or another operator. At 2020, assistance synchronized to the stage is rendered.

[0135] Additional notes and examples Example 1 is a surgical assistance system for a teleoperated surgical system (TSS) including a surgeon input interface, the surgeon input interface receiving surgical control inputs for enabling an electro-mechanical surgical system to perform a surgical procedure, the surgical assistance system having a virtual surgical environment engine and an assist engine; The virtual surgical environment engine includes computer hardware configured to be operable to implement: a surgical input evaluation engine that receives surgical input including monitored events of the TSS; and a segmentation engine that determines a current stage of the surgical procedure based on the surgical input; the assist engine includes computer hardware configured to be operable to implement: a TSS interface communicatively coupled to a surgeon input interface of the TSS; an assistance request detector that detects a request for surgeon assistance; and an assistance rendering engine that initiates context-related assistance via the TSS interface in response to the request for surgeon assistance; Context-related assistance is step-synchronized with the surgical procedure.

[0136] In Example 2, the subject matter of Example 1 optionally includes a TSS. In Example 3, any one or more of the subject matter of Examples 1-2 optionally includes, wherein the virtual surgical environment engine is further configured to implement a simulator engine that processes a computational model of the surgical procedure based on surgical input of the TSS. In Example 4, the subject matter of Example 3 optionally includes, wherein the simulator engine includes a TSS model for computationally representing a TSS in the virtual surgical environment; and a patient model for computationally representing a patient based on physical characteristics of the patient, and changes to the patient affected by manipulation of the TSS model.

[0137] In Example 5, the subject matter of any one or more of Examples 3-4 optionally includes, wherein the simulator engine is configured to operate in parallel with an actual surgical procedure being performed via the TSS, and surgical control inputs to the TSS cause virtual effects in a computational model of the simulator engine. In Example 6, the subject matter of any one or more of Examples 3-5 optionally includes, wherein the surgical input received by the surgical input evaluation engine includes simulated effects of the TSS control inputs from a computational model of the simulator engine. In Example 7, the subject matter of any one or more of Examples 3-6 optionally includes, wherein the simulator engine is configured to simulate a particular stage of the surgical procedure based on the determination of the current stage, and initiate the simulation.

[0138] In example 8, the subject matter of any one or more of examples 1-7 optionally includes, wherein the monitored events include a TSS control input that controls an electro-mechanical surgical system of the TSS. In Example 9, the subject matter of any one or more of Examples 1-8 optionally includes, wherein the monitored events include a TSS control input that controls a simulation of an electro-mechanical surgical system of the TSS. In Example 10, the subject matter of any one or more of Examples 1-9 optionally includes, wherein the monitored event includes a TSS detection action by a surgical assistant.

[0139] In Example 11, the subject matter of any one or more of Examples 1-10 optionally includes, wherein the segmentation engine includes a confidence measurement engine for determining a confidence score representing a probability of correct segment determination. In Example 12, the subject matter of Example 11 optionally includes, wherein the confidence score indicates a suitability of the corresponding sequence of surgical inputs for training a deep learning engine of the segmentation engine. In Example 13, the subject matter of any one or more of Examples 1-12 optionally includes, wherein the segmentation engine includes a real-time segment evaluator configured to implement a trained neural network that processes a portion of the surgical input during the surgical procedure. In Example 14, the subject matter of Example 13 optionally includes, wherein the segmentation engine further includes a post-processing segment evaluator configured to implement a clustering algorithm to process a portion of the surgical input.

[0140] In Example 15, the subject matter of any one or more of Examples 13-14 optionally includes, wherein the trained neural network is trained based on prior surgical inputs from multiple preceding surgical procedures. In Example 16, the subject matter of any one or more of Examples 13-15 optionally includes, wherein the trained neural network is trained based on prior surgical inputs from multiple previously simulated surgical procedures. In Example 17, the subject matter of any one or more of Examples 1-16 optionally includes, wherein the surgical input assessment engine includes a task assessment engine configured to determine a current surgical task being performed based on the surgical input, the current surgical task including a series of events that result in a defined surgical effect, and the segmentation engine determines a current stage of the surgical procedure further based on the current surgical task.

[0141] In Example 18, the subject matter of any one or more of Examples 1-17 optionally includes a surgeon evaluation engine including computer hardware configured to operate to implement a surgical technique evaluation engine, the surgical technique evaluation engine configured to: access surgical inputs and a current stage of the surgical procedure; calculate a plurality of temporal and spatial metrics of the surgical inputs corresponding to a plurality of different stages of the surgical procedure including the current stage; and generate a surgical stage specific performance score representative of the quality of the surgeon's surgical performance in producing the surgical control inputs. In Example 19, the subject matter of Example 18 optionally includes, wherein the surgical technique evaluator is further configured to perform a comparison between the temporal and spatial metrics and a benchmark metric based on stages performed by a surgical procedure expert, and the performance score is based on a result of the comparison. In Example 20, the subject matter of any one or more of Examples 18-19 optionally includes, wherein the surgical technique evaluator is further configured to access a confidence score representing a probability of accurate segment determination by the segmentation engine, and to generate a capability score further based on the confidence score.

[0142] In Example 21, the subject matter of any one or more of Examples 18-20 optionally includes, wherein the performance score is a running score that is updated throughout the surgical procedure. In Example 22, the subject matter of any one or more of Examples 18-21 optionally includes, wherein the assistance request detector is configured to detect a request for surgeon assistance based on the capability score. In Example 23, the subject matter of any one or more of Examples 1-22 optionally includes, wherein the context-relevant assistance includes an expert video segment including a demonstration of a stage-specific portion of the surgical procedure being performed.

[0143] In Example 24, the subject matter of any one or more of Examples 1-23 optionally includes, wherein the context-related assistance includes configuring and initiating a stage-specific simulation of a surgical procedure. In example 25, the subject matter of any one or more of examples 1-24 optionally includes, wherein the situation-related assistance includes adjusting control settings.

[0144] Example 26 is a machine-implemented method for assisting a teleoperated surgical system (TSS) including a surgeon input interface, the surgeon input interface receiving surgical control input for enabling an electro-mechanical surgical system to perform a surgical procedure, the method including the steps of receiving surgical input including monitored events of the TSS; determining a current stage of the surgical procedure based on the surgical input; detecting a request for surgeon assistance; and in response to the request for surgeon assistance, initiating context-related assistance to the surgeon input interface, wherein the context-related assistance is phase-synchronized to the surgical procedure. In Example 27, the subject matter of Example 26 optionally includes simulating a surgical procedure as a computational model based on surgical input of the TSS.

[0145] In Example 28, the subject matter of Example 27 optionally includes, wherein simulating the surgical procedure includes computationally representing the TSS in the virtual surgical environment as part of a computational model; and computationally representing changes to the patient based on physical characteristics of the patient and affected by modeled manipulation of the TSS computationally represented in the computational model. In Example 29, any one or more of the subjects of Examples 27-28 optionally include, wherein the simulation is performed in parallel with an actual surgical procedure performed via the TSS, and surgical control inputs to the TSS produce virtual effects in the computational model. In example 30, the subject matter of any one or more of examples 27-29 optionally includes, wherein the surgical input includes a simulated effect of the TSS control input from the computational model.

[0146] In Example 31, the subject matter of any one or more of Examples 27-30 optionally includes a step of simulating a particular stage of the surgical procedure using a computational model based on the determination of the current stage. In Example 32, the subject matter of any one or more of Examples 26-31 optionally includes, wherein the monitored events include a TSS control input that controls an electro-mechanical surgical system of the TSS. In Example 33, the subject matter of any one or more of Examples 26-32 optionally includes, wherein the monitored events include a TSS control input that controls a simulation of an electro-mechanical surgical system of the TSS.

[0147] In Example 34, the subject matter of any one or more of Examples 26 to 33 optionally includes, wherein the monitored event includes a TSS detection action by a surgical assistant. In Example 35, the subject matter of any one or more of Examples 26-34 optionally includes determining a confidence score representing a probability of correct segment determination. In Example 36, the subject matter of Example 35 optionally includes, wherein the confidence score indicates a suitability of the corresponding sequence of surgical inputs for training a deep learning algorithm.

[0148] In Example 37, the subject matter of any one or more of Examples 26-36 optionally includes the following, wherein the trained neural network is implemented to process a portion of the surgical input during the surgical procedure when determining a current stage of the surgical procedure. In Example 38, the subject matter of Example 37 optionally includes, wherein a clustering algorithm is implemented to process a portion of the surgical input when determining a current stage of the surgical procedure. In Example 39, the subject matter of any one or more of Examples 37-38 optionally includes, wherein the trained neural network is trained based on prior surgical input from multiple preceding surgical procedures.

[0149] In Example 40, the subject matter of any one or more of Examples 37-39 optionally includes, wherein the trained neural network is trained based on prior surgical inputs from multiple previously simulated surgical procedures. In Example 41, the subject matter of any one or more of Examples 26-40 optionally includes a step of determining a current surgical task being performed based on surgical input, the current surgical task including a series of events that result in a defined surgical effect, and the step of determining a current stage of the surgical procedure is further based on the current surgical task. In Example 42, the subject matter of any one or more of Examples 26-41 optionally includes the steps of: calculating multiple temporal and spatial metrics of surgical inputs corresponding to multiple different phases of the surgical procedure, including the current phase; and generating a surgical phase-specific performance score representative of the quality of the surgeon's surgical performance in producing the surgical control inputs.

[0150] In Example 43, the subject matter of Example 42 optionally includes a step of performing a comparison between the temporal and spatial metrics and a reference metric based on the stages performed by a surgical procedure expert, and the performance score is based on the results of the comparison. In example 44, the subject matter of any one or more of examples 42-43 optionally includes generating a capability score further based on the confidence score representing a probability of correct segment determination. In example 45, the subject matter of any one or more of examples 42-44 optionally includes, wherein the performance score is a running score that is updated throughout the surgical procedure.

[0151] In example 46, the subject matter of any one or more of examples 42-45 optionally includes, wherein the request for surgeon assistance is based on the performance score. In Example 47, the subject matter of any one or more of Examples 26-46 optionally includes, wherein the context-relevant assistance includes an expert video segment including a demonstration of a stage-specific portion of the surgical procedure being performed. In example 48, the subject matter of any one or more of examples 26-47 optionally includes, wherein the context-related assistance includes configuring and initiating a stage-specific simulation of a surgical procedure. In example 49, the subject matter of any one or more of examples 26-48 optionally includes, wherein the situation-related assistance includes adjusting control settings.

[0152] Example 50 is at least one non-transitory machine-readable storage medium including instructions that, when executed on a computer platform, cause the computer platform to implement a dedicated machine for supporting a teleoperated surgical system (TSS) including a surgeon input interface, the surgeon input interface receiving surgical control input for enabling an electro-mechanical surgical system to perform a surgical procedure, the instructions including: instructions for receiving surgical input including monitored events of the TSS; instructions for determining a current stage of the surgical procedure based on the surgical input; instructions for detecting a request for surgeon assistance; and instructions for initiating context-related assistance to the surgeon input interface in response to the request for surgeon assistance, wherein the context-related assistance is phase-synchronized to the surgical procedure.

[0153] In example 51, the subject matter of example 50 optionally includes instructions for simulating a surgical procedure as a computational model based on surgical input of the TSS. In Example 52, the subject matter of Example 51 optionally includes, wherein the instructions for simulating the surgical procedure include: instructions for computationally representing a TSS in the virtual surgical environment as part of a computational model; and instructions for computationally representing changes to the patient based on physical characteristics of the patient and affected by modeled manipulation of the TSS computationally represented in the computational model. In Example 53, any one or more of the subjects of Examples 51-52 optionally include, wherein the simulation is performed in parallel with an actual surgical procedure performed via the TSS, and surgical control inputs to the TSS produce virtual effects in the computational model.

[0154] In Example 54, the subject matter of any one or more of Examples 51-53 optionally includes, wherein the surgical input includes a simulated effect of the TSS control input from the computational model. In example 55, the subject matter of any one or more of examples 51-54 optionally includes instructions for simulating a particular stage of the surgical procedure using the computational model based on the determination of the current stage. In Example 56, the subject matter of any one or more of Examples 50 to 55 optionally includes, wherein the monitored events include a TSS control input that controls an electro-mechanical surgical system of the TSS.

[0155] In Example 57, the subject matter of any one or more of Examples 50 to 56 optionally includes, wherein the monitored events include TSS control inputs that control the simulation of the electro-mechanical surgical system of the TSS. In Example 58, the subject matter of any one or more of Examples 50 to 57 optionally includes the following, wherein the monitored event includes a TSS detection action by a surgical assistant. In example 59, the subject matter of any one or more of examples 50-58 optionally includes instructions for determining a confidence score representing a probability of correct segment determination.

[0156] In Example 60, the subject matter of Example 59 optionally includes, wherein the confidence score indicates a suitability of the corresponding sequence of surgical inputs for training a deep learning algorithm. In Example 61, the subject matter of any one or more of Examples 50-60 optionally includes, wherein the instructions for determining a current stage of the surgical procedure are implemented in a trained neural network to process a portion of the surgical input during the surgical procedure. In Example 62, the subject matter of Example 61 optionally includes, wherein the instructions for determining a current stage of the surgical procedure are performed by a clustering algorithm to process a portion of the surgical input.

[0157] In Example 63, the subject matter of any one or more of Examples 61-62 optionally includes, wherein the trained neural network is trained based on prior surgical input from multiple preceding surgical procedures. In Example 64, the subject matter of any one or more of Examples 61-63 optionally includes, wherein the trained neural network is trained based on prior surgical input from multiple previously simulated surgical procedures. In Example 65, the subject matter of any one or more of Examples 50-64 optionally includes instructions for determining a current surgical task being performed based on surgical input, the current surgical task including a series of events that result in a defined surgical effect, and the instructions for determining a current stage of the surgical procedure are based on the determination of the current surgical task.

[0158] In Example 66, the subject matter of any one or more of Examples 50 to 65 optionally includes instructions for calculating multiple temporal and spatial metrics of surgical inputs corresponding to multiple different phases of the surgical procedure including the current phase; and instructions for generating a surgical phase-specific performance score representative of the quality of the surgeon's surgical performance in producing the surgical control inputs. In Example 67, the subject matter of Example 66 optionally includes instructions for performing a comparison between the temporal and spatial metrics and a baseline metric based on steps performed by a surgical procedure professional, and the performance score is based on the results of the comparison. In example 68, the subject matter of any one or more of examples 66-67 optionally includes instructions for generating a capability score further based on the confidence score representing a probability of an accurate segment determination.

[0159] In Example 69, the subject matter of any one or more of Examples 66-68 optionally includes, wherein the performance score is a running score that is updated throughout the surgical procedure. In Example 70, the subject matter of any one or more of Examples 66-69 optionally includes, wherein the request for surgeon assistance is based on a performance score. In Example 71, the subject matter of any one or more of Examples 50-70 optionally includes, wherein the context-relevant assistance includes an expert video segment including a demonstration of a stage-specific portion of the surgical procedure being performed.

[0160] In Example 72, the subject matter of any one or more of Examples 50 to 71 optionally includes the following, wherein the context-related assistance includes configuring and initiating a stage-specific simulation of a surgical procedure. In Example 73, the subject matter of any one or more of Examples 50 to 72 optionally includes the following, wherein the situation-related assistance includes adjusting control settings.

[0161] Although exemplary embodiments have been shown and described, a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments can be used without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention is limited only by the claims that follow, and it is appropriate that the claims be interpreted in a manner and broadly consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A method of operating a surgical support system for supporting a teleoperated surgical system (TSS) including a surgeon input interface that receives control inputs for acting on an electro-mechanical surgical system to perform a surgical procedure, the method comprising: a surgical input evaluation engine receiving surgical input including monitored events of the TSS; a segmentation engine determining a current stage of the surgical procedure based on the surgical input; an assist engine detecting a request for surgeon assistance based on the surgeon's performance score generated by the surgical technique evaluator; and in response to the request for surgeon assistance, an assistance rendering engine initiates context-related assistance to the surgeon; the situation-related assistance is consistent with a stage of the surgical procedure; method.

2. and simulating, by a simulator engine, the surgical procedure as a computational model based on the surgical input of the TSS; The step of simulating a surgical procedure includes: computationally representing the TSS in a virtual surgical environment as part of the computational model; and computationally representing changes to the patient that are affected by the modeled electromechanical output of the TSS that are computationally represented in a computational model.

3. The method of claim 1 , wherein the monitored events include the control inputs that control an electro-mechanical surgical system of the TSS.

4. The method of claim 1 , wherein the monitored events include the control inputs that control a simulation of an electro-mechanical surgical system of the TSS.

5. The method of claim 1 , wherein the monitored events include detectable actions by a surgical assistant.

6. The method of claim 1 , wherein the context-related assistance includes an expert video segment including a demonstration of a portion of the surgical procedure being performed.

7. 2. The method of claim 1, wherein the context-related assistance includes at least one of a change in the orientation of a horizon of a visual display of a surgical scene, a change in a two-dimensional area shown within the visual display of a surgical scene, and a change in a zoom level of a visual display of a surgical scene.

8. 1. A surgical support system for supporting a teleoperated surgical system (TSS) including a surgeon input interface that receives control inputs for acting on an electro-mechanical surgical system to perform a surgical procedure, the surgical support system comprising: a surgical input evaluation engine for receiving surgical input including monitored events of the TSS; a segmentation engine that determines a current stage of the surgical procedure based on the surgical input; an assist engine that detects a request for surgeon assistance based on the surgeon's performance score generated by the surgical technique evaluator, and initiates context-relevant assistance to the surgeon via an assistance rendering engine in response to the request for surgeon assistance; the situation-related assistance is consistent with a stage of the surgical procedure; Surgical support system.

9. a virtual surgical environment engine that simulates the surgical procedure as a computational model based on the surgical input of the TSS; simulating the surgical procedure includes computationally representing the TSS in a virtual surgical environment as part of the computational model; 10. The surgical support system of claim 8, further comprising computationally representing changes to the patient that are affected by the patient based on physical characteristics of the patient and the modeled electromechanical output of the TSS that are computationally represented in a computational model.

10. The surgical support system of claim 8 , wherein the monitored events include the control input that controls an electro-mechanical surgical system of the TSS.

11. The surgical support system of claim 8 , wherein the monitored events include the control inputs that control a simulation of an electro-mechanical surgical system of the TSS.

12. The surgical support system of claim 8 , wherein the monitored events include detectable actions by a surgical assistant.

13. The surgical assistance system of claim 8 , wherein the context-related assistance includes an expert video segment including a demonstration of a portion of the surgical procedure being performed.

14. The surgical support system of claim 8, wherein the situation-related support includes at least one of a change in the orientation of the horizon of a visual display of a surgical scene, a change in a two-dimensional area shown within the visual display of a surgical scene, and a change in a zoom level of a visual display of a surgical scene.

Images