Surgical systems with training or assistance functions

Abstract

The present application relates to surgical systems with training or assistance functions. The present application discloses systems, methods and implementable software instructions for supporting a remotely operated surgical system (TSS). Surgical inputs including monitored events of the TSS are received. The current phase of the surgical routine is autonomously determined based on the surgical inputs. In response to what context-related assistance can be provided to a surgeon console of the TSS, a call for surgeon assistance can be detected, the context-related assistance being phase-synchronized with the surgical routine.

Description

[0001] This application is a divisional application of Chinese patent application 2016800760204 (PCT / US2016 / 061694) filed on November 11, 2016 and entitled “Surgical system with training or auxiliary function”.

[0002] Priority declaration

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 374,670, filed on August 12, 2016, and claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 254,556, filed on November 12, 2015, each of which is incorporated herein by reference in its entirety. Technical Field

[0004] Aspects of the present invention relate to medical devices for surgical procedures. More specifically, aspects of the present invention relate to controlling a robotic-assisted surgical system to collect and evaluate surgical performance data to automatically assist or provide advice to a surgical staff. Background Art

[0005] Minimally invasive teleoperated surgical systems (TSS) are developed to increase the surgeon's flexibility when working on internal surgical sites and allow surgeons to operate on patients from a remote location (outside the sterile field). In a TSS, the surgeon is usually provided with an image of the surgical site at a console. While viewing the image of the surgical site on an appropriate viewer or display, the surgeon performs a surgical routine on the patient by manipulating the main input device or control device of the console. Each main input device controls the movement of a servomechanically actuated / articulated surgical instrument. During a surgical routine, the TSS can provide mechanical actuation and control of various surgical instruments or tools with an end effector, which, in response to manipulation of the main input device, performs various functions of the surgeon, such as holding or driving a needle, grasping a blood vessel, dissecting tissue, suturing tissue, etc.

[0006] It is desirable to train surgeons to use TSS as well as to assist surgeons in performing surgical procedures using TSS. Practical solutions are needed to facilitate automated surgical assessment and automated surgical assistance where appropriate. Summary of the invention

[0007] The following summary introduces certain aspects of the subject matter of the present invention to provide a basic understanding. This summary is not an extensive overview of the subject matter of the present invention and is not intended to identify key or important elements or to delineate the scope of the subject matter of the present invention. Although this summary contains information related to various aspects and embodiments of the subject matter of the present invention, its sole purpose is to present some aspects and embodiments in an approximate form as a preface to the more detailed description below.

[0008] One aspect of the present invention relates to a surgeon support system for a teleoperated surgical system (TSS). The TSS generally includes a surgeon input interface that accepts surgical control inputs for implementing an electromechanical surgical system to perform a surgical routine.

[0009] In some embodiments, the surgeon support system includes a virtual surgical environment engine and an assistance engine, each of which can be implemented using computing hardware.

[0010] The virtual surgical environment engine includes a surgical input evaluator engine that receives surgical input including monitored events of the TSS and a segmenter engine that determines a current stage of a surgical routine based on the surgical input.

[0011] The assistance engine includes a TSS interface communicatively coupled to a surgeon input interface of the TSS, an assistance call detector for detecting a call for surgeon assistance, and an assistance rendering engine that sends context-dependent assistance via the TSS interface in response to the call for surgeon assistance. The context-dependent assistance is phase-synchronized with the surgical routine. Phase synchronization refers to alignment in terms of surgical process flow, rather than strict time alignment. In some embodiments, phase synchronization supports phase offsets so that assistance can be provided to prepare the surgeon or surgical team for an upcoming phase of the surgical routine.

[0012] Advantageously, phase synchronization provides the surgeon with context-relevant assistance that can be immediately available. Examples of assistance include automatically queued expert video segments of the current (or upcoming) portion of the surgical routine, on-demand simulations of the current or upcoming portion of the surgical routine that can be called up during surgery in some cases, surgical technique recommendations for specific situations from an automated expert system, and notification of human expert assistants previously assessed as having expertise in a particular portion of the surgical routine. The foregoing examples are provided for illustration and are not to be considered limiting in scope unless and to the extent they are expressly set forth in the appended claims.

[0013] In some embodiments, the virtual surgical environment engine is further configured to implement a computational model of a surgical routine that is processed by a simulator engine based on surgical input of a TSS. For example, the simulator engine may include a TSS model and a patient model, the TSS model computationally representing the TSS in the virtual surgical environment; and the patient model computationally representing the patient based on the patient's physical characteristics, and implementing changes to the patient through the operation of the TSS model.

[0014] In some embodiments, the surgeon support system includes a surgical technique evaluator engine configured to access surgical input and a current phase of a surgical routine and to compute a plurality of temporal and spatial metrics of the surgical input corresponding to a plurality of different phases of the surgical routine including the current phase. The surgical technique evaluator engine can generate a performance score for a particular phase of the surgery that represents the quality of the surgical performance of the surgeon generating the surgical control input. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a plan view of a minimally invasive tele-operated surgical system (TSS).

[0016] Figure 2 is a perspective view of the surgeon's console.

[0017] Figure 3 is a perspective view of an electronics cart.

[0018] Figure 4 is a schematic diagram of TSS.

[0019] Figure 5A This is an illustration of TSS.

[0020] Figure 5B is a perspective view of a patient side cart of a surgical system.

[0021] Figure 5C This is an illustration of a surgical operation scene.

[0022] Figure 6 It is a front view of the surgical instrument.

[0023] Figure 7 is a perspective view of an instrument manipulator.

[0024] Figure 8 is a schematic diagram of a surgical planning tool.

[0025] Fig. 9is a diagram illustrating an advanced surgical system according to various embodiments, the advanced surgical system featuring a TSS accompanied by additional support systems including a virtual surgical environment, a data recording system, a surgeon assessment system, and an auxiliary system.

[0026] Fig.10 is a diagram showing a method according to an embodiment Fig. 9 Diagram of the operating components of the system's virtual surgical environment and their interactions.

[0027] Fig.11 is a diagram showing a method according to an embodiment Fig.10 Diagram of the operating components and their interactions of a surgical input evaluator for a virtual surgical environment.

[0028] Fig.12 is a diagram showing a method according to an embodiment Fig.10 Diagram of a segmenter of a virtual surgical environment including its operating components and interactions between the operating components.

[0029] Fig.13 is a diagram showing a method according to an embodiment Fig.10 Diagram of the operating components of the virtual surgical environment simulator and their interactions.

[0030] Fig.14 is a diagram showing a method according to an embodiment Fig. 9 Diagram of the operating components of the surgeon evaluator of the system and their interactions.

[0031] Fig.15 is a diagram showing a method according to an exemplary embodiment Fig. 9 A system of data records a diagram of the operating components of the system and their interactions.

[0032] Fig.16 is a diagram showing a method according to some embodiments Fig. 9 A diagram of the operating components of the system's auxiliary engine and their interactions.

[0033] Fig.17 is a block diagram illustrating a computer system in the example form of a general purpose machine that may be converted to a special purpose machine to perform aspects of the embodiments described herein.

[0034] Fig.18 is a diagram showing a computing device (such as Fig.17 A diagram of an example hardware and software architecture for a computing device depicted in ), in which the various interfaces between hardware components and software components are shown.

[0035] Figure 19A-Figure 19Cis an illustrative diagram showing an example surgical instrument and actuator assembly according to some embodiments, in which the surgical instrument is shown in different example surgical instrument actuation states.

[0036] Fig. 20 is a diagram illustrating operation such as Fig. 9 A process flow diagram of an example method of a TSS support system for a system.

[0037] Figure 21A-Figure 21C It is shown by Fig.14 An illustrative diagram of changes in the surgical scene horizon implemented by the control setting adjuster of the surgeon evaluator to assist the surgeon in achieving a viewpoint that enables the surgeon to more precisely control the surgical instrument within the surgical scene of the TSS.

[0038] Figure 22A-22B It is shown by Fig.14 The surgeon evaluator's control setting adjuster implements an illustrative diagram of two-dimensional position changes of the surgical scene to assist the surgeon in achieving a viewpoint that enables the surgeon to better observe relevant information within the surgical scene of the TSS.

[0039] Figure 23A-23B It is shown by Fig.14 An illustrative diagram of changes in the zoom level of a surgical scene implemented by a control setting adjuster of a surgeon evaluator to assist the surgeon in achieving a viewpoint that enables the surgeon to better observe relevant information within the surgical scene of the TSS. DETAILED DESCRIPTION

[0040] The following description and drawings illustrating aspects, embodiments, implementations or applications of the invention should not be considered limiting - the claims define the protected invention. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the scope of the present specification and claims. In some cases, well-known circuits, structures or technologies are not shown or described in detail to avoid obscuring the present invention. The same numbers in two or more figures represent the same or similar elements.

[0041] The elements described in detail with reference to one embodiment, implementation or application may be included in any practice in other embodiments, implementations or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment, and the element is not described with reference to a second embodiment, the element may be included in the second embodiment to request protection. Therefore, in order to avoid unnecessary repetition in the following description, unless otherwise specifically described, unless one or more elements will make an embodiment or implementation inoperative, or unless two or more elements provide conflicting functions, one or more elements shown and described in association with one embodiment, implementation or application may be incorporated into other embodiments, implementations or aspects.

[0042] Aspects of the present invention, according to embodiments, primarily utilize the da Vinci surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California (specifically, Model IS4000, as However, the skilled artisan will appreciate that the inventive aspects disclosed herein may be embodied and implemented in a variety of ways, including robotic and (if applicable) non-robotic embodiments and implementations. In the case of the da Vinci surgical system (e.g. Surgical systems, The embodiments described herein on the invention are merely illustrative and are not to be construed as limiting the scope of the inventive aspects disclosed herein to any particular model or device, and are not to be construed as limiting it to the extent such limitations are expressly set forth in one or more claims.

[0043] Minimally invasive telesurgery system (TSS)

[0044] Teleoperation refers to operating a machine from a distance. In a minimally invasive teleoperation medical system, a surgeon can use an endoscope that includes a camera to view the surgical site inside the patient's body. In some embodiments, stereoscopic images can be captured, which enables depth perception during surgical routines.

[0045] Referring now to the drawings, in which like reference numerals refer to like parts throughout the several views, Figure 1is a plan view of a minimally invasive TSS 10 typically used to perform minimally invasive diagnostic or surgical routines on a patient 12 lying on an operating table 14. The system includes a surgeon's console 16 for use by a surgeon 18 during a surgical procedure. One or more assistants 20 may also participate in the procedure. The minimally invasive TSS 10 also includes (one or more) patient side carts 22 and an electronics cart 24. The patient side cart 22 can manipulate at least one surgical instrument 26 through a minimally invasive incision in the body of the patient 12 while the surgeon 18 observes the surgical site through the surgeon's console 16. Images of the surgical site can be acquired by a camera mounted with an endoscope 28, such as a stereo endoscope, which can be manipulated by the patient side cart 22 to position and orient the endoscope 28. A computer processor located on the electronics cart 24 can be used to process images of the surgical site for subsequent display to the surgeon 18 through the surgeon's console 16. It should be noted that although discrete system components (i.e., the patient side cart 22, the electronics cart 24, and the surgeon's console 16) are depicted and described for exemplary purposes, in various embodiments, the elements included herein may be combined and / or separated. For example, in some embodiments, the computer processor of the electronics cart 24 may be incorporated into the surgeon's console 16 and / or the patient side cart 22. The number of surgical instruments 26 used at one time will generally depend on the diagnostic or surgical procedure, as well as other factors such as space limitations within the surgical site. If one or more surgical instruments 26 in use need to be replaced during a routine, the assistant 20 can remove the surgical instrument 26 from the patient side cart 22 and replace it with another surgical instrument 26 from the tray 30 in the operating room.

[0046] Figure 2 1 is a perspective view of the surgeon's console 16. The surgeon's console 16 includes a viewer 31 including a left-eye display screen 32 and a right-eye display screen 34 for presenting a coordinated stereoscopic view of the surgical site with depth perception to the surgeon 18. In various other embodiments, a non-stereoscopic display may be provided to the surgeon 18. The console 16 also includes one or more control inputs 36. Mounted on the patient side cart 22 ( Figure 1 One or more surgical instruments on the operative area 26 (shown) are moved in response to the surgeon 18 manipulating one or more control inputs 36. The control inputs 36 may provide a control of the surgical instrument 26 ( Figure 1The surgeon 18 may have the same or greater mechanical degrees of freedom as shown) to provide telepresence to the surgeon 18, or the sense that the control input 36 is integral with the instrument 26 so that the surgeon has a strong sense of directly controlling the instrument 26. To this end, in some embodiments, position, force, and tactile feedback sensors (not shown) may be employed to convey position, force, and tactile sensations from the surgical instrument 26 back to the surgeon's hand through the control input 36, subject to communication latency limitations. It should be noted that although a physical console 16 having a fixed viewer 31 and a mechanically coupled control input 36 is depicted and described for exemplary purposes, in various other embodiments, an "ungrounded" control input and / or display structure may be used. For example, in some embodiments, the viewer 31 may be a head mounted display and / or the control input 36 may be mechanically independent of any infrastructure (e.g., wired, wireless, or gesture-based, such as Kinect from Microsoft).

[0047] The surgeon's console 16 is typically located in the same room as the patient so that the surgeon can directly monitor the surgical procedure, be physically present if necessary, and speak directly to the patient-side assistant rather than via a telephone or other communication medium. However, the surgeon may be located in a different room, an entirely different building, or other remote location away from the patient that allows for a telesurgery routine.

[0048] Figure 3 is a perspective view of the electronics cart 24. The electronics cart 24 can be coupled to the endoscope 28 and include a computer processor to process the captured images for subsequent display, such as display to the surgeon on the surgeon's console, or display on another suitable display located locally and / or remotely. For example, if a stereoscopic endoscope is used, the computer processor on the electronics cart 24 can process the captured images to present the surgeon with a coordinated stereoscopic image of the surgical site. Such coordination can include alignment between opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope. As another example, the image processing can include using previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. Optionally, the devices in the electronics cart can be integrated into the surgeon's console or patient side cart, or can be distributed in various other locations in the operating room.

[0049] Figure 4 The TSS 50 (such as Figure 1 The minimally invasive TSS 10). The surgeon's console 52 (such as Figure 1 The surgeon's console 16 in FIG. 1 may be used by the surgeon to control the patient side cart 54 (such as Figure 1The patient side cart 54 can capture images of the surgical site using an imaging device such as a stereo endoscope and output the captured images to an electronic cart 56 such as a Figure 1 The computer processor typically includes one or more data processing boards for executing computer readable code stored in a non-volatile memory device of the computer processor. In one aspect, the computer processor can process the captured image in various ways prior to any subsequent display. For example, the computer processor can overlay the captured image with a virtual control interface before displaying the combined image to the surgeon via the surgeon's console 52.

[0050] Additionally or alternatively, the captured images may be image processed by a computer processor located external to the electronics cart 56. In one aspect, the TSS 50 includes an optional computer processor 58 (shown in dashed lines) similar to the computer processor located on the electronics cart 56, and the patient side cart 54 outputs the captured images to the computer processor 58 for image processing prior to display on the surgeon's console 52. In another aspect, the captured images are first image processed by the computer processor on the electronics cart 56, and then additional image processing is performed by the computer processor 58 prior to display on the surgeon's console 52. As shown in dashed lines, the TSS 50 may include an optional display 60. The display 60 is coupled to the computer processor located on the electronics cart 56, and to the computer processor 58, and the captured images processed by these computer processors may be displayed on the display 60 in addition to being displayed on the display of the surgeon's console 52.

[0051] Figure 5A 1 is an illustrative simplified block diagram showing an arrangement of components of a teleoperated surgical system 10 according to some embodiments to perform surgical routines using one or more mechanical support arms 510. Aspects of the system 10 include robot-assisted and autonomous operation features. These mechanical support arms 510 typically support surgical instruments. For example, a mechanical surgical arm (e.g., a central mechanical surgical arm 510C) can be used to support an endoscope having a stereoscopic or three-dimensional surgical image capture device 101C. The mechanical surgical arm 510C can include a sterile adapter or clamp, clip, screw, slot / groove or other fastener mechanism to mechanically fix the endoscope including the image capture device 101C to the mechanical arm. In various other embodiments, the image capture device 101C (or any other surgical instrument) can be integrated into the mechanical surgical arm 510C.

[0052] A user or operator O (typically a surgeon) performs a surgical routine on a patient P by manipulating control input devices 36, such as hand grips and foot pedals at the main console 16. The operator can view video frames of images of the surgical site within the patient's body through a stereoscopic display viewer 31. The computer processor 58 of the console 16 directs the movement of the teleoperated controlled endoscopic surgical instruments 101A-101C via control lines 159, which in turn use the patient-side system 24 (also referred to as a patient-side cart) to achieve movement of the instruments.

[0053] The patient-side system 24 includes one or more mechanical support arms 510. Typically, the patient-side system 24 includes at least three mechanical surgical arms 510A-510C (commonly referred to as mechanical surgical support arms 510) supported by corresponding positioning assembly arms 156. The central mechanical surgical arm 510C can support or include an endoscopic camera 101C suitable for capturing images within the camera's field of view. The center left and right mechanical surgical support arms 510A and 510B can support or include instruments 101A and 101B, respectively, which can manipulate tissue.

[0054] Figure 5B 5 is a perspective view of a patient side cart 500 of a minimally invasive TSS 10 according to an embodiment. The patient side cart 500 includes one or more support arm assemblies 510. A surgical instrument manipulator 512 is mounted at the end of each support arm assembly 510. In addition, each support arm assembly 510 may optionally include one or more mounting joints (e.g., unpowered and / or lockable) for positioning the attached surgical instrument manipulator 512 with reference to a patient undergoing surgery. As depicted, the patient side cart 500 rests on the floor. In other embodiments, the operable portion of the patient side cart may be mounted to a wall, a ceiling, and may also be mounted to an operating table 526 that may support a patient's body 522, or to other operating room equipment. In addition, although a single patient side cart 500 is shown as including four surgical instrument manipulators 512, multiple patient side carts 500 may be provided and / or more or fewer surgical instrument manipulators 512 may be provided on the patient side cart 500.

[0055] A functional TSS will typically include a visual system portion that enables a user of the TSS to view the surgical site from outside the patient's body 522. The visual system typically includes a camera instrument 528 for capturing video images, and one or more video displays for displaying the captured video images. In some surgical system configurations, the camera instrument 528 includes an optical device that transmits the image from the distal end of the camera instrument 528 to one or more imaging sensors (e.g., CCD or CMOS sensors) outside the patient's body 522. Alternatively, the imaging sensor(s) may be positioned at the distal end of the camera instrument 528, and the signals generated by the sensor(s) may be transmitted along a wire or wirelessly for processing and display on one or more video displays. An example of a video display is a stereoscopic display on a surgeon's console in a surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.

[0056] Reference Figure 5A-5B , mounted to each surgical instrument manipulator 512 is a surgical instrument 520 that operates at a surgical site within a patient's body 522. Each surgical instrument manipulator 512 may be provided in a variety of forms that allow the associated surgical instrument to move in one or more mechanical degrees of freedom (e.g., all six Cartesian degrees of freedom, five or fewer Cartesian degrees of freedom, etc.). Typically, mechanical or control constraints restrict each manipulator 512 from moving its associated surgical instrument about a center of motion on the instrument that remains stationary relative to the patient, and which is typically located at the point where the instrument enters the body.

[0057] In one aspect, the surgical instrument 520 is controlled by computer-assisted teleoperation. The functional minimally invasive TSS includes a control input that receives input from a user of the TSS (e.g., a surgeon or other medical personnel). The control input communicates with one or more computer-controlled teleoperated actuators (such as one or more motors coupled to the surgical instrument 520). In this way, the surgical instrument 520 moves in response to the movement of the medical personnel at the control input. In one aspect, the one or more control inputs are included in a control panel such as a Figure 2 500. The surgeon can manipulate the control input device 36 of the surgeon's console 16 to operate the teleoperated actuator of the patient side cart 500. The force generated by the teleoperated actuator is transmitted via a drive train mechanism, which transmits the force from the teleoperated actuator to the surgical instrument 520.

[0058] Reference Figure 5A-5BIn one aspect, the surgical instrument 520 and the cannula 524 are removably coupled to the manipulator 512, wherein the surgical instrument 520 is inserted through the cannula 524. One or more teleoperated actuators of the manipulator 512 move the surgical instrument 512 as a whole. The manipulator 512 also includes an instrument bracket 530. The surgical instrument 520 is detachably connected to the instrument bracket 530. In one aspect, the instrument bracket 530 internally accommodates one or more teleoperated actuators, which provide multiple controller movements, and the surgical instrument 520 converts the multiple controller movements into various movements of the end effector on the surgical instrument 520. Therefore, the teleoperated actuators in the instrument bracket 530 only move one or more components of the surgical instrument 520 instead of moving the instrument as a whole. The input of the entire instrument or instrument assembly is controlled so that the input provided to the control input terminal by the surgeon or other medical personnel ("master" command) is converted into a corresponding action ("slave" response) by the surgical instrument.

[0059] According to some embodiments, the surgical system 10 can have multiple system actuation states, including park, follow, instrument type, head-in, camera movement, third arm control, ergonomic adjustment, table movement adjustment, etc. During the park system state, one or more manipulators 512 have been coupled to the sleeve 524. During subsequent system states, the surgical instruments ("slaves") track control inputs ("master" commands). During the instrument type system state, the system has installed therein a set of instruments suitable for performing a specific surgical routine or suitable for performing a specific surgical activity in a surgical routine. During the enter system state, the system is waiting for the surgeon to indicate that he / she has grasped the "master" control input device.

[0060] In an alternative embodiment, the instrument bracket 530 does not accommodate a remotely operated actuator. The remotely operated actuator that realizes various movements of the end effector of the surgical instrument 520 is accommodated in a position away from the instrument bracket 530, such as elsewhere on the patient side cart 500. A cable-based force transmission mechanism or the like is used to transmit the movement of each remotely located remotely operated actuator to the corresponding instrument interface actuator output located on the instrument bracket 530. In some embodiments, the surgical instrument 520 is mechanically coupled to a first actuator that controls a first movement of the surgical instrument, such as longitudinal (z-axis) rotation. The surgical instrument 520 is mechanically coupled to a second actuator that controls a second movement of the surgical instrument, such as two-dimensional (x, y) movement. The surgical instrument 520 is mechanically coupled to a third actuator that controls a third movement of the surgical instrument, such as opening and closing or a jaw end effector.

[0061] Figure 5C is an illustrative diagram showing a surgical scene 550, and also showing an endoscope 101C with a camera 528 mounted thereon for recording the scene, according to some embodiments. The scene 550 is disposed within a body cavity of a patient. The scene 550 includes an exemplary imaginary spherical anatomical structure 552 including a geometric outline 554. The scene 550 includes a surgical instrument 556. The camera 528 mounted on the endoscope 101C captures the scene, which is displayed within the viewer 31 and recorded for later playback.

[0062] Figure 6 6 is a side view of a surgical instrument 520, which includes a distal portion 650 and a proximal control mechanism 640 connected by a slender tube 610 having a slender tube centerline axis 611. The surgical instrument 520 is configured to be inserted into a patient's body and used to perform a surgical or diagnostic operation. The distal portion 650 of the surgical instrument 520 can provide any of a variety of end effectors 654, such as the forceps shown, a needle driver, a cauterization device, a cutting tool, an imaging device (e.g., an endoscope or an ultrasound probe), etc. The surgical end effector 654 may include functional mechanical degrees of freedom, such as a clamp that opens or closes or a knife that translates along a path. In the illustrated embodiment, the end effector 654 is connected to the slender tube 610 by a wrist 652, which allows the end effector to be oriented relative to the slender tube centerline axis 611. The surgical instrument 520 may also contain stored (e.g., on a semiconductor memory associated with the instrument) information that may be permanent or updateable by a surgical system configured to operate the surgical instrument 520. Thus, the surgical system may provide one-way or two-way communication of information between the surgical instrument 520 and one or more components of the surgical system.

[0063] Figure 7 5 is a perspective view of a surgical instrument manipulator 512. The instrument manipulator 512 is shown without a surgical instrument mounted thereon. The instrument manipulator 512 includes an instrument holder 530 to which a surgical instrument (e.g., surgical instrument 520) may be removably attached. The instrument holder 530 houses a plurality of remotely operated actuators. Each remotely operated actuator includes an actuator output 705. When a surgical instrument is mounted to the instrument manipulator 512, the instrument proximal end control mechanism (e.g., Figure 6One or more instrument inputs (not shown) of the proximal control mechanism 640 in the instrument manipulator 512 are mechanically coupled to corresponding actuator outputs 705. In one aspect, the mechanical coupling is direct, wherein the actuator output 705 directly contacts the corresponding instrument input. In another aspect, the mechanical coupling occurs through an intermediate interface, such as an intermediate interface of a component of a drape configured to provide a sterile barrier between the instrument manipulator 512 and the associated surgical instrument.

[0064] In one aspect, movement of one or more instrument inputs by corresponding teleoperated actuators results in movement of a surgical instrument's mechanical degrees of freedom. Figure 6 As shown, the surgical instrument mounted on the instrument manipulator 512 is surgical instrument 520. Figure 6 In one aspect, movement of one or more instrument inputs of the proximal control mechanism 640 by corresponding teleoperated actuators causes the elongated tube 610 (and the attached wrist 652 and end effector 654) to rotate relative to the proximal control mechanism 640 about the elongated tube centerline axis 611. In another aspect, movement of one or more instrument inputs by corresponding teleoperated actuators causes movement of the wrist 652, which in turn orients the end effector 654 relative to the elongated tube centerline axis 611. In another aspect, movement of one or more instrument inputs by corresponding teleoperated actuators causes movement of one or more movable elements of the end effector 654 (e.g., a jaw member, a knife member, etc.). Thus, various mechanical degrees of freedom of surgical instruments mounted to the instrument manipulator 512 may be moved by operation of the teleoperated actuators of the instrument bracket 530.

[0065] Surgical planning system

[0066] Figure 8 800. In one aspect, the surgical planning tool 800 includes a TSS 850 in data communication with an electronic medical device record database 830. The TSS 850 shown here is similar to Figure 4 TSS 50 shown. In one aspect, electronic medical records database 830 includes medical records of patients who have been treated at a particular hospital or at multiple hospitals. Database 830 can be implemented on a server located on-site at the hospital. The medical record entries contained in database 830 can be accessed from hospital computers via an intranet. Alternatively, database 830 can be implemented on a remote server located outside the hospital, such as using one of multiple cloud data storage services. In this case, the medical record entries of database 830 are stored on the cloud server and can be accessed by a computer system with Internet access.

[0067] In one aspect, a surgical procedure is performed on a first patient using TSS 850. An imaging device associated with TSS 850 captures images of the surgical site and displays the captured images as frames of a video on a display of the surgeon's console 52. In one aspect, medical personnel at the surgeon's console 52 use an input device of the surgeon's console 52 to highlight or annotate certain patient anatomical structures shown in the displayed video. Examples of such input devices are Figure 2 The left handle control input 36 and the right handle control input 38 are shown, and the input device is coupled to a cursor that operates in conjunction with a graphical user interface overlaid on the displayed video. The graphical user interface may include a QWERTY keyboard, a pointing device such as a mouse and an interactive screen display, a touch screen display, or other means for data or text input or voice annotations / or voice-to-text conversion through a microphone and processor. Thus, medical personnel can highlight certain tissues of interest in the displayed image or enter text annotations.

[0068] In one aspect, the surgical site video is additionally displayed on a display located on the electronic cart 56. In one aspect, the display of the electronic cart is a touch screen user interface that can be used by medical personnel to highlight and annotate certain portions of the patient's anatomy shown on the image displayed for viewing on the display on the electronic cart. The user can highlight portions of the displayed image by touching portions of the patient's anatomy displayed on the touch screen user interface. In addition, a graphical interface including a keyboard can be superimposed on the displayed image. The user can enter text annotations using a keyboard or other input device (such as a microphone and a speech-to-text conversion driver).

[0069] In one aspect, in addition to being displayed to a user in real time or near real time, a video of a surgical site captured by an imaging device associated with the TSS 850 is recorded by the TSS 850 and stored in a database 830. Highlights and / or annotations made by a user associated with the recorded video may also be stored on the database 830. In one aspect, the highlights made by the user are embedded in the recorded video before they are stored on the database 830. At a later time, the recorded video may be retrieved for viewing. In one aspect, a person viewing the recorded video may select whether the highlights are displayed or suppressed from view. Similarly, annotations associated with the recorded video may also be stored on the database 830. In one aspect, the annotations made by the user are used to mark the recorded video and may be used to provide a method of identifying the subject matter contained in the recorded video. For example, a comment may describe the condition of a certain disease state. This comment is used to mark the recorded video. A person who later wishes to view the recorded operation regarding the disease state may use a keyword search to locate the video.

[0070] Retrieval of stored videos

[0071] In some cases, it is desirable for medical personnel to be able to watch video records of past surgical routines performed on a given patient. In one aspect, a patient who previously received a first surgical routine to treat a medical condition subsequently requires a second surgical routine to treat a recurrence of the same medical condition or to treat an anatomical structure located near the surgical site of the first surgical routine. In one aspect, the surgical site events of the first surgical routine are captured in the surgical site video record, and the video record is archived in the database 830 as part of the patient's electronic medical record. In a related embodiment, the first surgical routine can be indexed according to the type of operation. Before performing the second surgical routine on the patient, medical personnel can perform a search of the database 830 to locate the video record of the patient's earlier surgical routine.

[0072] In some cases, for medical personnel who plan to perform a surgical routine on a patient, it is desirable to be able to view video records of similar surgical routines performed on people with certain characteristics similar to the patient. In one aspect, the surgical site video records of the surgical routine can be tagged with metadata information before each video record is archived in the database 830, such as the patient's age, gender, body mass index, genetic information, the type of surgical routine performed by the patient, etc. In one aspect, the metadata information used to tag the video record is automatically retrieved from the patient's then-existing medical record, and then used to tag the video record before the video record is archived in the database 830. Therefore, before performing a medical operation on the patient, the medical staff can search the database 830 to obtain video records of similar operations performed by patients who share certain common characteristics with the patient. For example, if the medical staff plans to perform a prostatectomy on a 65-year-old male patient with an elevated body mass index using TSS 850, the medical staff can search the database 830 to obtain surgical site video records of prostatectomies performed on other men of similar age and with a similar elevated body mass index using TSS 850.

[0073] In one aspect, the video recording of the surgical routine is transmitted via the database 830 to an optional personal computing device 820 (shown as a dashed line) such as a personal computer, tablet, smart phone, terminal or other electronic access device and is available for viewing by the medical staff planning to perform the surgical routine. Additionally or alternatively, the video recording of an earlier surgical routine can be transmitted to the TSS 850 via the database 830 and can be used for preoperative or intraoperative observation. In one aspect, the video recording is displayed by the TSS 850 on a display located on the surgeon's console 52. In another aspect, the video recording of the first surgical routine is displayed on a display located on the electronics cart 56.

[0074] In a related embodiment, as described in more detail below, the video recording of an earlier routine can be synchronized with the current surgical routine stage, or stage queued, which helps to immediately locate relevant surgical tasks or techniques, such as those related surgical tasks or techniques for the upcoming stage of the current surgical routine. Queuing the video playback to the current stage of the surgical routine allows the surgeon to scroll the playback very effectively to view the upcoming or recent steps. It is worth noting that stage synchronization or stage queuing is not strictly time synchronization, but there is a rough time ordering similarity between the previous surgical routine and the current surgical routine. However, as described above, stage synchronization is aligned according to the surgical process flow. The matching of direct visual features matched by video can be enhanced by one or more of (a) metadata, (b) stage synchronization and (c) time synchronization.

[0075] Alternatively, the video sequence from the current surgical procedure can be matched with video sequences of representative surgical procedures performed by a skilled surgeon based on visual similarity (e.g., color, time, edges, etc.), or based on content similarity (e.g., organs, anatomical structures, and tools) if the content can be identified using computer vision methods.

[0076] Cloud-based video database

[0077] In one aspect, database 830 is implemented on a remote server using a cloud data storage service and is accessible to multiple health care providers. Figure 8, as shown in dashed lines, surgical planning tool 800 optionally includes TSS 850 (shown in dashed lines) and personal computing device 840 (shown in dashed lines). In one aspect, TSS 850 is similar to TSS 850, and personal computing device 840 is similar to personal computing device 820, except that TSS 850 and personal computing device 820 are located at a first health care provider and TSS 850 and personal computing device 840 are located at a second location or even with a second health care provider. In one aspect, a first patient requires surgical treatment for a medical condition and a surgical routine is performed at the first health care provider using TSS 850. A video recording of the surgical routine is archived on database 830. At a later time, a second patient requires surgical treatment for the same medical condition and is scheduled to receive surgical treatment at the second health care provider using TSS 850. Before performing the surgical routine on the second patient, medical personnel access database 830 via a secure Internet connection and search database 830 for surgical site video recordings of similar routines. In one aspect, the medical staff treating the second patient is able to retrieve the video recording of the surgical procedure of the first patient from the database 830 without acquiring knowledge of the identity of the first patient. In this way, the privacy of the first patient is protected. In one aspect, the video recording of the surgical procedure of the first patient includes highlights and / or annotations made by the medical staff treating the first patient.

[0078] Computer-based pattern matching and analysis

[0079] The surgical planning tool 800 may include a pattern matching and analysis algorithm implemented in the form of a computer executable code. In one aspect, the pattern matching and analysis algorithm is stored in a non-volatile memory device of the surgical planning tool 800 and is configured to analyze video records archived in a database 830. As previously described, each video record archived in the database 830 may be tagged and / or embedded with certain metadata information. The metadata information may include patient information, such as patient age, gender, and other information describing the patient's health or medical history. In addition, as previously discussed, the metadata information may include highlights or annotations made by medical personnel. In one aspect, these highlights and annotations are embedded with the video record and archived in the database 830 together with the video.

[0080] In one aspect, the pattern matching and analysis algorithm includes an image analysis component that recognizes patterns shared between multiple video records stored on the database 830 by shape and color. The pattern matching and analysis algorithm then reviews the tagged metadata associated with a subset of the video records to determine if there are any words or phrases frequently associated with the videos within the subset. For example, metadata can be tagged by annotating text, converting audio to text. These analyses performed by the pattern matching and analysis algorithm can be used to help medical personnel determine patient anatomy, preferred surgical methods, disease states, potential complications, etc.

[0081] Virtual surgical environment, data recording and surgeon evaluator

[0082] TSS can record video of the surgical routine being performed as well as the surgeon's input through the console. In general, it is expected that this collected data will be used to train and assist the surgeon. However, system designers face many challenges in managing the large amount of acquired data, extracting relevant portions of the data, and providing useful and non-obtrusive assistance that surgeons welcome.

[0083] In various embodiments described in detail below, the TSS is augmented or supported with additional specialized machinery to provide assessment and assistance to medical personnel operating the TSS. Fig. 9 is a diagram showing an advanced surgical system 900 featuring a TSS 950 similar to the above combined Figure 4 The described TSS 50, in addition to the TSS 950 including a suitable interface to the additional support system, the additional support system includes a virtual surgical environment (VSE) 904, a data recording system 906, a surgeon evaluator 908 and an auxiliary system 910. These additional support systems can be implemented in various machine configurations. For example, one or more of the additional support systems can be set as a dedicated unit, or set as part of the computing platform through program instructions. The computing platform can be a physical machine, or can be distributed between multiple physical machines, such as by role or function distribution, or by processing thread distribution in the case of a cloud computing distributed model. In some embodiments, the computing platform can be implemented as part of the TSS 950 (for example, in the surgeon's console 52, electronic cart 58, etc.), or the computing platform can be implemented using hardware different from the TSS (for example, such as a tablet, a personal computer or laptop, a smart phone or a mobile computing device of some combination of the above). The computing platform can be implemented as a combination of local processing and "cloud" on a system or personal hardware.

[0084] In various embodiments, certain operations may be run in a virtual machine, which in turn is executed on one or more physical machines. It will be appreciated by those skilled in the art that the features of the embodiments may be implemented through a variety of different suitable machine implementations.

[0085] In various embodiments, these components are implemented as engines, circuits, components or modules, which are referred to as engines for consistency, but it should be understood that these terms can be used interchangeably. The engine can be hardware, software or firmware that is communicatively coupled to one or more processors to perform the operations described herein. The engine can be a hardware engine, and because such an engine can be considered as a tangible entity that can perform a specified operation and can be configured or arranged in some way. In one example, the circuit can be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuits) as an engine. In one example, all or part of one or more hardware processors can be configured by firmware or software (e.g., instructions, application parts or applications) to operate as an engine for performing a specified operation. In one example, the software can reside on a machine-readable medium. In one example, the software causes the hardware to perform a specified operation when executed by the underlying hardware of the engine. Therefore, the term hardware engine is understood to include a tangible entity, that is, an entity that is physically constructed, specially configured (e.g., hardwired) or temporarily (e.g., temporarily) configured (e.g., programmed) to operate or perform part or all of any operation described herein in a specified manner.

[0086] Consider an example of a temporary configuration of engines, where each engine need not be instantiated at any time. For example, where the engine comprises a general purpose hardware processor core configured using software; the general purpose hardware processor core can be configured as correspondingly different engines at different times. The software can configure the hardware processor core accordingly, such as to constitute a particular engine at one point in time and to constitute a different engine at a different point in time.

[0087] The TSS 950 interfaces with the surgical environment 902, facilitating the patient and medical staff in the surgical environment 902. Generally, the control inputs of the TSS 950 are fed to electromechanical systems that cause real-world effects in the surgical environment 902. On the other hand, according to some embodiments, the VSE 904 computationally models one or more portions of the actual surgical environment to create a virtual surgical environment. Thus, the same control inputs of the TSS 950 can be fed to the VSE 904, where the same control inputs produce computationally modeled effects in the virtual surgical environment.

[0088] In one type of embodiment, the virtual surgical environment includes a model of at least a portion of the patient, including physiological structures, fluids, disease states, hemodynamics, etc., in addition to the effects of simulated electromechanical outputs representing the outputs of the TSS 950 on the patient model. In another type of embodiment, the patient is not modeled; instead, the electromechanical outputs of the TSS 950 are represented. In another type of embodiment, the control signals to be fed to the electromechanical system of the TSS 950 are modeled by the VSE 904.

[0089] In a related embodiment, the virtual surgical environment 904 is programmed or otherwise configured to determine the phases or segments of the surgical routine based on an evaluation of surgical input received from the TSS 950 .

[0090] In another related embodiment, the VSE 904 is programmed or otherwise configured to determine the phase or segment of the surgical routine based on the VSE simulated effects of the surgical inputs on the TSS, including simulated effects in the patient (e.g., effects in tissue at the surgical site, hemodynamics, etc.), or other simulated features of the patient's interaction with the surgical instrument, rather than the surgical inputs themselves. In another related embodiment, a combination of surgical inputs and their simulated effects are used to evaluate the segment of the surgical routine.

[0091] Fig.10 904 and their interactions according to one embodiment. The surgical input evaluator 1002 is programmed or otherwise configured to collect event data such as control inputs, kinematic information, sensor data, system events or status indications from the TSS 950 and interpret their meaning in terms of surgical effects. The segmenter 1004 is programmed or otherwise configured to distinguish the stages of the surgical routine during the execution of the surgical routine, or in a post-processing mode. Based on the control input from the TSS 950, the simulator 1006 is programmed or otherwise configured to model the surgical routine in a virtual surgical environment. The simulator 1006 can operate in parallel with the actual surgical routine performed in the surgical environment 902, or replace the actual surgical routine, and can be used by the surgeon to practice certain operations or portions thereof. In some embodiments, the simulator 1006 can be called during a specific portion of the routine to allow the surgeon to practice a certain task before actually performing the task on the patient. More generally, simulator 1006 may be used for training and practice of surgeons, students, or other medical personnel.

[0092] Fig.11is a diagram illustrating the operational components of the surgical input evaluator 1002 and their interactions according to one embodiment. The event monitor 1102 receives event data from the TSS 950 and records event sequences, which event data may include system events, kinematics, and external inputs. In a related embodiment, the event monitor 1102 receives information about additional actions that occur in the surgical environment 902 other than with or by the TSS 950 (such as actions taken by one or more assistants or other devices / instruments associated with the operation). For example, these actions may be detected by one or more sensors of the TSS 950, or detected by environmental monitoring sensors such as video capture devices, or detected by those other devices / instruments themselves and transmitted to the TSS 950. As described above, events may also include any activation of instrument controls, such as, for example, button presses, instrument movement controls, instrument selections, video system controls, wrist movements, camera movement activations, master gestures / activations, and other inputs provided by the surgeon, assistants, or any other detectable actions / inputs / outputs. For example, other data that may be collected as events include kinematic data, eye movements, user authentication (e.g., iris recognition), muscle activity (e.g., measured using electromyography), sensed surgeon posture, images captured by the endoscope, system status (e.g., docked / deployed status of various instruments), engaged / disengaged status of arms with master controllers, types of instruments installed, number of instruments installed, operator's head engagement status, forces applied to controls such as master controls, forces detected by robotic instruments, camera movement and control, touch screen inputs, personnel position (e.g., provided by sensors in the surgical environment), and a history of any of these. The sequence of events may be recorded as time series data and may be unified and may represent coordinated actions of one or more operators of the TSS 950 and any personnel in the surgical environment 902, and / or any other data compiled from the surgical environment 902.

[0093] In some embodiments, gesture evaluator 1104 reads time series event data, and applies gesture criteria 1106 to determine whether the combination of events constitutes any defined gesture. In this context, a gesture is a sequence of two or more events that constitute the composite motion of a tool and a control. As an illustrative example, a single suture motion associated with a suturing task can be considered as a gesture. In this example, there is usually a sequence of separate motions, and control input data can be collected for the sequence of separate motions and the sequence of separate motions can be classified to identify gestures. In one embodiment, gesture criteria 1106 includes a library of event sequences that constitute various gestures. In a more basic environment, gesture criteria 1106 can simply define the time window of the most recent event (e.g., the most recent 40 seconds) so that for each time increment, the time window of the event sequence is considered to be a gesture. The time window or gesture in this example can overlap with other gestures.

[0094] instead, the monitored events are fed to the task evaluator 1108. In another type of embodiment where the gesture evaluator 1104 is available, a combination of detected gestures and monitored events are fed to the task evaluator 1108 in parallel.

[0095] Task evaluator 1108 is programmed or otherwise configured to evaluate the current surgical task being performed. In this context, a task is a series of events, gestures, or a combination thereof, which together produce a defined surgical effect. The task can be a clinically common operation (e.g., incision task, suturing task, etc.) or clinically relevant (one or more) steps / fragments of a routine (e.g., ultraviolet anastomosis of prostatectomy). Task evaluation is based on a series of recent events, and further based on data from a task standard database 1110, which associates certain specific series of events with tasks. In a related embodiment, the task standard database 1110 can also include a library of surgical routines in which those tasks are generally performed. In this type of embodiment, the identification of a surgical routine itself can be the input of a task evaluation algorithm. The surgical routine in this context is a high-level descriptor, such as prostatectomy, umbilical hernia repair, mastectomy, etc.

[0096] In various embodiments, the gesture evaluator 1104 and the task evaluator 1108 can respectively perform one or more classification operations, such as K-nearest neighbor (KNN) classification (e.g., clustering operations such as K-means clustering), association rule mining operations, support vector machine (SVM) operations, artificial neural network (ANN) operations (e.g., recursive neural networks, convolutional neural networks, etc.), etc. based on corresponding criteria 1106, 1110 used as training set data where applicable.

[0097] Events, gestures, and tasks may each have different parameter values, such as joint angles, speeds, forward / following idle, etc. It should be understood that during a teleoperated surgical routine, control inputs may occur that result in changes in the actuation state of the surgical system. For example, a surgeon may move his or her head in and out of a viewer, resulting in a change in the enter state. For example, a surgeon may move his or her hands or feet in or out of contact with a control input, resulting in a change in the follow state. For example, the combination of instruments in use may change, resulting in a change in the instrument type state.

[0098] Operation of the TSS 950 in support of surgical activities in one or more surgical states additionally results in the generation of control input information within the surgical system that indicates instrument motion that indicates the actual manner in which the surgeon performs the surgical activity. For example, the surgeon may have used control inputs to move instruments quickly or slowly. For example, the surgeon may have moved surgical instruments along one or another path in a direction toward or away from an anatomical structure. For example, the surgeon may have used control inputs to adjust the position of different instruments before activating a particular instrument.

[0099] Figures 19A-19C 1 is an illustrative diagram showing an example surgical instrument 1902 and an actuator assembly 1903 according to some embodiments, in which the surgical instrument is shown in three different example surgical instrument actuation states. The example instrument 1902 includes a jaw end effector 1904 that can be continuously converted between an open state and a closed state, and a partially open / partially closed state therebetween. The example instrument 1902 also includes a two-degree-of-freedom (2-dof) wrist 1906 that can move between different two-dimensional (x, y) position states. The example actuator assembly 1903 includes a first actuator 1908, which in some embodiments includes a jaw motor (JM) for actuating the jaw end effector 1904. The example actuator assembly 1903 includes a second actuator 1910, which in some embodiments includes a wrist motor (WM) for actuating the wrist 1906. During a surgical procedure, the surgical instrument 1902 may transition through a plurality of instrument actuation states corresponding to different activities during a surgical routine. Each transition results in the generation of control input information that is captured and stored by the event monitor 1102 and indicates the movement of the instrument when the instrument is commanded to transition from its physical orientation and configuration in one state (e.g., open or closed) to the physical orientation and configuration of its next state.

[0100] like Fig.19AAs shown, for example, the first gesture may include the first actuator 1908 (JM) setting the jaw end effector 1904 to a fully open state, and the second actuator 1910 (WM) setting the wrist 1906 to a first position state (x1, y1). Fig.19B As shown, for example, the surgical routine may include a second gesture in which the first actuator 1908 transitions the jaw end effector 1904 to a fully open state, and the second actuator 1910 transitions the wrist 1906 to a second position state (x2, y2). Fig.19C As shown, for example, the surgical routine may include a third surgical activity in which the first actuator 1908 sets the jaw end effector 1904 to a partially open / partially closed state and the second actuator 1910 transitions the wrist 1906 to a third position state (x3, y3).

[0101] According to one embodiment, in response to control input information (e.g., events, gestures, or a combination thereof), the surgical input evaluator 1002 cooperates with the simulator 1006 to model position states. The virtualized movement and position states can in turn be evaluated by the task evaluator 1108 to identify applicable tasks. In this type of embodiment, instead of or in addition to the processed control input data, the modeled motion information can be used to perform task evaluation.

[0102] Now go to Fig.12 , the segmenter 1004 is described in detail, including its operating components and their interactions, according to an illustrative embodiment. In the example shown, the segmenter 1004 is configured to selectively apply one of two types of algorithms for segment evaluation, namely, a real-time segment evaluator 1200 and a post-processing based segment evaluator 1220.

[0103] The real-time segmentation evaluator 1200 can be applied in surgery to perform segmentation evaluation with negligible delay within the range that the surgeon or medical staff can perceive. In an exemplary embodiment, the real-time segmentation evaluator 1200 is configured as a recurrent neural network (RNN) that uses a long short-term memory (LSTM) RNN architecture for deep learning. As depicted, the feature extractor 1202 applies filtering or other selection criteria to the evaluated task sequence and its corresponding parameters to create a feature vector as an input to the RNN 1204. Examples of feature vectors may include posture (e.g., position and rotation matrix information) and joint angles and speeds of the actuator. RNN 1204 can be a bidirectional RNN, in which there are actually two RNNs, one looking forward and the other looking backward. Therefore, the predicted future values ​​and past values ​​are considered in the RNN algorithm. In an exemplary embodiment, RNN 1204 has two layers, namely a bidirectional LSTM and a gated recurrent unit (GRU) layer, with hidden nodes in each layer.

[0104] As will be appreciated by those skilled in the relevant art, training data 1206 includes classification feature vectors for training RNN 1204 using a suitable training algorithm. The training data can be based on surgical inputs (e.g., including control input parameters from surgeon operating controls or events where an assistant can modify instrument configurations, events occurring in a surgical environment, etc.). In a related embodiment, the training data can be based on simulated effects of surgical inputs. In the latter type of embodiment, simulator 1006 can be the source of training data.

[0105] The post-processing segmentation evaluator can be applied at any point after surgery to perform segmentation evaluation using all data recorded from the surgery. In the example of the post-processing based segmentation evaluator 1220, clustering is performed using a hierarchical alignment clustering algorithm (HACA) according to the depicted embodiment. According to the depicted embodiment, the example segmentation evaluator 1220 architecture includes a feature extractor 1222, a preprocessor 1224, a frame kernel matrix 1226, and a clustering engine 1228. The feature extractor 1222 produces a feature vector having a specific dimension and representing a specific set of parameters. According to one embodiment, the preprocessor 1224 can use a K-means clustering algorithm.

[0106] Frame kernel matrix operation 1226 calculates the self-similarity (or Gram) matrix associated with the motion segment Each of these items, ij Define two samples x through the kernel function i and x j The similarity between:

[0107]

[0108] The HACA clustering engine 1228 executes a HACA algorithm consistent with "Hierarchical aligned cluster analysis for temporal clustering of human motion" by Zhou, Feng, Fernando de la Torre, and Jessica K. Hawkins as described in IEEE Transactions on Pattern Analysis and Machine Intelligence (2013), Vol. 35, No. 3, pp. 582-596.

[0109] In a related embodiment, the post-processing based segment evaluator 1220 is used post-operatively to update the training data 1206 and re-evaluate the segmentation determinations of the real-time segment evaluator 1200 .

[0110] The confidence measurement engine 1230 is programmed or otherwise configured to calculate a confidence score representing the probability of a correct segmentation. The confidence score can be calculated using cross entropy and log probability calculations. As will be described in more detail below, the calculated confidence can be used as a metric for evaluating the skill of a surgeon, or as a weighting for other metrics (such as time) to further improve the accuracy of the assessment.

[0111] Fig.13 1 is a diagram illustrating the operating components of the simulator 1006 and their interactions according to one embodiment. The patient model 1302 is programmed or otherwise configured to represent the patient based on measured or assessed physical characteristics 1304 of the patient and based on measured or diagnosed disease conditions 1306. Examples of relevant physical characteristics 1304 include height, weight, gender, age, body mass index (BMI), muscle tone, size and position of organs, etc., as well as any imaged anatomical structures, such as those obtained in a CT scan, MRI, ultrasound or other test. Similarly, disease conditions 1306 may include features such as tumors, cysts, hernias, torn connective tissue, fractures, etc. The TSS modeler 1310 is programmed or otherwise configured to represent the TSS 950, including tools, end effectors and other configuration parts, as represented by configuration information 1312. The TSS modeler 1310 receives control input information from the TSS 950 so that the positioning and movement of various surgical tools can be reflected in the virtual surgical environment.

[0112] The TSS modeler 1310 and the patient model 1302 can exchange state information during the simulation in which the virtual surgical routine is simulated. Thus, as changes are implemented through the surgical routine, one or both of these engines can maintain and update the patient's current state.

[0113] In one embodiment, simulator 1006 operates with surgical environment 902 to track the progress of TSS 950 and the patient in a surgical routine. In another embodiment, a given segment of a surgical routine can be initiated for simulation at the surgeon's request or in response to a trigger. Such intraoperative simulation can be requested if the surgeon encounters a critical or difficult moment in surgery and wishes to practice certain movements or various techniques in order to actually deploy these actions on a patient. In a related embodiment, simulations running in parallel can be autonomously analyzed to determine the current surgical stage by using segmenter 1004, which can be further trained with model-based contextual data in a set of feature vectors.

[0114] In another embodiment, simulator 1006 can be used for surgical training or practice in place of an actual patient. In some implementations, the user interface of TSS 950 for surgeons and other medical personnel can be used to realistically simulate the surgical environment and provide a machine interface experience as part of training or practice exercises.

[0115] Surgeon Evaluator

[0116] Fig.14 is a diagram illustrating in greater detail the operating components of the surgeon evaluator 908 and their interactions according to one embodiment. At the heart of the surgeon evaluator 908 is the surgical technique evaluator 1402, which is programmed or otherwise configured to analyze control input signals and is evaluated by the gesture evaluator 1104 (where applicable), the task evaluator 1108, and the segmenter 1004 to produce an evaluation or score representing the quality of the surgical performance of the surgeon operating the control inputs of the TSS 950, whether they are used in a surgical routine or during a simulation for practice or training.

[0117] The inputs to the surgical technique evaluator include video information V, event information E, gestures G, and assessed tasks T, and kinematic data K. This data may be provided by the TSS 950 or VSE 904 (e.g., from the surgical input evaluator 1002 and simulator 1006). In a related embodiment, the inputs may include segment confidence scores SC, as determined by the confidence measurement engine 1230 of the segmenter 1004 based on the segment determination of the RNN 1204 or the clustering engine 1228.

[0118] Segments correspond to surgical routine portions and are defined according to a combination of one or more features representing a combination of one or more of video, events, gestures, tasks, and kinematics. Segments include a combination of events / kinematics / video that can be represented as gestures / tasks or can be represented directly without an intermediate gesture / task representation.

[0119] In one embodiment, the surgical technique evaluator 1402 calculates various statistics for each of the various temporal and spatial metrics. For example, the duration of various surgical routine segments can be evaluated. Similarly, the duration of the task level can be calculated and summarized in some way (e.g., per segment). Motion economy analysis can also be performed, which examines the number of gestures or other movements associated with the completion of certain tasks or surgical segments, the number of events, or some combination of these quantities. Idle time (i.e., no movement) can also be considered. In the surgical technique evaluation, other metrics such as motion speed, input device wrist angle, idle time (e.g., stability), main workspace range, main clutch count, applied energy and other such parameters can also be considered. In the surgical technique evaluation, camera control metrics such as camera movement frequency, camera movement duration, and camera movement interval can also be considered. In some embodiments, camera clutch movements (CMFrq) are defined as the average number of endoscope movements made by the surgeon per time increment (e.g., per second) throughout the event or segment; camera movement duration (CMDur) is defined as the average time increment (e.g., in seconds) of all endoscope movements throughout the event or segment; and camera clutch interval (CMInt) is defined as the average time increment (e.g., in seconds) between endoscope movements throughout the event or segment. As non-limiting examples, the duration of camera use, the consistency of camera use, the duration of energy application (e.g., cutting tissue, cauterizing tissue, increasing applied energy), the frequency of use of the master clutch, the time control of different arms (e.g., the duration between the use of arm 1 and arm 3), the frequency of master alignment with end effector orientation, etc.

[0120] Evaluation criteria 1406 define one or more criteria for calculating the performance evaluation of the surgeon. In one type of embodiment, the execution of a surgical routine is given a series of scores corresponding to different identification stages of the surgical routine. Evaluation criteria 1406 may include filtering, masking and weighting information applied to various metrics to adjust their relative importance. It is worth noting that different weights / adjustments can be applied to metrics of different surgical routines or different stages of a given surgical routine. Moreover, evaluation criteria 1406 can define algorithms or formulas used in presenting surgical performance evaluations. In embodiments where the surgical technique evaluator 1402 uses a machine learning algorithm (such as an ANN, a classifier, a clustering agent, a support vector machine, etc.), evaluation criteria 1406 may include criteria for constructing a feature vector of a metric.

[0121] Benchmark skills database 1404 includes scores and metrics obtained by surgeons or other medical personnel who are considered experts in corresponding surgical routines.

[0122] For example, in some embodiments, the benchmark database 1404 includes a benchmark for the surgeon's skill level based on camera control time. The inventor has conducted a study in which the camera control time is measured for each of a plurality of exercises representing different surgical tasks (e.g., camera aiming, point and needle, energy decomposition, matching plate, needle aiming, nail plate, pick and place, ring and rail, loop, scaling, suturing, sponge, thread ring and tube) to rate the surgeon as experienced, intermediate or novice in using a teleoperated surgical system (TSS). The results of the exercise are used to create a benchmark for the surgeon's skill level based on camera control time. The study found that experienced surgeons usually perform camera control significantly faster than new surgeons. Intermediate surgeons usually perform exercises not significantly faster than new surgeons. However, the study found that there was no significant difference in the exercise completion time between intermediate and experienced surgeons.

[0123] The study revealed some surprising findings. For example, it was found that experienced surgeons had significantly higher CMFrq than new surgeons for most practices, intermediate surgeons had significantly higher CMFrq than new surgeons for most practices, and there was no significant difference in CMFrq between intermediate and experienced surgeons. The study also found that experienced surgeons had significantly shorter CMDur than new surgeons for most practices. In most practices, intermediate surgeons had significantly shorter CMDur than new surgeons. In most practices, experienced surgeons had significantly shorter CMDur than intermediate surgeons. In most practices, experienced surgeons had significantly shorter CMInt than new surgeons, while intermediate surgeons had significantly shorter CMInt than new surgeons in most practices. There was no significant difference in CMInt between intermediate and experienced surgeons.

[0124] In a related embodiment, a feature vector constructed for each task or each segment of a given surgical routine (which may also be done in real time) is compared to one or more benchmark scores corresponding to the tasks or segments of the surgical routine. The comparison may be a simple Cartesian distance measure between the feature vector in question and the benchmark feature vector, or some other similarity measure. The determined similarity may be returned as a surgical performance score corresponding to the task, segment, etc.

[0125] The various scores calculated for a given surgical routine for each surgeon are stored in the surgeon database 1408. The scores over time may be analyzed to detect any trends, such as the rate of improvement of surgeons as they gain experience and expertise in using the TSS 950 and performing certain routines. According to some embodiments, the surgeon database 1408 distinguishes performance scores between simulated-only operations of the TSS 950 at 1412 and those operations corresponding to routines performed on real patients at 1414.

[0126] Data Recording

[0127] Fig.15 15 is a diagram illustrating the operational components of a data logging system 906 and their interactions according to an exemplary embodiment. An indexer 1502 is programmed or otherwise configured to associate input data with other related data to facilitate data retrieval. In the depicted example, various data may be recorded during a surgical routine. For example, captured video and any added annotations 1504, an input log 1506 of an assessment, a task log 1508 of an assessment, a simulation log 1510, and a run surgical technique assessment log 1512. Each data item may also be associated with the date and time it was captured, the patient, surgeon, TSS, surgical procedure type, and other related data.

[0128] In one embodiment, for captured video and annotation data 1504, indexer 1502 may divide the video into segments corresponding to evaluation segments of surgical routines. Evaluation input log 1506 stores time series events and other input information and evaluated gestures. Evaluation task log 1508 contains evaluations of tasks and surgical routine segments performed in real time. In one embodiment, evaluation input log 1506 includes a short-term storage portion and a long-term storage portion, wherein the short-term storage portion stores all input information including events; and the long-term storage portion stores only those inputs for which task determination decisions have been made.

[0129] The simulation log 1510 contains time series state information describing the patient model 1302 and the TSS modeler 1310 of the simulator 1006. In one embodiment, the sampling rate of the simulation log is similar to the rate at which the segment evaluation is performed (e.g., 4-20 samples per second). Sufficient data is recorded to allow detailed inspection and analysis of the surgical routine performed.

[0130] Running surgical technique evaluation log 1512 stores a time series sample of surgeon performance scores that are continuously updated by surgical technique evaluator 1402. The data recorded in running surgical technique evaluation log 1512 may be specific to the current surgical routine (or simulation) being performed on the current patient. This data set may be associated with surgeon database 1408 ( Fig.14 ) in a surgeon database 1408 that stores surgeon-specific data across multiple different patients, routines, and dates.

[0131] Auxiliary Engine

[0132] Fig.16 is a diagram illustrating the operational components of the assistance engine 910 and their interactions according to some embodiments. As depicted, the assistance engine 910 includes an assistance call detector 1600 that is programmed or otherwise configured to detect a call for surgeon assistance. In some embodiments, the assistance call detector 1600, the routine performance sensor 1602, and the TSS user interface (UI) interface 1604, either of which can initiate assistance operations of the assistance engine 910 in response to their respective inputs. The routine performance sensor 1602 monitors the operation of the VSE 904 and the surgeon evaluator 908. For example, the output of the segmenter 1004, which overlaps with the surgical technique assessment performed (e.g., as recorded in the database 1512), provides information about the stage of the surgical routine, as well as the measured performance of the surgeon during that stage.

[0133] Assistance criteria 1603 include various threshold conditions that, when met, trigger the operation of an assistance action by one or more assistance rendering engines described below. Notably, different thresholds may be associated with different types of assistance. In operation, routine performance sensor 1602 compares a currently running surgical skill assessment for a current surgical routine and its segments to threshold conditions corresponding to the surgical routine and segment. If the threshold condition comparison is met - that is, if the currently running surgical skill assessment score is below any given threshold, then the corresponding assistance may be provided and / or given.

[0134] In a related embodiment, the threshold conditions of the assistance criteria 1603 may also be surgeon-specific or patient-specific. For example, a relatively more experienced surgeon may configure the system with a higher threshold before providing assistance than a novice surgeon. Similarly, a lower threshold may be selected when performing surgery on patients in more dangerous conditions (e.g., severely traumatized patients, elderly patients, newborns, or early-stage patients, etc.).

[0135] Additionally, for example, assistance criteria 1603 may be used to selectively request input from specific expert surgeons with expertise in specific steps of the routine. For example, certain surgeons may be very skilled in complex anatomy, so if someone is having trouble with that step, then they should be the one “on call” whereby other surgeons may be needed for simpler steps.

[0136] The TSS UI interface 1604 exchanges information with the surgeon's console of the TSS 950. The TSS UI interface 1604 may receive a request from the surgeon for a particular assistance, in which case the TSS UI interface may command the assistance rendering engine to provide the corresponding assistance. Context-related assistance information may also be displayed to the surgeon via the TSS UI interface, which provides the information to the surgeon's console.

[0137] In a related embodiment, in response to a routine performance sensor 1602 automated assistance request, the TSS UI interface 1604 provides a visual or audible notification to the surgeon to provide the assistance and provides the surgeon with control input to accept, reject, or defer the assistance.

[0138] According to the depicted embodiment, a plurality of different auxiliary rendering engines may be used to provide corresponding types of assistance to the surgeon or other medical personnel. In various other embodiments, more or fewer auxiliary rendering engines are provided. As depicted, the video queue 1606 is configured to retrieve one or more video segments from the database 1608 for playback. The video segments may have a specific correspondence with the current stage of the surgical routine (as determined by the segmenter 1004). For example, the video segments to be queued may represent the current stage or an upcoming stage of the surgical routine.

[0139] Notifier 1610 is configured to initiate communication with an on-call surgeon based on expert contact information 1612, who can provide direct guidance and / or assistance to the surgeon via a remote console or other interface.

[0140] The technical advisor 1614 can recommend a specific surgical technique for use in a given scenario, in response to a current or upcoming surgical phase, and in consideration of the surgeon's assessed operational surgical technique assessment, based on the surgical skill requirements database 1616. In a related embodiment, the technical advisor 1614 can provide applicable prompts or other instructions or recommendations to other medical personnel, such as to one or more assistants in the surgical environment.

[0141] In a related embodiment, the technical advisor 1614 includes a predictive function that predicts the upcoming surgical phase based on the duration of the steps, the sequence of the steps, the performance level of the steps (all based on the performance of the current session and surgeon, and optionally based on historical data corresponding to that surgeon.) This information can be used to predict future events, such as the end of an optimal OR turnaround operation, an upcoming instrument exchange, etc.

[0142] The simulation configurator 1618 is operable to autonomously build a simulation scenario corresponding to a current or upcoming stage of a surgical routine to be performed by the simulator 1006. This type of assistance may allow a surgeon to practice a certain technique in a virtual surgical environment intraoperatively and then return to the actual surgical environment and perform the technique on a patient.

[0143] The control setting adjuster 1620 is operable to automatically change one or more behaviors of the TSS in response to control inputs to assist in surgical tasks. The adjustment can be done according to control adjustment rules 1622, which can be surgical routine specific, task specific, surgeon specific, or some combination thereof. Examples of variable TSS behaviors include control-input sensitivity adjustment (e.g., a reduction in motion speed for a given input event or gesture), gesture filtering (e.g., allowing certain input motions to cause surgical effects, but not allowing other input motions to cause surgical effects), video capture setting adjustment, etc. Other adjustments include camera field of view / zoom / position, table position, image attributes (color, tint, overlay, etc.), loading preoperative images from the current patient or related patients for the current / upcoming step, master tool motion scaling, force feedback gain, mapping of control input button presses differently depending on the situation / segment, or some combination thereof.

[0144] Figures 21A-21C 16 is an illustrative diagram showing the change of the surgical scene field of view implemented by the control setting adjuster 1620 to help the surgeon achieve a viewpoint in which the surgeon can more accurately control the surgical instruments within the surgical scene of the remote surgical system (TSS). The surgeon stereoscopically views the surgical scene including the surgical instruments through the left eye display 32 and the right eye display 34 (it should be noted that although the stereoscopic surgeon view is depicted and described for exemplary purposes, in various other embodiments, a single view can be provided). For example, it is assumed that the TSS is equipped with three different surgical instruments 2110, 2112, 2114, plus a camera (not shown) for installing an endoscope. The surgeon can control two surgical instruments and the endoscope at once using the left handle control input 36 and the right handle control input 38. The surgeon located in the surgical scene view can see the instruments within the camera field of view. It is further assumed that, for example, the surgeon changes the instrument during the transition from one task to the next, and the surgeon changes. In addition, it is assumed that the change from the instrument is located in a different orientation within the surgical scene than the change to the instrument. In that case, the position and orientation of the surgical instruments in the surgical scene currently under the surgeon's control has changed despite the surgeon's grip on the left handle control input 36 and the right handle control input 38 not changing. The control setting adjuster 1620 may provide adjustments to the field of view of the surgical scene so that the surgeon's viewpoint of the instruments currently under the surgeon's control within the surgical scene is aligned with the surgeon's hand positions on the left handle control input 36 and the right handle control input 38 to improve or optimize the accuracy and control of the surgical instruments.

[0145] refer to Fig.21A, a first surgical scene 2100 is shown having a first field of view indicated by the orientation of the field of view icon 2120, the first surgical scene 2100 including a first surgical instrument 2110 and a second surgical instrument 2112 currently under the control of the surgeon. The field of view icon 2120 may be displayed as an optional overlay within the screen indicating that the surgical scene field of view is "horizontal" when the endoscope is in a neutral position in the manipulator on the patient side cart. It will be appreciated that the horizontal field of view is nominally defined as a neutral position within the patient's side arm - not relative to the left and right displays, as they never change position. In addition, it will be appreciated that the orientation of the surgical instruments 2110, 2112 and the first surgical scene field of view shown in FIG. 21 is consistent with the natural anatomical orientation of the surgeon's hands on the left handle control input 36 and the right handle control input 38 when viewing the scene directly in front of the surgeon's eyes. It should be noted that due to the master clutch function of the TSS, the 3D Cartesian position of the surgeon's hands may be different from the 3D Cartesian position of the instruments.

[0146] refer to Fig.21B, a second surgical scene 2102 is shown having a first field of view indicated by the orientation of the field of view icon 2120, the second surgical scene 2102 including a first surgical instrument 2110 and a third surgical instrument 2114 currently under the control of the surgeon. Assume that in the process of transitioning from the first scene 2100 to the second scene 2102, the surgeon has changed from using the second surgical instrument (shown in dashed lines) 2112 to using the third surgical instrument 2114. Thus, the surgeon's hands on the left and right handle control inputs 36, 38 control the first and third surgical instruments 2110, 2114 in the second surgical scene 2102, rather than controlling the first and second surgical instruments 2110, 2112 as in the first scene 2100. The second and third surgical instruments 2112, 2114 are positioned at different locations within the surgical scene. Additionally, the field of view icon 2120 indicates that the view of the second surgical scene 2102 from the perspective of the left and right eye displays 32, 34 is the same as the perspective of the first scene 2100. Thus, the surgeon now controls a third instrument 2114, which is located at a different orientation within the second surgical scene 2102 than the orientation of the second instrument 2112 previously controlled. Although the surgeon's hand positions on the left and right handle control inputs 36, 38 may not have changed, and the surgical scene field of view remains unchanged, the orientation of one of the currently controlled instruments is changed, namely, the orientation of the third instrument 2114. Additionally, if the surgeon needs to mirror the pose of the third instrument 2114 before gaining control of that instrument, the orientation of the right handle control input 38 will have to be changed to match the new orientation of the third instrument 2114, which is different from instrument 2112. For these and potentially other reasons, Fig.21B The positions of the surgical instruments 2110, 2114 and the second surgical scene view shown in FIG. 2 may not correspond to the natural anatomical positions of the surgeon's hands on the left and right handle control inputs 36, 38. Due to this inconsistency, the surgeon may not be able to move the third instrument 2114 with the accuracy and / or dexterity he or she might otherwise have.

[0147] refer to Fig. 21C , a third surgical scene 2104 is shown having a second field of view as indicated by the orientation of the field of view icon 2120, the third surgical scene 2104 including first and third surgical instruments 2110, 2114 currently under the control of the surgeon. The third scene 2104 is identical to the second scene 2102, except that the surgical scene field of view is changed, as indicated by the orientation of the field of view icon. It should be understood that due to the change in scene orientation, when viewing the scene directly in front of the surgeon's eyes, Fig. 21CThe positions / orientations of the surgical instruments 2110, 2114 and the third surgical scene field of view are shown to more closely align with the natural anatomical positions of the surgeon's hands on the left handle control input 36 and the right handle control input 38. Due to the changed orientation of the surgical scene field of view, the surgeon can move the third instrument 2114 more easily and with greater accuracy than he or she could under the surgical scene field of view of the second scene 2102.

[0148] The control setting adjuster 1620 imparts a change in the camera orientation control setting to the endoscope-mounted camera to change the field of view orientation from the field of view orientation shown in the second surgical scene 2102 to the field of view orientation shown in the third surgical scene 2104. For example, when controlling instruments at the orientations of the first and third instruments 2100, 2104 shown in the second and third scenes 2110, 2114, the surgical technique assessor 1402 can determine the field of view orientation used by the expert surgeon using the third scene 2114. The control setting adjuster 1620 can automatically implement such field of view orientation changes for the novice surgeon. Thus, for example, when the novice surgeon transitions to a task that can be improved by a change in the surgical scene field of view based on the accuracy of the expert surgeon's instrument movements, the control setting adjuster 1620 can automatically change the orientation of the surgical scene field of view to better correlate the surgical scene field of view orientation and the instrument orientation within the scene with the position of the surgeon's hands on the left handle control input 36 and the right handle control input 38.

[0149] Figures 22A-22B 16 is an illustration showing the two-dimensional position change of the surgical scene implemented by the control setting adjuster 1620 to help the surgeon achieve a viewpoint from which the surgeon can better observe relevant information within the TSS surgical scene. Fig.22A , a fourth surgical scene 2200 is shown, in which the end effector portions of the first and second surgical instruments 2110, 2112 are located near the edge of the surgical scene 2200. It should be understood that when the surgical instruments 2110, 2112 are located near the edge of the scene as in the fourth surgical scene 2200, items of interest such as body tissue (not shown) located near the edge of the scene may be located outside the visible scene and therefore not visible to the surgeon. Fig. 22B, a fifth surgical scene 2202 is shown in which the end effector portions of the first and second surgical instruments 2110, 2112 are positioned closer to the center of the surgical scene 2202. The control setting adjuster 1620 imparts a change in the camera control setting to the endoscope-mounted camera to change from the second surgical scene 2102, which displays a first two-dimensional area in which the instrument end effector is arranged at the edge of the scene, to the third surgical scene 2104, which displays a second two-dimensional area in which the instrument end effector is arranged near the center of the scene. For example, the surgical technique evaluator 1402 can determine that the expert surgeon prefers that the instrument be displayed in a particular area (e.g., the center) within the surgical scene. Thus, for example, when a novice surgeon transitions to a task involving the scene, when, according to the expert surgeon, the task can be better viewed by, for example, a change within the area of ​​the depicted scene to move the instrument to the center of the scene, the control setting adjuster 1620 can automatically change the position of the camera to change the position of the instrument and tissue structure (not shown) depicted within the scene.

[0150] Figures 23A-23B 16 is an illustrative diagram showing the change of the zoom level of the surgical scene implemented by the control setting adjuster 1620 to help the surgeon achieve a viewpoint where the surgeon can better observe relevant information within the surgical scene of the TSS. Fig.23A , a sixth surgical scene 2300 is shown in which the first and second surgical instruments 2110, 2112 are shown zoomed out so as to appear far away and small within the surgical scene 2200. It should be understood that the far and small appearance of the surgical instruments 2110, 2112 in the sixth surgical scene 2300 also causes other items of interest, such as body tissue (not shown), with which the surgical instruments 2110, 2112 interact, to appear far away and small, and therefore difficult for the surgeon to see. Fig. 23B, a seventh surgical scene 2302 is shown in which the first and second surgical instruments 2110, 2112 are displayed in an enlarged manner so as to appear closer and larger and easier to view in detail. The control setting adjuster 1620 imparts a change in the camera control setting to the camera mounted with the endoscope to change from displaying the fifth surgical scene 2300 at a first reduced zoom level to displaying the seventh surgical scene 2302 at a second enlarged zoom level. For example, the surgical technique evaluator 1402 may determine that an expert surgeon prefers instruments to be displayed at a particular zoom level for a particular task. Thus, for example, when a novice surgeon transitions to a task involving a scene, the control setting adjuster 1620 may automatically change the zoom level of the camera when, according to the expert surgeon, for example, the task may be better viewed with a change in zoom level.

[0151] Additionally, camera adjustments can involve estimating the patient's anatomical position and the orientation of the camera to optimize the camera position / orientation for movement and interaction. For example, if the surgeon needs to drive the needle in a specific direction based on the anatomical structure, the camera can align itself to that environmental cue.

[0152] In some embodiments, assistance from any of these assistance engines can be phase-synchronized with the current surgical operation phase based on the phase current assessment. As described above, phase synchronization refers to alignment according to the surgical procedure flow, rather than strict time alignment. In some embodiments, phase synchronization supports phase offsets so that assistance can be provided to prepare the surgeon or surgical team for an upcoming phase of the surgical routine.

[0153] The assistance may further be selected to match the assessed type of surgical technique being used, such as that assessed by the segmenter 1004, simulator 1006, surgeon evaluator 908, or a combination comprising two or more of these engines.

[0154] Fig.17 17 is a block diagram of a computer system in the form of an example general-purpose machine. In some embodiments, the computer system 1700 is programmed according to one or more specific operating architectures and algorithms described herein to produce a special-purpose machine that executes the programming, thereby forming a VSE 904, a data recording engine 906, a surgeon evaluator 908, an auxiliary engine 910, or any combination of these systems. In a networked deployment, the computer system can operate in the function of a server or a client in a server-client network environment, or it can act as a peer in a peer-to-peer (or distributed) network environment.

[0155] The example computer system 1700 includes at least one processor 1702 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both, a processor core, a computing node, etc.), a main memory 1704, and a static memory 1706, which communicate with each other via a link 1708 (e.g., a bus). The computer system 1700 may also include a video display unit 1710, an alphanumeric input device 1712 (e.g., a keyboard), and a user interface (UI) navigation device 1714 (e.g., a mouse). In one embodiment, the video display unit 1710, the input device 1712, and the UI navigation device 1714 are incorporated into a touch screen display. The computer system 1700 may additionally include a storage device 1716 (e.g., a drive unit), a signal generating device 1718 (e.g., a speaker), a network interface device (NID) 1720, and one or more sensors (not shown).

[0156] The storage device 1716 includes a machine-readable medium 1722 on which is stored one or more sets of data structures and instructions 1724 (e.g., software) that embody or are utilized by any one or more of the methodologies or functions described herein. The instructions 1724 may also reside, completely or at least partially, within the main memory 1704, the static memory 1706, and / or within the processor 1702 during execution of the instructions 1724 by the computer system 1700, with the main memory 1704, the static memory 1706, and the processor 1702 also constituting machine-readable media.

[0157] Although the machine-readable medium 1722 is illustrated as a single medium in the example embodiment, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more instructions 1724. The term "machine-readable medium" should also be understood to include any tangible medium described below that is capable of storing, encoding, or carrying instructions for execution by and causing the machine to perform any one or more methods of the present disclosure, or that is capable of storing, encoding, or carrying data structures used by or associated with these instructions. The term "machine-readable medium" should therefore be considered to include, but is not limited to, solid-state memory and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, including, but not limited to (by way of example) semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0158] According to various embodiments, NID 1730 can take any suitable form factor. In one such embodiment, NID 1720 takes the form of a network interface card (NIC) connected to processor 1702 via link 1708. In one example, link 1708 includes a PCI Express (PCIe) bus, which includes a slot into which the NIC form factor can be removably engaged. In another embodiment, NID 1720 is a network interface circuit that is arranged on a motherboard together with local link circuits, processor interface circuits, other input / output circuits, memory circuits, storage devices, and peripheral controller circuits. In another embodiment, NID 1720 is a peripheral device connected to link 1708 via a peripheral input / output port such as a universal serial bus (USB) port. NID 1720 transmits and receives data through a transmission medium 1726, which can be wired or wireless (e.g., radio frequency, infrared or visible spectrum, etc.), optical fiber, etc.

[0159] Fig.18 It shows that Fig.17 1804 and 1806. Memory management device 1804 provides mapping between virtual memory and physical memory used by processes being executed. Memory management device 1804 may be an integral part of a central processing unit that also includes processing device 1802.

[0160] The interconnect 1806 includes a backplane such as memory, data and control lines, and interfaces with input / output devices (e.g., PCI, USB, etc.). Memory 1808 (e.g., dynamic random access memory - DRAM) and non-volatile memory 1809 such as flash memory (e.g., electrically erasable read-only memory - EEPROM, NAND flash, NOR flash, etc.) are connected to the memory management device 1804 and the interconnect 1806 via a memory controller 1810. In some embodiments, the architecture can support direct memory access (DMA) through peripheral devices. I / O devices including video and audio adapters, non-volatile memory, external peripheral links such as USB, Bluetooth, etc., and network interface devices such as communication through Wi-Fi or LTE home interfaces are collectively referred to as I / O devices and networks 1812, which are connected to the interconnect 1806 via corresponding I / O controllers 1814.

[0161] In software terms, a pre-operating system (pre-OS) environment 1816 is executed at initial system startup and is responsible for initiating the booting of the operating system. A traditional example of a Pre-OS environment 1816 is a system basic input / output system (BIOS). In today's systems, a unified extensible firmware interface (UEFI) is implemented. The Pre-OS environment 1816 is responsible for initiating the start of the operating system and also provides an execution environment for embedded applications according to certain aspects of the present invention.

[0162] The operating system (OS) 1818 provides a kernel for controlling hardware devices, managing memory access to programs in memory, coordinating tasks and facilitating multitasking, organizing data to be stored, allocating memory space and other resources, loading program binary code into memory, initiating the execution of applications that subsequently interact with users and hardware devices, and detecting and responding to various defined interrupts. In addition, the operating system 1818 provides device drivers and various common services, such as services that facilitate connection to peripheral devices and networks, and services that provide abstractions for applications so that applications do not need to be responsible for handling the details of these common operations. The operating system 1818 additionally provides a graphical user interface (GUI) that facilitates interaction with the user via peripheral devices such as monitors, keyboards, mice, microphones, cameras, touch screens, etc.

[0163] The runtime system 1820 implements part of the execution model, including operations such as placing parameters on the stack before a function call, disk input / output (I / O) behavior, and parallel execution related behavior. The runtime system 1820 can also perform support services such as type checking, debugging, or code generation and optimization.

[0164] Libraries 1822 include a collection of program functions that provide further abstraction for applications. For example, these include shared libraries, dynamic link libraries (DLLs). Libraries 1822 can be integrated into operating system 1818, runtime system 1820, or can be additional features, or even remotely hosted. Libraries 1822 define application program interfaces (APIs) through which applications 1824 can make various function calls to enable services provided by operating system 1818. Applications 1824 are those programs that perform tasks useful to the user, in addition to tasks performed by lower-level system programs that coordinate the basic operability of the computing device itself.

[0165] Process Flow

[0166] Fig. 202 is a process flow diagram illustrating an example method 2000 of operating a TSS support system, such as system 900. It should be noted that method 2000 is a feature-rich embodiment that combines various operations for purposes of illustration. Fig. 20 The examples depicted in the drawings may be actual embodiments, and it should be understood that in various other embodiments, certain operations may be modified, omitted, or re-ordered.

[0167] The following description refers to Fig. 20 ,as well as Figure 9-16 1002. At 2002, event monitor 1102 of surgical input evaluator 1002 receives surgical input. The surgical input includes monitored events of the TSS. Additionally, the surgical input may include control inputs to a computational model of the TSS in simulator 1006. The surgical input may also include modeled events from the computational model of the TSS.

[0168] In 2004, TSS modeler 1310 models the TSS in a virtual surgical environment. This may include computationally representing tools, end effectors, and other configuration parts as defined in configuration information 1312. TSS modeler 1310 processes control inputs and generates virtual state changes in the model in response to those control inputs. The virtual state changes themselves may constitute event types that event monitor 1102 may receive.

[0169] At 2006, the patient model 1302 models the patient, including the effects of the operations of the TSS modeler 1310. At 2008, the surgical input evaluator determines the current surgical task being performed. At 2010, the segmenter 1004 determines the current stage of the surgical routine based on the surgical input and, optionally, the evaluated task.

[0170] At 2012, the confidence measurement engine 1230 of the segmenter 1004 determines a confidence score for the stage assessment. At 2014, the surgeon evaluator 908 calculates temporal and spatial metrics of the surgical input using the surgical technique evaluator 1402. Optionally, the confidence score is taken into account. At 2016, the surgical technique evaluator generates a performance score for the surgeon or surgical staff for the particular stage.

[0171] In 2018, a call for surgeon assistance may be detected based on the current phase of the surgical routine, the performance score, or some combination of these items. The call for assistance may be provided by the surgeon or other operator. In 2020, phase synchronization assistance is provided.

[0172] Additional Notes and Examples

[0173] Example 1 is a surgical support system for a teleoperated surgical system (TSS), the teleoperated surgical system including a surgeon input interface that accepts surgical control inputs to cause an electromechanical surgical system to perform a surgical routine, the surgical support system comprising: a virtual surgical environment engine comprising computing hardware, the computing hardware being operably configured to implement: a surgical input evaluator engine for receiving surgical inputs including monitored events of the TSS; and a segmenter engine for determining a current stage of the surgical routine based on the surgical input; and an assist engine comprising computing hardware, the computing hardware being operably configured to implement: a TSS interface communicatively coupled to the surgeon input interface of the TSS; an assist call detector for detecting a surgeon assist call; and an assist rendering engine for initiating context-related assists via the TSS interface in response to a surgeon assist call, the context-related assists being synchronized with the surgical routine stage.

[0174] In Example 2, the subject matter of Example 1 optionally includes TSS.

[0175] In Example 3, the subject matter of any one or more of Examples 1-2 optionally includes, wherein the virtual surgical environment engine is further configured to implement the simulator engine to process the computational model of the surgical routine based on the surgical input of the TSS.

[0176] In Example 4, the subject matter of Example 3 optionally includes, wherein the simulator engine comprises: a TSS model and a patient model, the TSS model computationally representing the TSS in a virtual surgical environment; and the patient model computationally representing the patient based on the patient's physical characteristics, and implementing changes to the patient through operations of the TSS model.

[0177] In Example 5, the subject matter of any one or more of Examples 3-4 optionally includes, wherein the simulator engine is configured to operate in parallel with an actual surgical routine performed via the TSS, wherein surgical control inputs to the TSS produce virtual effects in the computational model of the simulator engine.

[0178] In Example 6, the subject matter of any one or more of Examples 3-5 optionally includes, wherein the surgical input received by the surgical input evaluator engine comprises simulated effects of the TSS control input from a computational model of the simulator engine.

[0179] In Example 7, the subject matter of any one or more of Examples 3-6 may optionally include, wherein the simulator engine is configured and initiated to simulate a particular phase of the surgical routine based on the determination of the current phase.

[0180] In Example 8, the subject matter of any one or more of Examples 1-7 may optionally include wherein the monitored event comprises a TSS control input of an electromechanical surgical system that controls the TSS.

[0181] In Example 9, the subject matter of any one or more of Examples 1-8 may optionally include wherein the monitored event comprises a simulated TSS control input of an electromechanical surgical system that controls the TSS.

[0182] In Example 10, the subject matter of any one or more of Examples 1-9 may optionally include wherein the monitored event includes a surgical assistant performing a TSS detection action.

[0183] In Example 11, the subject matter of any one or more of Examples 1-10 may optionally include, wherein the segmenter engine includes a confidence measurement engine for determining a confidence score representing a probability of a correct segmentation determination.

[0184] In Example 12, the subject matter of Example 11 optionally includes, wherein the confidence score indicates suitability of the corresponding sequence of surgical inputs for a deep learning engine training a segmenter engine.

[0185] In Example 13, the subject matter of any one or more of Examples 1-12 optionally includes, wherein the segmenter engine comprises a real-time segmentation evaluator configured to implement the trained neural network to process portions of the surgical input during the surgical routine.

[0186] In Example 14, the subject matter of Example 13 optionally includes, wherein the segmenter engine further comprises a post-processing segment evaluator configured to implement a clustering algorithm to process the portion of the surgical input.

[0187] In Example 15, the subject matter of any one or more of Examples 13-14 may optionally include, wherein the trained neural network is trained based on prior surgical inputs from a plurality of previous surgical routines.

[0188] In Example 16, the subject matter of any one or more of Examples 13-15 optionally includes, wherein the trained neural network is trained based on prior surgical inputs from a plurality of previously simulated surgical routines.

[0189] In Example 17, the subject matter of any one or more of Examples 1-16 optionally includes, wherein the surgical input evaluator engine includes a task evaluator engine configured to determine a current surgical task being performed based on the surgical input, wherein the current surgical task includes a series of events that produce a defined surgical effect; and wherein the segmenter engine further determines a current stage of the surgical routine based on the current surgical task.

[0190] In Example 18, the subject matter of any one or more of Examples 1-17 optionally includes including a computing hardware surgeon evaluator engine, wherein the computing hardware is operably configured to implement a surgical technique evaluator engine configured to: access surgical inputs and a current stage of a surgical routine; calculate multiple temporal and spatial metrics of the surgical inputs corresponding to multiple different stages of the surgical routine including the current stage; and generate a surgical stage-specific performance score representing the quality of the surgical performance of the surgeon generating the surgical control inputs.

[0191] In Example 19, the subject matter of Example 18 optionally includes, wherein the surgical skill evaluator is further configured to perform a comparison between the time and space metrics and a baseline metric based on a skilled performance stage of the surgical routine; and wherein the performance score is based on the comparison result.

[0192] In Example 20, the subject matter of any one or more of Examples 18-19 optionally includes, wherein the surgical technique evaluator is further configured to: access a confidence score representing a probability of a correct segmentation determination made by the segmenter engine; and further generate the performance score based on the confidence score.

[0193] In Example 21, the subject matter of any one or more of Examples 18-20 may optionally include, wherein the performance score is a running score that is updated throughout the surgical routine.

[0194] In Example 22, the subject matter of any one or more of Examples 18-21 may optionally include, wherein the assistance call detector is configured to detect a call for surgeon assistance based on the performance score.

[0195] In Example 23, the subject matter of any one or more of Examples 1-22 may optionally include, wherein the context-related assistance includes an expert video segment containing a demonstration of a particular phase portion of a surgical routine being performed.

[0196] In Example 24, the subject matter of any one or more of Examples 1-23 may optionally include, wherein the context-dependent assistance includes configuration and initiation of a simulation of a particular phase of a surgical routine.

[0197] In Example 25, the subject matter of any one or more of Examples 1-24 may optionally include, wherein the context-dependent assistance includes adjustment of a control setting.

[0198] Example 26 is a machine-implemented method for supporting a teleoperated surgical system (TSS), the teleoperated surgical system including a surgeon input interface that accepts surgical control inputs to cause an electromechanical surgical system to complete a surgical routine, the method comprising: receiving surgical input comprising monitored events of the TSS; determining a current stage of the surgical routine based on the surgical input; detecting a call for surgeon assistance; and in response to the call for surgeon assistance, enabling context-related assistance to the surgeon input interface, the context-related assistance being synchronized with the surgical routine stage.

[0199] In Example 27, the subject matter of Example 26 optionally includes simulating the surgical routine as a computational model based on the surgical input of the TSS.

[0200] In Example 28, the subject matter of Example 27 optionally includes, wherein the simulated surgical routine includes: computationally representing the TSS in a virtual surgical environment as part of a computational model; and computationally representing the patient based on the patient's physical characteristics, and effecting changes to the patient through modeled manipulation of the computationally represented TSS in the computational model.

[0201] In Example 29, the subject matter of any one or more of Examples 27-28 optionally includes, wherein the simulation is performed in parallel with an actual surgical routine performed via the TSS, wherein surgical control inputs to the TSS produce virtual effects in the computational model.

[0202] In Example 30, the subject matter of any one or more of Examples 27-29 may optionally include, wherein the surgical input comprises a simulated effect of the TSS control input from the computational model.

[0203] In Example 31, the subject matter of any one or more of Examples 27-30 may optionally include simulating a particular phase of the surgical routine using the computational model based on the determination of the current phase.

[0204] In Example 32, the subject matter of any one or more of Examples 26-31 may optionally include wherein the monitored event comprises a TSS control input of an electromechanical surgical system that controls the TSS.

[0205] In Example 33, the subject matter of any one or more of Examples 26-32 optionally includes wherein the monitored event comprises a simulated TSS control input of an electromechanical surgical system that controls the TSS.

[0206] In Example 34, the subject matter of any one or more of Examples 26-33 optionally includes wherein the monitored event includes a surgical assistant performing a TSS detection action.

[0207] In Example 35, the subject matter of any one or more of Examples 26-34 may optionally include determining a confidence score representing a probability of a correct segment determination.

[0208] In Example 36, the subject matter of Example 35 optionally includes, wherein the confidence score indicates suitability of the corresponding sequence of surgical inputs for training the deep learning algorithm.

[0209] In Example 37, the subject matter of any one or more of Examples 26-36 may optionally include, wherein in determining the current stage of the surgical routine, implementing a neural network trained to process a portion of the surgical input during the surgical routine.

[0210] In Example 38, the subject matter of Example 37 optionally includes wherein in determining the current stage of the surgical routine, a clustering algorithm is executed to process a portion of the surgical input.

[0211] In Example 39, the subject matter of any one or more of Examples 37-38 may optionally include, wherein the trained neural network is trained based on prior surgical input from a plurality of previous surgical routines.

[0212] In Example 40, the subject matter of any one or more of Examples 37-39 may optionally include, wherein the trained neural network is trained based on prior surgical inputs from a plurality of previously simulated surgical routines.

[0213] In Example 41, the subject matter of any one or more of Examples 26-40 optionally includes determining a current surgical task being performed based on the surgical input, wherein the current surgical task includes a series of events that produce a defined surgical effect; and wherein a current stage of the surgical routine is further determined based on the current surgical task.

[0214] In Example 42, the subject matter of any one or more of Examples 26-41 optionally includes calculating multiple temporal and spatial metrics of surgical inputs corresponding to multiple different stages of a surgical routine including the current stage; and generating a surgical stage-specific performance score representing the quality of the surgical performance of the surgeon generating the surgical control inputs.

[0215] In Example 43, the subject matter of Example 42 optionally includes a comparison between the execution time and space metrics and a baseline metric based on a skilled performance stage of the surgical routine; wherein the performance score is based on the comparison result.

[0216] In Example 44, the subject matter of any one or more of Examples 42-43 optionally includes generating the performance score further based on a confidence score representing a probability of a correct segment determination.

[0217] In Example 45, the subject matter of any one or more of Examples 42-44 may optionally include, wherein the performance score is a running score that is updated throughout the surgical routine.

[0218] In Example 46, the subject matter of any one or more of Examples 42-45 may optionally include, wherein the call for surgeon assistance is based on a performance score.

[0219] In Example 47, the subject matter of any one or more of Examples 26-46 may optionally include, wherein the context-related assistance comprises an expert video segment containing a demonstration of a particular phase portion of a surgical routine being performed.

[0220] In Example 48, the subject matter of any one or more of Examples 26-47 may optionally include, wherein the context-dependent assistance includes configuration and initiation of a simulation of a particular phase of a surgical routine.

[0221] In Example 49, the subject matter of any one or more of Examples 26-48 may optionally include, wherein the context-related assistance comprises adjustment of a control setting.

[0222] Example 50 is at least one non-transitory machine-readable storage medium comprising instructions that, when executed on a computing platform, cause the computing platform to implement a dedicated machine for supporting a teleoperated surgical system (TSS), the teleoperated surgical system comprising a surgeon input interface that accepts surgical control inputs to cause an electromechanical surgical system to complete a surgical routine, the instructions comprising: instructions for receiving surgical inputs including monitored events of the TSS; instructions for determining a current stage of the surgical routine based on the surgical inputs; instructions for detecting a call for surgeon assistance; and instructions for initiating context-related assistance to the surgeon input interface in response to the call for surgeon assistance, the context-related assistance being synchronized with the surgical routine stage.

[0223] In Example 51, the subject matter of Example 50 optionally includes instructions for simulating a surgical routine as a computational model based on the surgical input of the TSS.

[0224] In Example 52, the subject matter of Example 51 optionally includes, wherein the instructions for simulating a surgical routine include: instructions for computationally representing the TSS in a virtual surgical environment as part of a computational model; and instructions for computationally representing the patient based on the patient's physical characteristics, and implementing changes to the patient through modeled operations of the TSS computationally represented in the computational model.

[0225] In Example 53, the subject matter of any one or more of Examples 51-52 optionally includes wherein the simulation is performed in parallel with an actual surgical routine performed via the TSS, wherein surgical control inputs to the TSS produce virtual effects in the computational model.

[0226] In Example 54, the subject matter of any one or more of Examples 51-53 optionally includes, wherein the surgical input comprises a simulated effect of the TSS control input from the computational model.

[0227] In Example 55, the subject matter of any one or more of Examples 51-54 may optionally include instructions for simulating a particular phase of the surgical routine using the computational model based on the determination of the current phase.

[0228] In Example 56, the subject matter of any one or more of Examples 50-55 optionally includes wherein the monitored event comprises a TSS control input of an electromechanical surgical system that controls the TSS.

[0229] In Example 57, the subject matter of any one or more of Examples 50-56 optionally includes wherein the monitored event comprises a simulated TSS control input of an electromechanical surgical system that controls the TSS.

[0230] In Example 58, the subject matter of any one or more of Examples 50-57 optionally includes wherein the monitored event includes a surgical assistant performing a TSS detection action.

[0231] In Example 59, the subject matter of any one or more of Examples 50-58 may optionally include instructions for determining a confidence score representing a probability of a correct segment determination.

[0232] In Example 60, the subject matter of Example 59 optionally includes, wherein the confidence score indicates suitability of the corresponding sequence of surgical inputs for training the deep learning algorithm.

[0233] In Example 61, the subject matter of any one or more of Examples 50-60 may optionally include, wherein the instructions for determining a current stage of a surgical routine cause implementation of a neural network trained to process a portion of a surgical input during the surgical routine.

[0234] In Example 62, the subject matter of Example 61 optionally includes wherein the instructions for determining a current stage of a surgical routine cause execution of a clustering algorithm to process a portion of the surgical input.

[0235] In Example 63, the subject matter of any one or more of Examples 61-62 may optionally include, wherein the trained neural network is trained based on prior surgical inputs from a plurality of previous surgical routines.

[0236] In Example 64, the subject matter of any one or more of Examples 61-63 optionally includes, wherein the trained neural network is trained based on prior surgical inputs from a plurality of previously simulated surgical routines.

[0237] In Example 65, the subject matter of any one or more of Examples 50-64 optionally includes instructions for determining a current surgical task being performed based on surgical input, wherein the current surgical task includes a series of events that produce a defined surgical effect; and wherein the instructions are used to determine a current stage of the surgical routine based on the determination of the current surgical task.

[0238] In Example 66, the subject matter of any one or more of Examples 50-65 optionally includes instructions for calculating multiple temporal and spatial metrics of surgical input corresponding to multiple different stages of a surgical routine including the current stage; and instructions for generating a surgical stage-specific performance score representing the quality of the surgical performance of a surgeon generating the surgical control input.

[0239] In Example 67, the subject matter of Example 66 optionally includes instructions for performing a comparison between the time and space metrics and a benchmark metric based on a skilled performance stage of the surgical routine; wherein the performance score is based on the comparison result.

[0240] In Example 68, the subject matter of any one or more of Examples 66-67 optionally includes instructions for generating the performance score further based on a confidence score representing a probability of a correct segment determination.

[0241] In Example 69, the subject matter of any one or more of Examples 66-68 may optionally include, wherein the performance score is a running score that is updated throughout the surgical routine.

[0242] In Example 70, the subject matter of any one or more of Examples 66-69 may optionally include, wherein the call for surgeon assistance is based on a performance score.

[0243] In Example 71, the subject matter of any one or more of Examples 50-70 may optionally include, wherein the context-related assistance comprises an expert video segment containing a demonstration of a particular phase portion of a surgical routine being performed.

[0244] In Example 72, the subject matter of any one or more of Examples 50-71 may optionally include, wherein the context-dependent assistance includes configuration and initiation of a simulation of a particular phase of a surgical routine.

[0245] In Example 73, the subject matter of any one or more of Examples 50-72 may optionally include, wherein the context-related assistance includes adjustment of a control setting.

[0246] Although illustrative embodiments have been shown and described, various modifications, changes and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions and modifications. Therefore, the scope of the present disclosure should be limited only by the appended claims, and the claims are to be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A surgical support system for a teleoperated surgical system (TSS), the TSS comprising a surgeon input interface that accepts surgical control inputs from a user to enable an electromechanical surgical system to perform a surgical routine, the surgical support system comprising: a segmenter engine for determining a current stage of the surgical routine based on the surgical control input; as well as A surgical control input evaluator engine comprising computing hardware operably configured to implement: calculating temporal and spatial metrics of the surgical control input corresponding to the current stage; generating a surgical stage-specific performance score corresponding to the current stage based on the temporal and spatial metrics representing a quality of the user's surgical performance, the surgical stage-specific performance score being included in a series of performance scores corresponding to different stages of the surgical routine; detecting, by an assistance call detector, a call for surgeon assistance based on the surgical phase-specific performance score; and Context-dependent assistance is initiated by the assistance rendering engine via a teleoperated surgical system interface (TSS) interface in response to the call for surgeon assistance.

2. The surgical support system according to claim 1, further comprising: a simulator engine comprising computing hardware operably configured to process a computational model of a surgical routine based on surgical control inputs of the TSS, the computational model comprising: A TSS model for computationally representing the TSS in a virtual surgical environment; and A patient model is used to computationally represent the patient based on physical characteristics of the patient and changes to the patient achieved through operation of the TSS model.

3. The surgical operation support system according to claim 2, Wherein the simulator engine is configured to operate in parallel with an actual surgical routine performed via the TSS, wherein the surgical control inputs to the TSS produce virtual effects in the computational model of the simulator engine.

4. The surgical operation support system according to claim 2, Wherein the surgical control input received by the surgical control input evaluator engine comprises a simulated effect of the surgical control input from the computational model of the simulator engine.

5. The surgical operation support system according to claim 1, wherein the user evaluator engine is further configured to perform a comparison between the temporal and spatial metrics and a benchmark metric based on a proficient performance stage of the surgical routine; and Wherein the surgical stage-specific performance score is based on a result of the comparing.

6. A surgical support system for a teleoperated surgical system (TSS), the TSS comprising a user input interface that accepts surgical control inputs from a user to enable an electromechanical surgical system to perform a surgical routine, the surgical support system comprising: a segmenter engine for determining a current stage of the surgical routine based on the surgical control input; as well as An auxiliary engine comprising computing hardware operably configured to implement: a TSS interface communicatively coupled to the user input interface of the TSS; an assistance call detector for detecting a call for user assistance; as well as an assistance rendering engine for initiating context-relevant assistance via said TSS interface in response to said call for surgeon assistance; A surgical control input evaluator engine comprising computing hardware operably configured to implement: Access camera control inputs, and; Computing spatial metrics for camera control inputs; and generating a camera performance score representing a quality of camera control by a user providing the camera control input; Wherein a control settings adjuster initiates adjustment of a camera control setting based at least in part on the camera performance score.

7. The surgical operation support system according to claim 6, The adjustment comprises at least one of: a change in a field of view orientation of the visual display of the surgical scene, a change in a two-dimensional area shown within the visual display of the surgical scene, and a change in a zoom level of the visual display of the surgical scene.

8. A non-transitory machine-readable storage medium containing instructions that, when executed by a computer, cause the computer to perform a method for supporting a teleoperated surgical system (TSS), the TSS comprising a surgeon input interface that accepts surgical control inputs from a user to enable an electromechanical surgical system to perform a surgical routine, the method comprising: determining a current stage of the surgical routine based on the surgical control input; calculating temporal and spatial metrics of the surgical control input corresponding to the current stage; generating a surgical stage-specific performance score based on the temporal and spatial metrics representing a quality of the user's surgical performance; detecting, by an assistance call detector, a call for surgeon assistance based on the surgical phase-specific performance score; and Context-dependent assistance is initiated by the assistance rendering engine via a teleoperated surgical system interface (TSS) interface in response to the call for surgeon assistance.

9. The non-transitory machine-readable storage medium of claim 8, further comprising instructions that, when executed by a computer, cause the computer to: a computational model for processing a surgical routine based on the surgical control input of the TSS, the computational model comprising: A TSS model for computationally representing the TSS in a virtual surgical environment; as well as A patient model is used to computationally represent the patient based on physical characteristics of the patient and changes to the patient achieved through operation of the TSS model.

10. The non-transitory machine-readable storage medium according to claim 8, wherein calculating the time and space metrics comprises comparing the time and space metrics to a benchmark metric based on a skilled performance stage of the surgical routine; and Wherein the performance score is based on a result of the comparing.

11. A non-transitory machine-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform a method for supporting a teleoperated surgical system (TSS), the TSS comprising a surgeon input interface that accepts surgical control inputs from a user to enable an electromechanical surgical system to perform a surgical routine, the method comprising: determining a current stage of the surgical routine based on the surgical control input; as well as Initiating context-dependent assistance via a teleoperated surgical interface (TSS interface) in response to a call for surgeon assistance; adjusting a control setting to automatically change one or more behaviors of the TSS interface; Access camera control inputs, and; Compute spatial metrics for camera control inputs; as well as generating a camera performance score representing a quality of camera control by a user providing the camera control input; detecting a call for user assistance; Camera control settings are adjusted based at least in part on the camera performance score.

12. The non-transitory machine-readable storage medium of claim 11, wherein the adjustment comprises at least one of: a change in the field of view orientation of the visual display of the surgical scene, a change in the two-dimensional area shown within the visual display of the surgical scene, and a change in the zoom level of the visual display of the surgical scene.

Images