Automatic endoscopic video enhancement

Abstract

Methods and systems for automatically enhancing endoscopic video of a task based on state data describing a state of a surgical robotic system used to at least partially perform the task. The system includes an event detector and an enhancer. The event detector is configured to: receive the status data; and identifying one or more patterns in the state data, the one or more patterns indicating events occurring during the task. The enhancer is configured to receive an endoscopic video captured during a task, the endoscopic video being time-synchronized with the status data; and in response to an event detector identifying a pattern in the state data indicative of an event occurring, automatically enhancing metadata forming a portion of the endoscopic video or associated with the endoscopic video with information identifying the event regarding the endoscopic video and a time at which the event occurs.

Description

[0001] The present disclosure is a divisional application of the invention patent application with application number 201980065312.1 filed on April 1, 2021 and invention name “Automatic Endoscopic Video Enhancement”. Background Art

[0002] It is known to use robots to assist and perform surgery. Figure 1 A typical surgical robot 100 is shown, comprising a base 108, an arm 102, and an instrument 105. The base supports the robot and is itself rigidly attached, for example, to an operating room floor, an operating room ceiling, or a cart. The arm extends between the base and the instrument. The arm is articulated along its length by means of a plurality of flexible joints 103, which are used to position the surgical instrument in a desired position relative to the patient. The surgical instrument is attached to the distal end 104 of the robot arm. The surgical instrument penetrates the body of the patient 101 at a port 107 in order to access the surgical site. At its distal end, the instrument includes an end effector 106 for participating in a medical procedure.

[0003] The surgical robot 100 is operated by an operator (eg, a surgeon) via Figure 2 The operator console 200 shown in is remotely controlled. The operator console 200 can be located in the same room as the surgical robot 100 (e.g., an operating room) or away from the surgical robot. The operator console 200 includes input devices 202, 204 for controlling the state of the arm 102 and / or the instrument 105 attached thereto. The input devices 202, 204 can be handles or manual controllers mounted on a parallelogram linkage. The control system converts the movement of the manual controller into control signals to move the arm joints and / or instrument end effectors of the surgical robot. The operator console 200 also includes a display 206. The display 206 is arranged to be visible to the user operating the input devices 202, 204. The display is used to display a video stream (e.g., endoscopic video) of the surgical site.

[0004] Some surgical procedures may require several surgical robots, each carrying an instrument or other implement that is used simultaneously with other implements at the surgical site. Figure 3A surgical robot system 300 is shown, wherein a plurality of robots 302, 304, 306 operate in a common workspace on a patient 308. For example, surgical robots are commonly used in endoscopic surgery (e.g., laparoscopic surgery), which may also be referred to as minimally invasive surgery. As known to those skilled in the art, during an endoscopic procedure, a surgeon inserts an endoscope through a small incision or natural opening of the body (e.g., but not limited to, the mouth or nostril). An endoscope is a rigid or flexible tube to which a tiny camera is attached, transmitting real-time images to a video monitor (e.g., display 206), which the surgeon uses to help guide his tools through the same incision / opening or through different incisions / openings. An endoscope allows a surgeon to view relevant areas of the body in detail without having to cut an opening and expose the relevant area. This technology allows a surgeon to see inside a patient's body and operate through a much smaller incision than is required for conventional open surgery. Therefore, in a typical robotic endoscopic surgery, an endoscope is attached to one surgical robot arm, and one or more surgical instruments (e.g., a pair of forceps and / or a scalpel) are attached to one or more other surgical robot arms.

[0005] Figure 4 An exemplary endoscope 400 that can be attached to the end of a robotic arm for minimally invasive surgery is shown. The endoscope 400 has a distal end 402 for insertion into a surgical site of a patient and a proximal end 404. The distal end 402 is connected to the proximal end 404 by an elongated shaft 406. The proximal end 404 includes an interface 408 for engaging the end of a robotic arm.

[0006] The endoscope 400 has a power supply and a light source for illuminating the surgical site. The endoscope 400 also has a data line for extracting image data from the surgical site. These can all be independent of the robotic arm and attached to the proximal end 404 of the endoscope 400 externally of the robotic arm, such as Figure 4 As shown in Figure 4 In the embodiment, power is applied by rod 412, image data is extracted by rod 412, and light is applied by optical rod 410. In alternative embodiments, any one or more of light input, power input, and data output may be applied to / extracted from the endoscope by a robotic arm.

[0007] Endoscope 400 is mounted to the end of the robotic arm. Endoscope interface 408 engages a complementary interface of the robotic arm. Endoscope 400 is attachable to and detachable from the robotic arm via the robotic arm and the endoscope interface. In some cases, endoscope 400 can be operated independently of the robotic arm in its detached state. In other words, in these cases, endoscope 400 can be manually operated by a member of the operating room staff when detached from the robotic arm.

[0008] In addition to using images captured by an endoscope during surgery (which may be collectively referred to herein as endoscopic video), images captured by an endoscope may be recorded and subsequently used for a variety of purposes, such as, but not limited to, learning and / or teaching surgical procedures, and evaluating and / or reviewing the surgeon's performance (either by a third party or by the surgeon himself).

[0009] One of the advantages of robotic-assisted surgery over conventional manual surgery is that it allows for easier collection of data about how a surgical procedure is performed. For example, the movement and / or position of a robot during surgery can be monitored and recorded. It would be advantageous to be able to link or otherwise connect endoscopic video with data generated / collected by or about a surgical robot to increase the usefulness of endoscopic video in teaching, performance assessment / auditing, etc.

[0010] The embodiments described below are provided by way of example only and are not intended to limit implementations that address any or all of the disadvantages of known surgical robotic systems. Summary of the invention

[0011] This Summary is provided to introduce a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0012] Described herein are methods and systems for automatically enhancing endoscopic video of a task performed at least in part by a surgical robotic system based on state data describing a state of the surgical robotic system during the task. The system includes an event detector and an enhancer. The event detector is configured to: receive the state data; and identify one or more patterns in the state data, the one or more patterns indicating events that occurred during the task. The enhancer is configured to: receive endoscopic video captured during the task, the endoscopic video being time-synchronized with the state data; and in response to the event detector identifying a pattern in the state data indicating an event that occurred, automatically enhance metadata forming a portion of or associated with the endoscopic video with information identifying an event about the endoscopic video and a time when the event occurred.

[0013] A first aspect provides an enhancement system for generating enhanced endoscopic video, the enhancement system comprising: an event detector, the event detector being configured to: receive state data describing the state of a surgical robot system during a task at least partially performed by the surgical robot system, the surgical robot system comprising at least one surgical robot, the at least one surgical robot having a base and an arm extending from the base to an attachment of an instrument, the arm comprising a plurality of joints, whereby the configuration of the arm can be changed; and identify one or more patterns in the state data, the one or more patterns indicating events occurring during the task; and an enhancer, the enhancer being configured to: receive endoscopic video captured during the task, the endoscopic video being time-synchronized with the state data; and in response to the event detector identifying a pattern in the state data indicating an event occurring, automatically enhance metadata forming part of or associated with the endoscopic video with information identifying events and event times about the endoscopic video.

[0014] When the enhanced endoscopic video is subsequently viewed by the user, information identifying the event and the time of the event may be visible to the user.

[0015] When the enhanced endoscopic video is subsequently viewed by the user, information identifying the event and the time of the event may be selectively visible to the user.

[0016] When the enhanced endoscopic video is subsequently viewed by the user based on permissions associated with the user, the information identifying the event and the time of the event may be automatically visible or not visible to the user.

[0017] Enhancing the metadata with information identifying an event and a time of the event regarding the endoscopic video may include adding a bookmark to the endoscopic video at the time of the event identifying the event.

[0018] The one or more patterns may include a plurality of patterns, each pattern corresponding to a different event, and the event detector may be configurable to selectively recognize only a subset of the plurality of patterns.

[0019] The one or more patterns may include a plurality of patterns, each pattern corresponding to a different event, and the event detector may be configured to identify a subset of the plurality of patterns based on user permissions associated with a user initiating the enhancement.

[0020] The status data may at least partially include data generated by the surgical robotic system.

[0021] The state data may at least partially include data generated by one or more sources external to the surgical robotic system.

[0022] The one or more modes can be configured to identify one or more instrument events, which may include one or more of the following: replacement of an instrument of a surgical robot arm attached to the surgical robot system; a change in state of a powered instrument when at least one instrument of the surgical robot arm attached to the surgical robot system is a powered instrument; cleaning of an endoscope when at least one instrument of the surgical robot arm attached to the surgical robot system is an endoscope; performing white balancing on an imaging system of the endoscope when at least one instrument of the surgical robot arm attached to the surgical robot system is an endoscope; the size or frequency of movement of the endoscope falls out of range when at least one of the instruments of the surgical robot arm attached to the surgical robot system is an endoscope; and replacement of an instrument that is being actively controlled by the surgical robot system.

[0023] The one or more modes can be configured to identify one or more collision events, which may include one or more of the following: a collision between at least two surgical robotic arms of the surgical robotic system; a collision between at least two instruments attached to different surgical robotic arms of the surgical robotic system; and a collision between a surgical robotic arm of the surgical robotic system and an instrument attached to another surgical robotic arm of the surgical robotic system.

[0024] The task may be a surgical procedure performed on a patient, and the one or more patterns may be configured to identify one or more patient events, which may include one or more of the following: one or more vital signs or one or more other health metrics of the patient falling outside of a range; and a change in one or more vital signs and / or one or more other health metrics of the patient.

[0025] The surgical robotic system can be controlled by an operator, and the one or more modes can be configured to identify one or more operator events, which may include one or more of the following: one or more vital signs and / or one or more other health metrics of the operator falling out of range; changes in one or more of the vital signs and / or other health metrics of the operator; changes in the grip force applied by the operator on one or more input devices used to control the surgical robotic system; and changes in the thrust applied by the operator on the one or more input devices.

[0026] The surgical robotic system may be controlled by an operator, and the one or more modes may be configured to identify one or more operator performance events, which may include one or more of: the operator's performance in performing the task falling below an acceptable level, and a change in the operator's performance in performing the task.

[0027] The operator's performance in performing the task can be based on one or more performance criteria, which may include one or more of the following: the path taken by the surgical robotic arm when performing the steps of the task; the smoothness of the surgical robotic arm; and the duration to complete the steps of the task.

[0028] The one or more patterns may be configured to identify emergency events.

[0029] The task may include a plurality of steps, and the one or more modes may be configured to automatically identify one or more steps of the task from the state data.

[0030] The state data may include instrument information describing instruments attached to the at least one surgical robot during the task and surgical robot position information describing the movement and / or position of the at least one surgical robot during the task; and the event detector may be configured to identify a suturing step in the task by identifying patterns in the instrument information and the surgical robot position information.

[0031] The event detector can be configured to identify a suturing step in the task when the event detector identifies from the instrument information that at least two needle drivers are attached to the surgical robot and the event detector identifies from the surgical robot position information that the surgical robot is moving in a repetitive circular motion.

[0032] The system may also be configured to link all or a portion of the enhanced endoscopic video and the status data such that the information identifying an event identifies a time when an event occurred with respect to the endoscopic video and the status data.

[0033] A portion of the endoscopic video and a portion of the status data related to the identified event may be retrievable from the information identifying the event.

[0034] The task may be a surgical procedure performed on a human.

[0035] The augmentation system may include the surgical robotic system.

[0036] A second aspect provides a computer-implemented method for generating enhanced endoscopic video, the method comprising, at one or more computing-based devices: receiving endoscopic video captured during a task performed at least in part by a surgical robotic system, the surgical robotic system comprising at least one surgical robot having a base and an arm extending from the base to an attachment for an instrument, the arm comprising a plurality of joints, whereby the configuration of the arm can be changed; receiving state data describing a state of the surgical robotic system during the task, the state data being time synchronized with the endoscopic video; identifying one or more patterns in the state data, the one or more patterns indicating events that occurred during the task; and in response to identifying patterns in the state data indicating events that occurred during the task, enhancing metadata forming part of or associated with the endoscopic video with information identifying events and event times about the endoscopic video.

[0037] The above features may be combined as appropriate, as will be apparent to the skilled person, and may be combined with any aspects of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0039] Figure 1 is a schematic diagram of an exemplary surgical robot performing an exemplary surgical procedure;

[0040] Figure 2 is a schematic diagram of an exemplary operator console;

[0041] Figure 3 is a schematic diagram of an exemplary surgical robot system having multiple surgical robots;

[0042] Figure 4 is a schematic diagram of an exemplary endoscope attachable to a surgical robot arm;

[0043] Figure 5 is a block diagram of an exemplary enhancement system;

[0044] Figure 6 is a schematic diagram of an exemplary surgical robotic system;

[0045] Figure 7 is a schematic diagram of an exemplary surgical robot;

[0046] Figure 8 is a block diagram of an exemplary method for enhancing endoscopic video;

[0047] Fig. 9 is a schematic diagram of an exemplary operator console display;

[0048] Fig.10 is a schematic diagram of an exemplary enhanced endoscopic video;

[0049] Fig.11 is a schematic diagram of an exemplary enhanced endoscopic video being viewed on a media player;

[0050] Fig.12 is a schematic diagram of an exemplary enhanced endoscopic video linked to corresponding status data; and

[0051] Fig.13 is a block diagram of an exemplary computing-based device.

[0052] The accompanying drawings illustrate various examples. The skilled person will appreciate that the element boundaries (e.g., boxes, boxes, or other shapes) shown in the drawings represent an example of boundaries. In some examples, an element may be designed as multiple elements, or multiple elements may be designed as one element. Where appropriate, common reference numerals are used throughout the accompanying drawings to indicate similar features. DETAILED DESCRIPTION

[0053] The following description is presented by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art. The embodiments are described only by way of example.

[0054] Methods and systems are described herein for automatically enhancing endoscopic video captured during a task (e.g., a surgery) performed at least in part by a surgical robotic system based on state data describing the state of the surgical robotic system during the task. The surgical robotic system includes at least one surgical robot having a base and an arm extending from the base to an attachment of an instrument, wherein the arm includes a plurality of joints whereby the configuration of the arm can be changed. The method includes: receiving an endoscopic video; receiving state data describing the state of the surgical robotic system during the task that is time-synchronized with the endoscopic video; identifying one or more patterns in the state data that indicate events that occurred during the task; and in response to identifying patterns in the state data that indicate events that occurred, enhancing the endoscopic video with information identifying events and event times about the endoscopic video.

[0055] First reference Figure 5, which shows an exemplary enhancement system 500 for automatically enhancing endoscopic video captured during a task (e.g., surgery) performed at least in part by a surgical robotic system 502. The surgical robotic system 502 includes at least one surgical robot having a base and an arm extending from the base to an attachment for a surgical instrument, the arm including a plurality of joints whereby the configuration of the arm can be changed. Figure 6 An exemplary surgical robotic system 502 is described.

[0056] Enhancement system 500 includes event detector 504 and enhancer 506. Event detector 504 and / or enhancer 506 may be implemented by one or more computing-based devices, such as, but not limited to, those described below with reference to Fig.12 An exemplary computing-based apparatus is described.

[0057] The event detector 504 is configured to receive state data indicating or describing the state of the surgical robotic system 502 during a robotic task (e.g., surgery). The state data is time-synchronized with the endoscopic video captured during the task. The state data can be time-synchronized with the endoscopic video in any suitable manner to allow events detected from the state data to be associated with a specific time or time period in the endoscopic video. For example, both the endoscopic video and the state data can include or be linked to a common timestamp or a common timeline.

[0058] As described in more detail below, the state data may include any data or information about the surgical robotic system 502, and / or any information about the environment in which the surgical robotic system 502 operates during a task. All or a portion of the state data may be generated by the surgical robotic system 502 itself (e.g., data related to the position and / or movement of the surgical robotic arm, instruments attached thereto, and any manual controllers that control the movement of the robotic arm). In some cases, at least a portion of the state data may be generated by one or more external sources.

[0059] For example, Figure 5As shown in , a portion of the state data may be generated by an audio and / or video recording device, such as, but not limited to, a camera 508 that captures audio and / or video in the room where the task is performed (e.g., an operating room); and / or a health monitoring device, such as, but not limited to, a heart rate monitor 510 that monitors one or more health metrics of the patient (if the task is performed on the patient) and / or an operator (e.g., a surgeon) controlling the surgical robotic system 502. It will be apparent to those skilled in the art that these are merely examples of external sources of state data, and that other external sources of state data may exist. For example, other external sources of state data may include a patient bed, where the patient bed is a 'smart' bed capable of generating, for example, position data; and an insufflation device. As is known to those skilled in the art, during laparoscopic surgery, a gas, such as, but not limited to, carbon dioxide, is typically insufflated into a body cavity via an insufflation device to inflate the cavity to allow for more working space.

[0060] The event detector 504 may receive status data generated by the surgical robotic system 502 and / or one or more external sources 508, 510 via any suitable means. For example, in some cases, the surgical robotic system 502 and / or one or more external sources 508, 510 may provide status data to the event detector 504 via a wireless or wired communication connection 512, such as, but not limited to, an Ethernet connection, connect, connection, near field communication (NFC) connection, etc. In these examples, all or a portion of the state data can be provided to the event detector 504 in real time (or substantially real time) while the task (e.g., surgery) is being performed. In other cases, the state data can be captured by another device (e.g., a computing-based device) during the task and stored in a storage device 514 (e.g., a memory) and subsequently provided to the event detector 504 via the communication connection 512 or by any other suitable means. In some cases, all or a portion of the state data can be stored on a portable storage medium at the task location (e.g., an operating room), such as, but not limited to, a USB (Universal Serial Bus) memory stick, and physically transported to the location of the event detector 504, where the portable storage medium is coupled to the event detector 504 so that the event detector 504 can read the state data from the portable storage medium.

[0061] The event detector 504 is configured to analyze the state data to identify events that occur during a task (e.g., surgery). Events that occur during a task (e.g., surgery) may include, but are not limited to: an instrument attached to a robotic arm of a surgical robot system is replaced; a collision occurs between multiple robotic arms or between multiple instruments attached to a robotic arm; one or more vital signs of a patient and / or one or more other patient health metrics have fallen out of range; one or more of the vital signs of an operator (e.g., a surgeon) and / or other operator health metrics have fallen out of range; using the surgical robot system 502 to perform a specific skill (e.g., suturing); and deviations from an expected sequence event (e.g., performing a cutting step when a suturing step is expected). A more extensive list of exemplary events that can be identified by the event detector 504 is described below. As described in more detail below, the event detector 504 can be configured to identify events that occur during a task by identifying patterns in the state data. For example, the event detector 504 can be configured to determine that the surgical robot system 502 is used to perform suturing if the path taken by the robotic arm or a needle instrument attached thereto indicates that suturing has been performed. The patterns indicating events that have occurred may be predetermined or may be learned from previously performed tasks.

[0062] In some cases, in response to detecting from the state data an event occurring during a task (e.g., a surgery), the event detector 504 can be configured to generate an output indicating that an event has been detected, the type of event, the time of the event, and optionally the duration of the event provided to the enhancer 506.

[0063] The enhancer 506 is configured to receive endoscopic video captured by an endoscope 516 during a mission. In some cases, the endoscopic video may have been captured by an endoscope 516 that is attached to and controlled by one of the robotic arms of the surgical robotic system 502. However, in other cases, the endoscopic video may have been captured by an endoscope that was manually controlled. For example, a user may be able to remove an endoscope that is attached to one of the robotic arms and manually control the movement and operation of the endoscope. In some cases, when the user removes the endoscope from the robotic arm and manually controls the endoscope, the surgical robotic system may no longer be able to accurately track the position and / or movement of the endoscope. Therefore, the surgical robotic system may be configured to limit the response to the control input when this occurs.

[0064] The enhancer 506 may receive the endoscopic video via any suitable means. For example, in some cases, the endoscope 516 itself may be configured to provide the endoscopic video to the enhancer 506 via a wireless or wired communication connection 512, such as, but not limited to, an Ethernet connection, connect, connection, near field communication (NFC) connection, etc. In these examples, all or a portion of the endoscopic video may be provided to the enhancer 506 in real time (or substantially real time) during a mission (e.g., surgery). In other cases, the endoscopic video may be stored in a storage device 514 (e.g., memory) and subsequently provided to the enhancer 506 via the communication connection 512 or by any other suitable means. For example, the endoscopic video may be stored at the location of the mission (e.g., an operating room) on a portable storage medium, such as, but not limited to, a USB memory stick, which is physically transported to the location of the enhancer 506, where it is coupled to the enhancer 506 so that the enhancer 506 can read the endoscopic video from the portable storage medium.

[0065] The enhancer 506 is configured to add information (e.g., a marker) to the endoscopic video that identifies the event that occurred and the time when the event occurred and optionally the duration of the event in response to the event detector 504 determining from the state data that the event occurred during the mission to generate an enhanced endoscopic video. In some cases, the information or marker may not be visible to the user when the enhanced endoscopic video is subsequently played or viewed in a video player. For example, in some cases, the information or marker may be detectable by a special system or software. The special system or software may be configured to determine the identified events from the information or marker and present a list of the identified events to the user. If the user selects one of the identified events, the system may present or output a portion of the endoscopic video related to the identified event.

[0066] In other cases, the information or mark may be visible to the user when the enhanced endoscopic video is subsequently played or viewed in a video player. In some cases, the mark may take the form of a bookmark that allows the user to easily jump to a certain point in the endoscopic video when the enhanced endoscopic video is played by a media player. For example, when the user starts the enhanced endoscopic video in a media player, a list of bookmarks may be presented to the user in the video, and the user may go directly to one of the points or sections identified by the bookmark by selecting the bookmark. In some cases, the enhancer 506 may be configured to automatically group the marks and / or bookmarks based on, for example, event type and / or any other criteria. In some cases, the enhancer 506 may also be configured to automatically identify additional events from the events identified via the state data, and enhance the endoscopic video with information / marks identifying each additional event. In some cases, information related to a particular event (e.g., a mark) may be included in metadata that constitutes a part of or is associated with the endoscopic video.

[0067] The augmentation system 500 may be remote from the surgical robotic system 502. For example, the surgical robotic system 502 may be located in an operating room, and the augmentation system 500 may be in another room of a hospital or treatment center.

[0068] Reference now Figure 6 , which shows an exemplary surgical robot system 502. In this example, the surgical robot system 502 includes two surgical robots 602 and 604 driven by a control unit 606. The control unit 606 receives commands from an operator console 610 (such as, but not limited to, Figure 2 The control unit 606 receives inputs 608 from an operator console 200 of the present invention, including inputs from a first manual controller 612 and a second manual controller 614. The control unit 606 may receive other inputs from the operator console 610, such as foot pedal inputs, voice recognition inputs, gesture recognition inputs, eye recognition inputs, etc. The control unit 606 also receives inputs 616 from the surgical robots 602, 604. These inputs include sensor data from position sensors and torque sensors located on the robot arm joints. The control unit 606 may receive other inputs 616 from each robot, such as force feedback, data from or about surgical instruments, etc. The control unit 606 drives the robots 602, 604 in response to the inputs it receives from the robots 602, 604 and the operator console 610. The control unit 606 includes one or more processors 618 and a memory 620. The memory 620 stores software code that can be executed by one or more processors 618 to control the drive in a non-transitory manner.

[0069] Although Figure 6 The exemplary surgical robot system 502 includes two surgical robots, but it will be apparent to those skilled in the art that the methods and techniques described herein are equally applicable to surgical robot systems having only one surgical robot and surgical robot systems having more than two surgical robots.

[0070] Reference now Figure 7 , which shows an exemplary embodiment of a surgical robot 700. Figure 6 One or more of the surgical robots 602, 604 may be implemented as Figure 7 The surgical robot 700 includes an arm 702 extending from a base 704 that is fixed in place when a task (e.g., a surgical procedure) is being performed. In some cases, the base 704 can be mounted to a chassis. The chassis can be a cart, such as a bedside cart for mounting the robot at bed height. Alternatively, the chassis can be a ceiling mounted device or a bed mounted device.

[0071] The arm 702 extends from the base 704 of the robot to an attachment 706 for a surgical instrument 708. The arm is flexible. It articulates with the aid of a number of flexible joints 710 along its length. Between the joints are rigid arm members 712. Figure 7 The arm in has seven joints. The joints include one or more roll joints (which have an axis of rotation on either side of the joint along the longitudinal direction of the arm member), one or more pitch joints (which have an axis of rotation transverse to the longitudinal direction of the previous arm member) and one or more yaw joints (which also have an axis of rotation transverse to the longitudinal direction of the previous arm member and also transverse to the co-located pitch joint). However, the arm can be joined differently. For example, the arm can have fewer or more joints. The arm can include joints that allow movement other than rotation between corresponding sides of the joint, such as telescopic joints. The robot includes a set of drives 714, each of which drives one or more joints 710.

[0072] Attachment 706 enables surgical instrument 708 to be releasably attached to the distal end of the arm. Surgical instrument 708 has a linear rigid shaft and a working tip at the distal end of the shaft. The working tip includes an end effector for participating in a medical procedure. The surgical instrument can be configured to extend linearly parallel to the axis of rotation of the distal joint of the arm. For example, the surgical instrument can extend along an axis that coincides with the axis of rotation of the distal joint of the arm. Surgical instrument 708 can be, for example, a cutting, grasping, cauterizing or imaging device (e.g., an endoscope).

[0073] The robot arm includes a series of sensors 716, 718. For each joint, these sensors include a position sensor 716 for sensing the position of the joint, and a torque sensor 718 for sensing the torque applied around the rotation axis of the joint. One or both of the position sensor and torque sensor for a joint can be integrated with the motor for that joint. The outputs of the sensors are passed to the control unit 606, where they form the input of one or more processors 618.

[0074] Reference now Figure 8 , which illustrates an exemplary method 800 for automatically enhancing endoscopic video captured during a task (e.g., a surgery) performed at least in part by a surgical robotic system 502 based on state data describing the state of the surgical robotic system during the task. Specifically, in the method 800, events occurring during the task are detected from the state data, and the endoscopic video is enhanced with information (e.g., one or more markers) identifying the detected events and the time when the events occurred in the endoscopic video.

[0075] The method 800 begins at block 802, where the augmentation system 500 (e.g., the augmentor 506) receives endoscopic video captured by an endoscope during a task (e.g., a surgery) performed at least in part by the surgical robotic system 502. As described above, in some cases, the endoscopic video may have been captured by an endoscope that is attached to and controlled by one of the robotic arms of the surgical robotic system 502. In other cases, the endoscopic video may have been captured by a manually operated endoscope. As described above, the endoscopic video may be received at the augmentation system 500 (e.g., the augmentor 506) via any suitable means. For example, the endoscopic video may be transmitted via a wired or wireless communication connection (e.g., an Ethernet connection, a wireless communication network ... connect, Connection, NFC connection, etc.) is provided by the endoscope directly to the enhancement system 500 (e.g., enhancer 506). In other cases, the endoscopic video can be stored in the memory of another device and provided to the enhancement system 500 via the other device.

[0076] At block 804, the augmentation system 500 (e.g., the event detector 504) receives state data describing the state of the surgical robotic system 502 during a task (e.g., surgery). The state data may include any information describing the surgical robotic system 502 during a task (e.g., surgery) or describing the environment in which the surgical robotic system 502 operates. As described above, the state data is time-synchronized with the endoscopic video so that events detected from the state data can be associated with time points or time periods in the endoscopic video.

[0077] All or part of the state data may be generated by the surgical robot system 502 itself (e.g., the surgical robot system may generate data related to the position and / or movement of the surgical robot arm and / or instruments attached thereto). In some cases, part of the state data may be generated by one or more external sources. The state data may include one or more data streams, each of which is individually synchronized with the endoscopic video time. The term "data stream" is used herein to refer to a set of data generated by the same source and describing the same features of the surgical robot system 502. For example, as described above, each joint of the surgical robot system 502 may include a position sensor 716 for sensing the position of the joint, and a torque sensor 718 for sensing the torque around the axis of rotation of the joint. The data generated by each position sensor 716 can be considered as a data stream. Similarly, the data generated by each torque sensor 718 can be considered as a data stream.

[0078] The state data may include information or data describing the current state of the robotic arm of the surgical robotic system, such as, but not limited to, position and / or torque information indicating the position and / or movement of the robotic arm or its joints. Specifically, the state data may include position and / or torque data generated by the position sensor 716 and / or the torque sensor 718. For example, each position sensor 716 may report the joint angle of the joint it is sensing, and / or each torque sensor may report the torque acting on the joint it is sensing. It will be apparent to those skilled in the art that this is merely an example, and the surgical robot may include other sensors that provide information about the state of the robotic arm.

[0079] The state data may also or alternatively include data or information describing the state of the instrument attached to the robot arm. For example, the state data may include information identifying whether the instrument is attached to the robot arm, and if so, the type of instrument attached to the robot arm. In particular, the surgical robot system 502 may include a device for detecting whether the instrument is attached to the arm (for example, the surgical robot system may include an instrument engagement device described in, for example, the applicant's published patent application GB 2552855 A, which is incorporated herein by reference), a device for automatically detecting the type of instrument (for example, each instrument may include an RFID or other component that is configured to automatically provide information about its identity to the surgical robot system when it is attached to the robot arm), and this information may be provided to the event detector 504 as state data. The state data may also or alternatively include, for example, but not limited to, information indicating the position and / or movement of the instrument attached to the robot arm and / or torque information. It is obvious to those skilled in the art that this is just an example, and the surgical robot or instrument itself may include other sensors that provide information about the state of the instrument.

[0080] When at least one of the instruments attached to the robotic arm is a powered instrument, such as, but not limited to, an electrocautery instrument or an electrosurgical instrument to which an electric current is supplied to perform a surgical function (e.g., cauterization, cutting, etc.), the status data may include information about the status of the powered instrument, such as, but not limited to, whether the powered instrument is currently powered, and if so, the waveform of the current used to power the instrument.

[0081] Where at least one of the instruments attached to the robotic arm is an endoscope, the status data may include information indicative of the position, movement, status and / or field of view of the endoscope.

[0082] The state data may also or alternatively include data or information describing the state of an operator input device (e.g., a manual controller 612, 614) used to control the operation of the robot arm and the apparatus. As described above, the operator input device may be, for example, a manually operable mechanical input device, such as a manual controller or a joystick. In these cases, the state data may include, for example, but not limited to, data indicating the position and / or movement of the input device (e.g., the manual controller 612, 614) and / or torque data. For example, one or more of the input devices may include a position sensor configured to sense the position of the input device and / or a torque sensor configured to sense the torque or force applied to the input device. In some cases, the manual controller may be mounted on a parallelogram linkage via a universal joint. In these cases, the linkage and / or universal joint may include position and / or torque sensors that measure the position and torque of the joints of the linkage and / or universal joint. These measurements may then be provided as state data to the augmentation system 500 (e.g., the event detector 504).

[0083] In some cases, when the input device is a manual controller, each manual controller may include one or more hand contact sensors, etc., which are configured to detect whether a person's hand is in contact with the manual controller or is grasping the manual controller, which indicates whether the manual controller is in use. In these cases, the hand contact detection information generated by the hand contact sensor can be provided to the enhancement system 500 (e.g., event detector 504) as state data. In some cases, when the input device is a manual controller, each manual controller may also or alternatively include one or more grip sensors, which are configured to detect the grip force applied to the manual controller (i.e., how tightly the operator grasps the manual controller). In these cases, the grip force information generated by the grip force sensor can be provided to the enhancement system 500 (e.g., event detector 504) as state data. In some cases, the hand contact sensor and the grip sensor can be the same sensor or integrated in other ways. In some cases, the operator's hand can be equipped with a motion sensor that detects the operator's hand movement. In these cases, the hand motion data generated by the hand motion sensor can be provided to the enhancement system 500 (e.g., event detector 504) as state data.

[0084] The state data may also or alternatively include information about the operating mode of the surgical robotic system 502. For example, in some cases, the surgical robotic system 502 is capable of operating in one of a plurality of modes, such as, but not limited to, an active mode, in which an operator (e.g., a surgeon) controls one or more of the robotic arms to perform a task (e.g., surgery); a selection mode, in which an operator (e.g., a surgeon) selects or switches the robotic arm they will control from the operator console 610; and an instrument change mode, in which at least one of the instruments attached to the robotic arm is removed, attached, and / or changed. There may also be several types of active modes. For example, in some cases, there may be: an endoscope active mode, in which an operator controls an endoscope attached to a robotic arm; and an instrument active mode, in which an operator controls one or more other instruments attached to one or more robotic arms. In these cases, the state data may include information about the current operating mode of the surgical robotic system 502. It will be apparent to one skilled in the art that these are exemplary operating modes of the surgical robotic system, and that the surgical robotic system 502 may have additional or different operating modes.

[0085] In the case where the task being performed is a surgical procedure performed on a patient, the state data may also or alternatively include data related to the patient. For example, the state data may include patient health information or data describing the patient's health status during the task. For example, the state data may include information describing the patient's vital signs (e.g., body temperature, pulse rate, respiratory rate) or other health metrics (e.g., blood pressure, oxygen saturation, blood glucose / sugar) during the task (e.g., surgery). Typically, during surgery, the patient's vital signs and other health metrics are measured by special medical measurement devices, such as heart rate monitors, pulse oximeters, continuous glucose monitors (CGM) devices, etc. In these cases, the medical measurement device can be configured to send vital sign information or other patient measurement information to the enhancement system 500 (e.g., event detector 504) via, for example, a communication network (e.g., a wired or wireless communication network).

[0086] However, in other cases, information about patient vital signs and / or other patient health metrics can be obtained in another way. For example, in some cases, the operating room where the operation is being performed can be equipped with a video and / or audio recording device, and vital sign data and / or other patient medical data can be obtained from the video and / or audio recording. For example, in the case where the medical measurement device produces an audible representation of the measurement (e.g., a set of audible sounds or "beeps"), an estimated value of the measurement can be obtained by analyzing the audio recorded by the video and / or audio recording device. In another example, in the case where the medical measurement device visually displays the current or previous measurement value, the measurement value can be obtained by image analysis of the video recorded by the video recording device (e.g., the measurement value can be read from the medical measurement device). In other cases, one or more of the patient's vital signs can be obtained from the endoscopic video. For example, the patient's pulse rate or blood oxygen level can be identified from the endoscopic video by video analysis technology. In particular, the patient's blood oxygen level can be determined from the color of the patient's blood. The determination of the patient's vital signs or other health metrics from the endoscopic video can be facilitated by lighting the relevant part of the patient's body.

[0087] The status data may also or alternatively include other information about the patient, such as, but not limited to, demographic information, such as age and weight, and other medical information, such as previous scans (e.g., X-rays, MRIs), previous results, medical history, diagnostic history, and / or treatment plans.

[0088] In some cases, the state data may also include surgeon health information or data describing the health status of an operator (e.g., a surgeon) controlling the surgical robotic system. For example, the state data may include information or data describing one or more vital signs (e.g., body temperature, pulse rate, respiratory rate) and / or other health metrics (e.g., blood pressure) of an operator (e.g., a surgeon) during a task (e.g., surgery). For example, in some cases, where the input device is a manual controller, the manual controller may include measuring the heart rate of the operator (e.g., a surgeon), the tremor of the operator's (e.g., a surgeon's) movements, and / or the sweat rate of the operator (e.g., a surgeon), and providing information or data related to the measured heart rate, tremor, and / or sweat rate as state data to the event detector 504. In other cases, there may be an external medical device, etc., configured to measure the vital signs or other health metrics of an operator (e.g., a surgeon) during a task. As described above with respect to patient vital signs and other health metrics, a medical device that measures the vital signs or other health metrics of an operator (e.g., a surgeon) can be configured to provide data about the measured metric to event detector 504 via, for example, a communications network, or an estimate of the measured metric can be obtained by other means (e.g., by analyzing video and / or audio from an operating room).

[0089] As described above, in some cases, the room where the task is being performed (e.g., an operating room) may be equipped with audio and / or video recording equipment to record the sounds in the room and / or the visual images of the room while the task is being performed. In these cases, the state data may also or alternatively include video and audio data captured by the audio and / or video recording equipment.

[0090] In some cases, the state data may also or instead include information about the task being performed. For example, the state data may include information about a task (e.g., a surgical procedure) that is at least partially performed by the surgical robotic system 502. In some cases, where the task includes several steps, the state data may include information indicating the steps of the task that have been performed and the time at which they were performed. For example, an operator (e.g., a surgeon) of the surgical robotic system 502 may be able to indicate during a task which step or stage of the task he / she is currently performing, and this information may be provided as state data to the augmentation system 500 (e.g., the event detector 504). For example, if Fig. 9As shown in , the operator console 610 may include or be associated with a display 902 that displays an endoscopic video 904 and also provides the operator with a list of steps in a task 906. The listed steps may be automatically selected based on the task, or may be manually input by an operator (e.g., a surgeon). For example, an operator (e.g., a surgeon) can manually input the steps or stages of a task that the operator expects to perform. The operator (e.g., a surgeon) can use an input device (e.g., a manual controller 612, 614) to select or otherwise indicate which steps or stages are currently being listed by, for example, pressing a button on a manual controller or via other means (e.g., verbally or via a display when the display is a touch screen). Any selections made by the operator (e.g., a surgeon) can then be provided to the enhancement system 500 (e.g., an event detector 504) as state data.

[0091] In some cases, the status data may also or alternatively include information about a user of the surgical robotic system 502 (e.g., an operator (e.g., a surgeon) controlling the surgical robot and / or other members of a task group (e.g., a surgical team)). For example, in some cases, information indicating the user who will perform the task may be provided to the surgical robotic system 502 and / or the event detector 504 before the task begins. In other cases, the user (e.g., a surgical team) performing the task (e.g., surgery) may be equipped with a device (e.g., an RFID device, a mobile phone, a tablet computer, etc.) that automatically transmits information identifying the user when the user is in the vicinity of the surgical robotic system to the surgical robotic system 502 and / or the event detector 504. An example of such a system is described in the applicant's co-pending UK patent application entitled "DEVICE INTEROPERATION" filed on the same day as the present application, which is incorporated herein by reference.

[0092] In some cases, an operator (e.g., a surgeon) controlling the surgical robotic system or another member of the task group (e.g., a surgical team) can manually indicate that an event has occurred. In particular, in some cases, the operator (e.g., a surgeon) can be presented with a list of predetermined events that may occur during the task, and the operator (e.g., a surgeon) can indicate that one of the predetermined events has occurred by selecting one of the predetermined events. Any selection made by the operator (e.g., a surgeon) can then be provided to the augmentation system 500 (e.g., an event detector 504) as part of the state data. For example, if Fig. 9As shown in , the operator console 610 may include a display 902 or may be associated with a display on which an endoscopic video 904 is presented to an operator (e.g., a surgeon). The display 902 may also display one or more predetermined events 908 that may occur during the mission, and the operator (e.g., a surgeon) may indicate that one of the predetermined events has occurred by selecting one of the listed events. The predetermined events may be automatically selected based on the type of event, the specific operator, and / or any other criteria; or the operator or another user may be able to manually input a set of events that may occur during the mission. In some cases, the augmented system 500 or another computing-based device may be configured to automatically generate high-probability events or important events based on state data of previously performed missions, for example, which are presented to the operator (e.g., a surgeon) as events that can be manually identified by the operator (e.g., a surgeon) or other members of the mission group.

[0093] An operator (e.g., a surgeon) can select one of the predetermined events using an input device (e.g., a hand controller) by, for example, pressing a button on one of the hand controllers or by other means (e.g., verbally or by touching the display 902 when the display 902 is a touch screen). In other cases, another member of the task group (e.g., a surgical team) may have a device, such as, but not limited to, a computer or tablet to which they are presented with a list of predetermined events, and they can indicate that one of the predetermined events has occurred by selecting one of the predetermined events. Any selection can then be provided to the augmented system 500 (e.g., event detector 504) as state data. An event that can be manually identified by the operator or another member of the task group can still be automatically detected by the augmented system from other state data.

[0094] return Figure 8 At block 806, the augmentation system 500 (e.g., event detector 504) analyzes the state data to identify events that occurred during a task (e.g., surgery). The term "event" is used herein to refer to an identifiable event that occurred during a task. Events may include, but are not limited to, instrument events, collision events, task events, patient events, operator (e.g., surgeon) events, performance events, and / or emergency events. Examples of each of these event types are described below.

[0095] The enhancement system 500 (e.g., event detector 504) can be configured to identify events from the state data by identifying patterns in the state data. One or more events can be identified by identifying patterns in a single data stream of the state data or by identifying patterns in multiple streams of the state data. The enhancement system 500 is typically preconfigured with patterns that identify that an event has occurred and which specific event the pattern identifies. For example, the pattern can be stored in a memory of the enhancement system 500. The event and / or a specific pattern indicating that the event has occurred can be manually determined, or can be learned from the state data of a previously performed task. For example, in some cases, the enhancement system 500 or another computing-based device can be configured to analyze the state data of a previously performed task that has been manually enhanced with event information to identify common patterns before the event has been identified in the state data.

[0096] In some cases, the enhancement system 500 (e.g., the event detector 504) may also or alternatively be configured to detect instrument events from the state data. Instrument events are events related to instruments used by (e.g., attached to) the surgical robot system 502 during a task. Surgical robotic tasks (e.g., surgery) typically require multiple different surgical instruments. Multiple instruments can be used simultaneously (e.g., there can be multiple robotic arms to which different instruments are attached and controlled) and / or multiple instruments can be used sequentially (e.g., a first instrument can be attached to a robotic arm to perform a specific function, the first instrument can be removed, and a second instrument is attached to the robotic arm to perform a different function). An exemplary set of instruments that can be used during a surgical robotic task may include, but are not limited to, scissors, graspers of various forms and geometries, monopolar and bipolar electrosurgical instruments, other energy-driven instruments (e.g., laser or ultrasonic instruments), staplers, and tissue sealers. Instrument events may include, but are not limited to: instrument replacement; a change in the state of a powered instrument when at least one of the instruments attached to the robotic arm during a task (e.g., surgery) is a powered instrument; cleaning the endoscope when one of the instruments is an endoscope; performing white balancing on the imaging system when one of the instruments is an endoscope; the size or frequency of movement falls out of range when one of the instruments is an endoscope; and a change in the active instrument.

[0097] As described above, the state data may include information describing the state of the instruments attached to the robotic arms. For example, the state data may include information identifying whether an instrument is attached to each arm, the type of device attached to each robotic arm, and when the instrument is a powered instrument, the state of the powered instrument (e.g., whether the instrument is powered, and optionally the waveform of the current supplied to the powered instrument). In these cases, the enhancement system 500 (e.g., the event detector 504) can be configured to detect an instrument replacement event when it is detected that the instrument is removed from the robotic arm or the instrument is attached to the arm. In these cases, the enhancement system 500 (e.g., the event detector 504) can be configured to detect a change in the state of the powered instrument when it is detected from the state data that the state of the powered instrument changes from powered to non-powered or vice versa, or when it is detected that the waveform of the current supplied to the powered instrument changes.

[0098] As described above, the state data may include information about the state of the endoscope, such as whether the endoscope is in a cleaning state or whether the operator is performing white balancing on the imaging system. In these cases, the enhancement system 500 (e.g., the event detector 504) can be configured to determine that an instrument event has occurred if the enhancement system 500 (e.g., the event detector 504) detects from the endoscope state information that the endoscope is in an endoscope cleaning state or that white balancing is being performed on the imaging system. In the event that the state data does not include information about the state of the endoscope, the enhancement system 500 (e.g., the event detector 504) can be configured to automatically detect from other state data that the endoscope is in a cleaning state or that white balancing is being performed on the imaging system. For example, the enhancement system 500 (e.g., the event detector 504) can be configured to detect that the endoscope is in a cleaning state or that white balancing is being performed on the imaging system when the enhancement system 500 (e.g., the event detector 504): identifies a pattern of particular forces on the endoscope; identifies that the position of the tip of the endoscope is outside of the patient; identifies that the endoscope is being cleaned or that white balancing is being performed from video from the operating room; and / or identifies a combination thereof.

[0099] As described above, the state data may include information about the position and / or movement of the endoscope. In the case where the endoscope is attached to a robotic arm and controlled by an operator at an operator console, the endoscope position and / or movement information may be generated, for example, by a position and / or torque sensor on the robotic arm. In these cases, the enhancement system 500 (e.g., the event detector 504) may be configured to detect that an instrument event has occurred when it is detected from the position and / or movement information that the size (i.e., magnitude) or frequency of the endoscope movement has fallen outside of a range.

[0100] As described above, in some cases, the surgical robotic system 502 may include multiple robotic arms, each of which may have an instrument attached thereto, and an operator (e.g., a surgeon) may be able to selectively control one or more of the robotic arms and the instruments attached thereto from an operator (e.g., a surgeon) console. The operator may be able to select which robotic arm and, therefore, which instruments will be controlled by the operator through the operator console by connecting an input device (e.g., a hand controller) to a particular instrument. The instruments controlled by the operator console at any one time may be referred to as "active" instruments. In these cases, the state data may include information indicating which instruments are active at which points during the mission (this may be referred to as active instrument information). In these cases, the augmentation system 500 (e.g., event detector 504) may be configured to detect that an instrument event has occurred when the augmentation system 500 (e.g., event detector 503) detects from the active instrument information that one or more active instruments have changed. By detecting active instrument changes and augmenting the endoscopic video, such information allows the endoscopic video to include a recording of active instruments during the mission.

[0101] In some cases, the enhancement system 500 (e.g., the event detector 504) can be configured to additionally or alternatively detect a collision event. The collision event may include, but is not limited to, a collision between robotic arms of the surgical robotic system 502, a collision between instruments attached to different robotic arms of the surgical robotic system 502, and a collision between a robotic arm of the surgical robotic system 502 and an instrument attached to another robotic arm of the surgical robotic system 502.

[0102] As described above, in some cases, the state data may include data or information indicating the position and / or motion of the robotic arms and / or instruments attached thereto. In these cases, the augmentation system 500 (e.g., the event detector 504) may, for example, be configured to detect that a collision event has occurred (i.e., that the robotic arm or instrument has collided with another robotic arm or instrument) based on the data or information indicating the position and / or motion of the robotic arms and / or instruments. For example, the augmentation system 500 (e.g., the event detector 504) may be configured to determine that a collision has occurred between arms, between instruments, or between arms and instruments, for example by (i) determining or estimating the position of the arms and / or instruments from the position and / or motion data, and determining that the positions of one or more instruments and / or one or more arms are the same or substantially similar; or (ii) determining from the position and / or motion information (e.g., torque data) that forces of equal magnitude and opposite direction have been detected on two of the arms and / or instruments, as described in the applicant's UK patent application No. 1810754.0, which is incorporated herein by reference.

[0103] In some cases, the enhancement system 500 (e.g., the event detector 504) can be configured to detect task events in addition or alternatively. Task events can include, for example, changes in the steps of a task being performed. For example, a task (e.g., a surgery) can be associated with a predetermined set of steps, all or a portion of which can be performed during the task, and the enhancement system 500 (e.g., the event detector 504) can be configured to detect that an event has occurred when a change in the steps of the task being performed is detected from the state data. Automatically identifying the steps of the task being performed and enhancing the endoscopic video with this information can allow the endoscopic video to be automatically segmented based on the steps of the task. This can allow a user to easily identify portions of the endoscopic video that are associated with specific steps of the task.

[0104] As described above, in some cases, the state data may include information that explicitly identifies which steps and times of performing a task (which may be referred to herein as step identification information). For example, as described above, an operator (e.g., a surgeon) or other user (e.g., other surgical team member) can manually indicate which steps of a task are being performed during a task. In these cases, the enhancement system 500 (e.g., event detector 504) can identify changes in the task steps from the step identification information. When information that explicitly identifies the task steps is not received as part of the state data, the enhancement system 500 (e.g., event detector 504) can be configured to dynamically detect the steps of the task from other state data. For example, the enhancement system 500 (e.g., event detector 504) can be configured to detect the steps of a task based on a pattern detected in the state data that matches a pattern in the state data of a previously performed task. In particular, augmented system 500 (e.g., event detector 504) or another computing-based device can be configured to analyze state data related to previously performed tasks of the same type as the current task (e.g., state data related to the same surgical procedure or the same type of surgical procedure); identify a common set of steps in the task based on the analysis; and generate a map connecting the identified steps. In some cases, the analysis can include analyzing at least the endoscopic video.

[0105] In other cases, the augmentation system 500 (e.g., event detector 504) or another computing-based device may be provided with state data for one or more previously performed tasks that have been manually divided into segments associated with the steps of the task, and the augmentation system 500 (e.g., event detector 504) or other computing-based device may identify characteristics or features of the state data associated with each step. For example, each step may be characterized by, for example, one or more of the following features: instrument type, robot arm motion, and / or manual controller motion. For example, the augmentation system 500 (e.g., event detector 504) or other computing-based device may be configured to receive state data for a previously performed task, wherein segments of the state data corresponding to suturing steps are identified as such. The augmentation system 500 (e.g., event detector 504) or other computing-based device may then analyze the portions of the state data identified as corresponding to suturing steps and identify features, such as: the needle driver instrument is attached to two of the robot arms; the operator performs repetitive small bending motions with the manual controller (and therefore with the end effector); and high forces are applied to the end effector and end effector jaws. The characteristics and / or features of the step may then be used to identify the step from the state data for a subsequent task. In some cases, the enhancement system 500 (e.g., event detector 504) or other computing-based device may be able to generate a feature or feature that sufficiently describes a task step from a single state data set (i.e., state data associated with a single previously performed task) so that it can be identified in another state data set (i.e., state data associated with another task). In other cases, the enhancement system 500 (e.g., event detector 504) or other computing-based device may require multiple state data sets to generate a feature or feature that sufficiently describes a task step so that it can be identified in another state data set.

[0106] An exemplary set of steps for a robotic-assisted surgical procedure that may be automatically detected by the augmentation system 500 (eg, the event detector 504) is shown in Table 1. A robotic-assisted surgical procedure may include all or only a portion (ie, a subset) of these steps.

[0107] Table 1

[0108]

[0109]

[0110] In many cases, each task has a set of predefined steps that can be performed during the task, and there are only a few possible combinations of steps that may occur. In these cases, once the enhancement system 500 (e.g., event detector 504) detects a specific step, a fairly limited number of steps may usually occur after the step. This can alleviate the burden of the enhancement system 500 (e.g., event detector 504) in the steps of the detection task. It can also allow the enhancement system 500 (e.g., event detector 504) to detect the time when there is a deviation from the expected step set (e.g., the time when the steps in the expected step list are executed out of order). As described below, when the enhancement system 500 (e.g., event detector 504) detects that there is a deviation from the expected step set, the enhancement system 500 (e.g., event detector 504) can be configured to detect an emergency.

[0111] In some cases, the augmentation system 500 (e.g., event detector 504) can be configured to additionally or alternatively detect patient events. A patient event can include one or more vital signs or one or more other health metrics of a patient falling outside of a range; and / or one or more vital signs and / or other health metrics of a patient changing (e.g., exceeding a certain amount).

[0112] As described above, in some cases, the status data may include patient health information or data indicating the status of one or more vital signs and / or one or more other health metrics of the patient during the mission. In these cases, the augmentation system 500 (e.g., event detector 504) may be configured to detect a patient event when the patient health information or data indicates that one or more of the patient's health metrics (e.g., vital signs) have fallen outside of a range. The ranges may vary between health metrics (e.g., there may be different ranges for respiratory rate and heart rate).

[0113] In some cases, the ranges may be predetermined. In other cases, the ranges may be learned from state data for previously performed tasks. For example, the augmentation system 500 (e.g., event detector 504) or another computing-based device may identify from state data for previously performed tasks the levels of different health metrics just before a negative outcome occurred, and generate ranges from the identified levels (e.g., a range covering the lowest and highest identified levels may be selected). The ranges may be determined across all tasks, or different ranges may be determined for different tasks (i.e., different surgical procedures). For example, it may be determined from state data for previously performed tasks that a heart rate within a range of X to Y is normal for surgical type A, but abnormal for surgical type B.

[0114] In some cases, the augmentation system 500 (e.g., event detector 504) can be configured to additionally or alternatively detect operator (e.g., surgeon) events. Operator (e.g., surgeon) events may include, but are not limited to, one or more of the following: the vital signs and / or other health metrics of the operator (e.g., surgeon) fall outside of a range; one or more changes in the vital signs and / or other health metrics of the operator (e.g., surgeon) (e.g., exceed a specific amount); a change in grip force applied by the operator (e.g., surgeon) on one or more input devices (e.g., manual controllers) (e.g., exceed a specific amount); and / or a change in thrust force applied by the operator (e.g., surgeon) on one or more of the input devices (e.g., manual controllers) (e.g., exceed a specific amount). For example, the augmentation system 500 (e.g., event detector) can be configured to detect that an operator event has occurred when the operator's blood pressure exceeds a certain range; or an operator event is detected if the operator's heart rate exceeds a certain range and the grip force applied by the operator exceeds a certain threshold / amount.

[0115] As described above, in some cases, the state data may include operator (e.g., surgeon) health information or data indicating the state of one or more vital signs of the operator (e.g., surgeon) and / or one or more other health metrics of the operator (e.g., surgeon) during the task. In these cases, the augmentation system 500 (e.g., event detector 504) may be configured to detect an operator event when the operator health information or data indicates that one or more of the operator's health metrics (e.g., vital signs) have fallen out of range or changed by more than a specific amount. The range and / or specific amount may vary between health metrics (e.g., there may be different ranges for respiratory rate and heart rate), operators, and / or tasks.

[0116] In some cases, ranges and / or specific quantities may be learned from state data for previously performed tasks. For example, the augmentation system 500 (e.g., event detector 504) or another computing-based device may identify levels of different health metrics for an operator just prior to a negative event (e.g., a negative event associated with an operator or a patient) from state data for previously performed tasks, and generate ranges from the identified levels (e.g., a range that may selectively cover the lowest and highest identified levels). Ranges may be determined for all operators, or different ranges may be determined for different operators. For example, it may be determined from state data for previously performed tasks that a heartbeat within a range of X to Y is normal for one operator, but abnormal for another operator.

[0117] As described above, in some cases, the state data may include manual controller grip information that describes the grip force applied by the operator to one or more manual controllers. In these cases, the augmentation system 500 (e.g., event detector 504) can be configured to detect an operator event when the augmentation system 500 (e.g., event detector 504) detects from the manual controller grip information that the grip force has changed (or changed by more than a certain amount).

[0118] As described above, in some cases, the manual controller may be mounted on a parallelogram linkage via a gimbal, and the state data may include position and / or torque data indicative of the position and / or torque data of the joints of the linkage and / or gimbal. In these cases, the augmentation system 500 (e.g., event detector 504) may be configured to determine the thrust applied by the operator on one or more input devices (e.g., the manual controller) from the position and / or torque linkage and / or gimbal information, and detect that an operator event has occurred when the augmentation system 500 detects that the thrust has changed (or has changed by more than a certain amount).

[0119] In some cases, the augmentation system 500 (e.g., event detector 504) may be configured to additionally or alternatively detect a performance event. A performance event is an event related to the performance of an operator (e.g., a surgeon) in performing a task (or a portion of a task), and may include, but is not limited to, that the performance of the operator (e.g., a surgeon) has fallen below an acceptable level, and / or that the performance of the operator (e.g., a surgeon) has changed (or changed by more than a certain amount).

[0120] In some cases, the augmentation system 500 (e.g., event detector 504) can be configured to detect that the performance of an operator (e.g., a surgeon) has fallen below a threshold by comparing the state data of the task with the state data of a previously performed task (e.g., a task of the same type). For example, the augmentation system 500 can access a data repository in which state data of previously performed tasks is stored and in which the performance of the operator (e.g., a surgeon) has been evaluated or identified (e.g., the performance can be scored on a scale such as, but not limited to, from 1 to 10); and the state data can be compared with the state data stored in the repository to determine the performance level of the operator (e.g., a surgeon) for the task.

[0121] In some cases, the augmentation system 500 (e.g., event detector 504) can be configured to determine the performance level of the operator (e.g., surgeon) in performing the task by comparing one or more performance metrics of the operator (e.g., surgeon) to performance metrics of tasks previously performed by the operator (e.g., surgeon). For example, the augmentation system 500 (e.g., event detector 504) can be configured to determine the performance level of the operator (e.g., surgeon) based on the performance metrics at multiple time points during the task (e.g., determine a level such as 1 to 10), and the augmentation system 500 (e.g., event detector 504) can detect the performance level when the performance level changes or the performance level drops outside of an acceptable range. In some cases, the performance metrics can include one or more of the following: the path taken by the robotic arm or instrument when performing the task or task step; the smoothness of the movement of the robotic arm and / or instrument in performing the task or task step; the time taken to perform the task or task step; and / or the amplitude or frequency of the movement of the robotic arm.

[0122] As described above, in some cases, the state data may include information describing the position and / or movement of the arms and / or (multiple) instruments of the surgical robotic system 502. In these cases, the augmentation system 500 (e.g., the event detector 504) can be configured to determine the path of the robotic arm and / or instrument from the information describing the position and / or movement of the arm and / or instrument. Once the path taken has been identified, the augmentation system 500 (e.g., the event detector) can be configured to evaluate the user's performance by comparing the path taken with the "best" or "optimal" path for the task or steps of the task. The "best" path can be determined from the state data of previously performed tasks. For example, the augmentation system 500 or another computing-based device can be configured to determine the "best" path from the state data of previously performed tasks (where the paths have been sorted). In addition to all sources of the above-mentioned state data, the state data of previously performed tasks may also include result data. As known to those skilled in the art, result data is information about the outcome (e.g., success / failure) of the operation. In some cases, the closer the path taken is to the "best" or "optimal" path, the better the performance, and the further the path is from the "best" or "optimal" path, the worse the performance.

[0123] The enhancement system 500 (e.g., event detector 504) may also be capable of measuring the smoothness of a path based on whether the path has smooth curves or jagged edges, and the enhancement system 500 (e.g., event detector 504) may be configured to evaluate the operator's performance based on, for example, the number of smooth curves and / or the number of jagged edges.

[0124] As described above, in some cases, the state data may include information that explicitly identifies the tasks and / or steps of the tasks being performed. In other cases, the enhancement system 500 can dynamically identify the tasks and / or steps of the tasks based on other state data. In either case, the enhancement system 500 (e.g., the event detector 504) can be configured to determine the time taken to perform the tasks or steps of the tasks by measuring the time between tasks or between steps of the tasks. The enhancement system 500 (e.g., the event detector 504) can evaluate the operator's performance by comparing the time taken to complete the tasks or steps of the tasks with the time taken to complete the same tasks or steps of the tasks during the previous execution of the tasks (as generated from the state data).

[0125] As described above, in some cases, the state data may include information describing the position and / or movement of an input device (e.g., a manual controller). In these cases, the augmentation system 500 may be configured to determine the amplitude (e.g., magnitude) and frequency of the manual controller movement from the manual controller position and / or movement data. The augmentation system 500 may then be configured to evaluate the operator's performance based on the measured amplitude and frequency of the movement. Specifically, large and / or large variations in movement may indicate poor performance, while small, infrequent movements may indicate good performance.

[0126] For example, the augmentation system 500 (e.g., event detector 504) can be configured to convert the manual controller position and / or movement data from the time domain to the frequency domain (e.g., via Fourier transform) and identify patterns in the frequency domain that indicate an event has occurred. For example, if the augmentation system 500 detects a large number of low-frequency components in the frequency domain, this may indicate fatigue or intoxication; while a large number of high-frequency components in the frequency domain may indicate a high level of tremors or panic movements. In some cases, the low-frequency and high-frequency components may have acceptable ranges, and if one or both fall outside the corresponding acceptable ranges, the augmentation system 500 (e.g., event detector 504) can be configured to determine that a manifestation event has occurred.

[0127] In some cases, the augmentation system may be configured to determine a single operator performance score or descriptor based on multiple performance criteria. In other cases, the augmentation system 500 (e.g., event detector 504) may be configured to determine multiple operator performance scores based on different performance criteria. The augmentation system 500 (e.g., event detector 504) may be configured to determine that a performance event has occurred if any performance criterion has fallen below an acceptable level or if any performance criterion has changed (e.g., changed by at least a certain amount).

[0128] In some cases, the augmentation system 500 (e.g., the event detector 504) can be configured to additionally or alternatively detect an emergency event. An emergency event refers to an event that indicates that an emergency situation has occurred or may occur during a task. The augmentation system 500 (e.g., the event detector 504) can be configured to detect an emergency event when the augmentation system 500 detects: one or more individuals present for the task (e.g., one or more people on the surgical team) speaking loudly; one or more individuals present for the task (e.g., one or more people on the surgical team) speaking a warning word or phrase; a sudden movement of the surgical robot arm; and / or a deviation or deviation from a predetermined step of the task.

[0129] As described above, in some cases, the status data may include a video and / or audio recording of the room (e.g., an operating room) where the task is performed. In these cases, the enhancement system 500 (e.g., event detector 504) can be configured to detect an emergency event when the enhancement system 500 (e.g., event detector 504) detects from the video and / or audio recording that an individual in the room has spoken in a raised voice. Specifically, if a person speaks in a raised voice, this may indicate that there is some friction between the people in the room / operating room and / or that there is a dispute or problem in the room / operating room. The enhancement system 500 (e.g., event detector 504) is able to detect the elevated voice using any known speech analysis technology.

[0130] In the case where the status data includes video and / or audio recordings, the enhancement system 500 (e.g., event detector 504) can also or alternatively be configured to detect an emergency event when the enhancement system 500 (e.g., event detector 504) detects from the video and / or audio recording that an individual in the room has spoken a warning word or phrase (i.e., a word or phrase indicating that there may be a problem). The enhancement system 500 (e.g., event detector 504) can be configured to detect warning words and / or phrases in the video and / or audio recording using any known speech recognition or analysis technology. Before starting the analysis, a list of warning words and / or phrases can be preloaded into the enhancement system 500 (e.g., event detector 504). In some cases, the list of warning words and / or phrases can be learned from the status data of previously performed tasks. For example, the enhancement system 500 (e.g., event detector 504) or another computing-based device can identify any words or phrases spoken a certain period of time before the negative result from the status data of each of a plurality of previously performed tasks, and generate a list of the identified words or phrases that are common between several previously performed tasks.

[0131] It will be apparent to one skilled in the art that this is an exemplary set of events that the enhancement system 500 (e.g., event detector 504) can detect, and in other instances, the enhancement system (e.g., event detector 504) can be configured to detect additional events, only a subset of events, or different events. For example, other events that the enhancement system 500 (e.g., event detector 504) can detect include, but are not limited to, a pressure drop in the insufflation device.

[0132] In some cases, once a set of events has been identified from the state data, the augmentation system 500 (e.g., event detector 504) can be configured to automatically identify additional events based on event patterns. For example, certain event patterns can indicate that another event has occurred. Event patterns that indicate that a particular event has occurred can be learned (e.g., learned from state data for a previously performed task—the previously performed task can be the same as the current task, a similar task group to the current task, or a different task group relative to the current task).

[0133] In some cases, the enhancement system 500 (e.g., event detector 504) is capable of detecting a predetermined set of events from the state data. The predetermined set of events that the enhancement system 500 (e.g., event detector 504) is capable of detecting from the state data may be based on the type of state data received and / or one or more other criteria. In some cases, the enhancement system 500 may be configured to detect only a subset of the predetermined set of events that it is capable of detecting. For example, a user may be able to configure the enhancement system 500 (e.g., event detector) to detect only a subset of the predetermined set of events. This may be useful, for example, when a user is looking for something specific in an endoscopic video. For example, there may be a user interface (e.g., a graphical user interface (GUI)) connected to the enhancement system 500 (e.g., event detector 504) that allows the user to configure the enhancement system 500. The user interface may, for example, present the predetermined set of events supported by the enhancement system 500 to the user, and the user may select which supported events are to be detected by the enhancement system (e.g., event detector 504).

[0134] Alternatively or in addition, in addition to the user being able to manually select which events the enhancement system 500 (e.g., event detector 504) is to detect, the enhancement system 500 (e.g., event detector 504) may be configured to automatically select events to detect based on the user logged into the enhancement system 500. For example, in some cases, to generate enhanced endoscopic video based on status data a user may have to log in or otherwise authenticate at the enhancement system 500, and the events detected by the enhancement system 500 (e.g., event detector 504) may be based on, for example, the permissions of the user. For example, an operator performing a task may have access to more events than another operator (e.g., a surgeon). For example, an operator performing a task may be able to see performance events, while another operator (e.g., a surgeon) may not be able to see performance events for that particular task. As described in more detail below, instead of the enhancement system 500 (e.g., event detector 504) being configured to detect only a subset of a predetermined set of events from the state data, the enhancement system 500 can be configured to detect all predetermined sets of events, and the events available to a user when viewing the enhanced video can be configurable and / or based on user permissions.

[0135] In some cases, for each event detected from the state data, the enhancement system 500 (e.g., event detector 504) can generate event information for enhancing the endoscopic video. The event information can include, but is not limited to, an event identifier identifying the detected event, an event start time identifying when the event started, and optionally, an event duration indicating when the event stopped (in the case where the event has a duration) or an event end time. The event identifier can be a name or number identifying the detected event. In some cases, a predetermined set of events can be detected, which are listed in a lookup table, for example, with a number and a text description. The start time and / or end time can be an absolute time (e.g., 3:45pm) or can be a relative time (e.g., 35 minutes after the start of the mission).

[0136] Once the augmentation system 500 (e.g., the event detector 504) has analyzed the state data and identified events that occurred during the mission, the method 800 proceeds to block 808 where the endoscopic video is augmented. Figure 8 In the method 800 of the embodiment, the event detector analyzes the state data to identify all events that occurred during the mission, and then the method 800 proceeds to block 808 for enhancing the endoscope video, but in other exemplary methods, the endoscope video can be enhanced on a per-event basis. In other words, in other cases, the endoscope video can be enhanced as long as an event is detected, rather than having to wait until all events are detected.

[0137] At block 808, the enhancement system 500 (e.g., enhancer 506) generates an enhanced endoscopic video by enhancing the endoscopic video received at block 802 with information that identifies, for each event detected at block 806, the detected event and the time at which the event occurred with respect to the endoscopic video. As described above, in some cases, the information added to the endoscopic video may take the form of a marker that is placed at a relevant time in the endoscopic video (e.g., when the event begins or within the duration of the event). When the enhanced endoscopic video is subsequently played or viewed via, for example, a media player, the information or marker may be visible to the user. In some cases, the marker may take the form of a bookmark that allows the user to easily jump to a certain point in the endoscopic video when the enhanced endoscopic video is played or viewed via, for example, a media player. For example, when the user launches the enhanced endoscopic video in a media player, a list of bookmarks in the video representing detected events may be presented to the user, and the user may go directly to one of the points identified by the bookmark by selecting the bookmark. In some cases, information related to a particular event (eg, a marker) may be included in metadata that is part of or associated with the endoscopic video.

[0138] For example, Table 2 shows a list of events that may be detected by the enhancement system 500 (e.g., event detector 504) in block 806. It will be apparent to one skilled in the art that these are exemplary events, start times, end times, and durations, and that other events with other state times, end times, and durations may be detected.

[0139] Table 2

[0140]

[0141] Fig.10 is an exemplary enhanced endoscopic video 1000 that has been enhanced based on the events shown in Table 2 detected by the enhancement system 500 (eg, event detector) in block 806. Fig.10It can be seen that in this example, the enhancement system 500 (e.g., enhancer 506) adds one marker 1002, 1004, 1006, 1008 to the original endoscopic video 1010 for each identified event to generate the enhanced endoscopic video 1000. Specifically, the enhancement system 500 (e.g., enhancer 506) adds a first marker 1002 indicating that the suturing step occurs between 25 minutes and 45 minutes; a second marker 1004 indicating that the needle driver instrument is attached to the robot arm 0 between 25 minutes and 30 minutes; a third marker 1006 indicating that the needle driver instrument is attached to the robot arm 1 between 30 minutes and 35 minutes; and a fourth marker 1008 indicating that the operator's performance changes to level 7 at 65 minutes. It will be apparent to those skilled in the art that this is merely an example of how an endoscopic video may be enhanced, and in other examples, the endoscopic video may be enhanced in different ways.

[0142] Once the enhanced endoscopic video has been generated, the enhanced endoscopic video may be stored in the memory of the enhancement system 500 or in the memory of another device for later use. For example, the stored enhanced endoscopic video may be subsequently viewed or played using, for example, a media player to view the operator's performance, teach the operator how to perform a procedure, etc. The information or markings added to the enhanced endoscopic video may be viewed by the user while viewing or playing the enhanced endoscopic video. In particular, in some cases, the information or markings may be displayed as bookmarks. For example, Fig.11 As shown in FIG. 1 , when a user views the Fig.10 When viewing the exemplary enhanced endoscopic video 1000 of FIG. 1006 , a list (or bookmark menu) of bookmarks 1104 may be presented to the user alongside the endoscopic video 1106 . Fig.11 In Table 2, there is a bookmark for each event detected - a bookmark for the suturing step, a bookmark for each instrument change, and a bookmark for a change in performance level. The user can then automatically move to a specific portion of the endoscopy video 1106 by selecting one of the bookmarks 1104. In some cases, such as Fig.11 As shown in , bookmarks 1104 can be presented to the user in a group form based on, for example, type or time. However, it will be apparent to those skilled in the art that this is just an example, and bookmarks can be presented to the user in another manner.

[0143] In some cases, in addition to adding information or indicia for each detected event as a bookmark to the endoscopic video presented to the user, the information added to the endoscopic video for the detected events may be presented to the user when the endoscopic video is played back. For example, the enhanced endoscopic video may be enhanced so that a banner at the top, bottom, or side of the screen displays the event information when the video is played back - that is, it displays information about the detected events at appropriate or relevant times in the endoscopic video.

[0144] In some cases, the information or marks visible when viewing or playing enhanced endoscopic video using, for example, a media player may be configurable. For example, a list of events marked in the enhanced endoscopic video may be presented to the user of the enhanced endoscopic video, and the user may be able to select which of these events are visible. The user may be able to select individual events or event types. For example, the user may be able to select all instrument events and / or all task events; or some instrument events and / or some task events. In some cases, the information or marks visible when viewing or playing enhanced endoscopic video may be based on the user viewing the enhanced endoscopic video. For example, in some cases, in order to view the enhanced endoscopic video, the user may have to log in to the endoscopic video viewing system or otherwise authenticate in the endoscopic video viewing system. In these cases, the information or marks visible when viewing or playing the enhanced endoscopic video may be determined based on the permissions associated with the user playing the video. For example, an operator performing a task may be able to view all information or marks, while other users may only be able to view a subset of the information or marks. For example, other users may not be able to view any information or marks related to the performance event.

[0145] In some cases, in addition to being configured to automatically enhance the endoscopic video with information or markers identifying events that occurred during the mission, the augmentation system 500 can also be configured to generate a three-dimensional (3D) virtual reality (VR) reconstruction of the instrument and its position and / or movement from the state data. The augmentation system 500 can then synchronize and connect the augmented endoscopic video with the 3D VR reconstruction of the instrument so that the 3D VR reconstruction can be viewed next to the augmented endoscopic video when the endoscopic video is viewed or played. For example, as described above, in some cases, the state data may include information describing the position and / or movement of the robotic arm and / or the instrument attached thereto during the mission, as well as information describing the instrument attached to each robotic arm during the mission. In these cases, the augmentation system 500 can be configured to determine the position and movement of the instrument from the robotic arm / instrument position / movement information. Once the position and movement of the instrument have been identified, the movement can be mapped to a 3D model of the relevant instrument based on the type of instrument attached to each arm (as identified by the instrument information of the state data). Once the movement is mapped to the 3D model, the 3D model can be merged to form a 3D VR reconstruction of the instrument during the mission. The 3D VR reconstruction can then be time synchronized and linked to the enhanced endoscopic video. In some cases, the perspective or view of the instrument in the 3D VR reconstruction can be configurable (i.e., the user can set or configure the viewpoint). This can allow the user to observe the instrument from a different perspective than the endoscope perspective. In some cases, the viewpoint can be dynamically reconfigured during playback. Specifically, the 3D VR reconstruction can allow the user to adjust the viewpoint during playback or viewing.

[0146] In some cases, in addition to being configured to automatically enhance the endoscopic video with information or markers identifying events that occurred during a mission and the time or period of time when the endoscopic video occurred, the enhancement system 500 can also be configured to link the enhanced endoscopic video to all or a portion of the state data in a manner that links the identified events not only to the relevant portion of the endoscopic video but also to the relevant portion of all or a portion of the state data. For example, the enhanced endoscopic video can be linked to all or only a subset of the data streams that form the state data. For example, if the state data includes a patient heart rate data stream and a surgical robot arm position information data stream, the enhanced endoscopic video can be linked to one or both of the data streams. The linked enhanced endoscopic video and state data can be referred to as enhanced combined data.

[0147] As described above, the state data is time synchronized with the endoscopic video via, for example, a common timestamp or a common timeline. Thus, enhanced combined data may be generated by linking the enhanced endoscopic video with all or a portion of the state data such that corresponding portions of the endoscopic video and the state data are aligned or linked. For example, Fig.12Links to corresponding status data 1202 are shown. Fig.10 1208. The exemplary enhanced endoscopic video 1000 is shown in FIG. 1204, so that corresponding portions of the endoscopic video 1000 and the state data 1202 are aligned or linked. In this example, the state data 1202 includes two data streams - a patient heart rate data stream 1206 and a surgical robot position information data stream - and both data streams are linked to the enhanced endoscopic video 1000. This means, for example, that the heart rate data and the surgical robot position information at 30 minutes (into the mission) are linked to the endoscopic video at 30 minutes (into the mission). However, it will be apparent to those skilled in the art that this is merely an example and that the state data 1202 may include any combination of the above data streams. The linked enhanced endoscopic video 1000 and state data 1202 form enhanced combined data 1208.

[0148] Once the enhanced endoscopic video is linked to the state data to form the enhanced combined data, the information and / or markers in the enhanced endoscopic video can be used to identify the portion of the endoscopic video and state data associated with each identified event. In this way, the information or markers in the enhanced endoscopic video can be regarded as a time slice that identifies the combined data associated with the identified event. For example, the marker 1002 identifying the suturing step identifies the portion of the endoscopic video, the patient's heart rate data stream, and the surgical robot position information data stream corresponding to the suturing step. Then, the marker can be used to retrieve data from one or more other data streams of the endoscopic video and / or state data associated with the event. For example, the system or software can be configured to receive the enhanced combined data, which can present a list of identified events to the user based on the information or markers in the enhanced combined data. Then, by selecting one of the events, the portion of the endoscopic video and state data associated with the event can be presented to the user. The portion of the endoscopic video and state data associated with the event can be presented to the user in any suitable form. For example, by selecting a listed event, the portion of the endoscopic video related to that event can be played along with a display of relevant status data (e.g., the patient's heart rate during the relevant portion can be displayed above, below, or to the side of the endoscopic video during playback).

[0149] Although the above has been described as generating enhanced combined data by first generating enhanced endoscopic video and then linking the enhanced endoscopic video, in other instances, the endoscopic video may be linked to all or a portion of the status data to generate combined data prior to enhancement, such that the enhancer 506 enhances the combined data with information or markers identifying the identified events and the time of occurrence to generate enhanced combined data.

[0150] Reference now Fig.13, which illustrates various components of an exemplary computing-based device 1300, which may be implemented as any form of computing and / or electronic device and may implement embodiments of the methods and enhancement systems described herein.

[0151] The computing-based device 1300 includes one or more processors 1302, which may be microprocessors, controllers, or any other suitable type of processor for processing computer-executable instructions to control the operation of the device in order to verify the hardware design of the data conversion pipeline. In some examples, such as where a system-on-chip architecture is used, the processor 1302 may include one or more fixed function blocks (also referred to as accelerators) that implement a portion of the method of enhancing endoscopic video in hardware (rather than software or firmware). Platform software including an operating system 1304 or any other suitable platform software may be provided on the computing-based device to enable, for example, implementation of the system. Figure 8 The method software 1305 is an application software capable of executing on the device.

[0152] Computer executable instructions can be provided using any computer readable medium accessible by the computing-based device 1300. Computer readable media can include, for example, computer storage media, such as memory 1306 and communication media. Computer storage media (i.e., non-transitory machine-readable media), such as memory 1306, include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device, or any other non-transmission medium that can be used to store information for access by computing-based devices. In contrast, communication media can contain computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, computer storage media does not include communication media. Although computer storage media (i.e., non-transitory machine-readable media such as memory 1306) are shown within computing-based device 1300, it should be appreciated that the storage device may be remotely distributed or located and accessed via a network or other communication link (e.g., using communication interface 1308).

[0153] The computing-based device 1300 also includes an input / output controller 1210 that is arranged to output display information to a display device 1312, which may be separate from or integrated with the computing-based device 1300. The display information may provide a graphical user interface. The input / output controller 1310 is also arranged to receive and process input from one or more devices, such as a user input device 1314 (e.g., a mouse or keyboard). This user input may be used to initiate authentication. In one embodiment, if the display device 1312 is a touch-sensitive display device, the display device may also serve as the user input device 1314. The input / output controller 1310 may also output data to a device other than the display device, such as a locally connected printing device ( Fig.13 not shown).

[0154] The applicant hereby independently discloses each individual feature described herein and any combination of two or more such features, as long as these features or combinations can be implemented according to the common knowledge of those skilled in the art based on this specification as a whole, regardless of whether these features or combinations of features solve any problems disclosed herein. In view of the above description, it is obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. An enhancement system for generating enhanced endoscopic video, the enhancement system comprising: An event detector, the event detector being configured to: receiving state data describing a surgical robotic system during a task at least partially performed by the surgical robotic system, wherein the surgical robotic system includes at least one surgical robot and an arm extending from the base to an attachment for an instrument, the at least one surgical robot having a base, the arm including a plurality of joints such that a configuration of the arm can be changed, wherein the state data includes instrument information describing at least one instrument attached to the at least one surgical robot during the task and surgical robot position information describing movement and / or position of the at least one surgical robot during the task; identifying one or more patterns in the state data, the one or more patterns indicating events occurring during the task, wherein the one or more patterns in the state data include one or more patterns in the instrument information and one or more patterns in the surgical robot position information; and identifying a suturing step in the task by identifying the one or more patterns in the instrument information and the one or more patterns in the surgical robot position information; and An enhancer, the enhancer being configured to: receiving endoscopic video captured during the mission, the endoscopic video being time synchronized with the status data; and In response to the event detector identifying a pattern in the status data indicative of an event that occurred, metadata forming part of or associated with the endoscopic video is automatically enhanced with information identifying events and event times regarding the endoscopic video.

2. The enhancement system according to claim 1, wherein: When the enhanced endoscopic video is viewed by a user after the task, information identifying the event and the time of the event is visible to the user.

3. The enhancement system according to claim 1, wherein: When the enhanced endoscopic video is viewed by a user after the task, information identifying the event and the time of the event is selectively visible to the user.

4. The enhancement system according to claim 1, wherein: When the enhanced endoscopic video is viewed by the user after the task based on permissions associated with the user, the information identifying the event and the time of the event is automatically visible or not visible to the user.

5. The enhancement system according to any one of claims 1 to 4, wherein: The one or more patterns include a plurality of patterns, each pattern corresponding to a different event, and the event detector is configurable to selectively identify only a subset of the plurality of patterns.

6. The enhancement system according to any one of claims 1 to 4, wherein: The one or more patterns include a plurality of patterns, each pattern corresponding to a different event, and the event detector is configured to identify a subset of the plurality of patterns based on user permissions associated with a user initiating the enhancement.

7. The enhancement system according to any one of claims 1 to 4, wherein: The state data at least partially comprises data generated by the surgical robotic system; and / or wherein the state data at least partially comprises data generated by one or more sources external to the surgical robotic system.

8. The enhancement system according to any one of claims 1 to 4, wherein: The one or more modes are configured to identify one or more instrument events, the one or more instrument events including one or more of the following: instrument replacement of a surgical robot arm attached to the surgical robot system; when at least one instrument attached to the surgical robot arm of the surgical robot system is a powered instrument, a state change of the powered instrument; when at least one instrument attached to the surgical robot arm of the surgical robot system is an endoscope, cleaning the endoscope; when at least one instrument attached to the surgical robot arm of the surgical robot system is an endoscope, performing white balancing on the imaging system of the endoscope; when at least one of the instruments attached to the surgical robot arm of the surgical robot system is an endoscope, the size or frequency of the endoscope movement falls out of range; and replacement of instruments being actively controlled by the surgical robotic system.

9. The enhancement system according to any one of claims 1 to 4, wherein: The one or more modes are configured to identify one or more collision events, including one or more of the following: a collision between at least two surgical robot arms of the surgical robot system; a collision between at least two instruments attached to different surgical robot arms of the surgical robot system; and a collision between a surgical robot arm of the surgical robot system and an instrument attached to another surgical robot arm of the surgical robot system.

10. The enhancement system according to any one of claims 1 to 4, wherein: The task is a surgical procedure performed on a patient, and the one or more modes are configured to identify one or more patient events, the one or more patient events comprising one or more of the following: one or more vital signs or one or more other health metrics of the patient falling outside of a range; and changes in one or more vital signs and / or one or more other health metrics of the patient.

11. The enhancement system according to any one of claims 1 to 4, wherein: The surgical robotic system is controlled by an operator, and the one or more modes are configured to identify one or more operator events, the one or more operator events comprising one or more of the following: one or more vital signs and / or one or more other health metrics of the operator falling out of range; changes in one or more of the operator's vital signs and / or other health metrics; a change in grip force applied by the operator on one or more input devices used to control the surgical robotic system; and a change in thrust force applied by the operator on the one or more input devices.

12. The enhancement system according to any one of claims 1 to 4, wherein: The surgical robotic system is controlled by an operator, and the one or more modes are configured to identify one or more operator performance events, including one or more of the following: the operator's performance in performing the task falls below an acceptable level; and a change in the operator's performance in performing the task.

13. The enhancement system of claim 12, wherein: The operator's performance in performing the task is based on one or more performance criteria, including one or more of the following: the path taken by the surgical robotic arm when performing the steps of the task; the smoothness of the surgical robotic arm; and the duration to complete the steps of the task.

14. The enhancement system according to any one of claims 1 to 4, wherein: The task includes a plurality of steps, and the one or more modes are configured to automatically identify one or more steps of the task from the state data.

15. The enhancement system according to any one of claims 1 to 4, wherein: The event detector is configured to identify a suturing step in the task when the event detector identifies from the instrument information that at least two needle drivers are attached to the surgical robot and the event detector identifies from the surgical robot position information that the surgical robot is moving in a repetitive circular motion.

16. An enhancement system according to any one of claims 1 to 4, wherein: The system is also configured to link all or a portion of the enhanced endoscopic video and the status data such that the information identifying an event identifies a time when an event occurred with respect to the endoscopic video and the status data.

17. The enhancement system of claim 16, wherein: A portion of the endoscopic video and a portion of the status data related to the identified event may be retrieved from the information identifying the event.

18. An enhancement system according to any one of claims 1 to 4, wherein: The augmentation system includes the surgical robot system.

19. A computer-implemented method of generating enhanced endoscopic video, the method comprising, at one or more computing-based devices: receiving endoscopic video captured during a task performed at least in part by a surgical robotic system, wherein The surgical robot system comprises at least one surgical robot and an arm extending from the base to an attachment of an instrument, the at least one surgical robot having a base, the arm comprising a plurality of joints so that the configuration of the arm can be changed, wherein the state data comprises instrument information describing at least one instrument attached to the at least one surgical robot during the task and surgical robot position information describing the movement and / or position of the at least one surgical robot during the task; receiving state data describing the surgical robotic system during the task, the state data being time synchronized with the endoscopic video; identifying one or more patterns in the state data, the one or more patterns indicating events occurring during the task, wherein the one or more patterns in the state data include one or more patterns in the instrument information and / or one or more patterns in the surgical robot position information, and identifying a suturing step in the task by identifying the one or more patterns in the instrument information and the one or more patterns in the surgical robot position information; and In response to identifying a pattern in the state data indicating that an event has occurred during the task, metadata forming part of or associated with the endoscopic video is enhanced with information identifying events and event times regarding the endoscopic video.

Images