Endoscopic surgical navigation with a 3D surgical workspace

Abstract

Provided are systems and methods for generating graphical user interfaces representing teleoperation of robotic systems. A system receive at least one data stream of a medical procedure. The system splits the at least one data stream into a plurality of segments. The system receives a selection of a segment of the plurality of segments. The system generates a first graphical user interface (GUI) element and a second GUI element. The second GUI element is generated based at least on kinematics data and displays at least one path taken by the at least one instrument during the segment. The system provides, for display via a graphical user interface, an arrangement comprising the first GUI element and the second GUI element.

Description

ENDOSCOPIC SURGICAL NAVIGATION WITH A 3D SURGICAL WORKSPACECROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 649,737, filed May 20, 2024, which is hereby incorporated by reference herein in its entirety.BACKGROUND

[0002] Robotic systems can be used to perform various tasks, such as medical procedures. However, training individuals to operate robotic systems can be difficult, particularly when significant amounts of time are spent training to perform highly-specialized techniques manually before training to perform the techniques using the robotic systems. As a result of the shift in spatial domains (e.g., from a three-dimensional workspace to control of instruments in the three-dimensional workspace at a different location), individuals can spend an excessive amount of additional training time to transfer their skills from a particular domain (e.g., the traditional surgical domain) to the robotics domain.SUMMARY

[0003] Technical solutions disclosed herein are generally related to systems and methods for the generation of graphical user interfaces representing teleoperation of robotic systems. These solutions can involve obtaining data generated during operation of a robotic system such as, for example, a robotic surgical system; generating a first graphical user interface element that displays a segment of a video stream; generating a second graphical user interface element that displays at least one path taken by at least one instrument corresponding to (e.g., synchronized with) the video stream, and providing data associated with an arrangement. In examples, the data associated with the arrangement can be configured to cause a display device to generate a graphical user interface in accordance with the arrangement.

[0004] Aspects of the technical solution are directed to a system. The system can include one or more processors, coupled with memory. The one or more processors can receive at least one data stream of a medical procedure performed with a robotic medical system. The at least one data stream can include kinematics data or event data associated with at least one instrument of the robotic medical system and video data of the medical procedure. The one ormore processors can split the at least one data stream into a plurality of segments of the medical procedure. The one or more processors can receive a selection of a segment of the plurality of segments. The one or more processors can generate a first graphical user interface element that displays a segment of a video stream corresponding to the segment of the medical procedure. The one or more processors can generate, using the kinematics data, a second graphical user interface element that displays at least one path taken by the at least one instrument during the segment. The one or more processors can provide, for display via a graphical user interface, an arrangement comprising the first graphical user interface element and the second graphical user interface element. The at least one path taken by the instrument displayed in the second graphical user interface can be synchronized with the segment of the video stream displayed in the first graphical user interface element.

[0005] In some aspects, the one or more processors can generate the arrangement based at least in part on the first graphical user interface element and the second graphical user interface element. The arrangement can identify a position of the first graphical user interface relative to a position of the second graphical user interface element.

[0006] In some aspects, the at least one path can include a plurality of paths. The one or more processors can generate the second graphical user interface element, where each path of the plurality of paths are positioned in spaced relation to each other one or more other paths of the plurality of paths.

[0007] In some aspects, the each path of the plurality of paths is associated with a plurality of areas corresponding to each path of the plurality of paths. In some aspects, the plurality of areas comprise at least one selectable region associated with portions of the instrument path.

[0008] In some aspects, each path of the plurality of paths is associated with a plurality of areas corresponding to each path of the plurality of paths, and the plurality of areas comprise at least one selectable region that, upon selection, are configured to cause generation of the first graphical user interface element and the second graphical user interface element in accordance with a point in time corresponding to the selectable region.

[0009] In some aspects, the one or more processors can receive an indication of input selecting a selectable region of the at least one selectable region. In some aspects, the one or more processors can generate a third graphical user interface element based at least in part on a portion of the at least one path taken by the at least one instrument during the medical procedure. The portion of the at least one path can correspond to the point in time. In some aspects, the one or more processors can provide, for display via the graphical user interface,the arrangement, where the arrangement comprises at least a portion of the third graphical user interface element.

[0010] In some aspects, the one or more processors can split the at least one data stream into the plurality of segments of the medical procedure based at least in part on one or more periods of time or one or more phases of the medical procedure. In some aspects, the at least one path taken by the at least one instrument comprises a plurality of paths. In some aspects, the one or more processors can provide, for display via the graphical user interface, the arrangement comprising the first graphical user interface element and the second graphical user interface element, the second graphical user interface element representing the plurality of paths. In some aspects, the at least one instrument comprises at least one surgical robotic instrument or at least one master tool manipulator (MTM).

[0011] In some aspects, the one or more processors can determine one or more objective performance indicators based at least in part on movement of the at least one instrument during the segment of the medical procedure. In some aspects, the one or more processors can provide, for display via the graphical user interface, the arrangement, where the arrangement comprises a representation of the one or more objective performance indicators. In some aspects, the representation of the one or more objective performance indicators can include a graph of changes in the one or more objective performance indicators during a period of time.

[0012] In some aspects, the one or more processors can receive input indicating one or more criteria associated with the medical procedure. In some aspects, the one or more processors can generate the first graphical user interface element and the second graphical user interface element based at least in part on the one or more criteria associated with the medical procedure. In some aspects, the one or more criteria comprise at least one of: a period of time during the medical procedure, identifiers of one or more of the segments of the medical procedure, or a playback speed to control playback of the medical procedure.

[0013] In some aspects, the one or more processors can provide, for display via the graphical user interface, data that is configured to cause the arrangement comprising the first graphical user interface element and the second graphical user interface element to be displayed via a display device.

[0014] Aspects of the technical solution are directed to a method. The method can include one or more processors receiving, by one or more processors, at least one data stream of a medical procedure performed with a robotic medical system, the at least one data stream comprising kinematics data associated with at least one instrument of the robotic medicalsystem and video data of the medical procedure. In some aspects, the method can include splitting, by the one or more processors, the at least one data stream into a plurality of segments of the medical procedure. In some aspects, the method can include receiving, by the one or more processors a selection of a segment of the plurality of segments. In some aspects, the method can include generating, by the one or more processors, a first graphical user interface element that displays the segment of the video stream corresponding to the selected segment. In some aspects, the method can include generating, by the one or more processors and using the kinematics data, a second graphical user interface element that displays a path taken by the instrument during the selected segment. In some aspects, the method can include providing, by the one or more processors, an arrangement comprising the first graphical user interface element and the second graphical user interface element.

[0015] In some aspects, the method can include generating, by the one or more processors, the arrangement based at least in part on the first graphical user interface element and the second graphical user interface element. The arrangement cam identify a position of the first graphical user interface relative to a position of the second graphical user interface element.

[0016] In some aspects, the at least one path can include a plurality of paths. In some aspects, the method can include generating, by the one or more processors, the second graphical user interface element, each path of the plurality of paths positioned in spaced relation to one or more other paths of the plurality of paths.

[0017] In some aspects, each path of the plurality of paths is associated with a plurality of areas corresponding to each path of the plurality of paths. In some aspects, the plurality of areas include at least one selectable region associated with portions of the instrument path. In some aspects, the plurality of areas include at least one selectable region that, upon selection, are configured to cause generation of the first graphical user interface element and the second graphical user interface element in accordance with a point in time corresponding to the selectable region.

[0018] Aspects of the technical solution are directed to a non-transitory computer- readable medium. The non-transitory computer-readable medium can store processorexecutable instructions that, when executed by one or more processors, cause the one or more processors to receive at least one data stream of a medical procedure performed with a robotic medical system. The at least one data stream can include kinematics data associated with at least one instrument of the robotic medical system and video data of the medical procedure. When executed by the one or more processors, the processor-executableinstructions can cause the one or more processors to split the at least one data stream into a plurality of segments of the medical procedure. When executed by the one or more processors, the processor-executable instructions can cause the one or more processors to receive a selection of a segment of the plurality of segments. When executed by the one or more processors, the processor-executable instructions can cause the one or more processors to generate a first graphical user interface element that displays the segment of the video stream corresponding to the selected segment. When executed by the one or more processors, the processor-executable instructions can cause the one or more processors to generate, using the kinematics data, a second graphical user interface element that displays a path taken by the instrument during the selected segment. When executed by the one or more processors, the processor-executable instructions can cause the one or more processors to provide, for display via a graphical user interface, an arrangement comprising the first graphical user interface element and the second graphical user interface element, wherein the path taken by the instrument displayed in the second graphical user interface is synchronized with the segment of the video stream displayed in the first graphical user interface element.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. 1 depicts an example system to generate graphical user interfaces representing teleoperation of robotic systems, according to some embodiments;

[0020] FIG. 2 illustrates a flowchart diagram illustrating an example method for generating graphical user interfaces representing teleoperation of robotic systems, according to some embodiments;

[0021] FIGS. 3 A-3F illustrates an example sequence of graphical user interfaces, according to some embodiments;

[0022] FIG. 4 illustrates a diagram of a medical environment; according to some embodiments;

[0023] FIG. 5 illustrates a block diagram depicting an architecture for a computer system that can be employed to implement elements of the systems and methods described and illustrated herein.DETAILED DESCRIPTION

[0024] Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems for the generation of graphical user interfaces representing teleoperation of robotic systems. The various concepts introducedabove and discussed in greater detail below can be implemented in any of numerous ways. Although the present disclosure is discussed in the context of a surgical procedure, in some embodiments, the present disclosure can be applicable to other medical sessions or environments or activities, as well as non-medical activities where the determination of forces applied is desired.

[0025] Systems, methods, apparatuses, and non-transitory computer-readable media are disclosed involving generation of graphical user interfaces representing teleoperation of robotic systems. Examples described herein can be directed to the post-operative analysis of data generated by robotic systems (e.g., robotic medical systems) during surgical procedures and, in some examples, to the visual reconstruction of movements performed by the robotic systems remotely controlled by surgeons, the visual reconstruction involving instruments positioned at different angles relative to a camera as both the camera and instruments move in relative to a reference frame (e.g., a world space). In an example, as surgeons engage teleoperated surgical robots, the positions of the instruments (e.g., positions representing the distal portions of the instruments) or the surgeon’s hand controllers (also referred to as master tool manipulators (MTMs)) can be tracked. Tracking can occur using imaging devices (such as cameras) or based on kinematic data generated by the surgical robots as registered with respect to a patient cart of the robotic system. Visualizations of the movement of the instruments during the surgical procedure can then be generated based on the tracked positions, stored in a datacenter, or later displayed together with surgical video to support the review of surgical performance and training.

[0026] In some examples, the surgical procedures can be annotated (e.g., tagged) based on operation of the instruments during the surgical procedures in accordance with one or more ontologies associated with the type of surgical procedure. For example, individuals can review and annotate portions of the surgical procedure related to predetermined techniques involved in a given procedure. Annotated surgical procedures can then be reproduced visually along with corresponding representations of metrics determined at segments during the surgical procedure. In one example, a surgical procedure can be replayed (e.g., by displaying a video captured during the surgical procedure of the instruments engaging with an insufflated portion of the body of the patient) such that surgeons (e.g., the surgeon that performed the procedure, junior surgeons reviewing surgeries performed by a senior or expert surgeon, etc.) can review the procedure and compare the procedure to one or more other procedures. This can provide graphical user interfaces for junior surgeons to view whencomparing actions taken during surgical procedures that they are involved in with actions taken by senior or expert surgeons in similar procedures.

[0027] In examples, the surgical procedures can be represented visually as a plot (also referred to as a “noodle plot”). When generating a plot, a system can reproduce the motion of the instrument or of the hand controller during a surgical procedure based on the kinematic data generated or derived during the surgical procedure. The plot can be represented as a continuous path or segmented path that corresponds to movement of a distal portion of the instrument during the surgical procedure or segments of the surgical procedure. In examples, plots can include multiple paths, each path representing a specific instrument or MTM that is used during the surgical procedure. The paths can be augmented during the reproduction (e.g., through the application of various colors, etc.) to indicate which hand of the surgeon was controlling the instrument at a given point in time. In examples, the paths can be augmented during the reproduction to indicate one or more actions taken such as, for example, whether energy was applied (e.g., during cauterization, electroporation, etc.), an amount of force applied by the instrument, when the instrument was in contact with certain anatomical structures, etc.

[0028] Aspects of the presently-disclosed techniques improve on existing techniques in various ways, including, for example, in that they provide surgeons with the ability to visualize the entire 3D workspace of a given surgery. For example, and in contrast with techniques of reviewing surgical procedures that involve reviewing only two-dimensional (2D) endoscopic video footage, the present techniques allow surgeons involved in postoperative review to visualize the three-dimensional (3D) positions of the surgical instruments involved in a procedure relative to a reference frame apart from, or in addition to, the traditional 2D images captured by an imaging device supported by an endoscope. This allows for an improved understanding of the position of the instruments during post-operative review relative to the patient with respect to the reference frame as opposed to just the image frame of the imaging device(s) present during the surgical procedure. Advantageously, this assists more junior surgeons in visualizing the position of the instrument s) involved in the surgical procedure within a 3D workspace, enabling them to better appreciate the positioning of similar instruments relative to the 3D workspace in subsequent surgical procedures, reducing the amount of time involved in adapting skills from the surgical domain to the robotics domain.

[0029] FIG. 1 depicts an example system 100 to generate graphical user interfaces representing teleoperation of robotic systems such as, for example, robotic medical systemsused in robot-assisted surgeries. The example system 100 can include a combination of hardware and software for generating graphical user interfaces representing aspects of teleoperation of robotic systems. For example, the example system 100 can include a network 105, a medical environment 110, a data processing system 130, and a computing device 150 as described herein.

[0030] The example system 100 can include a medical environment 110 (e.g., a medical environment that is the same as, or similar to, the example medical environment 400 of FIG. 4) including one or more data capture devices 112, medical instruments 114, visualization tools 116, displays 118 and robotic medical systems (RMSs) 120. RMS 120 can include or generate various types of data streams 122 that are described herein, and can operate using system configurations 124. One or more RMSs 120 can be communicatively coupled with one or more data processing systems 130.

[0031] The RMS 120 can be deployed in any medical environment 110. The medical environment 110 can include any space or facility for performing medical procedures, such as a surgical facility or an operating room. The medical environment 110 can include medical instruments 114 (e.g., surgical tools used for specialized tasks) that the RMS 120 can use for performing operational procedures, such as surgical patient procedures, whether invasive, non-invasive, or any in-patient or out-patient procedures. The RMS 120 can be centralized or distributed across a plurality of computing devices or systems, such as computing devices 500 (e.g., used on servers, network devices or cloud computing products) to implement various functionalities of the RMS 120, including communicating or processing data streams 122 across various devices via the network 105.

[0032] The medical environment 110 can include one or more data capture devices 112 (e.g., optical devices, such as cameras or sensors or other types of sensors or detectors) for capturing data streams 122. The data streams 122 can include any sensor data, such as images or videos of a surgery, kinematics data on any movement of medical instruments 114, or any events data, such as installation, configuration or selection events corresponding to medical instruments 114. The medical environment 110 can include one or more visualization tools 116 to gather the captured data streams 122 and process it for display to the user (e.g., a surgeon, a medical professional or an engineer or a technician configuring RMS) via one or more displays 118 (e.g., a touchscreen, an LCD display, or the like). A display 118 can present data stream 122 (e.g., images or video frames) of a medical procedure (e.g., surgery) being performed using the RMS 120 while handling, manipulating, holding or otherwise utilizing medical instruments 114 to perform surgical tasks at the surgical site. RMS 120 caninclude system configurations 124 based at least on which RMS 120 can operate, and the functionality of which can impact the data flow of the data streams 122. As will be described herein, the data streams 122 can be divided into multiple data streams.

[0033] The system 100 can include one or more data capture devices 112 (e.g., video cameras, sensors or detectors) for collecting data streams 122, that can be used for machine learning, including detection of objects from sensor data (e.g., video frames or force or feedback data), detection of particular events (e.g., user interface selection of, or a surgeon’s engaging of, a medical instrument 114) or detection of kinematics (e.g., movements of the medical instrument 114). The data capture devices 112 can include cameras or other image capture devices for capturing videos or images from a particular viewpoint within the medical environment 110. The data capture devices 112 can be positioned, mounted, or otherwise located to capture content from any viewpoint that facilitates the data processing system capturing various surgical tasks or actions.

[0034] The data capture devices 112 can include any of a variety of detectors, sensors, cameras, video imaging devices, infrared imaging devices, visible light imaging devices, intensity imaging devices (e.g., black, color, grayscale imaging devices, etc.), hyperspectral imaging devices (e.g., a hyperspectral camera, etc.), depth imaging devices (e.g., stereoscopic imaging devices, time-of-flight imaging devices, etc.), medical imaging devices such as endoscopic imaging devices, ultrasound imaging devices, etc., non-visible light imaging devices, any combination or sub-combination of the above mentioned imaging devices, or any other type of imaging devices that can be suitable for the purposes described herein. The data capture devices 112 can include cameras that a surgeon can use to perform a surgery and observe manipulation components within a purview of field of view suitable for the given task performance. The data capture devices can output any type of data streams 122, including data streams 122 of kinematics data (e.g., kinematics data streams), data streams 122 of events data (e.g., events data streams) and data streams 122 of sensor data (e.g., sensors data streams).

[0035] For example, data capture devices 112 can capture, detect, or acquire sensor data such as videos or images, including for example, still images, video images, vector images, bitmap images, other types of images (e.g., Raman hyperspectral images, etc.), or combinations thereof. The data capture devices 112 can capture the images at any suitable predetermined capture rate or frequency. Settings, such as zoom settings or resolution, of each of the data capture devices 112 can vary as desired to capture suitable images from any viewpoint. For instance, data capture devices 112 can have fixed viewpoints, locations,positions, or orientations. The data capture devices 112 can be portable, or otherwise configured to change orientation or telescope in various directions. The data capture devices 112 can be part of a multi-sensor architecture including multiple sensors, with each sensor being configured to detect, measure, or otherwise capture a particular parameter (e.g., sound, images, or pressure).

[0036] The data capture devices 112 can generate sensor data from any type and form of a sensor, such as a positioning sensor, a biometric sensor, a velocity sensor, an acceleration sensor, a vibration sensor, a motion sensor, a pressure sensor, a light sensor, a distance sensor, a current sensor, a focus sensor, a temperature or pressure sensor or any other type and form of sensor used for providing data on the medical instruments 114, or the data capture devices (e.g., optical devices). For example, the data capture device 112 can include a location sensor, a distance sensor or a positioning sensor providing coordinate locations of a medical instrument 114 (e.g., kinematics data). The data capture device 112 can include a sensor providing information or data on a location, position or spatial orientation of an object (e.g., medical instrument 114 or a lens of data capture device 112) with respect to a reference point for kinematics data. The reference point can include any fixed, defined location used as the starting point for measuring distances and positions in a specific direction, serving as the origin from which all other points or locations can be determined.

[0037] The display 118 can show, illustrate or play the data stream 122, such as a video stream, in which the medical instruments 114 at or near surgical sites are shown. For example, the display 118 can display a rectangular image of a surgical site along with at least a portion of the medical instruments 114 being used to perform surgical tasks. The display 118 can provide compiled or composite images generated by the visualization tool 116 from a plurality of data capture devices 112 to provide a visual feedback from one or more points of view.

[0038] The visualization tool 116 can be configured or designed to receive any number of different data streams 122 from any number of data capture devices 112 and combine them into one or more data streams displayed on a display 118. The visualization tool 116 can be configured to receive a plurality of data stream components and combine the plurality of data stream components into a single data stream 122. For instance, the visualization tool 116 can receive visual sensor data from one or more of the medical instruments 114, sensors or cameras with respect to a surgical site or an area in which a surgery is performed. The visualization tool 116 can incorporate, combine or utilize multiple types of data (e.g., positioning data of a medical instrument 114 along sensor readings of pressure, temperature,vibration or any other data) to generate an output to present on a display 118. The visualization tool 116 can present locations of medical instruments 114 along with locations of any reference points or surgical sites, including locations of anatomical parts of the patient (e.g., organs, glands or bones).

[0039] The medical instruments 114 can be any type and form of tool or medical instrument used for surgery, medical procedures or a tool in an operating room or environment. The medical instrument 114 can be imaged by, associated with, or include an image capture device. For instance, a medical instrument 114 can be a tool for making incisions, a tool for suturing a wound, an endoscope for visualizing organs or tissues, an imaging device, a needle and a thread for stitching a wound, a surgical scalpel, forceps, scissors, retractors, graspers, or any other tool or medical instrument to be used during a surgery. The medical instruments 114 can include hemostats, trocars, surgical drills, suction devices or any medical instruments for use during a surgery. The medical instrument 114 can include other or additional types of therapeutic or diagnostic medical imaging implements. The medical instrument 114 can be configured to be installed in, coupled with, or manipulated by an RMS 120, such as by manipulator arms or other components for holding, using and manipulating the medical instruments. In some embodiments, the medical instruments 114 can be the same as, or similar to, the medical instruments discussed with respect to FIG. 4.

[0040] The RMS 120 can be a computer-assisted system configured to perform a surgical or medical procedure or activity on a patient via, or using or with the assistance of, one or more robotic components or the medical instruments 114. The RMS 120 can include any number of manipulator arms for grasping, holding or manipulating various medical instruments 114 and performing computer-assisted medical tasks using the medical instruments 114 controlled by the manipulator arms.

[0041] The data streams 122 can be generated by the RMS 120. For instance, sensor data associated with the data streams 122 can include images (e.g., video images) captured by a medical instrument 114 and can be sent to the visualization tool 116. For instance, a display 118 (e.g., a touchscreen) can be used by a surgeon to select, engage, or configure a particular medical instrument 114, thereby triggering an event that can be indicated or included in data packets of a data stream 122. The RMS 120 can include one or more input ports to receive direct or indirect connection of one or more auxiliary devices. For example, the visualization tool 116 can be connected to the RMS 120 to receive the images from the medical instrument 114 when the medical instrument 114 is installed in the RMS 120 (e.g., on a manipulator armfor handing medical instruments 114). For example, the data stream 122 can include data indicative of positioning and movement of the medical instruments 114 that can be captured or identified by data packets of a kinematics data. The visualization tool 116 can combine the data stream components from the data capture devices 112 and the medical instrument 114 into a single combined data stream 122 which can be indicated or presented on a display 118. The RMS 120 can provide the data streams 122 to the data processing system 130 periodically, continuously, or in real-time.

[0042] Data packets can include a unit of data in a data stream 122. The data packets can include the actual information being sent and metadata, such as a source and a destination address, a port identifier or any other information for transmitting data. The data packets can include a data (e.g., a payload) corresponding to an event (e.g., installation, uninstallation, engagement or setup of a medical instrument 112). The data packets can include data corresponding to sensor information (e.g., a video frame captured by a camera), or data on movement of a medical instrument 114. The data packets can be transmitted in the data streams 122 that can be separated or combined. For instance, a data stream 122 for kinematics data (e.g., a kinematics data stream) can include a plurality of data packets indicative of movement of robotic system components or features.

[0043] Data packets can include one or more timestamps, which can indicate a particular time when particular events took place. Timestamps can include time indications expressed in any combination of nanoseconds, microseconds, milliseconds, seconds, hours, days, months or years. Timestamps can be included in the payload or metadata of data packets and can indicate the time when a data packet was generated, the time when the data packet was transmitted from the device that generated the data packet, the time when the data packet was received by another device (e.g., a system within the RMS 120, the data processing system 130, the computing device 150 or another device on a network) or a time when the data packet is stored into a data repository 132.

[0044] The data repository 132 can include one or more data files, data structures, arrays, values, or other information that facilitates operation of the data processing system 130. The data repository 132 can include one or more local or distributed databases and can include a database management system. The data repository 132 can include, maintain, or manage one or more data streams 122. The data streams 122 can include or be formed from one or more of a video stream, image stream, stream of sensor measurements, event stream, or kinematics stream. The data streams 122 can include data collected by one or more data capture devices112, such as a set of 3D sensors from a variety of angles or vantage points with respect to the procedure activity (e.g., point or area of surgery).

[0045] The data stream 122 can include any stream of data. The data stream 122 can include a video stream, including a series of video frames or organized into video fragments, such as video fragments of about 1, 2, 3, 4, 5, 10 or 15 seconds of a video. Each second of the video can include, for example, 30, 45, 60, 90, 120, 240, or the like video frames per second. The data streams 122 can include an event stream which can include a stream of event data or information, such as packets, which identify or convey a state of the RMS 120 or an event that occurred in association with the RMS 120. For example, data stream 158 can include any portion of system configuration 122, including information on operations on data streams 122, data on installation, uninstallation, calibration, set up, attachment, detachment or any other action performed by or on an RMS 120 with respect to the medical instruments 114.

[0046] The data stream 122 can include data about an event, such as a state of the RMS 120 indicating whether the medical instrument 114 is calibrated, adjusted or includes a manipulator arm installed on the RMS 120. A data stream 122 representing event data (e.g., event data stream) can include data on whether an RMS 120 was fully functional (e.g., without errors) during the procedure. For example, when a medical instrument 114 is installed on a manipulator arm of the RMS 120, a signal or data packet(s) can be generated indicating that the medical instrument 114 has been installed on the manipulator arm of the RMS 120.

[0047] The data stream 122 can include a stream of kinematics data which can refer to or include data associated with one or more of the manipulator arms or medical instruments 114 attached to the manipulator arms, such as arm locations or positioning. The data corresponding to the medical instruments 114 can be captured or detected by one or more displacement transducers, orientational sensors, positional sensors, or other types of sensors and devices to measure parameters or generate kinematics information. The kinematics data can include sensor data along with time stamps and an indication of the medical instrument 114 or type of medical instrument 114 associated with the data stream 122.

[0048] The data repository 132 can store sensor data having video frames that can include one or more static images or frames extracted from a sequence of images of a video file. Video frame can represent a specific moment in time and can be identified by a metadata including a timestamp. Video frames can display visual content of the video of a medical procedure being analyzed by the data processing system 130 to form a composite video along with performance metrics indicative of the performance of the surgeon performing theprocedure. For example, in a video file capturing a robotic surgical procedure, a video frame can depict a snapshot of the surgical task, illustrating a movement or usage of a medical instrument 114 such as a robotic arm manipulating a surgical tool within the patient's body.

[0049] The data streams 122 corresponding to sensor data (e.g., videos), events, and kinematics can include related, corresponding or duplicate information that can be used for cross-data comparisons and verification that all three data sources are in agreement. For instance, the detection function can implement a check for consistency between diverse data types and data sources by mapping and comparing timestamps between different data types to facilitate if they consistently progress over time, such as in accordance with expected flow and correlation of events, video stream details and kinematics values.

[0050] For example, an installation of a medical instrument 114 can be recorded as a system event and provided in a data stream 122 of events data. At the same or similar expected time frame, the installed medical instrument 114 can shows up in a sensor data (e.g., in a video) which can be detected by the data processing system 130, which can include a computer vision model. Kinematics data can confirm movements of the medical instrument 114 according to the movements detected by the data processing system 130. Using these cross-data stream correlation techniques, the data processing system 130 can verify time synchronization across the three data sources (e.g., three data streams 122).

[0051] With continued reference to FIG. 1, the data processing system 130 can include any combination of hardware or software that performs one or more of the functions described herein. For example, the data processing system 130 can include any combination of hardware and software for generation of graphical user interfaces representing teleoperation of robotic systems. The data processing system 130 can include any computing device (e.g., a computing device that is the same as, or similar to, the computing device 500 of FIG. 5) and can include one or more servers, virtual machines, or can be part of or include a cloud computing environment. The data processing system 130 can be provided via a centralized computing device or be provided via distributed computing components, such as including multiple, logically grouped servers and facilitating distributed computing techniques. The logical group of servers can be referred to as a data center, server farm or a machine farm. The servers, which can include virtual machines, can also be geographically dispersed. A data center or machine farm can be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous - one or more of the servers or machines can operate according to one or more type of operating system platform.

[0052] The data processing system 130, or components thereof can include a physical or virtual computer system operatively coupled, or associated with, the medical environment 110. The data processing system 130, or components thereof, can be coupled, or associated with, the medical environment 110 via a network 105, either directly or indirectly through an intermediate computing device or system. The network 105 can be any type or form of network. The geographical scope of the network can vary widely and can include a body area network (BAN), a personal area network (PAN), a local-area network (LAN) (e.g., Intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 105 can assume any form such as point-to-point, bus, star, ring, mesh, tree, etc. The network 105 can utilize different techniques and layers or stacks of protocols, including, for example, the Ethernet protocol, the internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, the SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP / IP internet protocol suite can include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network 105 can be a type of a broadcast network, a telecommunications network, a data communication network, a computer network, a Bluetooth network, or other types of wired and wireless networks.

[0053] The data processing system 130, or components thereof, can be located at least partially at the location of the surgical facility associated with the medical environment 110 or remotely therefrom. Elements of the data processing system 130, or components thereof can be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The data processing system 130, or components thereof, can include other or additional elements that can be considered desirable to have in performing the functions described herein. The data processing system 130, or components thereof, can include, or be associated with, one or more components or functionality of a computing device including, for example, one or more processors coupled with memory that can store instructions, data or commands for implementing the functionalities of the data processing system 130 discussed herein.

[0054] The data processing system 130 can include one or more of a data repository 132, data collector 134, a data stream parser 136, a graphical user interface (GUI) element generator 138, and or an arrangement generator 140. The data processing system 130 can be communicatively coupled with one or more data processing systems 130. For example, the data processing system 130 can be communicatively coupled with one or more other dataprocessing systems 130 that operate in cooperation to perform one or more of the operations described herein.

[0055] The data processing system 130 can be implemented by one or more components of the medical environment 110. For example, the data processing system 130 can be implemented by one or more components of the RMS 120. The data processing system 130 can receive one or more data streams 122 that are described herein, and can monitor operation of the RMS 120 using the system configurations 124. One or more RMSs 120 can be communicatively coupled with the one or more data processing systems 130. The data repository 132 can be configured to receive, store, and provide the data streams 122 (e.g., one or more data packets associated with the data streams 122) before, during, or after a medical procedure.

[0056] The data repository 132 can be implemented by the data processing system 130 or can be a device that is the same as, or similar to, the computing device 500 of FIG. 5. The data repository 132 can receive data from any of the devices of FIG. 1, either directly or indirectly (e.g., via the data processing system 130). The data can include the data streams 122. In examples, the data stored by the data repository 132 is associated with a previously- performed medical procedure. In some examples, the data stored by the data repository 132 is associated with a current medical procedure. In some embodiments, the data repository 132 can receive the data streams 122 or the system configurations 124 and store the data streams 122 or the system configurations 124. The data repository 132 can provide the data streams 122 or the system configurations 124 (e.g., one or more data packets thereof) to the one or more of the components of the data processing system 130. In some embodiments, the data repository 132 stores data associated with one or more of data streams 122 generated during operation of at least one component of the RMS 120 or data generated by the data stream parser 136, the GUI element generator 138, or the arrangement generator 140.

[0057] The data collector 134 can be implemented by the data processing system 130 or can be a device that is the same as, or similar to, the computing device 500 of FIG. 5. In some embodiments, the data collector 134 can receive the data streams 122. For example, the data collector 134 can receive the data streams 122 via the network 105. In some embodiments, the data collector 134 can receive the data streams 122 of a medical procedure performed with the RMS 120. For example, the data collector 134 can receive the data streams 122 where the data streams 122 include data (e.g., packets or the like) generated by one or more devices of the RMS 120 during the medical procedure. In some embodiments, the one or more packets associated with the data streams 122 can include (e.g., represent) one or moreimages (e.g., single images or sets of images representing a video stream) during a medical procedure or one or more movements of the medical instruments 114 or MTMs of the RMS 120 engaged by an individual during the medical procedure to at least in part cause movement of the medical instruments 114. The MTMs can include one or more devices configured to be grasped by the individual and moved within an area along six degrees of freedom and the movements can be tracked by the RMS 120 to at least in part cause the medical instruments 114 to move in accordance with the movements of the MTMs.

[0058] In some embodiments, the one or more images of the medical procedure can be captured or otherwise obtained by the data capture devices 112 or the visualization tool 116. In some embodiments, the images included in the data streams 122 can be generated by an imaging device that is supported by a distal portion of a medical instrument 114 (e.g., an endoscope or the like). The one or more images can represent one or more anatomical structures or portions of the one or more medical instruments 114 as describe herein. The one or more movements of the medical instruments 114 or the MTMs can be associated with kinematics data. For example, as the medical instruments 114 are moved based at least in part on motion of one or more MTMs, the changes in position of the instruments or MTMs relative to one or more reference frames can be recorded as positions at points in time during a period of time. In this example, the RMS 120 can generate the kinematics data, where the kinematics data is associated with the movements (e.g., changes in position) of the medical instruments 114 or the MTMs. The movements of the medical instruments 114 or the MTMs can also be associated with (e.g., correspond to) the one or more images. In some embodiments, the RMS 120 can include the kinematics data in the data streams 122. In some embodiments, the RMS 120 can include an indication of a type of medical procedure performed (e.g., a cholecystectomy or the like) in the data streams 122. In some embodiments, the RMS 120 can include an indication of a skill level of a surgeon involved in the medical procedure in the data streams 122. In some embodiments, the data collector 134 can provide the data streams 122 (e.g., one or more packets of the data streams 122) to the data repository 132, the data stream parser 136, the GUI element generator 138, or the arrangement generator 140.

[0059] In some embodiments, the kinematics data can represent movement of the medical instruments 114 or the MTMs. For example, the kinematics data can be associated with one or more of: position data associated with positions of portions (e.g., distal portions) of the medical instruments 114 or MTMs involved in the medical procedurejoint data associated with the joint angle usable to derive a position and location of a distal portion of a medicalinstrument 114 or MTMs involved in the medical procedure, or the like. In some embodiments, the kinematics data can be derived by the data processing system 130. For example, the kinematics data can be derived by the data processing system 130 based at least in part on camera calibration parameters and the application of computer vision-based techniques to detect instruments captured by the monoscopic views on one or more monoscopic views. In some embodiments, the kinematics data can be derived from one or more stereoscopic views to reconstruct the 3D workspace from inferred camera calibration parameters. In some embodiments, the kinematics data can be derived from one or more deep learning models to infer the 3D workspace from monoscopic or stereoscopic views.

[0060] In some embodiments, the data stream can include a system event stream. The system event stream can be determined based at least in part on indications received from one or more sensors of the RMS 120 indicating one or more of: a medical instrument 114 (e.g., tool) installation or uninstallation, an identifier of a medical instrument 114, camera movement of one or more cameras of the data capture devices 112, indications of electrocautery use initiated by a surgeon by engaging one or more buttons of the MTMs, ultrasonic or other energy initiated by a surgeon by engaging one or more buttons of the MTMs, tissue sampling, or a swap between medical instruments 114 (e.g., a swap between one or more instrument arms).

[0061] In some embodiments, the data collector 134 can update (e.g., add to) the data streams 122 by generating one or more three-dimensional (3D) plots (also referred to as “plots”), the plots representing one or more paths of the medical instruments 114 or MTMs during at least portions of the medical procedure. For example, the data collector 134 can generate the plots based at least in part on positions of the medical instruments 114 or MTMs represented by the data included in the data streams 122. In examples, the data collector 134 can execute one or more clustering algorithms to determine one or more regions within a reference frame where movement of at least a portion of the medical instruments 114 or the MTMs is clustered. For example, during a segment of a medical procedure, the data collector 134 can generate the plots based at least in part on the medical instruments 114 operating in a region of the reference frame (e.g., in a particular region such as within an insufflated portion of a patient). In examples, during the segment of a medical procedure, the data collector 134 can generate the plots based at least in part on the MTMs operating in a region of a workstation supporting the MTMs.

[0062] In some embodiments, the data collector 134 can implement one or more models that generate the plots described herein. For example, the data collector 134 can implementone or more generative machine learning models such as generative adversarial networks (GANs) or the like. In examples, the data collector can provide data associated with the data streams 122 to the models to cause the models to generate outputs, the outputs representing two-dimensional (2D) or 3D representations of at least portions of the data streams 122. The 2D or 3D representations of the data streams can include paths traversed by the medical instruments 114 or the MTMs during segments or the entire the medical procedure. In some embodiments, the data collector 134 can provide inputs to the models to update (e.g., control) the outputs of the models. For example, the data collector 134 can provide data indicating speeds, rates of acceleration, or the like of the medical instruments 114 or MTMs to cause the models to update the paths associated with the outputs of the models. In these examples, the updates to the paths associated with the outputs of the model can represent a heatmap for movement of the medical instruments 114 or MTMs.

[0063] In some embodiments, the data collector 134 can overlay information (e.g., as text, icons, or the like) onto one or more plots. For example, the data collector 134 can overlay information to indicate points in time at which one or more events occurred. An example can include the data collector 134 overlaying information to indicate points in time when energy was applied by the medical instruments 114 and when energy application ended.

[0064] In some embodiments, the data collector 134 can generate the one or more 3D plots based at least in part on the data collector 134 associating portions of the data streams 122 with one or more phases of the medical procedure. The one or more phases can correspond to a surgical ontology describing, for example, setup (e.g., initialization of the RMS 120), performance of operations at certain steps of the medical procedure (e.g., ligation during dissection of Calot’s triangle, or the like), and closing (e.g., suturing anatomical structures of the patient involved in the medical procedure). In some embodiments, the data collector 134 can associate portions of the data included in the data streams 122 with an instrument type of one or more medical instruments 114 involved in the medical procedure. For example, the data collector 134 can associate portions of the data included in the data streams 122 with an instrument type of one or more medical instruments 114 as specified by the data streams 122.

[0065] In some embodiments, the data collector 134 can update the data included in the data streams 122 based at least in part on registration of devices of the medical environment 110 with a reference frame. For example, the data collector 134 can update the data included in the data streams 122 based at least in part on registration of portions (e.g., the linkages or the like) of the medical instruments 114 relative to a global coordinate system such thatmedical instruments 114 can move relative to the patient and such motion can be determined during a replay. In examples, the data collector 134 can update the data included in the data streams 122 based at least in part on registration of devices in a medical environment 110 where the devices are fixed or not fixed relative to one another prior to, during, or after the medical procedure.

[0066] In some embodiments, the data collector 134 can preprocess the data included in the data streams 122. For example, the data collector 134 can preprocess the data included in the data streams 122 based at least in part on the data collector 134 determining the data used to generate plots and calculate metrics. In this example, the data used to generate the plots and calculate the metrics can be preprocessed based at least in part on camera calibration parameters and object detection techniques for identifying the 3D position of the instruments within the workspace. In examples, the data collector 134 can preprocess the data based at least in part on stereoscopic imaging devices being included in the data capture devices 112. In this example, stereoscopic imaging devices (and corresponding techniques for determining, for example, distances to objects relative to the data capture devices 112) can be included in the data capture devices 112. In these examples, the data collector 134 can determine positions of the medical instruments 114 relative to a reference frame based at least in part on the data generated or derived from the stereoscopic imaging devices.

[0067] In some embodiments, the data collector 134 can preprocess the data included in the data streams 122 based at least in part on the data collector 134 determining a location of one or more of the medical instruments 114 relative to a reference frame. For example, the data collector 134 can preprocess the data included in the data streams 122 based at least in part on one or more simultaneous localization and mapping (SLAM) techniques. In this example, the data collector 134 can derive positions of the medical instruments 114, anatomical structures, or the like relative to the reference frame by implementing the SLAM techniques.

[0068] In some embodiments, the data collector 134 can preprocess the data included in the data streams 122 by updating the data (e.g., the data associated with the images, kinematics, etc.) generated during the medical procedure. For example, the data stream collector 134 can preprocess the data included in the data streams 122 by updating (e.g., removing) data associated with movement (e.g., of the medical instruments 114 when clutched or the like) as the data capture devices 112 are moved. In examples, the data stream collector 134 can preprocess the data included in the data streams 122 by removing OPI values generated during movement that exceed a predetermined threshold (e.g., a spike inrotations per minute of one or more drives as the one or more drives initially start movement).

[0069] The data stream parser 136 can be implemented by the data processing system 130 or can be a device that is the same as, or similar to, the computing device 500 of FIG. 5. In some embodiments, the data stream parser 136 can receive the data streams 122. For example, the data stream parser 136 can receive the data streams 122 from the RMS 120, the data repository 132, or the data collector 134. In examples, the data stream parser 136 can split the data streams into a plurality of segments. For example, the data stream parser 136 can split the data streams 122 into the plurality of segments based at least in part on one or more periods of time during which the medical procedure was performed. The periods of time can include predetermined periods of time or dynamically determined periods of time (e.g., periods of time corresponding to phases of medical procedures, described herein).

[0070] In some embodiments, the data stream parser 136 can split the data streams 122 into the plurality of segments based at least in part on one or more phases of a medical procedure. For example, the data stream parser 136 can split the data streams 122 into the plurality of segments based at least in part on periods of time for each phase of the medical procedure. In some embodiments, the data stream parser 136 can identify a type of an anatomical structure on which a medical procedure is performed or one or more operations being performed by the medical instruments 114 and determine the phases of the medical procedure based at least in part on the anatomical structure or the operations being performed. In this example, the data stream parser 136 can split the data streams 122 into the plurality of segments based at least in part on the phases of the medical procedure.

[0071] In some embodiments, the data stream parser 136 can identify the type of the anatomical structure on which a medical procedure is performed based at least on the data streams 122 and an ML model. For example, the data stream parser 136 can provide the data streams 122 (or portions thereof such as images included in the data streams 122) to the ML model to cause the ML model to provide an output, the output representing the type of the anatomical structure on which the medical procedure is performed. In some embodiments, the data stream parser 136 can parse the data streams 122 based at least in part on the anatomical structures or the portions of the medical procedure determined by the data stream parser 136. For example, the data stream parser 136 can parse the data streams 122 based at least in part on the phases of the medical procedure as determined by the data stream parser. In this example, the beginning and end of each segment of the data stream can correspond to inflection points reflecting changes in the phase of the medical procedure.

[0072] In some embodiments, the ML model can be trained to generate outputs through one or more supervised, semi-supervised, or unsupervised training techniques. For example, the ML model can be trained based at least in part on the data stream parser 136 receiving a plurality of images or video segments and corresponding tags indicating the type(s) of anatomical structures, staring points and stopping points of one or more segments of the medical procedure corresponding to specific tasks involved in the medical procedure, etc., as represented by the plurality of images or video segments and providing the images or video segments, and corresponding tags, to the ML model to cause the ML model to generate an output. The output can be compared to the one or more corresponding tags and the data stream parser 136 can update weights of the ML model based at least in part on a difference between the output and the corresponding tags. This process can be iteratively performed until the ML model achieves a threshold degree of accuracy when generating outputs indicating the type(s) of anatomical structures represented by the plurality of images or video segments.

[0073] In some embodiments, the data processing system 130 can receive data associated with a selection of a segment of the plurality of segments of the data stream 122. For example, the data processing system 130 can receive the data associated with the selection of the segment from the computing device 150. In this example, the computing device 150 can generate the data associated with the selection of the segment based at least in part on the computing device 150 receiving input. In some embodiments, the computing device 150 can receive input based at least in part on an individual interacting with the input device 154. For example, the computing device 150 can receive input based at least in part on an individual interacting with the input device 154 to provide one or more of: a selection of a segment from among the plurality of segments, selection of criteria associated with the medical procedure, or the like. In some embodiments, the criteria associated with the medical procedure can include a period of time during the medical procedure, identifiers of one or more segments of the medical procedure, a playback speed to control playback of the medical procedure, or the like. The input can be provided in response to display of one or more icons on the display device 152 of the computing device 150. As described herein, the GUI element generator 138 can generate the first GUI element or the second GUI element based at least in part on the data streams 122, the plurality of segments, the selected segment from among the plurality of segments, or the criteria associated with the medical procedure.

[0074] The GUI element generator 138 can generate a first GUI element. For example, the GUI element generator 138 can generate the first GUI element based at least in part onthe data associated with the segment of the data stream 122 selected from among the plurality of segments. In some embodiments, the first GUI element can include a portion of a GUI that is configured to display one or more images. For example, the first GUI element can include a portion of a GUI that is configured to display the images or the video stream corresponding to the segment of the plurality of segments.

[0075] In some embodiments, the GUI element generator 138 can generate a second GUI element. For example, the GUI element generator 138 can generate the second GUI element based at least in part on the data associated with the segment of the data stream 122 selected from among the plurality of segments. In some embodiments, the second GUI element can include a representation of motion of one or more of the medical instruments 114 or the MTMs involved in controlling the medical instruments during the corresponding segment of the medical procedure. In this example, the second GUI element can include a representation of at least one path taken by one or more medical instruments 114 or MTMs during at least a segment of the medical procedure.

[0076] In some embodiments, the second GUI element includes a representation of a plurality of paths. For example, the second GUI element can include a representation of a plurality of paths such that each of the paths are located in spaced relation relative to each of the other paths of the plurality of paths (e.g., in a vertical list of paths). In an example, the second GUI element can include a plurality of areas (e.g., regions within the second GUI element), and each area can correspond to a path of the plurality of paths. In some embodiments, each area that corresponds to a path of the plurality of paths can include a selectable region. For example, each area that corresponds to a path of the plurality of paths can include a selectable region that, when selected by an individual operating the input device 154 of the computing device, cause the first GUI element or the second GUI element to be generated in accordance with the data associated with the data streams 122 representing a corresponding point or period of time. In some embodiments, as described herein, in response to input received by the input device 154 indicating selection of one or more of the selectable regions, the data processing system 130 can generate a third GUI associated with the plots of the medical procedure.

[0077] The arrangement generator 140 can generate an arrangement. The arrangement can identify a position of the first GUI element and the second GUI element. In some embodiments, the arrangement can identify the position of the first GUI element and the second GUI element relative to one another within a GUI to be displayed on a display device. For example, the arrangement can identify the position of the first GUI element and thesecond GUI element relative to (e.g., juxtaposed with or the like) one another within a GUI to be displayed on the display device 152 of the computing device 150. In some embodiments, the arrangement generator 140 can provide the arrangement for display via a GUI. For example, the arrangement generator 140 can provide the arrangement for display via the GUI where the arrangement includes the first GUI element and the second GUI element. In this example, the first GUI element and the second GUI element can be synchronized. In examples, the first GUI element and portions of the second GUI element can be synchronized such that, as the GUI continues through the progression of one or more time steps, the path(s) represented by the second GUI element can be generated or updated so as to indicate the path taken by (e.g., movement of) the instruments or MTMs involved in the medical procedure at the one or more time steps.

[0078] In some embodiments, the arrangement generator 140 can generate data associated with the arrangement, where the data is configured to cause the arrangement to be displayed (e.g., as a GUI). In an example, the arrangement generator 140 can generate data associated with the arrangement, where the data is configured to cause the arrangement including the first GUI element and the second GUI element to be displayed via the display device 152 of the computing device 150. In examples, the data can be configured to cause the arrangement including the first GUI element and the second GUI element to be displayed via the display device 152 of the computing device 150 along with a third GUI element. As described herein, the third GUI element can include a region (e.g., pop-up window or the like) at least in part overlaid on the first GUI element or the second GUI element, the region including a representation of one or more paths involved in the segment of the plurality of segments. In some embodiments, the arrangement generator 140 can generate the third GUI element based at least in part on input received from an individual via the input device 154. For example, the arrangement generator 140 can generate the third GUI element based at least in part on input received via the input device 154, the input indicating selection of a selectable region of the one or more selectable regions.

[0079] In some embodiments, the arrangement generator 140 can generate the arrangement based at least in part on one or more objective performance indicators (OPIs) associated with the medical procedure. For example, the arrangement generator 140 can generate the arrangement based at least in part on one or more OPIs associated with the medical procedure such that the OPIs are included in at least a portion of a GUI as indicated by the arrangement. In this example, the arrangement generator 140 can determine the OPIs based at least in part on the data associated with the data streams 122. In examples, thearrangement generator 140 can obtain the OPIs from one or more other components of the data processing system 130. The OPIs can represent metrics associated with the operation of the at least one medical instrument 114 or the MTMs such as, for example, a rate of motion, a range of motion, a speed at which the medical instruments 114 are moving, a speed at which the MTMs are moving, a smoothness of movement of the medical instruments 114 or the MTMs, force applied by the medical instruments 114 to one or more objects (e.g., tissue or the like), an amount of distance traveled by the medical instruments 114 or the MTMs, a change in angles measured at one or more joints of the medical instruments 114, a rate of motion of a data capture device 112, a rate at which the MTMs are clutched (e.g., motion is suspended), a rate at which medical instruments 114 are swapped, a rate at which an energy pedal is engaged indicating a number of times the pedal is engaged or a duration of the engagement of the pedal), a console workspace volume, a rate (e.g., a proportion) of time spent by the individual(s) operating the medical instruments 114 with the RMS 120 suspended (e.g., “head-out”) as compared to not suspended, or the like. In some embodiments, the OPIs can be updated (e.g., adjusted) to indicate an average of one or more of the OPIs, a range (e.g., high / low) of the OPIs, a sum (total) of the OPIs, or the like. In some embodiments, the OPIs can be represented as numerical values, graphs charting the numerical values, or the like and can continuously be updated based at least in part on a point or period of time (e.g., during the playback of the medical procedure). The OPIs can be displayed for each medical instrument 114 or each MTM involved in the medical procedure. As described herein, the GUI can include a GUI that is displayed on a display device, such as the display device 152 of the computing device 150.

[0080] With continued reference to FIG. 1, the computing device 150 can include any combination of hardware or software that perform one or more of the functions described herein. For example, the computing device 150 can include any combination of hardware and software that receive and generate data associated with input received via the input device 154 and display a GUI in accordance with an arrangement via the display device 152. The computing device 150 can be the same as, or similar to, the computing device 500 of FIG. 5 or other computing devices described herein, and can include one or more tablets, laptops, desktops, servers, or virtual machines, or can be part of or include a cloud computing environment.

[0081] The computing device 150, or components thereof, can include a display device 152. The display device 152 can include any suitable display device such as a monitor, a touchscreen monitor, a liquid crystal display (LCD) monitor, a light emitting diode (LED)monitor, or the like. The computing device 150 can include an input device 154. The input device 154 can include any suitable input device 154 such as a keyboard, a mouse, a touchscreen, combinations thereof, or the like.

[0082] The computing device 150, or components thereof, can be located at least partially at the location of the surgical facility associated with the medical environment 110 or remotely therefrom. Elements of the computing device 150, or components thereof, can be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The computing device 150, or components thereof, can include other or additional elements that can be considered desirable to have in performing the functions described herein. The computing device 150, or components thereof, can include, or be associated with, one or more components or functionality of a computing including, for example, one or more processors coupled with memory that can store instructions, data or commands for implementing the functionalities of the computing device 150 discussed herein.

[0083] FIG. 2 is a flowchart diagram illustrating an example method 200 for the generation of graphical user interfaces representing teleoperation of robotic systems, according to some embodiments. The method 200 can be performed by one or more systems, devices, or components depicted in FIGS. 1, 3A-3E, 4, 5, and 6 including, for example, the data processing system 130 of FIG. 1.

[0084] At operation 210, a data stream of a medical procedure is received. For example, a data processing system (e.g., a data processing system that is the same as, or similar to, the data processing system 130 of FIG. 1) can receive the data stream of the medical procedure. In this example, a data collector of the data processing system can be implemented to receive the data stream of the medical procedure. In some implementations, the data stream can be stored in a data repository of the data processing system.

[0085] At operation 220, a data stream can be split into a plurality of segments. For example, the data processing system can split the data stream into a plurality of segments. In this example, a data stream parser of the data processing system can be implemented to split the plurality of segments as described herein. In some implementations, the plurality of segments can be stored in the data repository. For example, the plurality of segments can be stored in the data repository in associated with the data stream of the medical procedure.

[0086] At operation 230, a selection of a segment of the plurality of segments can be received. For example, the data processing system can receive data associated with (e.g., representing) the selection of the segment of the plurality of segments. In some implementations, the data processing system can receive the data associated with theselection of the segment from a computing device (e.g., a computing device that is the same as, or similar to, the computing device 150 of FIG. 1). For example, the computing device can receive input based on an individual interacting with an input device of the computing device. In this example, the computing device can generate the data associated with the selection of the segment and provide (e.g., transmit) the data to the data processing system 130.

[0087] At operation 240, a first GUI element and a second GUI element can be generated. For example, the data processing system can generate the first GUI element and the second GUI element. In this example, a GUI element generator of the data processing system can be implemented to generate the first GUI element and the second GUI element.

[0088] At operation 250, an arrangement including the first GUI element and the second GUI element can be provided (e.g., transmitted or otherwise made available for download by a computing device). For example, the arrangement generator 140 can generate the arrangement including the first GUI element and the second GUI element. In this example, an arrangement generator 140 of the data processing system 130 can provide the arrangement to one or more components illustrated in FIG. 1. In some embodiments, providing the arrangement can cause a display device (e.g., a display device that is the same as, or similar to, the display device 152 of FIG. 1) to output one or more of: images or a video stream of the medical procedure (e.g., via the first GUI element); one or more paths forming a plot that represents motion of the medical instruments 114 or the MTMs an entire medical procedure or during segments of the medical procedure; or one or more OPIs associated with the medical procedure (e.g., as strings of text, graphs, or the like). In some embodiments, the arrangement can cause the display device to provide the arrangement based at least in part on a playback speed (e.g., at half-speed (l / 2x), at a real-time speed (lx), at two-times (2x), or the like).

[0089] In some implementations, the data processing system 130 can generate data associated with the arrangement, where the data is configured to cause the display device to generate an output based at least in part on the arrangement. The output can visually include plots having one or more paths that represent movement of the medical instruments 114 or the MTMs during the medical procedure during an entire medical procedure or during segments of the medical procedure. In some implementations, the output can include one or more visual indications of the OPIs determined by the data processing system 130. For example, the output can include one or more strings of text, one or more graphs, or the likethat represent the OPIs of the entire medical procedure or of segments of the medical procedure.

[0090] In some embodiments, the data processing system 130 can receive data associated with input based at least in part on (e.g., in response to) the display device 152 generating the output in accordance with the arrangement. For example, the data processing system 130 can receive the data associated with the input where the input represents selection of one or more of the criteria associated with the medical procedure described herein. In some embodiments, the input can include criteria associated with the medical procedure where the criteria is configured to cause the data processing system 130 to update the GUI elements. For example, the input can involve an individual adjusting a period of time (e.g., by providing numerical updates or by toggling sliders displayed on the display device 152) such that portions of the replayed video stream or portions of the paths forming the plot are extended or trimmed to match the period of time.

[0091] FIGS. 3A-3E are an example sequence of GUIs 300a-300f, according to some embodiments. In some embodiments, the GUIs can be generated by a data processing system that is the same as, or similar to, the data processing system 130 of FIG. 1. The GUIs 300a- 300f can be displayed via a display device. For example the GUIs 300a-300f can be displayed via a display device that is the same as, or similar to, the display 118 or the display device 152 of FIG. 1. In some embodiments, one or more of the GUIs 300a-300f can be updated based at least in part on inputs provided by an individual such as a surgeon reviewing the medical procedure represented by the GUIs 300a-300f. The inputs can be received via an input device that is the same as, or similar to, the input device 154 of FIG. 1. In some embodiments, the GUIs 300a-300f can be the same as, similar to, or share similar features between one another.

[0092] FIG. 3 A illustrates a first GUI 300a. The first GUI 300a can include a navigation region 302. The navigation region can include one or more selectable strings of text that, when selected, cause the GUI 300a to transition to a different GUI of the one or more GUIs 300a-300f. The first GUI 300a can include one or more selectable regions 304. The one or more selectable regions 304 can be associated with (e.g., correspond to) one or more medical procedures that were performed (e.g., medical procedures involving an RMS that is the same as, or similar to, the RMS 120 of FIG. 1). In some embodiments, in response to selection of a selectable region 304 (e.g., “Cholecystectomy #36”) via the input at the input device, the data processing system can cause the display device to transition to GUI 300b of FIG. 3B.

[0093] FIG. 3B illustrates a second GUI 300b. The second GUI 300b can include the navigation region 302, and an overview region 306. The overview region 306 can include one or more sub-regions 306a-306d that correspond to aspects (e.g., segments, OPIs, or the like) of the medical procedure. For example, a first sub-region 306a can include an identifier of the medical procedure represented by the overview region 306. A second sub-region 306b can include identifiers of one or more OPIs such as a console time (a total time in which a workstation was operated by an individual), an instrument active time (a total time during which instruments were operated by the workstation), an instrument count (a number of instruments involved in the medical procedure that were controlled by the workstation), and an instrument exchange count (a number of time the instruments were swapped). A third subregion 306c can include indications of aspects of the operation. For example, the third subregion 306c can include one or more timelines 306d corresponding to operation of MTMs. As illustrated, the timelines 306d include a first timeline corresponding to an MTM associated with a left hand of a surgeon and a second timeline corresponding to an MTM associated with a right hand of a surgeon. The timelines 306d can include one or more segments (delineated by color) indicating an instrument from among a plurality of instruments controlled by the respective MTMs. For example, the upper timeline 306d illustrates the MTM associated with the left hand as being associated with an instrument titled “FFB Maryland Bipolar” and the lower timeline 306d illustrates the MTM associated with the right hand as being associated with instruments titled “FFB ProGrasp Forcepts”, “Perm Cautery Hook”, and “30° endoscope”. The overview region 306 can include a selectable button 308 that, in response to selection via the input at the input device, causes the data processing system to transition to GUI 300c of FIG. 3C.

[0094] FIG. 3C illustrates a third GUI 300c. The third GUI 300c can include a first GUI element 310, a second GUI element 312, and a selectable button 314. The first GUI element 310 can include a region of the GUI 300c that is configured to display images or video representing at least segments of the medical procedure. The second GUI element 312 is configured to display one or more OPIs determined based at least in part on a data stream that is the same as, or similar to, the data streams 122 of FIG. 1 and generated for the medical procedure. As illustrated, the OPIs include a duration of the medical procedure, an average duration for similar medical procedures that are performed by the surgeon reviewing the medical procedure presented on the GUIs shown in FIGS. 3A-3F, and a reference average duration indicating an average duration for similar medical procedures performed based at least in part by one or more other surgeons. The OPIs also include an indication of forcesexerted during the medical procedure by one or more of the instruments involved in the medical procedure such as a maximum force (“3.74N” for a first instrument, “2.49N” for a second instrument) for the medical procedure, an average force (e.g., “1.15N” for the first instrument, “0.88N” for the second instrument) for the medical procedure, and a percentage of time during which the force(s) exerted exceeded a threshold force. The OPIs also include an indication of a count indicating the number of times one or more endoscopes were clutched. The third GUI 300c can include a selectable button 314 that, in response to selection via the input at the input device, causes the data processing system to transition to GUI 300d of FIG. 3D.

[0095] FIG. 3C illustrates a fourth GUI 300d. The fourth GUI 300d can include a first GUI element 316 and a second GUI element 318. The second GUI element 318 can include selectable regions 318a-318e. The first GUI element 316 can include a region of the GUI 300c that is configured to display images or video representing at least segments of the medical procedure. In some embodiments, the first GUI element 316 of FIG. 3D is the same as, or similar to, the first GUI element 310 of FIG. 3C. The second GUI element 318 can include an ordered list of selectable regions 318a-318e. Each region of the selectable regions 318a-318e can correspond to segments of the medical procedure. For example, the selectable region 318a can correspond to initial exposure, the selectable region 318b can correspond to dissection of Calot’s triangle, the selectable region 318c can correspond to ligation / division of a cystic duct, the selectable region 318d can correspond to ligation / division of a cystic artery, and the selectable region 318e can correspond to dissection of a gallbladder. In some embodiments, each selectable region 318a-318e can include a representation of one or more paths corresponding to a plot, the plot generated based at least in part on the segment of the medical procedure for the respective selectable region 318a-318e. In some embodiments, the selectable regions 318a-318e can include an indication of the periods of time (e.g., time stamps) during the medical procedure corresponding to each selectable region 318a-318e. In some embodiments, in response to selection via the input at the input device of a selectable region 318a-318e, the data processing system is caused to transition to GUI 300e of FIG. 3E.

[0096] FIG. 3E illustrates a fifth GUI 300e. The fifth GUI 300e can be the same as, or similar to, the fourth GUI 300d. In some embodiments, the fifth GUI 300e can include a third GUI element 320. The third GUI element 320 can include a region that is overlaid on top of at least a portion of the first GUI element 316 and the second GUI element 318. The third GUI element 320 can include a 3D representation of a plot including a plurality of paths. The plurality of paths can correspond to paths of medical instruments or MTMs involved with asegment of a medical procedure. In some embodiments, the third GUI element 320 can include a representation of a 3D graph indicating the movement of the medical instruments or MTMs within a reference frame (e.g., a world view frame). The 3D graph can indicate measurements (e.g., in centimeters, millimeters, or the like) along an X-axis, a Y-axis, and a Z-axis. In some embodiments, selection of a start button 322 can cause the third GUI element to display the video or the plot in accordance with a point in time associated with the selection of the start button 322. For example, the third GUI element can be updated such that the paths are generated in accordance with a playback speed to visually represent motion of the medical instruments or MTMs in space and time. In some embodiments, the third GUI element 320 can include a close button 324 that, in response to selection via the input at the input device, causes the data processing system to transition to the sixth GUI 300f of FIG. 3F.

[0097] In some embodiments, the sixth GUI 300f can be the same as, or similar to, the fourth GUI 300d of FIG. 3D. In some embodiments, the sixth GUI 300f can be advanced or reversed as compared to the fourth GUI 300d. For example, the sixth GUI 300f can be advanced to indicate a different point in time in a same segment as represented by the fourth GUI 300d or a different point in time of a different segment.

[0098] FIG. 4 is a diagram of a medical environment 400, according to some embodiments. The medical environment 400 can refer to or include a surgical environment or surgical system. The medical environment 400 can include a robotic medical system 424 (e.g., a robotic medical system that is the same as, or similar to, the RMS 120 of FIG. 1), a user control system 410, and an auxiliary system 415 communicatively coupled one to another. A visualization tool 420 can be connected to the auxiliary system 415, which in turn can be connected to the robotic medical system 424. Thus, when the visualization tool 420 is connected to the auxiliary system 415 and this auxiliary system is connected to the robotic medical system 424, the visualization tool can be considered connected to the robotic medical system. In some embodiments, the visualization tool 420 can be directly connected to the robotic medical system 424. A metric prediction system 162 can be connected to the user control system 410 which in turn can be connected to the robotic medical system 424.

[0099] The medical environment 400 can be used to perform a computer-assisted medical procedure with a patient 425. In some embodiments, surgical team can include a surgeon 430A and additional medical personnel 430B-430D such as a medical assistant, nurse, and anesthesiologist, and other suitable team members who can assist with the surgical procedure or medical session. The medical session can include the surgical procedure being performed on the patient 425, as well as any pre-operative (e.g., which can include setup of the medicalenvironment 400, including preparation of the patient 425 for the procedure), and postoperative (e.g., which can include clean up or post care of the patient), or other processes during the medical session. Although described in the context of a surgical procedure, the medical environment 400 can be implemented in a non-surgical procedure, or other types of medical procedures or diagnostics that can benefit from the accuracy and convenience of the surgical system.

[0100] The robotic medical system 424 can include a plurality of manipulator arms 435A-435D to which a plurality of medical instruments (e.g., the instruments described herein) can be coupled to, installed to, or supported by. In some embodiments, the plurality of manipulator arms 435A-435D can include one or more linkages. Each medical instrument can be any suitable surgical tool (e.g., a tool having tissue-interaction functions), imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or other suitable instrument that can be used for a computer-assisted surgical procedure on the patient 425 (e.g., by being at least partially inserted into the patient and manipulated to perform a computer-assisted surgical procedure on the patient). Although the robotic medical system 424 is shown as including four manipulator arms (e.g., the manipulator arms 435A-435D), in other embodiments, the robotic medical system can include greater than or fewer than four manipulator arms. Further, not all manipulator arms can have a medical instrument installed thereto at all times of the medical session. Moreover, in some embodiments, a medical instrument installed on a manipulator arm can be replaced with another medical instrument as suitable.

[0101] One or more of the manipulator arms 435A-435D or the medical instruments attached to manipulator arms can include one or more displacement transducers, orientational sensors, positional sensors, or other types of sensors and devices to measure parameters or generate kinematics information. One or more components of the medical environment 400 can be configured to use the measured parameters or the kinematics information to track (e.g., determine poses of) or control the medical instruments, as well as anything connected to the medical instruments or the manipulator arms 435A-435D.

[0102] The user control system 410 can be used by the surgeon 430 A to control (e.g., move) one or more of the manipulator arms 435A-435D or the medical instruments connected to the manipulator arms. To facilitate control of the manipulator arms 435A-435D and track progression of the medical session, the user control system 410 can include a display that can provide the surgeon 430A with imagery (e.g., high-definition 3D imagery) of a surgical site associated with the patient 425 as captured by a medical instrument installed toone of the manipulator arms 435A-435D. The user control system 410 can include a stereo viewer having two or more displays where stereoscopic images of a surgical site associated with the patient 425 and generated by a stereoscopic imaging system can be viewed by the surgeon 430A. In some embodiments, the user control system 410 can also receive images from the auxiliary system 415 and the visualization tool 420.

[0103] The surgeon 430A can use the imagery displayed by the user control system 410 to perform one or more procedures with one or more medical instruments attached to the manipulator arms 435A-435D. To facilitate control of the manipulator arms 435A-435D or the medical instruments installed thereto, the user control system 410 can include a set of controls. These controls can be manipulated by the surgeon 430Ato control movement of the manipulator arms 435A-435D or the medical instruments installed thereto. The controls can be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 430Ato allow the surgeon to intuitively perform a procedure on the patient 425 using one or more medical instruments installed to the manipulator arms 435A-435D.

[0104] The auxiliary system 415 can include one or more computer systems (e.g., computing devices that are the same as, or similar to the computing device 600 of FIG. 6) configured to perform processing operations within the medical environment 400. For example, the one or more computer systems can control or coordinate operations performed by various other components (e.g., the robotic medical system 424, the user control system 410) of the medical environment 400. A computer systems included in the user control system 410 can transmit instructions to the robotic medical system 424 by way of the one or more computing devices of the auxiliary system 415. The auxiliary system 415 can receive and process image data representative of imagery captured by one or more imaging devices (e.g., medical instruments) attached to the robotic medical system 424, as well as other data stream sources received from the visualization tool. For example, one or more image capture devices can be located within the medical environment 400. These image capture devices can capture images from various viewpoints within the medical environment 400. These images (e.g., video streams) can be transmitted to the visualization tool 420, which can then passthrough those images to the auxiliary system 415 as a single combined data stream. The auxiliary system 415 can then transmit the single video stream (including any data stream received from the medical instrument s) of the robotic medical system 424) to present on a display of the user control system 410.

[0105] In some embodiments, the auxiliary system 415 can be configured to present visual content (e.g., the single combined data stream) to other team members (e.g., themedical personnel 43 OB-43 OD) who can not have access to the user control system 410. Thus, the auxiliary system 415 can include a display 640 configured to display one or more user interfaces, such as images of the surgical site, information associated with the patient 425 or the surgical procedure, or any other visual content (e.g., the single combined data stream). In some embodiments, display 440 can be a touchscreen display or include other features to allow the medical personnel 43 OB-43 OD to interact with the auxiliary system 415.

[0106] The robotic medical system 424, the user control system 410, and the auxiliary system 415 can be communicatively coupled one to another in any suitable manner. For example, in some embodiments, the robotic medical system 424, the user control system 410, and the auxiliary system 415 can be communicatively coupled by way of control lines 445, which can represent any wired or wireless communication link as can serve a particular implementation. Thus, the robotic medical system 424, the user control system 410, and the auxiliary system 415 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0107] It is to be understood that the medical environment 400 can include other or additional components or elements that can be needed or considered desirable to have for the medical session for which the surgical system is being used.

[0108] FIG. 5 is a block diagram depicting an architecture for a computing device 500 that can be employed to implement elements of the systems and methods described and illustrated herein, including aspects of the systems depicted in FIGS. 1, 3, or 4, and the method depicted in FIG. 2. For example, the data processing system 130, the data collector 134, the data stream parser 136, the GUI element generator 138, the arrangement generator 140, the computing device 150, or the devices described with respect to medical environment 400 can include one or more component or functionality of computing device 500. The computing device 500 can be any computing device used herein and can include or be used to implement a data processing system or its components. The computing device 500 includes at least one bus 505 or other communication component or interface for communicating information between various elements of the computer system. The computer system further includes at least one processor 510 or processing circuit coupled to the bus 505 for processing information. The computing device 500 also includes at least one main memory 515, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 505 for storing information, and instructions to be executed by the processor 510. The main memory 515 can be used for storing information during execution of instructions by the processor510. The computing device 500 can further include at least one read only memory (ROM) 520 or other static storage device coupled to the bus 505 for storing static information and instructions for the processor 510. A storage device 525, such as a solid-state device, magnetic disk or optical disk, can be coupled to the bus 505 to persistently store information and instructions.

[0109] The computing device 500 can be coupled via the bus 505 to a display 530, such as a liquid crystal display, or active-matrix display, for displaying information. An input device 535, such as a keyboard or voice interface can be coupled to the bus 505 for communicating information and commands to the processor 510. The input device 535 can include a touch screen display (e.g., the display 530). The input device 535 can include sensors to detect gestures. The input device 535 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 510 and for controlling cursor movement on the display 530.

[0110] The processes, systems and methods described herein can be implemented by the computing device 500 in response to the processor 510 executing an arrangement of instructions contained in the main memory 515. Such instructions can be read into the main memory 515 from another computer-readable medium, such as the storage device 525. Execution of the arrangement of instructions contained in the main memory 515 causes the computing device 500 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 515. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.[OHl] The processor 510 can execute one or more instructions associated with the system 100. The processor 510 can include an electronic processor, an integrated circuit, or the like including one or more of digital logic, analog logic, digital sensors, analog sensors, communication buses, volatile memory, nonvolatile memory, and the like. The processor 510 can include, but is not limited to, at least one microcontroller unit (MCU), microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC), or the like. The processor 510 can include, or be associated with, a main memory 515 operable to store or storing one or more non- transitory computer-readable instructions for operating components of the system 100 andoperating components operably coupled to the processor 510. The one or more instructions can include at least one of firmware, software, hardware, operating systems, or embedded operating systems, for example. The processor 510 or the system 100 generally can include at least one communication bus controller to effect communication between the system processor and the other elements of the system 100.

[0112] The main memory 515 can include one or more hardware memory devices to store binary data, digital data, or the like. The main memory 515 can include one or more electrical components, electronic components, programmable electronic components, reprogrammable electronic components, integrated circuits, semiconductor devices, flip flops, arithmetic units, or the like. The main memory 515 can include at least one of a non-volatile memory device, a solid-state memory device, a flash memory device, a NAND memory device, a volatile memory device, etc. The main memory 515 can include one or more addressable memory regions disposed on one or more physical memory arrays.

[0113] Although an example computing system has been described in FIG. 5, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0114] The subject matter described herein can illustrate different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are illustrative, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable or physically interacting components or wirelessly interactable or wirelessly interacting components or logically interacting or logically interactable components.

[0115] With respect to the use of plural or singular terms herein, those having skill in the art can translate from the plural to the singular or from the singular to the plural as is appropriate to the context or application. The various singular / plural permutations can be expressly set forth herein for sake of clarity.

[0116] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).

[0117] Although the figures and description can illustrate a specific order of method steps, the order of such steps can differ from what is depicted and described, unless specified differently above. Also, two or more steps can be performed concurrently or with partial concurrence, unless specified differently above. Such variation can depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0118] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the following appended claims can contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

[0119] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general, such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0120] Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent.

[0121] The foregoing description of illustrative implementations has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or can be acquired from practice of the disclosed implementations. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: one or more processors, coupled with memory, to: receive at least one data stream of a medical procedure performed with a robotic medical system, the at least one data stream comprising kinematics data and event data associated with at least one instrument of the robotic medical system and video data of the medical procedure; split the at least one data stream into a plurality of segments of the medical procedure; receive a selection of a segment of the plurality of segments; generate a first graphical user interface element that displays a segment of a video stream corresponding to the segment of the medical procedure; generate, using the kinematics data, a second graphical user interface element that displays at least one path taken by the at least one instrument during the segment; and provide, for display via a graphical user interface, an arrangement comprising the first graphical user interface element and the second graphical user interface element, wherein the at least one path taken by the instrument displayed in the second graphical user interface is synchronized with the segment of the video stream displayed in the first graphical user interface element.

2. The system of claim 1, wherein the one or more processors are further to: generate the arrangement based at least in part on the first graphical user interface element and the second graphical user interface element, the arrangement identifying a position of the first graphical user interface relative to a position of the second graphical user interface element.

3. The system of claim 1, wherein the at least one path comprises a plurality of paths; and wherein the one or more processors are further to:generate the second graphical user interface element, where each path of the plurality of paths are positioned in spaced relation to each other one or more other paths of the plurality of paths.

4. The system of claim 3, wherein each path of the plurality of paths is associated with a plurality of areas corresponding to each path of the plurality of paths, and wherein the plurality of areas comprise at least one selectable region associated with portions of the instrument path.

5. The system of claim 3, wherein each path of the plurality of paths is associated with a plurality of areas corresponding to each path of the plurality of paths, and wherein the plurality of areas comprise at least one selectable region that, upon selection, are configured to cause generation of the first graphical user interface element and the second graphical user interface element in accordance with a point in time corresponding to the selectable region.

6. The system of claim 5, wherein the one or more processors are further to: receive an indication of input selecting a selectable region of the at least one selectable region; generate a third graphical user interface element based at least in part on a portion of the at least one path taken by the at least one instrument during the medical procedure, the portion of the at least one path corresponding to the point in time; and provide, for display via the graphical user interface, the arrangement, where the arrangement comprises at least a portion of the third graphical user interface element.

7. The system of claim 1, wherein the one or more processors are further to: split the at least one data stream into the plurality of segments of the medical procedure based at least in part on one or more periods of time or one or more phases of the medical procedure.

8. The system of claim 1, wherein the at least one path taken by the at least one instrument comprises a plurality of paths, and wherein the one or more processors are further to:provide, for display via the graphical user interface, the arrangement comprising the first graphical user interface element and the second graphical user interface element, the second graphical user interface element representing the plurality of paths.

9. The system of claim 1, wherein the at least one instrument comprises at least one surgical robotic instrument or at least one master tool manipulator (MTM).

10. The system of claim 1, wherein the one or more processors are to: determine one or more objective performance indicators based at least in part on movement of the at least one instrument during the segment of the medical procedure; and provide, for display via the graphical user interface, the arrangement, where the arrangement comprises a representation of the one or more objective performance indicators.

11. The system of claim 10, wherein the representation of the one or more objective performance indicators comprises a graph of changes in the one or more objective performance indicators during a period of time.

12. The system of claim 1, wherein the one or more processors are to: receive input indicating one or more criteria associated with the medical procedure; generate the first graphical user interface element and the second graphical user interface element based at least in part on the one or more criteria associated with the medical procedure.

13. The system of any of claims 1-12, wherein the one or more criteria comprise at least one of: a period of time during the medical procedure, identifiers of one or more of the segments of the medical procedure, or a playback speed to control playback of the medical procedure.

14. The system of any of claims 1-13, wherein the one or more processors are to: provide, for display via the graphical user interface, data that is configured to cause the arrangement comprising the first graphical user interface element and the second graphical user interface element to be displayed via a display device.

15. A method, comprising: receiving, by one or more processors, at least one data stream of a medical procedure performed with a robotic medical system, the at least one data stream comprising kinematics data associated with at least one instrument of the robotic medical system and video data of the medical procedure; splitting, by the one or more processors, the at least one data stream into a plurality of segments of the medical procedure; receiving, by the one or more processors, a selection of a segment of the plurality of segments; generating, by the one or more processors, a first graphical user interface element that displays the segment of the video stream corresponding to the selected segment; generating, by the one or more processors and using the kinematics data, a second graphical user interface element that displays a path taken by the instrument during the selected segment; and providing, by the one or more processors, an arrangement comprising the first graphical user interface element and the second graphical user interface element.

16. The method of claim 15, further comprising: generating, by the one or more processors, the arrangement based at least in part on the first graphical user interface element and the second graphical user interface element, the arrangement identifying a position of the first graphical user interface relative to a position of the second graphical user interface element.

17. The method of claim 15, wherein the at least one path comprises a plurality of paths; and wherein the method further comprises: generating, by the one or more processors, the second graphical user interface element, each path of the plurality of paths positioned in spaced relation to one or more other paths of the plurality of paths.

18. The method of claim 17, wherein each path of the plurality of paths is associated with a plurality of areas corresponding to each path of the plurality of paths, and wherein the plurality of areas comprise at least one selectable region associated with portions of the instrument path.

19. The method of claim 17, wherein each path of the plurality of paths is associated with a plurality of areas corresponding to each path of the plurality of paths, and wherein the plurality of areas comprise at least one selectable region that, upon selection, are configured to cause generation of the first graphical user interface element and the second graphical user interface element in accordance with a point in time corresponding to the selectable region.

20. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to: receive at least one data stream of a medical procedure performed with a robotic medical system, the at least one data stream comprising kinematics data associated with at least one instrument of the robotic medical system and video data of the medical procedure; split the at least one data stream into a plurality of segments of the medical procedure; receive a selection of a segment of the plurality of segments; generate a first graphical user interface element that displays the segment of the video stream corresponding to the selected segment; generate, using the kinematics data, a second graphical user interface element that displays a path taken by the instrument during the selected segment; and provide, for display via a graphical user interface, an arrangement comprising the first graphical user interface element and the second graphical user interface element, wherein the path taken by the instrument displayed in the second graphical user interface is synchronized with the segment of the video stream displayed in the first graphical user interface element.

21. A system, comprising: one or more processors, coupled with memory, to: receive data associated with operation of a robotic medical system, generate a graphical user interface (GUI) based at least in part on the operation of the robotic medical system, the GUI comprising:a first area associated with a two-dimensional (2D) image or video frame generated during operation of the robotic medical system, and a second area associated with at least one three-dimensional (3D) instrument path; and provide data associated with the GUI to a display device, the data associated with the GUI configured to cause the display device to generate the GUI as an output of the display device.

22. The system of claim 21, wherein the first area comprises a first GUI element representing the 2D image or video frame; and the second area comprises a second GUI element representing the at least one 3D instrument path.

23. The system of claim 22, wherein, in response to obtaining input indication selection of a 3D instrument path from among the at least one 3D instrument path, the one or more processors are further to: update the GUI to comprise a third area, the third area overlapping at least a portion of the first area or the second area, wherein the third area is associated with a third GUI element representing a plot of at least one path corresponding to the 3D instrument path selected from among the at least one 3D instrument path.

Images