Endoscopic surgical navigation with automated anatomy measurement

Abstract

Automated measurement of objects in a field of view of a robotic system are provided. A system can identify a frame capturing a scene in a medical procedure. The system can generate one or more masks that segment an anatomical structure in the frame. Each mask of the one or more masks can include a plurality of labels indicating the anatomical structure captured in the frame, the system can determine a correspondence between the one or more masks and a depth map. The depth map can include depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure. The system can determine at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map.

Description

Atty Dkt. No. 135039-0428 (P06967-WO)ENDOSCOPIC SURGICAL NAVIGATION WITH AUTOMATED ANATOMY MEASUREMENTCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 690,099, filed September 3, 2024, which is hereby incorporated by reference herein in its entirety.BACKGROUND

[0002] During teleoperation of robotic systems, precise visual replication of the spaces in which the systems operate is important in allowing for successful system operation. As an example, in minimally invasive surgeries that involve teleoperation of robotic medical systems, such precise visual replication of spaces within the body of the patient can allow for faster and more precise manipulation of instruments at least partially disposed within the body of the patient. This can, in turn, allow for reduced manipulation of tissues or other anatomical structures of the patients and reduced recovery periods after the surgery.

[0003] But precisely determining distances in the space in which a teleoperated robotic system is operating can be difficult for remote operators. Unlike human vision, robotic cameras often lack depth perception due to the reliance on a two-dimensional (2D) image feed. This can make it difficult to judge how far away objects are from the robotic systems, leading to imprecise movements. Additionally, factors like camera positioning and zoom limitations can further distort the perceived size and distance of objects within the environment. This sensory deprivation can cause operators to rely on visual cues like overlapping objects and estimations based on past experience to determine distances within space, making remote manipulation a constant exercise in spatial reasoning and educated estimation.SUMMARY

[0004] Technical solutions disclosed herein are generally related to systems and methods for automated measurement of objects in a field of view of a robotic system. These solutions can involve identifying a frame that captures a scene in a medical procedure, generating one or more masks that segment an anatomical structure in the frame, determine a correspondence between the one or more masks and a depth map of the scene, determine at least one-1-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) measurement of the anatomical structure, and generate display data associated with a visual representation of the at least one measurement.

[0005] Aspects of the technical solution are directed to a system. The system can include one or more processors, coupled with memory. The one or more processors can identify a frame that captures a scene in a medical procedure performed with a robotic medical system. The one or more processors can generate one or more masks that segment an anatomical structure in the frame. Each mask of the one or more masks can include a plurality of labels indicating the anatomical structure captured in the frame. The one or more processors can determine a correspondence between the one or more masks and a depth map, the depth map corresponding to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure. The one or more processors can determine at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map. The one or more processors can generate display data associated with a visual representation of the at least one measurement, the display data configured to cause a display device to output the visual representation of the at least one measurement.

[0006] In some aspects, the one or more processors can be further configured to determine at least one measurement including a distance of the anatomical structure extending along the axis.

[0007] In some aspects, the at least one measurement can include a first measurement and the axis defined by the anatomical structure can include a first axis, and the one or more processors can be further configured to: determine a second measurement along a second axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth.

[0008] In some aspects, the first axis and the second axis can be substantially transverse to each other along a plane defined by the first axis and the second axis. In some aspects, the first axis and the second axis are non-transverse to each other along a plane defined by the first axis and the second axis.

[0009] In some aspects, the one or more processors configured to determine the at least one measurement along the axis defined by the anatomical structure can be configured to determine the at least one measurement based on a surface area of the anatomical structure.

[0010] In some aspects, the anatomical structure can be associated with a first patient, and the one or more processors can be further configured to: compare the at least one measurement along the axis defined by the anatomical structure to at least one measurement -2-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) along an axis defined by a second anatomical structure, the second anatomical structure associated with a second patient. In some aspects, the one or more processors can be configured to determine a difference between the at least one measurement along the axis defined by the anatomical structure and the at least one measurement along the axis defined by the second anatomical structure. In some aspects, the one or more processors can be configured to determine a metric based on the difference.

[0011] In some aspects, the at least one measurement along the axis can be defined by the anatomical structure and correspond to operations performed by robotic surgical system when engaged by a first clinician. In some aspects, the at least one measurement along the axis defined by the second anatomical structure can correspond to operations performed by the robotic surgical system when engaged by a second clinician, and the one or more processors can be further configured to determine a clinician score for the first clinician or the second clinician based on the metric.

[0012] In some aspects, the one or more processors are further configured to receive an input indicating a reference measurement corresponding to the anatomical structure. The one or more processors can be configured to search a dataset including data associated with a first medical procedure and a second medical procedure. The first medical procedure associated with the at least one measurement along the axis defined by the anatomical structure. The second medical procedure associated with at least one second measurement along an axis defined by a second anatomical structure. The one or more processors can be configured to determine the reference measurement satisfies the at least one measurement or the at least one second measurement. The one or more processors configured to generate the display data associated with the visual representation of the at least one measurement can be configured to generate the display data based on the visual representation of the at least one measurement and the determination that the reference measurement satisfies the at least one measurement or the at least one second measurement.

[0013] In some aspects, the one or more processors can be configured to receive an input at the robotic medical system to generate the display data associated with a visual representation of the at least one measurement. The one or more processors configured to generate the display data associated with the visual representation of the at least one measurement can be configured to: generate the display data associated with the visual representation of the at least one measurement based on the input at the robotic medical system to generate the display data.-3-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0014] In some aspects, the one or more processors can be further configured to determine the frame that captures a scene in a medical procedure corresponds to a phase of the medical procedure. The one or more processors configured to generate the display data associated with the visual representation of the at least one measurement can be configured to generate the display data associated with the visual representation of the at least one measurement based on the phase of the medical procedure.

[0015] In some aspects, the one or more processors can be configured to: for each patient of a plurality of patients involved in at least one medical procedure performed by a robotic medical system: identify a frame that captures a scene in a medical procedure performed with a robotic medical system. The one or more processors can be configured to generate one or more masks that segment an anatomical structure in the frame. Each mask of the one or more masks can include a plurality of labels indicating the anatomical structure captured in the frame. The one or more processors can be configured to determine a correspondence between the one or more masks and a depth map, the depth map corresponding to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure. The one or more processors can be configured to determine at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map. The one or more processors can be configured to update at least one dataset based on the at least one measurement along the axis defined by the anatomical structure.

[0016] In some aspects, the anatomical structure of each of the patients is associated with a type of anatomical structure; and the one or more processors can be further configured to: receive an input indicating a reference measurement corresponding to the anatomical structure. The one or more processors can be configured to compare the reference measurement to the at least one measurement along the axis defined by the anatomical structure for each patient of the plurality of patients. The one or more processors can be configured to determine that the reference measurement satisfies the at least one measurement along the axis defined by the anatomical structure of at least one patient of the plurality of patients. The one or more processors can be configured to generate display data associated with a visual representation of the at least one measurement of the at least one patient, the display data configured to cause a display device to output the visual representation of the at least one measurement.

[0017] In some aspects, the one or more processors can be further configured to determine that the at least one measurement of an outlier patient of the plurality of patients -4-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) satisfies an outlier threshold. The one or more processors can be configured to update the at least one dataset based on the at least one measurement of the outlier patient satisfying the outlier threshold.

[0018] In some aspects, the one or more processors configured to update the at least one dataset based on the at least one measurement of the outlier patient satisfying the outlier threshold can be configured to update a second dataset to include the at last one measurement of the outlier patient.

[0019] Aspects of the technical solution are directed to a method. In some aspects, the method includes identifying, by at least one processor coupled with memory, a frame that captures a scene in a medical procedure performed with a robotic medical system. The method can include generating, by the at least one processor, one or more masks that segment an anatomical structure in the frame. Each mask of the one or more masks can include a plurality of labels indicating the anatomical structure captured in the frame. The method can include determining, by the at least one processor, a correspondence between the one or more masks and a depth map. The depth map can correspond to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure. The method can include determining, by the at least one processor, at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map. The method can include generating, by the at least one processor, display data associated with a visual representation of the at least one measurement, the display data configured to cause a display device to output the visual representation of the at least one measurement.

[0020] In some aspects, the method can further include determining, by the at least one processor, at least one measurement including a distance of the anatomical structure extending along the axis.

[0021] In some aspects, the at least one measurement can include a first measurement and the axis defined by the anatomical structure can include a first axis. In some aspects the method can include determining, by the at least one processor, a second measurement along a second axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map.

[0022] In some aspects, determining the at least one measurement along the axis defined by the anatomical structure can include determining the at least one measurement based on a surface area of the anatomical structure.-5-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0023] Aspects of the technical solution are directed to a non-transitory computer- readable medium. The non-transitory computer-readable medium can store processorexecutable instructions that, when executed by one or more processors, cause the one or more processors to identify a frame that captures a scene in a medical procedure. In some aspects, the instructions can cause the one or more processors to generate one or more masks that segment an anatomical structure in the frame. In some aspects, the instructions can cause the one or more processors to determine a correspondence between the one or more masks and a depth map. The depth map can correspond to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure. In some aspects, the instructions can cause the one or more processors to determine at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map. In some aspects, the instructions can cause the one or more processors to generate display data associated with a visual representation of the at least one measurement, the display data configured to cause a display device to output the visual representation of the at least one measurement.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG. 1 depicts an example system to automated measurement of objects in a field of view of a robotic system, according to some embodiments;

[0025] FIG. 2 depicts a flowchart diagram illustrating an example method for automated measurement of objects in a field of view of a robotic system, according to some embodiments;

[0026] FIG. 3 depicts an example combination of models for automated measurement of objects in a field of view of a robotic system, according to some embodiments;

[0027] FIG. 4 depicts example graphical user interfaces generated to indicate measurements of objects in a field of view of a robotic system, according to some embodiments;

[0028] FIG. 5 depicts a diagram of a medical environment; according to some embodiments;

[0029] FIG. 6 depicts a block diagram depicting an architecture for a computer system that can be employed to implement elements of the systems and methods described and illustrated herein.-6-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)DETAILED DESCRIPTION

[0028] Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems for automated measurement of objects in a field of view of a robotic system. The various concepts introduced above and discussed in greater detail below can be implemented in any of numerous ways. Although the present disclosure is discussed in the context of a surgical procedure, the present disclosure can be applicable to other medical sessions or environments or activities, as well as nonmedical activities where the measurement of objects in the field of view of a robotic system is desired.

[0029] Over the course of a surgery, robotic surgical systems can allow clinicians to view 2D or 3D representations of a work area (e.g., an area within a body of a patient) and preform maneuvers that involve estimation of the dimensions anatomical structures involved in the operation. Examples of these maneuvers include suture closure of anastomosis, ligation of dissected regions, determination of the length of small intestines to further determine where to transect. To perform these maneuvers, clinicians can develop estimations of measurements of anatomical structures involved in the surgery in coordination with performance of tool gestures like hand-over-hand tool manipulation or by insertion of a surgical ruler into the work area. These techniques can add time and difficulty to the surgery and provide limited measurements that would have to be manually obtained and recorded (if used at future points in time).

[0030] When clinicians review or perform new or complex surgeries using certain robotic surgical systems, such clinicians can rely on prior knowledge and experience to estimate measurements of various anatomical structures involved in such surgeries and visualized during the medical procedure. In one example, clinicians can develop prior knowledge and experience manipulating anatomical structures (e.g., a uterus or the like) that is in an abnormal state. But learning to operate the robotic surgical system in cases where the size of the anatomical structure is atypical (e.g., enlarged, deformed) can take time, particularly in cases where clinicians rarely encounter such cases. This can allow for inconsistent estimations from clinician to clinician and, in some cases, extend the amount of time involved during surgeries performed by less-experienced clinicians.

[0031] In other examples, during and after a surgery physical measurements of anatomical structures can be obtained to annotate data associated with the surgery (e.g., the video feed captured during the surgery). Performing these physical measurements can add-7-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) time to the surgeries performed by clinicians with less experience than other, more senior clinicians. Further, experienced clinicians training more junior clinicians can have difficulty communicating their estimated measurements to the junior clinicians. Further, because human observation can be prone to error, it can be difficult for clinicians to communicate, annotate, or document the data associated with the surgery for subsequent review by different clinicians with an established degree of consistency.

[0032] Aspects of the disclosure that address the technical problems described relate to techniques for measuring and indicating the metric size of various anatomical structures during a medical procedure or during post-operative review of the procedure. The metric size can include, for example, a length of the anatomical structure (e.g., along a longest dimension that is in the current view angle) or a surface area of the anatomical structure. A first machine learning (ML) model can be trained to segment anatomical structures represented in a video stream of a medical procedure. For example, a transformer-based neural network (such as a Vision Transformer (ViT)) can be trained to receive input images (e.g., monocular images) included in the video stream, and output anatomy masks based on the images. The anatomy masks can include arrays of tags that correspond to the pixels in the images. The output can then be combined with an output of different ML models to generate the indications discussed.

[0033] A second ML model can include a metric depth estimation model that is trained to determine depths to objects in a field-of-view (FoV) of a camera generating the video stream during a medical procedure. For example, the second ML model can include a combination of layers that perform convolution, attention, and normalization functions to map each pixel of an input image to a metric distance value. The output of the first ML model (e.g., the segmented masks) can be transformed based on intrinsic parameters of the camera and aligned to the corresponding depths included in the output of the second model. Once aligned, the distance of the anatomical structures along varying axis can be determined with millimeter degrees of precision. This can enable downstream applications such as, for example, calculation of objective performance indicators (OPIs) involving the distance of the anatomical structures, improved indexing and searching of archived video footage of medical procedures, or intraoperative measurement of anatomical structures.

[0030] Techniques implemented by the systems and methods described involve the measurement of objects in a field of view of a robotic system by a data processing system in communication with the robotic system. These measurements can be determined based on anatomical structures observed both in real-time (e.g., intra-operative) and on review of the -8-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) operation of the robotic system (e.g., post-operative). In the case of post-operative measurement, the systems and methods described herein can further determine one or more objective performance indicators based on the determined measurements, allow for targeted searching, and updating of entries in databases storing images (or videos) with measurement information. By virtue of the implementation of the systems and methods described, anatomical structures can be identified, segmented (e.g., within a scene), and measured during a surgery, reducing or eliminating the need for manual, intra- or post-operative annotation. This can likewise reduce or eliminate the introduction of human error in generating such measurements as well as the time involved in performing the surgical procedure, and allow for the generation of alerts or warnings to provide real-time feedback to clinicians indicating whether one or more maneuvers are appropriate for a given surgical procedure. Further, the measurements generated by the systems and methods described herein can be determined in accordance with predetermined tolerances for error, allowing for more accurate and consistent measurements of anatomical structures (or portions thereof). And by establishing indexes of anatomical structures and their corresponding measurements for postoperative retrieval (e.g., when determining OPIs), the system and methods described herein can reduce the query time involved in identifying relevant entries of post-operative medical procedures stored in databases.

[0034] FIG. 1 depicts an example system 100 to generate automated measurement of objects in a field of view of a robotic system such as, for example, robotic medical systems used in robot-assisted surgeries. The example system 100 can include a combination of hardware and software for generating graphical user interfaces representing aspects of teleoperation of robotic systems. For example, the example system 100 can include a network 105, a medical environment 110, a data processing system 130, and a computing device 150 as described herein.

[0035] The example system 100 can include a medical environment 110 (e.g., a medical environment that is the same as, or similar to, the example medical environment 500 of FIG. 5) including one or more data capture devices 112, medical instruments 114, visualization tools 116, displays 118 and robotic medical systems (RMSs) 120. RMS 120 can include or generate various types of data streams 122 that are described herein, and can operate using system configurations 124. One or more RMSs 120 can be communicatively coupled with one or more data processing systems 130.

[0036] The RMS 120 can be deployed in any medical environment 110. The medical environment 110 can include any space or facility for performing medical procedures, such as -9-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) a surgical facility or an operating room. The medical environment 110 can include medical instruments 114 (e.g., surgical tools used for specialized tasks) that the RMS 120 can use for performing operational procedures, such as surgical patient procedures, whether invasive, non-invasive, or any in-patient or out-patient procedures. The RMS 120 can be centralized or distributed across a plurality of computing devices or systems, such as computing devices 600 (e.g., used on servers, network devices or cloud computing products) to implement various functionalities of the RMS 120, including communicating or processing data streams 122 across various devices via the network 105.

[0037] The medical environment 110 can include one or more data capture devices 112 (e.g., optical devices, such as cameras or sensors or other types of sensors or detectors) for capturing data streams 122. The data streams 122 can include any sensor data, such as images or videos of a surgery, kinematics data on any movement of medical instruments 114, or any events data, such as installation, configuration or selection events corresponding to medical instruments 114. The medical environment 110 can include one or more visualization tools 116 to gather the captured data streams 122 and process it for display to the user (e.g., a surgeon, a medical professional or an engineer or a technician configuring RMS) via one or more displays 118 (e.g., a touchscreen, an LCD display). A display 118 can present data stream 122 (e.g., images or video frames) of a medical procedure (e.g., surgery) being performed using the RMS 120 while handling, manipulating, holding or otherwise utilizing medical instruments 114 to perform surgical tasks at the surgical site. RMS 120 can include system configurations 124 based at least on which RMS 120 can operate, and the functionality of which can impact the data flow of the data streams 122. As will be described herein, the data streams 122 can be divided into multiple data streams.

[0038] The system 100 can include one or more data capture devices 112 (e.g., video cameras, sensors or detectors) for collecting data streams 122, that can be used for machine learning, including detection of objects from sensor data (e.g., video frames or force or feedback data), detection of particular events (e.g., user interface selection of, or a surgeon’s engaging of, a medical instrument 114) or detection of kinematics (e.g., movements of the medical instrument 114). The data capture devices 112 can include cameras or other image capture devices for capturing videos or images from a particular viewpoint within the medical environment 110. The data capture devices 112 can be positioned, mounted, or otherwise located to capture content from any viewpoint that facilitates the data processing system capturing various surgical tasks or actions.-10-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0039] The data capture devices 112 can include any of a variety of detectors, sensors, cameras, video imaging devices, infrared imaging devices, visible light imaging devices, intensity imaging devices (e.g., black, color, grayscale imaging devices, etc.), hyperspectral imaging devices (e.g., a hyperspectral camera, etc.), depth imaging devices (e.g., stereoscopic imaging devices, time-of-flight imaging devices, etc.), medical imaging devices such as endoscopic imaging devices, ultrasound imaging devices, etc., non-visible light imaging devices, any combination or sub-combination of the above mentioned imaging devices, or any other type of imaging devices that can be suitable for the purposes described herein. The data capture devices 112 can include cameras that a surgeon can use to perform a surgery and observe manipulation components within a purview of field of view suitable for the given task performance. The data capture devices can output any type of data streams 122, including data streams 122 of kinematics data (e.g., kinematics data streams), data streams 122 of events data (e.g., events data streams) and data streams 122 of sensor data (e.g., sensors data streams).

[0040] For example, data capture devices 112 can capture, detect, or acquire sensor data such as videos or images, including for example, still images, video images, vector images, bitmap images, other types of images (e.g., Raman hyperspectral images, etc.), or combinations thereof. The data capture devices 112 can capture the images at any suitable predetermined capture rate or frequency. Settings, such as zoom settings or resolution, of each of the data capture devices 112 can vary as desired to capture suitable images from any viewpoint. For instance, data capture devices 112 can have fixed viewpoints, locations, positions, or orientations. The data capture devices 112 can be portable, or otherwise configured to change orientation or telescope in various directions. The data capture devices 112 can be part of a multi-sensor architecture including multiple sensors, with each sensor being configured to detect, measure, or otherwise capture a particular parameter (e.g., sound, images, or pressure).

[0041] The data capture devices 112 can generate sensor data from any type and form of a sensor, such as a positioning sensor, a biometric sensor, a velocity sensor, an acceleration sensor, a vibration sensor, a motion sensor, a pressure sensor, a light sensor, a distance sensor, a current sensor, a focus sensor, a temperature or pressure sensor or any other type and form of sensor used for providing data on the medical instruments 114, or the data capture devices (e.g., optical devices). For example, the data capture device 112 can include a location sensor, a distance sensor or a positioning sensor providing coordinate locations of a medical instrument 114 (e.g., kinematics data). The data capture device 112 can include a sensor-11-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) providing information or data on a location, position or spatial orientation of an object (e.g., medical instrument 114 or a lens of data capture device 112) with respect to a reference point for kinematics data. The reference point can include any fixed, defined location used as the starting point for measuring distances and positions in a specific direction, serving as the origin from which all other points or locations can be determined.

[0042] The display 118 can show, illustrate or play the data stream 122, such as a video stream, in which the medical instruments 114 at or near surgical sites are shown. For example, the display 118 can display a rectangular image of a surgical site along with at least a portion of the medical instruments 114 being used to perform surgical tasks. The display 118 can provide compiled or composite images generated by the visualization tool 116 from a plurality of data capture devices 112 to provide a visual feedback from one or more points of view.

[0043] The visualization tool 116 can be configured or designed to receive any number of different data streams 122 from any number of data capture devices 112 and combine them into one or more data streams displayed on a display 118. The visualization tool 116 can be configured to receive a plurality of data stream components and combine the plurality of data stream components into a single data stream 122. For instance, the visualization tool 116 can receive visual sensor data from one or more of the medical instruments 114, sensors or cameras with respect to a surgical site or an area in which a surgery is performed. The visualization tool 116 can incorporate, combine or utilize multiple types of data (e.g., positioning data of a medical instrument 114 along sensor readings of pressure, temperature, vibration or any other data) to generate an output to present on a display 118. The visualization tool 116 can present locations of medical instruments 114 along with locations of any reference points or surgical sites, including locations of anatomical parts of the patient (e.g., organs, glands or bones).

[0044] The medical instruments 114 can be any type and form of tool or medical instrument used for surgery, medical procedures or a tool in an operating room or environment. The medical instrument 114 can be imaged by, associated with, or include an image capture device. For instance, a medical instrument 114 can be a tool for making incisions, a tool for suturing a wound, an endoscope for visualizing organs or tissues, an imaging device, a needle and a thread for stitching a wound, a surgical scalpel, forceps, scissors, retractors, graspers, or any other tool or medical instrument to be used during a surgery. The medical instruments 114 can include hemostats, trocars, surgical drills, suction devices or any medical instruments for use during a surgery. The medical instrument 114 can -12-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) include other or additional types of therapeutic or diagnostic medical imaging implements. The medical instrument 114 can be configured to be installed in, coupled with, or manipulated by an RMS 120, such as by manipulator arms or other components for holding, using and manipulating the medical instruments. The medical instruments 114 can be the same as, or similar to, the medical instruments discussed with respect to FIG. 5, among others.

[0045] The RMS 120 can be a computer-assisted system configured to perform a surgical or medical procedure or activity on a patient via, or using or with the assistance of, one or more robotic components or the medical instruments 114. The RMS 120 can include any number of manipulator arms for grasping, holding or manipulating various medical instruments 114 and performing computer-assisted medical tasks using the medical instruments 114 controlled by the manipulator arms.

[0046] The data streams 122 can be generated by the RMS 120. For instance, sensor data associated with the data streams 122 can include images (e.g., video images) captured by a medical instrument 114 and can be sent to the visualization tool 116. For instance, a display 118 (e.g., a touchscreen) can be used by a surgeon to select, engage, or configure a particular medical instrument 114, thereby triggering an event that can be indicated or included in data packets of a data stream 122. The RMS 120 can include one or more input ports to receive direct or indirect connection of one or more auxiliary devices. For example, the visualization tool 116 can be connected to the RMS 120 to receive the images from the medical instrument 114 when the medical instrument 114 is installed in the RMS 120 (e.g., on a manipulator arm for handing medical instruments 114). For example, the data stream 122 can include data indicative of positioning and movement of the medical instruments 114 that can be captured or identified by data packets of a kinematics data. The visualization tool 116 can combine the data stream components from the data capture devices 112 and the medical instrument 114 into a single combined data stream 122 which can be indicated or presented on a display 118. The RMS 120 can provide the data streams 122 to the data processing system 130 periodically, continuously, or in real-time.

[0047] Data packets can include a unit of data in a data stream 122. The data packets can include the actual information being sent and metadata, such as a source and a destination address, a port identifier or any other information for transmitting data. The data packets can include a data (e.g., a payload) corresponding to an event (e.g., installation, uninstallation, engagement or setup of a medical instrument 114). The data packets can include data corresponding to sensor information (e.g., a video frame captured by a camera), or data on -13-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) movement of a medical instrument 114. The data packets can be transmitted in the data streams 122 that can be separated or combined. For instance, a data stream 122 for kinematics data (e.g., a kinematics data stream) can include a plurality of data packets indicative of movement of robotic system components or features.

[0048] Data packets can include one or more timestamps, which can indicate a particular time when particular events took place. Timestamps can include time indications expressed in any combination of nanoseconds, microseconds, milliseconds, seconds, hours, days, months or years. Timestamps can be included in the payload or metadata of data packets and can indicate the time when a data packet was generated, the time when the data packet was transmitted from the device that generated the data packet, the time when the data packet was received by another device (e.g., a system within the RMS 120, the data processing system 130, the computing device 150 or another device on a network) or a time when the data packet is stored into a data repository 132.

[0049] The data repository 132 can include one or more data files, data structures, arrays, values, or other information that facilitates operation of the data processing system 130. The data repository 132 can include one or more local or distributed databases and can include a database management system. The data repository 132 can include, maintain, or manage one or more data streams 122. The data streams 122 can include or be formed from one or more of a video stream, image stream, stream of sensor measurements, event stream, or kinematics stream. The data streams 122 can include data collected by one or more data capture devices 112, such as a set of 3D sensors from a variety of angles or vantage points with respect to the procedure activity (e.g., point or area of surgery).

[0050] The data stream 122 can include any stream of data. The data stream 122 can include a video stream, including a series of video frames or organized into video fragments, such as video fragments of about 1, 2, 3, 4, 5, 10 or 15 seconds of a video. Each second of the video can include, for example, 30, 45, 60, 90, 120, 240 video frames per second. The data streams 122 can include an event stream which can include a stream of event data or information, such as packets, which identify or convey a state of the RMS 120 or an event that occurred in association with the RMS 120. For example, data stream 122 can include any portion of system configuration 124, including information on operations on data streams 122, data on installation, uninstallation, calibration, set up, attachment, detachment or any other action performed by or on an RMS 120 with respect to the medical instruments 114.

[0051] The data stream 122 can include data about an event, such as a state of the RMS 120 indicating whether the medical instrument 114 is calibrated, adjusted or includes a-14-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) manipulator arm installed on the RMS 120. A data stream 122 representing event data (e.g., event data stream) can include data on whether an RMS 120 was fully functional (e.g., without errors) during the procedure. For example, when a medical instrument 114 is installed on a manipulator arm of the RMS 120, a signal or data packet(s) can be generated indicating that the medical instrument 114 has been installed on the manipulator arm of the RMS 120.

[0052] The data stream 122 can include a stream of kinematics data which can refer to or include data associated with one or more of the manipulator arms or medical instruments 114 attached to the manipulator arms, such as arm locations or positioning. The data corresponding to the medical instruments 114 can be captured or detected by one or more displacement transducers, orientational sensors, positional sensors, or other types of sensors and devices to measure parameters or generate kinematics information. The kinematics data can include sensor data along with time stamps and an indication of the medical instrument 114 or type of medical instrument 114 associated with the data stream 122.

[0053] The data repository 132 can store sensor data having video frames that can include one or more static images or frames extracted from a sequence of images of a video file. Video frame can represent a specific moment in time and can be identified by a metadata including a timestamp. Video frames can display visual content of the video of a medical procedure being analyzed by the data processing system 130 to form a composite video along with performance metrics indicative of the performance of the surgeon performing the procedure. For example, in a video file capturing a robotic surgical procedure, a video frame can depict a snapshot of the surgical task, illustrating a movement or usage of a medical instrument 114 such as a robotic arm manipulating a surgical tool within the patient's body.

[0054] The data streams 122 corresponding to sensor data (e.g., videos), events, and kinematics can include related, corresponding or duplicate information that can be used for cross-data comparisons and verification that all three data sources are in agreement. For instance, the detection function can implement a check for consistency between diverse data types and data sources by mapping and comparing timestamps between different data types to facilitate if they consistently progress over time, such as in accordance with expected flow and correlation of events, video stream details and kinematics values.

[0055] For example, an installation of a medical instrument 114 can be recorded as a system event and provided in a data stream 122 of events data. At the same or similar expected time frame, the installed medical instrument 114 can shows up in a sensor data (e.g., in a video) which can be detected by the data processing system 130, which can include a-15-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) computer vision model. Kinematics data can confirm movements of the medical instrument 114 according to the movements detected by the data processing system 130. Using these cross-data stream correlation techniques, the data processing system 130 can verify time synchronization across the three data sources (e.g., three data streams 122).

[0056] With continued reference to FIG. 1, among others, the data processing system 130 can include any combination of hardware or software that performs one or more of the functions described herein. For example, the data processing system 130 can include any combination of hardware and software for automated measurement of objects in a field of view of a robotic system. The data processing system 130 can include any computing device (e.g., a computing device that is the same as, or similar to, the computing device 600 of FIG. 6) and can include one or more servers, virtual machines, or can be part of or include a cloud computing environment. The data processing system 130 can be provided via a centralized computing device or be provided via distributed computing components, such as including multiple, logically grouped servers and facilitating distributed computing techniques. The logical group of servers can be referred to as a data center, server farm or a machine farm. The servers, which can include virtual machines, can also be geographically dispersed. A data center or machine farm can be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous - one or more of the servers or machines can operate according to one or more type of operating system platform.

[0057] The data processing system 130, or components thereof can include a physical or virtual computer system operatively coupled, or associated with, the medical environment 110. The data processing system 130, or components thereof, can be coupled, or associated with, the medical environment 110 via a network 105, either directly or indirectly through an intermediate computing device or system. The network 105 can be any type or form of network. The geographical scope of the network can vary widely and can include a body area network (BAN), a personal area network (PAN), a local-area network (LAN) (e.g., Intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 105 can assume any form such as point-to-point, bus, star, ring, mesh, tree, etc. The network 105 can utilize different techniques and layers or stacks of protocols, including, for example, the Ethernet protocol, the internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, the SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP / IP internet protocol suite can include application layer, transport layer, internet layer (including, -16-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) e.g., IPv6), or the link layer. The network 105 can be a type of a broadcast network, a telecommunications network, a data communication network, a computer network, a Bluetooth network, or other types of wired and wireless networks.

[0058] The data processing system 130, or components thereof, can be located at least partially at the location of the surgical facility associated with the medical environment 110 or remotely therefrom. Elements of the data processing system 130, or components thereof can be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The data processing system 130, or components thereof, can include other or additional elements that can be considered desirable to have in performing the functions described herein. The data processing system 130, or components thereof, can include, or be associated with, one or more components or functionality of a computing device including, for example, one or more processors coupled with memory that can store instructions, data or commands for implementing the functionalities of the data processing system 130 discussed herein.

[0059] The data processing system 130 can include one or more of a data repository 132 configured to store one or more datasets, a frame identification system 134, a mask generation system 136, a depth estimation system 137, a correspondence system 138, a measurement generation system 140, a graphical user interface (GUI) element generator 142, or a post-processing system 144. While each of the systems of the data processing system 130 are described as being configured to perform one or more operations, the components can cooperate with one or more different components of the data processing system 130 to perform the one or more described operations. The data processing system 130 can be communicatively coupled with one or more data processing systems that operate in cooperation to perform one or more of the operations described herein.

[0060] The data processing system 130 can be implemented by one or more components of the medical environment 110. For example, the data processing system 130 can be implemented by one or more components of the RMS 120. The data processing system 130 can receive one or more data streams 122 that are described herein, and can monitor operation of the RMS 120 using the system configurations 124. One or more RMSs 120 can be communicatively coupled with the one or more data processing systems 130. The data repository 132 can be configured to receive, store, and provide the data streams 122 (e.g., one or more data packets associated with the data streams 122) before, during, or after a medical procedure to one or more other devices of FIG. 1, among others.-17-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0061] The data repository 132 can be implemented by the data processing system 130 or can be a device that is the same as, or similar to, the computing device 600 of FIG. 6. The data repository 132 can receive data from any of the devices of FIG. 1, among others, either directly or indirectly (e.g., via the data processing system 130). The data can include the data streams 122. In examples, the data stored by the data repository 132 is associated with a currently- or previously-performed medical procedure involving the RMS 120 or another robotic system. The data repository 132 can receive the data streams 122 or the system configurations 124 and store the data streams 122 or the system configurations 124 therein. The data repository 132 can provide the data streams 122 or the system configurations 124 (e.g., one or more data packets thereof) to the one or more of the components of the data processing system 130. The data repository 132 stores data associated with one or more of data streams 122 generated during operation of at least one component of the RMS 120 or data generated by the components of the data processing system 130.

[0062] The frame identification system 134 can be implemented by the data processing system 130 or can be a device that is the same as, or similar to, the computing device 600 of FIG. 6. The frame identification system 134 can receive the data streams 122. For example, the frame identification system 134 can receive the data streams 122 via the network 105. In examples, the frame identification system 134 can receive the data streams 122 via the data repository 132. The frame identification system 134 can receive the data streams 122 of a medical procedure performed with the RMS 120. For example, the frame identification system 134 can receive the data streams 122, where the data streams 122 include data (e.g., packets or the like) generated by one or more devices of the RMS 120 or in communication with the RMS 120 during the medical procedure. The one or more packets associated with the data streams 122 can include (e.g., represent) one or more images (e.g., single images or sets of images representing a video stream) during a medical procedure or one or more movements of the medical instruments 114 or MTMs of the RMS 120 engaged by an individual during the medical procedure to at least in part cause movement of the medical instruments 114. The MTMs can include one or more devices configured to be grasped by the individual and moved within an area along six degrees of freedom and the movements can be tracked by the RMS 120 to at least in part cause the medical instruments 114 to move in accordance with the movements of the MTMs.

[0063] The one or more images of the medical procedure can be captured as frames or otherwise obtained by the data capture devices 112 or the visualization tool 116. The images included in the data streams 122 can be generated by an imaging device that is supported by a -18-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) distal portion of a medical instrument 114 (e.g., an endoscope or the like). The one or more images can represent one or more anatomical structures (or portions thereof) or portions of the one or more medical instruments 114.

[0064] The frame identification system 134 can obtain frame data associated with one or more frames generated during one or more medical procedures. For example, the frame identification system 134 can obtain the frame data during operation of one or more medical instruments 114 in coordination with the RMS 120. In this example, the frame data can be associated with one or more images, videos captured by the data capture devices 112 and represent one or more anatomical structures (or portions thereof), one or more medical instruments 114 involved in given surgical procedures.

[0065] The frame identification system 134 can identify (e.g., determine) one or more frames that capture a scenario in a medical procedure performed in association with the RMS 120. For example, the frame identification system 134 can identify one or more frames involved in a particular type of medical procedure (e.g., a hysterectomy, a colectomy). In this example, the frame identification system 134 can identify the one or more frames involved in the particular type of medical procedure based on (e.g., during) performance of one or more phases of the medical procedure. During the medical procedure, the RMS 120 can add metadata (e.g., tags or the like) indicating the type of medical procedure, the anatomical structures involved in the medical procedure to the data associated with the frames. In another example, the frame identification system 134 can identify the one or more frames that capture the scene based on the data processing system 130 receiving a query indicating a type of medical procedure. For example, a clinician can provide input via the computing device 150 identifying one or more medical procedures, one or more phases of a medical procedure, one or more scenarios encountered during a medical procedure to be replayed or analyzed. In this example, the computing device 150 can generate input data associated with the input (e.g., representing a query) and provide the input data to the data processing system 130. The input data can be configured to cause the frame identification system 134 to search the data repository 132 and identify frames responsive to the query. These frames can be individual frames (e.g., corresponding to individual images) or groups of frames corresponding to videos of medical procedures.

[0066] The mask generation system 136 can obtain data associated with frames from the frame identification system 134. For example, the mask generation system 136 can obtain the data associated with the frames from the frame identification system 134 to generate one more segmentation masks (generally referred to as masks) based on each frame. The mask -19-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) generation system 136 can generate the one or more masks where the one or more masks segment one or more anatomical structures in the frame. As described herein, a mask output by the mask generation system 136 (e.g., by an anatomy segmentation model that is the same as, or similar to, a segmentation model such as the anatomy segmentation model 306 of FIG. 3) can include a binary image or a set of labels that represents segmented regions or objects within a frame (e.g., or an image represented by a frame). The mask can be an output of a segmentation algorithm or model, which assigns each pixel in the image to a specific class based on characteristics of one or more pixels in the image, such as color, texture, shape. The mask can include a discrete representation of the image, where each pixel is assigned a value indicating its membership to a particular segment or object.

[0067] In one example, the mask generation system 136 can implement one or more machine learning models (referred to as a first model) when generating the one or more masks. For example, the mask generation system 136 can implement at least a portion of the environment 300 of FIG. 3, among others (e.g., the anatomy segmentation model 306) to generate the masks. In examples, the mask generation system 136 can provide data associated with the frames to the first model to cause the first model to generate an output. The output can include data associated with (e.g., identifying) the one or more masks. The first model that generates the masks can implement one or more layers that are associated with compression of the frames into a compact feature representation. This feature representation can then be provided to a decoder head that is trained to receive the compact feature representation and generate a segmentation mask corresponding to one or more anatomical structures represented by the frame. The first model can also generate one or more annotations indicating a type of an anatomical structure corresponding to a given mask. For example, the first model can output a mask (e.g., via a decoder head of a segmentation model, a fully-connected layer) comprising a plurality of pixels, where each pixel is further associated with a tag indicating the anatomical structure (e.g., liver, lung, colon) that corresponds to the masks.

[0068] The mask generation system 136 can receive one or more depth maps from the depth estimation system 137. For example, the mask generation system 136 can receive one or more depth maps from the depth estimation system 137, where each depth map corresponds to a frame provided to the mask generation system 136 (e.g., by the frame identification system 134. In this example, the mask generation system 136 can include (e.g., append or associate) the depth maps (e.g., depths corresponding to each pixel of the depth maps) to the masks generated by the mask generation system 134.-20-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0069] The mask generation system 136 can generate one or more meshes (sometimes referred to as 3D meshes). For example, the mask generation system 136 generate one or more 3D meshes. The mask generation system 136 can construct a 3D mesh of a scene represented by (e.g., captured in) a frame based on one or more masks including one or more labels and depth maps including depth values corresponding to the pixels (e.g., individual pixels or groups of pixels) of a frame that can be color coded or shaded based on their metric distance from the camera. For example, the mask generation system 136 can create a 3D model of the patient's internal anatomy structures, allowing clinicians to visualize the exact locations and relationships of various anatomical structures. The 3D model representation in the 3D mesh can be implemented with respect to a frame of reference of a data capture device 112 (e.g., the camera that captured the frame). The 3D mesh can represent relative locations of various anatomical structures (or portions thereof) from other anatomical structures. This can greatly aid in preoperative planning, intraoperative navigation, and postoperative analysis. The 3D mesh can include any 3D representation of a scene captured in an image frame. The 3D mesh can be a graphic representation of a 3D area. For example, the 3D mesh can include a file having one or more vertices, edges and faces that provide a 3D representation of the anatomical structures in a given frame.

[0070] The depth estimation system 137 (sometimes referred to as a depth estimator) can obtain data associated with frames from the frame identification system 134. For example, the depth estimation system 137 can obtain the data associated with the frames from the frame identification system 134 to generate one more depth maps based on each frame. In examples, the data associated with the frames can be the same as the data obtained by the mask generation system 136. The depth estimation system 137 can generate the one or more depth maps (e.g., by a model that is the same as, or similar to, a depth estimation model such as the metric depth estimation model 308 of FIG. 3) where the one or more depth maps include an array of values corresponding to distances between a camera that captured the frame and objects in a field of view of the camera.

[0071] In one example, the depth estimation system 137 can implement one or more machine learning models (referred to as a second model) when generating the one or more depth maps. For example, the depth estimation system 137 can implement at least a portion of the environment 300 of FIG. 3, among others (e.g., a model that is the same as, or similar to, the metric depth estimation model 308) to generate the depth maps. In examples, the depth estimation system 137 can provide data associated with the frames to the second model to cause the second model to generate an output. The output can include data associated with the-21-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) one or more depth maps corresponding to a given frame. The second model that generates the depth maps can use one or more neural networks (e.g., attention-based networks, convolutional neural networks (CNNs), combinations thereof) to learn hierarchical features from the input frame, which are then used to estimate the depth of each pixel in the frame. The second model can be trained on datasets containing frames and their corresponding depth maps, allowing the second model to learn the relationships between the visual features in the image and the depth information. During inference, the second model can take a single image as input, processes the input by providing the data associated with the input through the one or more neural networks forming the second model, and generate an output including data associated with a depth map that represents the estimated depth of each pixel in the image.

[0072] The depth estimation system 137 can include any functionality for generating depth maps (sometimes referred to as a metric depth map or a depth mask) including a plurality of depth values representing a depth for each pixel of a given frame. The depth estimation system 137 can generate labels for each pixel of the depth maps. For instance, the depth estimation system 137 can provide data associated with one or more machine learning models (sometimes referred to as a label annotator) to assign labels to various pixels of the depth map.

[0073] The depth estimation system 137 can provide one or more frames as input the second model, where the second model includes one or more layers forming an encoderdecoder structure and one or more layers forming a transformer to map the frame to a corresponding depth values. The depth estimation system 137 can generate a depth map for each frame. Each pixel in the resulting depth map can represent a real metric distance (e.g., a measurement of 1cm, 2cm, 5cm) from the camera center to a point in the scene as represented by the frame. The depth estimation system 137 can use camera intrinsic parameters for calibration during training, allowing for accurate depth estimation. The output depth map can be color-coded, with different colors or shades indicating various distances.

[0074] In examples, the depth estimation system 137 can generate a depth map that includes a distance for each of the one or more pixels of the frame received from the frame identification system 134. In this example, each depth can represent a distance from a lens of a camera to a surface along one or more objects within the medical environment 110 that are visualized by the frame. The depth estimation system 137 can implement the second model to maps frames to corresponding metric depths. In examples where the depths are represented as pixel intensities, each pixel intensity can correspond to a depth value that represents a real metric distance between a reference point of a camera (e.g., the camera lens center) to a point -22-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) of the scene. In examples, the intensities can be represented using one or more colors that color code the depth map, such that different distances are colored or shaded. Each coloring or shade intensity can represent a real distance from the shaded region (e.g., one or more pixels) to the lens of the that captured the frame. In some examples, the depth estimation system 137 can provide one or more depth maps to the mask generation system 136 to allow the mask generation system 136 to generate one or more masks based on the depth maps.

[0075] The correspondence system 138 can determine a correspondence between the one or more masks and a depth map. For example, the correspondence system 138 can determine the correspondence between the one or more masks and the depth map based on the mask generation system 136 generating the one or more masks that segment the anatomical structures for a given frame and the depth estimation system 137 generating the depth map for the given frame. In this example, the correspondence system 138 can determine the correspondence based on the correspondence system 138 localizing (e.g., matching) the points of the mask with the points in the depth map. In an example, the correspondence system 138 can directly compare the one or more masks and the depth map for a given frame to determine a correlation between the one or more masks and the depth map, the direct correlation being based on one or more intrinsic properties (e.g., focal length, principle point offset, skew factor, aspect ratio) of the imaging device that captured the frame. This can allow the correspondence system 138 to identify the correspondence between the segmented objects associated with the one or more masks and their corresponding depths in the depth map. This can also allow the correspondence system 138 to determine one or more transformations for the depth map such that the depth map is adjusted to correspond to the one or more masks.

[0076] The measurement generation system 140 can determine at least one measurement for at least one object in the frame. For example, the measurement generation system 140 can determine the at least one measurement for the at least one object in the frame based on the one or more masks generated by the mask generation system 136 and the one or more correspondences generated by the correspondence system 138 between the one or more masks and the one or more depth maps. In examples, the measurement generation system 140 can determine the at least one measurement between two or more points in the frame based on the correspondence between the pixels of the frame and the depth map. The measurements generated by the measurement generation system 140 described herein can be 2D or 3D where contextually appropriate.-23-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0077] The measurement generation system 140 can determine the at least one measurement where the at least one measurement includes a distance along an axis of the anatomical structure associated with a mask. For example, the measurement generation system 140 can determine the at least one measurement, where the at least one measurement corresponds to a visible portion of the anatomical structure covered by a mask. In some examples, the measurement can extend between two or more points along the mask (e.g., along an exterior portion of the mask). The two or more points can extend along an axis associated with the anatomical structure and be specified by a clinician operating the RMS 120, by a clinician reviewing the data generated by the RMS 120 after the surgical procedure, or by the data processing system 130 when processing the data generated by the RMS 120. In examples where the data processing system 130 specifies the two or more points, the data processing system 130 can specify points that are associated with predetermined positions or features of the anatomical structure (e.g., points corresponding to an iliac crest or the like). In some examples, the data processing system 130 can specify the two or more points, where the two or more points are located at left-most, right-most, upper-most, or lower-most portions of the mask viewed from the frame (e.g., the extents of the anatomical structure as represented in the frame).

[0078] The measurement can extend along one or more axis associated with (e.g., defined by) the anatomical structure. For example, where the anatomical structure includes a portion of a cystic duct, the at least one measurement can include a measurement along an axis associated with midpoints for the cystic duct. In this example, the measurement generation system 140 can fit a line along an axis extending along an upper portion of the cystic duct toward a lower portion of the cystic duct based on edges formed by the mask segmenting the cystic duct. In examples, the measurement can be along a straight line or a spline formed by one or more of the points along the mask.

[0079] The measurement generation system 140 can determine the at least one measurement where the at least one measurement includes a first measurement and a second measurement, the first measurement extending between a first point and a second point associated with the mask and the second measurement extending between a third point and the first point, the second point, or a fourth point along the mask. For example, the measurement generation system 140 can determine the first measurement, the first measurement extending along a first axis of the anatomical structure. The measurement generation system 140 can then determine a second measurement along a second axis. In some examples, the second measurement can extend along a second axis that is transverse -24-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) relative to the first axis when viewed in the frame (e.g., along a plane defined by the first axis and the second axis). In other examples, the second measurement can extend along a second axis that is non-transverse relative to the first axis when viewed in the frame (e.g., along the plane defined by the first axis and the second axis). The first axis or the second axis can be positioned in accordance with a predetermined configuration relative to the anatomical structure. For example, the measurement generation system 140 can determine one or more predetermined axis along a given anatomical structure (e.g., a left or right lung of a patient). The measurement generation system 140 can then determine an orientation of the anatomical structure as represented in the frame by the mask and a correspondence between the orientation of the anatomical structure and one or more normal orientations (e.g., a front view, a top view of the anatomical structure that are further associated with the predetermined axis).

[0080] In examples, the measurement generation system 140 can determine the measurement when extending along one or more axis associated with (e.g., defined by) the anatomical structure. For example, where the anatomical structure includes a portion of a cystic duct, the at least one measurement can include a measurement along an axis associated with midpoints for the cystic duct. In this example, the measurement generation system 140 can fit a line along an axis extending along an upper portion of the cystic duct toward a lower portion of the cystic duct. The measurement generation system 140 can fit the line based on edges formed by the mask segmenting the cystic duct from one or more other anatomical structures or one or more instruments captured by the frame.

[0081] The measurement generation system 140 can determine at least one measurement based on a surface area of the anatomical structure. For example, the measurement generation system 140 can determine one or more points associated with a mask and measure the distances between the points of interest in the frame. The measurement generation system 140 can then divide the image into triangles (or other suitable shapes) formed by connecting the points of interest and calculate the area of each triangle using the distances between the points as the sides of the triangle. In examples, the measurement generation system 140 can add the areas of all the triangles to obtain the total surface area of the region of interest in the frame.

[0082] The measurement generation system 140 can determine at least one measurement based on a perimeter of the anatomical structure. For example, the measurement generation system 140 can determine one or more points associated with an edge of the anatomical structure. In this example, the measurement generation system 140 can then determine a-25-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) length of the edge of the anatomical structure. In some examples, the measurement generation system 140 can infer estimate lengths of portions of the edge of the anatomical structure that are not visible and combine the estimated lengths with the lengths of the visible portions of the anatomical structure to determine the at least one measurement of the perimeter of the anatomical structure.

[0083] The measurement generation system 140 can determine at least one measurement based on an inferred volume of the anatomical structure. For example, the measurement generation system 140 can determine one or more measurements (e.g., a length, width, surface area) of the anatomical structure and infer a volume of the anatomical structure. In examples, the measurement generation system 140 can infer the volume of the anatomical structure based on the measurement generation system 140 comparing the one or more measurements to one or more models (e.g., predetermined models) of the anatomical structure. In these examples, the measurement generation system 140 can determine the inferred volume of the anatomical structure by matching the volume of the model of the anatomical structure to the volume of the anatomical structure being measured in accordance with a scale associated with the differences in the measurements.

[0084] The measurement generation system 140 can provide the one or more measurements for a given frame to the GUI element generator 142. For example, the measurement generation system 140 can provide the one or more measurements to the GUI element generator 142 to allow the GUI element generator 142 to generate one or more GUIs illustrating the measurements. The one or more GUIs can include representations of the frames and corresponding representations of the measurements generated by the measurement generator system 140. The data processing system 130 can be configured to cause the GUI element generator 142 to generate the GUIs described herein based on input provided by a clinician at the RMS 120 requesting the measurements. For example, as a clinician interacts with the RMS 120 during a surgical procedure, the clinician can provide input via the RMS 120 requesting measurements of one or more anatomical structures involved in the surgical procedure. The RMS 120 can then communicate with the data processing system 130 to obtain the one or more GUIs output by the GUI element generator 142 and display the one or more GUIs via the display 118. In examples, the clinician can provide input to stop the display of the one or more GUIs and the RMS 120 can forgo obtaining the data associated with the GUIs from the data processing system 130. In this way, the mask or the measurement can be visually toggled by clinicians during or after surgical procedures for preselected anatomical structures. In other examples, as the clinician interacts -26-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) with the RMS 120 during the surgical procedure, the RMS 120 can determine one or more predetermined phases of the surgical procedure during which the measurements are to be displayed. In these examples, the RMS 120 can perform the above-described operations in the background and obtain the data associated with the GUIs from the data processing system 130 and display the GUIs via the display 118 to automatically display the measurements as predetermined for the phases of the surgical procedure.

[0085] The data processing system 130 can be configured to monitor measurements of anatomical structures generated by the measurement generation system 140 and generate one or more alerts, alarms, warnings or other actions or operations based on the measurements satisfying one or more thresholds. For example, the data processing system 130 can identify one or more measurements based on the frames obtained from the data repository 132 and determine that one or more measurements exceed a range of measurements (e.g., corresponding to an acceptable amount of tissue engagement or the like). The data processing system 130 can then generate warning data associated with a warning to be displayed to a clinician during a medical procedure and provide the warning data to the RMS 120. The RMS 120 can then cause a display device to provide the warning to the clinician during the medical procedure.

[0086] The data processing system 130 can be configured to cause the GUI element generator 142 to generate the GUIs based on input received by the post-processing system 144. For example, the computing device 150 can receive input from a clinician that causes the computing device 150 to obtain data associated with the GUIs and display the GUIs on the display device 152. In this way, the computing device 150 can generate data that causes one or more systems of the data processing system 130 (e.g., the GUI element generator 142, the post-processing system 144) to identify frames responsive to a request (e.g., a query) input by a clinician and provide data to the computing device 150 allowing for the display of frames responsive to the request.

[0087] In an example, the data processing system 130 can cause the GUI element generator 142 to generate the one or more GUIs based on the data processing system 130 receiving data associated with input from the computing device 150. In examples, the data processing system 130 can cause the GUI element generator 142 to generate the one or more GUIs at predetermined points in time corresponding to phases of a surgical procedure. In some examples, the data processing system 130 can cause the GUI element generator 142 to generate the one or more GUIs when processing data associated with one or more surgical procedures stored in the data repository 132.-27-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0088] The GUI element generator 142 can generate the one or more GUIs based on the one or more measurements obtained from the measurement generation system 140. For example, the GUI element generator 142 can generate the one or more GUIs by including an indication of the one or more measurements on the one or more GUIs as a visual representation of the at least one measurement. The visual representation can be based on the data generated by the mask generation system 136 (e.g., the one or more masks) or the data associated with the measurement generation system 140 (e.g., the one or more measurements). For example, the GUI element generator 142 can obtain the data associated with the frame that is identified by the frame identification system 134 and the one or more masks that are generated by the mask generation system 136. The GUI element generator 142 can then overlay the one or more masks onto the frame and overlay the indication of the one or more measurements on the one or more GUIs in association with the one or more masks. In some examples, the indication of the one or more measurements can include a numerical representation of the one or more measurements that are obtained from the measurement generation system 140. In examples, the indication of the one or more measurements can include an icon indicating the one or more measurements such as a line having one or more segments, where each segment represents a predetermined length. The mask can include a box that is drawn around the anatomical structure and indicate a desired threshold surface area of the anatomical structure for which a given task (e.g., ligation or the like) is to be performed.

[0089] The GUI element generator 142 can generate display data associated with the visual representation of the at least one measurement. For example, the GUI element generator 142 can generate display data associated with the visual representation of the at least one measurement such that the display data is configured to cause display devices to output the visual representation of the at least one measurement. In an example, during a surgical procedure involving the RMS 120, the RMS 120 can receive input from one or more devices in the medical environment 110 operated by a clinician requesting the visual representation of the at least one measurement. In this example, the data processing system 130 can obtain data associated with the input from the RMS 120 and cause one or more of the systems of the data processing system 130 to perform one or more of the operations described herein. This can allow the GUI element generator 142 to generate the display data and provide the display data to the RMS 120 during a surgical procedure. The RMS 120 can then provide the display data to the display 118 of the medical environment 110 to cause the-28-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) display to generate the GUI including the visual representation of the at least one measurement.

[0090] With continued reference to FIG. 1, among others, the post-processing system 144 can be configured to process data stored in the data repository 132 corresponding to surgical procedures to include one or more of the measurements with the one or more frames. For example, the post-processing system 144 can be configured to obtain the data from the data repository 132 and annotate one or more frames associated with the data from the data repository 132 based on (e.g., with) the one or more masks or the one or more measurements generated by the systems described herein. In this example, the post-processing system 144 can obtain the data involved in annotating the one or more frames from the mask generation system 136 or the measurement generation system 140 and annotate the frames. The postprocessing system 144 can continuously (e.g., as surgical procedures are performed) or periodically (e.g., at predetermined time intervals or on request from a clinician) process the data stored in the data repository 132 by adding the masks generated by the mask generation system 136 or the measurements generated by the measurement generation system 140 to the frames. The post-processing system 144 can then store the frame data that is updated to include the masks or the measurements in the data repository 132 for later retrieval and display by the computing device 150.

[0091] The post-processing system 144 can determine one or more objective performance indicators (OPIs) based on the frames processed by the post-processing system 144 and stored in the data repository 132 (e.g., as part of an endoscope log stored in the data repository 132). For example, the post-processing system 144 can compare one or more of the measurements for an anatomical structure included in a given frame to one or more measurements that are expected for similar anatomical structures in the same or similar scenarios. In an example, the post-processing system 144 can determine a measurement of a size of an anatomical structure being researched (e.g., a bladder neck, urethra, tumor resections in wedges, partial nephrectomies). The measurement can then be compared to one or more predetermined values representing e.g., an average height, length, width, surface area (or ranges thereof) for the anatomical structure as represented in the data repository 132, and the post-processing system 144 can determine a difference between the measurement and the metric. The post-processing system 144 can then correlate the difference between the measurement and the metric to determine the one or more OPIs. The measurement of the size of the anatomical structure can be associated with a given patient involved in a surgical procedure, and the one or more predetermined values can be determined based on-29-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) measurements of anatomical structures of other patients. The post-processing system 144 can cause a GUI to be generated representing the OPIs. For example, the post-processing system 144 can cause a GUI to be generated representing the OPIs during replay of one or more surgical procedures, as part of a report analyzing the one or more surgical procedures, a report analyzing the performance of a surgeon that performed one or more surgical procedures using the RMS 120. In this example, the OPI can be displayed as an indication included in the GUI.

[0092] In examples, the post-processing system 144 can determine one or more OPIs representing a degree to which movements of medical instruments 114 and subsequent engagement of tissue of a patient during surgical procedures. For example, the postprocessing system 144 can determine the one or more OPIs based on comparing the measurements representing e.g., engagement of tissue by a clinician during a phase of a surgical procedure with similar engagements of tissue by the clinician or other clinicians during similar phases of surgical procedures. The post-processing system 144 can then cause the GUI element generator 142 to output an indication of the performance of the clinician for a given frame as compared to earlier-generated frames.

[0093] In examples, the post-processing system 144 can determine one or more measurements that were generated during a surgical procedure and then determine an OPI (e.g., a clinician score representing a skill level of a patient, a complexity score representing a complexity of a maneuver of a surgical procedure or groupings of maneuvers) based on the one or more measurements. For example, the post-processing system can determine a first measurement associated with a clinician providing input via the RMS 120 during a surgical procedure such that, prior to ligation, 5cm2of a cystic duct of a patient is exposed. The postprocessing system 144 can determine a second measurement based on a second clinician (clinician B) exposing 7cm2during a similar procedure and at a similar stage of the procedure. The post-processing system 144 can then determine a difference (e.g., 2cm2) between the portion of the cystic duct exposed by clinician A and clinician B and determine an OPI (e.g., a clinician score) based on the difference. In this example, the OPI can represent an amount by which a clinician (clinician B) exceeds exposing the portion of the cystic duct as compared to other clinicians (clinician A).

[0094] The post-processing system 144 can be configured to receive data associated with a query and search a dataset stored in the data repository 132 for one or more frames or one or more groups of frame responsive to the query. For example, the post-processing system 144 can receive data generated based on input provided by a clinician or researcher at the -30-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) computing device 150 representing the query. In this example, the query can specify an anatomical structure for which a mask was generated during a surgical procedure by the mask generation system 136, an example measurement or range of measurements of an anatomical structure processed by the data processing system 130 or the like, an example set of measurements or sets of ranges of measurements (e.g., along one or more axis of an anatomical structure, along a surface of an anatomical structure, a surface area, or combinations thereof). The post-processing system 144 can then compare one or more aspects of the query to the data associated with the frames stored in the data repository and identify the one or more frames that satisfy the query. In some examples, the post-processing system 144 can generate a GUI to display the one or more frames responsive to the query and transmit data associated with the GUI to the computing device 150. In these examples, the data associated with the GUI can be configured to cause the display device 152 to generate the GUI as an output. In these examples, the GUI can be configured to display one or more of the frames identified by the post-processing system as satisfying the query either individually (e.g., as an image) or collectively (e.g., as a video where desired measurements (e.g., portions of the video with a largest viewable surface area) are represented in the frames). In examples, the query can also specify one or aspects that indicate one or more filters to be applied. For example, the post-processing system 144 can receive data generated based on input provided by a clinician representing the query, where the query specifies one or more measurements or ranges of measurements for a given anatomical structure to be excluded. The post-processing system 144 can then compare one or more aspects of the query to the data associated with the frames stored in the data repository and identify the one or more frames responsive to the query (e.g., one or more frames that do not include the anatomical structure where a measurement of the anatomical structure satisfies the filter). In examples, the post-processing system 144 can overlay one or more measurements when generating the GUI such that the one or more measurements are included in the reproduced frames.

[0095] The post-processing system 144 can be configured to obtain the data stored in the data repository 132 and parse the data in the data repository into one or more datasets. For example, the post-processing system 144 can parse the data in the data repository into one or more datasets based on the measurements associated with the anatomical features represented in the frames. In an example, the post-processing system can determine that one or more measurements corresponding to one or more anatomical structures are outliers (e.g., outside of a range of measurements for similar portions of anatomical structures as represented by a dataset in the post-processing system 144) and store the frames associated with the outliers in -31-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) a second dataset. In this example, the post-processing system 144 can determine that the one or more measurements are outliers where the measurements satisfy one or more outlier thresholds that are based on one or more measurements or ranges of measurements corresponding to average measurements for the anatomical structures. In this way, the postprocessing system 144 can normalize datasets stored in the data repository 132 and include outliers in datasets associated with edge cases.

[0096] The post-processing system 144 can be configured to obtain the data stored in the data repository 132 and generate visual indications (e.g., plots) of metrics determined by the components of the data processing system 130 over time. For example, the post-processing system 144 can determine an aspect of a metric (e.g., measurements of a surface area of an anatomical structure) over a period of time (e.g., the course of a surgical procedure). The post-processing system 144 can generate GUIs (e.g., single images or videos) representing changes in the metric over time (e.g., a line graph that is overlaid onto the frame of the image of frames of the videos) to indicate the state of the metric at one or more points in time during the surgical procedure. The post-processing system 144 can then store data associate with the GUIs in the data repository 132 or provide the data to the computing device 150 to cause the computing device to display the GUI via the display device 152 of the computing device 150.

[0097] With continued reference to FIG. 1, among others, the computing device 150 can include any combination of hardware or software that perform one or more of the functions described herein. For example, the computing device 150 can include any combination of hardware and software that receive and generate data associated with input received via the input device 154 and display a GUI via the display device 152. The computing device 150 can be the same as, or similar to, the computing device 600 of FIG. 6 or other computing devices described herein, and can include one or more tablets, laptops, desktops, servers, or virtual machines, or can be part of or include a cloud computing environment.

[0098] The computing device 150, or components thereof, can include a display device 152. The display device 152 can include any suitable display device such as a monitor, a touchscreen monitor, a liquid crystal display (LCD) monitor, a light emitting diode (LED) monitor. The computing device 150 can include an input device 154. The input device 154 can include any suitable input device 154 such as a keyboard, a mouse, a touchscreen, combinations thereof.

[0099] The computing device 150, or components thereof, can be located at least partially at the location of the surgical facility associated with the medical environment 110 or remotely therefrom. Elements of the computing device 150, or components thereof, can be -32-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The computing device 150, or components thereof, can include other or additional elements that can be considered desirable to have in performing the functions described herein. The computing device 150, or components thereof, can include, or be associated with, one or more components or functionality of a computing including, for example, one or more processors coupled with memory that can store instructions, data or commands for implementing the functionalities of the computing device 150 discussed herein.

[0100] FIG. 2 depicts a flowchart diagram illustrating an example method 200 for automated measurement of objects in a field of view of a robotic system, according to some embodiments. The method 200 can be performed by one or more systems, devices, or components depicted in FIGS. 1, 5, or 6 including, for example, the data processing system 130 of FIG. 1, among others.

[0101] At operation 210, a frame that captures a scene in a medical procedure is identified. For example, one or more components of a data processing system (e.g., that is the same as, or similar to, the data processing system 130 of FIG. 1) can identify a frame of a scene captured during a surgical procedure. In this example, the frame can include a representation of one or more anatomical structures involved in the surgical procedure.

[0102] At operation 220, one or more masks that segment an anatomical structure in the frame can be generated. For example, one or more components of the data processing system can generate the one or more masks based on the data associated with the frame. In examples, the data processing system can cause one or more components to implement a first model (e.g., that is the same as, or similar to, the anatomy segmentation model 306 of FIG. 3) to generate the one or more masks. In these examples, the one or more components of the data processing system can provide the data associated with the frames to the first model to cause the first model to generate an output, the output representing the one or more masks.

[0103] At operation 230, a correspondence is determined between the one or more masks and a depth map. For example, one or more components of the data processing system can determine the correspondence between the one or more masks and the depth map. In examples, the depth map can be generated by the data processing system and the correspondence determined based on the generation of the depth map.

[0104] At operation 240, at least one measurement along an axis defined by the anatomical structure can be determined. For example, the one or more components of the data processing system can determine the at least one measurement. The at least one measurement can be between two or more points associated with the mask. In examples, the at least one-33-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) measurement can include a measurement of a surface area represented by the mask. In these examples, the surface area can include an amount of the surface visible to the clinician during the surgical procedure.

[0105] At operation 250, display data associated with a visual representation of the at least one measurement can be generated. For example, the one or more components of the data processing system can generate the display data. The one or more components of the data processing system can generate the display data in response to receiving input indicating input (e.g., from a robotic medical system) to generate the display data. The display data can be configured to cause a display device (e.g., of a computing device that is the same as, or similar to, the computing device 150 of FIG. 1) to display the visual representation. The visual representation can include a GUI having the one or more masks and the at least one measurement included therein.

[0106] FIG. 3 depicts an example environment 300 for automated measurement of objects (e.g., anatomical structures or the like) in a field of view of a robotic system. The combination of models in the environment 300 can include an anatomy segmentation model 306 and a metric depth estimation model 308 that are combined to perform one or more operations in combination when measuring anatomical structures illustrated in frames 302 of a surgical procedure. Each frame 302 can represent a scene 303 of a surgical procedure. As described herein, the anatomy segmentation model 306 can be configured to identify and localize anatomical structures of interest from frames (e.g., surgical images) generate during surgical procedures. The metric depth estimation model 308 can be configured to infer the metric depth (e.g., distances between a camera and respective points along a surface of a visualized anatomical structure such as a cystic duct or the like) of the view in the image. The geometric dimensions of the anatomical structures represented by a given frame can be determined (e.g., derived) by combining the mask (also referred to as a segmentation mask) and the depth map to determine metric (length) measurements. One or more of the components of the example environment 300 can be implemented by one or more components of a data processing system (e.g., that is the same as, or similar to, the data processing system 130 of FIG. 1).

[0107] Data associated with a frame 302 generated during a surgical procedure can be discretized by the data processing system. The data associated with each frame 302 can then be sequentially processed by the data processing system and provided to the anatomy segmentation model 306 and the metric depth estimation model 308. Data associated with one or more imaging device configurations (e.g., camera intrinsic properties or the like) can -34-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) be combined with the frame 302 and provided to the anatomy segmentation model 306 or the metric depth estimation model 308.

[0108] The data processing system can implement the anatomy segmentation model 306. For example, the anatomy segmentation model 306 can be configured to receive the data associated with a frame 302 or the data associated with one or more imaging device configurations. The anatomy segmentation model 306 can implement one or more components associated with a vision transformers (ViT). For example, the anatomy segmentation model 306 can be configured to receive the frame 302 (or a series of frames 302) as input and output corresponding segmentation masks 310 including one or more individual segmentation masks 314. The anatomy segmentation model 306 can include a type of neural network architecture that implements self-attention mechanisms to model global dependencies between the input data and the output data. When segmenting each frame 302, the anatomy segmentation model 306 can process the frames 302 individually or as a group (to maintain temporal dependencies) to identify and delineate specific anatomical structures. The anatomy segmentation model 306 can be trained to map the input frames 302 to the corresponding segmentation masks 310, where each pixel in the segmentation mask 310 represents the probability that the corresponding pixel in the frame 302 belongs to a specific region associated with an anatomical structure. The anatomy segmentation model 306 can include a series of self-attention and feed-forward layers, which enable the model to capture long-range dependencies and contextual information in the medical images, leading to accurate segmentation masks 310. As illustrated, the segmentation mask 310 can include individual masks 314.

[0109] The data processing system can implement the metric depth estimation model 308. The metric depth estimation model 308 include a neural network that is trained on a dataset of frames 302 paired with corresponding depth maps 312. The metric depth estimation model 308 can include convolutional layers to extract relevant features from the frames 302, followed by fully connected layers to predict the depth values for each pixel of the frame 302. The metric depth estimation model 308 can be trained using a loss function that measures the difference between the predicted depth map 312 and a ground truth depth map, and the weights of the metric depth estimation model 308 can be iteratively adjusted to minimize loss. The metric depth estimation model 308 can also be calibrated based on intrinsic parameters associated with the input device 304 that generated the frames 302. Once trained, the metric depth estimation model 308 can receive frames 302 as input and output a depth maps 312.-35-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)

[0110] The anatomy segmentation mask 310 can indicate the anatomical structure(s) in a given frame 302. To measure the size / dimension of the anatomical structure(s), the data processing system can localize the points of the individual masks 314 and find a correspondence in the depth map 312. The data processing system can then determine one or more shapes formed by the mask based on the correlation between the mask 314 and the depth map 312. The data processing system can also update the correlation (e.g., calibrate) based on the one or more intrinsic parameters of the imaging device involved in generating the frame 302. The data processing system can determine distances between two or more points in the metric depth map 312 when determining corresponding measurements. The data processing system can then determine (e.g., derive) the size of the anatomical structure from the measurements. The size of the anatomical structure can be represented in a user interface 316 generated by the data processing system based on the metric depth map 312 or the mask 314.[OHl] FIG. 4 depicts example graphical user interfaces (GUIs) 400, 400', 400" generated to indicate measurements of objects in a field of view of a robotic system, according to some embodiments. The GUIs 400, 400', 400" can be generated based on performance of one or more operations by a data processing system that is the same as, or similar to, the data processing system 130 of FIG. 1. In examples, the GUIs can be generated by one or more other components such as one or more components of a robotic medical system that is the same as, or similar to, the RMS 120 of FIG. 1, one or more components of a medical environment that is the same as, or similar to, the medical environment 110 of FIG. 1, or one or more computing devices that are the same as, or similar to, the computing device 150 of FIG. 1.

[0112] The GUI 400 includes a representation of a first instrument 402a and a second instrument 402b. The first instrument 402a and the second instrument 402b can be represented when engaging (e.g., grasping) an anatomical feature 404. For example, the first instrument 402a and the second instrument 402b can be engaging the anatomical feature 404 when exposing the anatomical feature 404 to allow for measurement of extents of the anatomical feature represented by the GUI 400. The GUI 400 can include a bounding box 406 that encloses the anatomical feature 404. For example, extents of the anatomical feature 404 can be enclosed by the bounding box 406. In examples, the length and width of the bounding box 406 can be indicated as described herein.

[0113] The GUI 400' includes a representation of a first instrument 402a and a second instrument 402b engaging the anatomical feature 404. For example, the first instrument 402a -36-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) and the second instrument 402b can be engaging the anatomical feature 404 when exposing the anatomical feature 404 to allow for measurement of the surface area of the anatomical feature. The GUI 400 can include a mask 408 that encloses the anatomical feature 404. For example, the mask 408 can be overlaid onto portions of the anatomical feature 404 visible in the representation of the GUI 400'. In examples, a measurement of the surface area of the visible or non-visible portions of the mask 408 can be indicated as described herein.

[0114] The GUI 400" includes a representation of a first instrument 402a and a second instrument 402b engaging the anatomical feature 404. For example, the first instrument 402a and the second instrument 402b can be engaging the anatomical feature 404 when exposing the anatomical feature 404 to allow for measurement of the extents and the surface area of the anatomical feature 404. The GUI 400 can include a bounding box 410 enclosing the anatomical feature 404 and a mask 408 overlaid onto the visible portions of the anatomical feature 404. One or more measurements that are associated with the bounding box 410 and the mask 412 are overlaid onto the GUI 400". For example, a measurement 414 indicating a maximum diagonal length associated with a diagonal measurement of the bounding box 410 and a measurement indicating the surface area of the mask 412 can be overlaid onto a portion (e.g., a bottom-right portion) of the GUI 400".

[0115] FIG. 5 depicts a diagram of a medical environment 500, according to some embodiments. The medical environment 500 can refer to or include a surgical environment or surgical system. The medical environment 500 can include a robotic medical system 524 (e.g., a robotic medical system that is the same as, or similar to, the RMS 120 of FIG. 1), a user control system 510, and an auxiliary system 515 communicatively coupled one to another. A visualization tool 520 can be connected to the auxiliary system 515, which in turn can be connected to the robotic medical system 524. Thus, when the visualization tool 520 is connected to the auxiliary system 515 and this auxiliary system is connected to the robotic medical system 524, the visualization tool can be considered connected to the robotic medical system. The visualization tool 520 can be directly connected to the robotic medical system 524.

[0116] The medical environment 500 can be used to perform a computer-assisted medical procedure with a patient 525. Surgical teams can include a surgeon 530A and additional medical personnel 530B-530D such as a medical assistant, nurse, and anesthesiologist, and other suitable team members who can assist with the surgical procedure or medical session. The medical session can include the surgical procedure being performed on the patient 525, as well as any pre-operative (e.g., which can include setup of the medical environment 500, -37-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) including preparation of the patient 525 for the procedure), and post-operative (e.g., which can include clean up or post care of the patient), or other processes during the medical session. Although described in the context of a surgical procedure, the medical environment 500 can be implemented in a non-surgical procedure, or other types of medical procedures or diagnostics that can benefit from the accuracy and convenience of the surgical system.

[0117] The robotic medical system 524 can include a plurality of manipulator arms 535A-535D to which a plurality of medical instruments (e.g., the instruments described herein) can be coupled to, installed to, or supported by. The plurality of manipulator arms 535A-535D can include one or more linkages. Each medical instrument can be any suitable surgical tool (e.g., a tool having tissue-interaction functions), imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or other suitable instrument that can be used for a computer-assisted surgical procedure on the patient 525 (e.g., by being at least partially inserted into the patient and manipulated to perform a computer-assisted surgical procedure on the patient). Although the robotic medical system 524 is shown as including four manipulator arms (e.g., the manipulator arms 535A-535D), in other embodiments, the robotic medical system can include greater than or fewer than four manipulator arms. Further, not all manipulator arms can have a medical instrument installed thereto at all times of the medical session. Moreover, a medical instrument installed on a manipulator arm can be replaced with another medical instrument as suitable.

[0118] One or more of the manipulator arms 535A-535D or the medical instruments attached to manipulator arms can include one or more displacement transducers, orientational sensors, positional sensors, or other types of sensors and devices to measure parameters or generate kinematics information. One or more components of the medical environment 500 can be configured to use the measured parameters or the kinematics information to track (e.g., determine poses of) or control the medical instruments, as well as anything connected to the medical instruments or the manipulator arms 535A-535D.

[0119] The user control system 510 can be used by the surgeon 530Ato control (e.g., move) one or more of the manipulator arms 535A-535D or the medical instruments connected to the manipulator arms. To facilitate control of the manipulator arms 535A-535D and track progression of the medical session, the user control system 510 can include a display that can provide the surgeon 530A with imagery (e.g., high-definition 3D imagery) of a surgical site associated with the patient 525 as captured by a medical instrument installed to one of the manipulator arms 535A-535D. The user control system 510 can include a stereo -38-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) viewer having two or more displays where stereoscopic images of a surgical site associated with the patient 525 and generated by a stereoscopic imaging system can be viewed by the surgeon 530A. The user control system 510 can also receive images from the auxiliary system 515 and the visualization tool 520.

[0120] The surgeon 530A can use the imagery displayed by the user control system 510 to perform one or more procedures with one or more medical instruments attached to the manipulator arms 535A-535D. To facilitate control of the manipulator arms 535A-535D or the medical instruments installed thereto, the user control system 510 can include a set of controls. These controls can be manipulated by the surgeon 530Ato control movement of the manipulator arms 535A-535D or the medical instruments installed thereto. The controls can be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 530Ato allow the surgeon to intuitively perform a procedure on the patient 525 using one or more medical instruments installed to the manipulator arms 535A-535D.

[0121] The auxiliary system 515 can include one or more computer systems (e.g., computing devices that are the same as, or similar to the computing device 600 of FIG. 6) configured to perform processing operations within the medical environment 500. For example, the one or more computer systems can control or coordinate operations performed by various other components (e.g., the robotic medical system 524, the user control system 510) of the medical environment 500. A computer systems included in the user control system 510 can transmit instructions to the robotic medical system 524 by way of the one or more computing devices of the auxiliary system 515. The auxiliary system 515 can receive and process image data representative of imagery captured by one or more imaging devices (e.g., medical instruments) attached to the robotic medical system 524, as well as other data stream sources received from the visualization tool. For example, one or more image capture devices can be located within the medical environment 500. These image capture devices can capture images from various viewpoints within the medical environment 500. These images (e.g., video streams) can be transmitted to the visualization tool 520, which can then passthrough those images to the auxiliary system 515 as a single combined data stream. The auxiliary system 515 can then transmit the single video stream (including any data stream received from the medical instrument s) of the robotic medical system 524) to present on a display of the user control system 510.

[0122] The auxiliary system 515 can be configured to present visual content (e.g., the single combined data stream) to other team members (e.g., the medical personnel 530B- 530D) who can not have access to the user control system 510. Thus, the auxiliary system -39-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)515 can include a display 640 configured to display one or more user interfaces, such as images of the surgical site, information associated with the patient 525 or the surgical procedure, or any other visual content (e.g., the single combined data stream). The display 540 can be a touchscreen display or include other features to allow the medical personnel 530B-530D to interact with the auxiliary system 515.

[0123] The robotic medical system 524, the user control system 510, and the auxiliary system 515 can be communicatively coupled one to another in any suitable manner. For example, the robotic medical system 524, the user control system 510, and the auxiliary system 515 can be communicatively coupled by way of control lines 545, which can represent any wired or wireless communication link as can serve a particular implementation. Thus, the robotic medical system 524, the user control system 510, and the auxiliary system 515 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0124] It is to be understood that the medical environment 500 can include other or additional components or elements that can be needed or considered desirable to have for the medical session for which the surgical system is being used.

[0125] FIG. 6 depicts a block diagram depicting an architecture for a computing device 600 that can be employed to implement elements of the systems and methods described and illustrated herein, including aspects of the systems depicted in FIGS. 1, 3, or 5, and the method depicted in FIG. 2, among others. For example, some or all of the components of the network 105, the medical environment 110, the RMS 120, the data processing system 130, the computing device 150, or the devices described with respect to medical environment 500 can include one or more component or functionality of computing device 600. The computing device 600 can be any computing device used herein and can include or be used to implement a data processing system or its components. The computing device 600 includes at least one bus 605 or other communication component or interface for communicating information between various elements of the computer system. The computer system further includes at least one processor 610 or processing circuit coupled to the bus 605 for processing information. The computing device 600 also includes at least one main memory 615, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 605 for storing information, and instructions to be executed by the processor 610. The main memory 615 can be used for storing information during execution of instructions by the processor 610. The computing device 600 can further include at least one read only memory (ROM) 620 or other static storage device coupled to the bus 605 for storing static information and-40-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) instructions for the processor 610. A storage device 625, such as a solid-state device, magnetic disk or optical disk, can be coupled to the bus 605 to persistently store information and instructions.

[0126] The computing device 600 can be coupled via the bus 605 to a display 630, such as a liquid crystal display, or active-matrix display, for displaying information. An input device 635, such as a keyboard or voice interface can be coupled to the bus 605 for communicating information and commands to the processor 610. The input device 635 can include a touch screen display (e.g., the display 630). The input device 635 can include sensors to detect gestures. The input device 635 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 610 and for controlling cursor movement on the display 630.

[0127] The processes, systems and methods described herein can be implemented by the computing device 600 in response to the processor 610 executing an arrangement of instructions contained in the main memory 615. Such instructions can be read into the main memory 615 from another computer-readable medium, such as the storage device 625. Execution of the arrangement of instructions contained in the main memory 615 causes the computing device 600 to perform the illustrative processes described herein. One or more processors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 615. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0128] The processor 610 can execute one or more instructions associated with the system 100. The processor 610 can include an electronic processor, an integrated circuit including one or more of digital logic, analog logic, digital sensors, analog sensors, communication buses, volatile memory, nonvolatile memory, and the like. The processor 610 can include, but is not limited to, at least one microcontroller unit (MCU), microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC). The processor 610 can include, or be associated with, a main memory 615 operable to store or storing one or more non-transitory computer-readable instructions for operating components of the system 100 and operating components operably coupled to the processor 610. The one or more instructions can include at least one of firmware, software, hardware, operating systems, or embedded operating-41-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) systems, for example. The processor 610 or the system 100 generally can include at least one communication bus controller to effect communication between the system processor and the other elements of the system 100.

[0129] The main memory 615 can include one or more hardware memory devices to store binary data, digital data. The main memory 615 can include one or more electrical components, electronic components, programmable electronic components, reprogrammable electronic components, integrated circuits, semiconductor devices, flip flops, arithmetic units. The main memory 615 can include at least one of a non-volatile memory device, a solid-state memory device, a flash memory device, a NAND memory device, a volatile memory device, etc. The main memory 615 can include one or more addressable memory regions disposed on one or more physical memory arrays.

[0130] Although an example computing system has been described in FIG. 6, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0131] The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are illustrative, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable or physically interacting components or wirelessly interactable or wirelessly interacting components or logically interacting or logically interactable components.

[0132] With respect to the use of plural or singular terms herein, those having skill in the art can translate from the plural to the singular or from the singular to the plural as is-42-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) appropriate to the context or application. The various singular / plural permutations can be expressly set forth herein for sake of clarity.

[0133] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).

[0134] Although the figures and description can illustrate a specific order of method steps, the order of such steps can differ from what is depicted and described, unless specified differently above. Also, two or more steps can be performed concurrently or with partial concurrence, unless specified differently above. Such variation can depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0135] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the following appended claims can contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

[0136] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having -43-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) skill in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general, such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0137] Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent.

[0138] The foregoing description of illustrative implementations has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or can be acquired from practice of the disclosed implementations. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.4861 -0507-3621.3

Claims

Atty Dkt. No. 135039-0428 (P06967-WO)CLAIMSWhat is claimed is:

1. A system, comprising: one or more processors, coupled with memory, to: identify a frame that captures a scene in a medical procedure performed with a robotic medical system; generate one or more masks that segment an anatomical structure in the frame, each mask of the one or more masks comprising a plurality of labels indicating the anatomical structure captured in the frame, determine a correspondence between the one or more masks and a depth map, the depth map corresponding to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure; determine at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map; and generate display data associated with a visual representation of the at least one measurement, the display data configured to cause a display device to output the visual representation of the at least one measurement.

2. The system of claim 1, wherein the one or more processors are further configured to: determine at least one measurement comprising a distance of the anatomical structure extending along the axis.

3. The system of claim 2, wherein the at least one measurement comprises a first measurement and the axis defined by the anatomical structure is a first axis, and wherein the one or more processors are further configured to: determine a second measurement along a second axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth.

4. The system of claim 3, wherein the first axis and the second axis are substantially transverse to each other along a plane defined by the first axis and the second axis.-45-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)5. The system of claim 3, wherein the first axis and the second axis are non-transverse to each other along a plane defined by the first axis and the second axis.

6. The system of claim 1, wherein the one or more processors to determine the at least one measurement along the axis defined by the anatomical structure are to: determine the at least one measurement based on a surface area of the anatomical structure, a perimeter of the anatomical structure, or a volume of the anatomical structure.

7. The system of claim 1, wherein the anatomical structure is associated with a first patient, and wherein the one or more processors are further configured to: compare the at least one measurement along the axis defined by the anatomical structure to at least one measurement along an axis defined by a second anatomical structure, the second anatomical structure associated with a second patient; determine a difference between the at least one measurement along the axis defined by the anatomical structure and the at least one measurement along the axis defined by the second anatomical structure; and determine a metric based on the difference.

8. The system of claim 7, wherein the at least one measurement along the axis defined by the anatomical structure corresponds to operations performed by robotic surgical system when engaged by a first clinician, wherein the at least one measurement along the axis defined by the second anatomical structure corresponds to operations performed by the robotic surgical system when engaged by a second clinician, and wherein the one or more processors are further configured to: determine a clinician score for the first clinician or the second clinician based on the metric.

9. The system of claim 1, wherein the one or more processors are further configured to: receive an input indicating a reference measurement corresponding to the anatomical structure;-46-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) search a dataset comprising data associated with a first medical procedure and a second medical procedure, the first medical procedure associated with the at least one measurement along the axis defined by the anatomical structure and the second medical procedure associated with at least one second measurement along an axis defined by a second anatomical structure; and determine the reference measurement satisfies the at least one measurement or the at least one second measurement, wherein the one or more processors configured to generate the display data associated with the visual representation of the at least one measurement are configured to: generate the display data based on the visual representation of the at least one measurement and the determination that the reference measurement satisfies the at least one measurement or the at least one second measurement.

10. The system of claim 1, wherein the one or more processors are further configured to: receive an input at the robotic medical system to generate the display data associated with a visual representation of the at least one measurement, and wherein the one or more processors configured to generate the display data associated with the visual representation of the at least one measurement are configured to: generate the display data associated with the visual representation of the at least one measurement based on the input at the robotic medical system to generate the display data.

11. The system of claim 1, wherein the one or more processors are further configured to: determine the frame that captures a scene in a medical procedure corresponds to a phase of the medical procedure; and wherein the one or more processors configured to generate the display data associated with the visual representation of the at least one measurement are configured to: generate the display data associated with the visual representation of the at least one measurement based on the phase of the medical procedure.-47-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO)12. A system comprising: one or more processors configured to: for each patient of a plurality of patients involved in at least one medical procedure performed by a robotic medical system: identify a frame that captures a scene in a medical procedure performed with a robotic medical system; generate one or more masks that segment an anatomical structure in the frame, each mask of the one or more masks comprising a plurality of labels indicating the anatomical structure captured in the frame, determine a correspondence between the one or more masks and a depth map, the depth map corresponding to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure; determine at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map; and update at least one dataset based on the at least one measurement along the axis defined by the anatomical structure.

13. The system of claim 12, wherein the anatomical structure of each of the patients is associated with a type of anatomical structure; and wherein the one or more processors are further configured to: receive an input indicating a reference measurement corresponding to the anatomical structure; compare the reference measurement to the at least one measurement along the axis defined by the anatomical structure for each patient of the plurality of patients; determine that the reference measurement satisfies the at least one measurement along the axis defined by the anatomical structure of at least one patient of the plurality of patients; and generate display data associated with a visual representation of the at least one measurement of the at least one patient, the display data configured to cause a display device to output the visual representation of the at least one measurement.

14. The system of claim 12, wherein the one or more processors are further configured to:-48-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) determine that the at least one measurement of an outlier patient of the plurality of patients satisfies an outlier threshold; and update the at least one dataset based on the at least one measurement of the outlier patient satisfying the outlier threshold.

15. The system of claim 14, wherein the one or more processors configured to update the at least one dataset based on the at least one measurement of the outlier patient satisfying the outlier threshold are configured to: updating a second dataset to comprise the at least one measurement of the outlier patient.

16. A method, comprising: identifying, by at least one processor coupled with memory, a frame that captures a scene in a medical procedure performed with a robotic medical system; generating, by the at least one processor, one or more masks that segment an anatomical structure in the frame, each mask of the one or more masks comprising a plurality of labels indicating the anatomical structure captured in the frame, determining, by the at least one processor, a correspondence between the one or more masks and a depth map, the depth map corresponding to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure; determining, by the at least one processor, at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map; and generating, by the at least one processor, display data associated with a visual representation of the at least one measurement, the display data configured to cause a display device to output the visual representation of the at least one measurement.

17. The method of claim 16, further comprising: determining, by the at least one processor, at least one measurement comprising a distance of the anatomical structure extending along the axis.

18. The method of claim 17, wherein the at least one measurement comprises a first measurement and the axis defined by the anatomical structure is a first axis, and the method further comprising:-49-4861 -0507-3621.3Atty Dkt. No. 135039-0428 (P06967-WO) determining, by the at least one processor, a second measurement along a second axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth.

19. The method of claim 16, wherein determining the at least one measurement along the axis defined by the anatomical structure comprises: determining, by the at least one processor, the at least one measurement based on a surface area of the anatomical structure.

20. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to: identify a frame that captures a scene in a medical procedure; generate one or more masks that segment an anatomical structure in the frame, determine a correspondence between the one or more masks and a depth map, the depth map corresponding to depth values as measured between a sensor used to capture the frame and points along a surface of the anatomical structure; determine at least one measurement along an axis defined by the anatomical structure based on the correspondence between the one or more masks and the depth map; and generate display data associated with a visual representation of the at least one measurement, the display data configured to cause a display device to output the visual representation of the at least one measurement.-50-4861 -0507-3621.3

Images