Unified modeling using segment probabilities for robotic medical procedures

Abstract

A unified model from multiple constituent model predictions for a robotic medical procedure is described. A system can receive data of a robotic medical system and select, based on a number of corrupt samples in the data, a first one or more models from a plurality of models trained with machine learning. The system can update, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure. The system can provide, for display via a graphical user interface, an indication of one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

Description

UNIFIED MODELING USING SEGMENT PROBABILITIESFOR ROBOTIC MEDICAL PROCEDURESCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 665,863, filed June 28, 2024, which is hereby incorporated by reference herein in its entirety.BACKGROUND

[0002] Medical procedures can be performed in an operating room. As the amount and variety of equipment in the operating room increases, or medical procedures become increasingly complex, it can be challenging to accurately and reliably analyze robotic medical procedures.SUMMARY

[0003] Robotic medical procedures (RMPs) conducted using robotic medical systems (RMSs) can vary based on RMP segments, such as the types of procedures performed, individual phases of such procedures, individual steps of such phases or individual actions to perform such steps. ML models can be trained, using RMP data sets, to identify each of these RMP segments using data from the RMS. However, RMS data (e.g., sensor, video, events or kinematics data streams) can encounter various issues, such as data errors, corruption, or loss, complicating accurate and reliable segment determination using ML models utilizing such data as input for their determinations. To address these challenges, technical solutions can monitor and analyze incoming RMS data characteristics to select the most suitable ML models for accurate determination. The technical solutions can provide a unified ML model that can utilize relationships and dependencies between different RMP segment determinations to generate determinations with an improved accuracy and reliability. By selecting the most suitable ML segment models based on data characteristics and employing a unified analysis with a holistic probability-based approach (e.g., Bayesian prediction), the technical solutions provide a more accurate and reliable segment determination at a level that is beyond that of individual ML models alone.

[0004] At least one aspect of the technical solutions is directed to a system. The system can include one or more processors coupled with memory. The one or more processors can be configured to receive data collected by a plurality of sensors of a robotic medical system associated with performance of a medical procedure. The one or more processors can be configured to select, based at least in part on a characteristic of the data and a probability of a procedure type, a probability of a phase type, and a probability of a step type established for at least a segment of a plurality of segments of the medical procedure, a first one or more models from a plurality of models trained with machine learning. The one or more processors can be configured to update, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure. The one or more processors can be configured to provide, for display via a graphical user interface, an indication of at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

[0005] The one or more processors can be configured to determine the characteristic of the data based at least in part on a quality of the data. The one or more processors can be configured to determine the quality of the data based at least in part on a comparison of a count of invalid, corrupt, or missing samples in the data with a quality threshold. The one or more processors can be configured to select the quality threshold based on at least one of a type of the data, the probability of the procedure type, the probability of the phase type, or the probability of the step type for the at least the segment of the plurality of segments.

[0006] The one or more processors can be configured to determine a type of the data, wherein the type comprises at least one of video, kinematics, system events. The one or more processors can be configured to select the first one or more models based on the type of the data. The one or more processors can be configured to determine the data comprises a video stream, a kinematics stream, and an event stream. The one or more processors can be configured to determine a quality of each of the video stream, the kinematics stream, and the event stream. The one or more processors can be configured to select the first one or more models based on the quality of each of the video stream, the kinematics stream, and the event stream.

[0007] The one or more processors can be configured to determine, for a second segment of the plurality of segments of the medical procedure, that the quality of the event stream fails a quality threshold. The one or more processors can be configured to select, responsive to thequality of the event stream failing the quality threshold for the second segment, a second one or more models of the plurality of models. The one or more processors can be configured to determine, using the second one or more models, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more second segments.

[0008] The probability of the procedure type, the probability of the phase type, and the probability of the step type established for a first segment of the plurality of segments can correspond to a default probability. The one or more processors can be configured to select a second one or more models from the plurality of models based on the update made using the first one or more models to at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

[0009] The one or more processors can be configured to receive, via the graphical user interface, an indication of a metric to compute for the medical procedure, wherein the metric indicates performance of the medical procedure. The one or more processors can be configured to select the first one or more models based on the indication of the metric. The one or more processors can be configured to estimate a first duration to process the data using the first one or more models and a second duration to process the data using a second one or more models of the plurality of models, wherein the first duration is less than the second duration. The one or more processors can be configured to provide, for display via the graphical user interface, a graphical user interface element that indicates the first duration and the second duration. The one or more processors can be configured to receive, via the graphical user interface, a selection of the first one or more models corresponding to the first duration less than the second duration.

[0010] The one or more processors can be configured to determine a first confidence score associated with predictions made using the first one or more models, and a second confidence score associated with predictions made using the second one or more models. The second confidence score can be greater than the first confidence score. The one or more processors can be configured to provide, for display via the graphical user interface, the graphical user interface element comprising the first confidence score and the second confidence score. The one or more processors can be configured to select, for a second segment of the medical procedure subsequent to the segment, a second one or more models from the plurality of models using a confusion matrix established based on performance of the plurality of models during training. The one or more processors can be configured to use a probabilistic graphicalmodel and the probability of the procedure type, the probability of the phase type, and the probability of the step type to generate the indication.

[0011] An aspect of the technical solutions relates to a method. The method can include one or more processors coupled with memory receiving data collected by a plurality of sensors of a robotic medical system associated with performance of a medical procedure. The method can include selecting, by the one or more processors, based on a probability of a procedure type, a probability of a phase type, and a probability of a step type established for at least a segment of a plurality of segments of the medical procedure, a first one or more models from a plurality of models trained with machine learning. The method can include updating, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure. The method can include providing, for display via a graphical user interface, an indication of at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

[0012] The method can include selecting, by the one or more processors, the first one or more models based on a characteristic of the data, the characteristic comprising at least one of a quality of the data or a type of the data. The method can include determining, by the one or more processors, for a second segment of the plurality of segments of the medical procedure, that a quality of the event stream fails a quality threshold. The method can include selecting, by the one or more processors, responsive to the quality of the event stream failing the quality threshold for the second segment, a second one or more models of the plurality of models. The method can include determining, by the one or more processors, using the second one or more models, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more second segments. The probability of the procedure type, the probability of the phase type, and the probability of the step type established for a first segment of the plurality of segments can correspond to a default probability.

[0013] An aspect of the technical solutions relates to a non-transitory computer-readable medium storing processor executable instructions. The instructions, when executed by one or more processors, can cause the one or more processors to receive data collected by a plurality of sensors of a robotic medical system associated with performance of a medical procedure. The instructions, when executed by one or more processors, can cause the one or more processors to select, based at least in part on a number of corrupt samples in the data and aprobability of at least a segment of the medical procedure corresponding to a procedure type, a phase type, or a step type, a first one or more models from a plurality of models trained with machine learning. The instructions, when executed by one or more processors, can cause the one or more processors to update, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure. The instructions, when executed by one or more processors, can cause the one or more processors to provide, for display via a graphical user interface, an indication of at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

[0014] The instructions, when executed by one or more processors, can cause the one or more processors to select, for a second segment of the medical procedure subsequent to the segment, a second one or more models from the plurality of models using a probabilistic graphical model and based on the probability of the procedure type, the probability of the phase type, and the probability of the step updating using the first one or more models.

[0015] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations and are incorporated in and constitute a part of this specification. The foregoing information and the following detailed description and drawings include illustrative examples and should not be considered as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component can be labeled in every drawing. In the drawings:

[0017] FIG. 1 depicts an example system for providing a unified modeling using multiple model predictions for a robotic medical procedure.

[0018] FIG. 2 illustrates an example representation of a probabilistic graphical model that can leverage statistical dependencies among different categories or procedure segments.

[0019] FIG. 3 illustrates an example of a phase confusion matrix of an example set of determinations from an example ML segment model.

[0020] FIG. 4 illustrates an example of a step confusion matrix corresponding to steps or tasks of a phase based on determinations or detections.

[0021] FIGS 5-7 illustrate examples of data of determinations of procedure type probabilities, phase type probabilities and step probabilities of a particular RMP whose data is processed using a unified ML model.

[0022] FIG. 8 illustrates an example plot of cross entropy and weighted determinations of procedure type, phase and steps for the unified ML model and the ML segment models.

[0023] FIGS. 9-14 illustrate example plots of determinations of procedure type probabilities, phase probabilities and step probabilities by unified ML model when errors with ML segment models are simulated.

[0024] FIGs. 15-16 illustrate examples of graphical model examples associated with a unified ML model of a unified segment analyzer.

[0025] FIG. 17 illustrates an example flow diagram of a method for providing a unified modeling using multiple constituent model predictions for a robotic medical procedure.

[0026] FIG. 18 illustrates an example of a surgical system, in accordance with some aspects of the technical solutions.

[0027] FIG. 19 illustrates an example block diagram of an example computer system is shown, in accordance with some aspects of the technical solutions.DETAILED DESCRIPTION

[0028] Following below are more detailed descriptions of various concepts related to, and implementations of, systems, methods, apparatuses for providing a unified model analysis of robotic medical procedure (RMP) types and segments using constituent ML modeling outputs. The various concepts introduced above and discussed in greater detail below can be implemented in any of numerous ways.

[0029] Although the present disclosure is discussed in the context of a surgical procedure, in various aspects, the technical solutions of this disclosure can be applicable to other medicalor non-medical applications, treatments, sessions, environments or activities, in which model analysis of robotic procedure types, phases, steps or actions are determined using combined outputs of ML models are desired.

[0030] Robotic medical procedures, such as robotic surgeries, conducted using robotic medical systems (RMSs) can vary based on the types of the procedures performed and differences in RMP segments (e.g., individual procedure phases, steps or actions) that may be performed. ML models can be trained to determine or identify the RMP segments by training individual models to detect individual RMP segments. However, the RMS data (e.g., sensor, video, kinematics or events data streams) can experience challenges, such as errors or data corruption or loss, making it difficult to achieve an accurate and reliable determination of the individual RMP segments using ML modeling.

[0031] To address these and other technical challenges, the technical solutions of this disclosure can monitor and analyze characteristics of the incoming RMS data to select, based on the characteristics of the received RMS data, those ML models that are most suitable to provide the accurate determination. The technical solutions can further utilize a unified ML model that can utilize relationships and dependencies between various RMP segment determinations to improve the accuracy and reliability of these determinations. By selecting the most suitable ML segment models, based on data characteristics and utilizing a unified analysis with a wholistic probability-based approach (e.g., Bayesian prediction) to improve the RMP segment determinations, the technical solutions can improve the accuracy and reliability of the determinations beyond those of the individual ML models trained to perform such tasks alone.

[0032] FIG. 1 depicts an example system 100 for unified modeling using multiple ML predictions for a robotic medical system, including by combining determinations from constituent ML models. Example system 100 can include a medical environment 102 having a robotic medical system (RMS) 120, data capture devices 110, medical instruments 112, visualization tool 114 and display 116. RMS 120 can be in communication, via a network 101, with a data processing system (DPS) 130. DPS 130 can include one or more ML frameworks 132, model selectors 146, data characteristics determiners 148, performance determiners 152, data repositories 160, unified segment analyzers 164 and interfaces 180. ML framework 132 can include one or more unified ML models 140, ML model trainers 142, confusion matrices 144 and ML segment models 134 for determining segment types of the RMPs and their respective probabilities. The ML segment models 134 can be trained use the RMS 120 data to detect, determine or predict one or more procedure segments 136 (e.g., procedure types, typesof phases of a procedure, types of steps or tasks of a phase or types of actions of step or task) and segment probabilities 138 for the segment types. Unified segment analyzer 164 can use the segment probabilities 138 and procedure segments 136 from the ML segment models 134 as inputs into Unified ML models 140 to identify, determine or detect the combined procedure segments 136 and segment probabilities 138. Data characteristics determiner 148 can determine one or more data characteristics 150. Performance determiner 152 can include, detect, determine or predict one or more metrics 154. Data repository 160 can include or store one or more data streams 162, which can include any one or more of kinematics data 172, sensor data 174, events data 176 and video data 178. Interface 180 can include, generate or provide one or more indications 182.

[0033] Example system 100 can include an RMS 120, which a surgeon can use to perform actions, tasks, phases or procedure types, otherwise referred to as procedure segments 136 of the robotic medical procedure (RMP). RMS 120 can be deployed in a medical environment 102, such as a space or a facility for performing medical procedures, including a surgical operating room. Medical environment 102 can include a space in a hospital or a surgical facility in which RMS 120 can engage and utilize various medical instruments 112 to perform actions, tasks, phases or any other portion of any RMP, whether invasive, non-invasive, inpatient, or out-patient.

[0034] The medical environment 102 can include data capture devices 110. Data capture devices 110 can include, for example, any devices capturing data (e.g., measurements or recordings), including optical devices, such as cameras or sensors or other types of sensors or detectors. Data capture devices can take measurements or recordings to generate data streams 162. Data streams 162 can include video data 178 of images or a video stream of a surgery as well as other sensor data 174 with measurements of various sensors or detectors, events data 176 indicative of any events (e.g., engagement or disengagement of a medical instrument 112) and kinematics data 172 (e.g., indicative of movement of medical instruments 112). The medical environment 102 can include one or more visualization tools 114 to gather the captured data streams 162 and process it for display to the user (e.g., a surgeon or other medical professional) at one or more displays 116. A display 116 can present data stream 162 (e.g., video frames, kinematics or sensor data) of an ongoing medical procedure (e.g., an ongoing surgery) performed using the robotic medical system 120 handling, manipulating, holding or otherwise utilizing medical instruments or tools 112 to perform surgical tasks at the surgical site.

[0035] Data capture devices 110 can include any of a variety of sensors, cameras, video imaging devices, infrared imaging devices, visible light imaging devices, intensity imaging devices (e.g., black, color, grayscale imaging devices, etc.), depth imaging devices (e.g., stereoscopic imaging devices, time-of-flight imaging devices, etc.), medical imaging devices such as endoscopic imaging devices, ultrasound imaging devices, etc., non-visible light imaging devices, any combination or sub-combination of the above mentioned imaging devices, or any other type of imaging devices that can be suitable for the purposes described herein. Data capture devices 110 can include cameras that a surgeon can use to perform a surgery and observe manipulation components within a purview of field of view suitable for the given task performance.

[0036] Data capture devices 110 can include sensors or detectors to capture, detect, or acquire sensor data, such as videos or images, including for example, still images, video images, vector images, bitmap images, other types of images, or combinations thereof. The data capture devices 110 can capture the images at any suitable predetermined capture rate or frequency. Settings, such as zoom settings or resolution, of each of the data capture devices 110 can vary as desired to capture suitable images from any viewpoint. For instance, data capture devices 110 can have fixed viewpoints, locations, positions, or orientations. The data capture devices 110 can be portable, or otherwise configured to change orientation or telescope in various directions. The data capture devices 110 can be part of a multi-sensor architecture including multiple sensors, with each sensor being configured to detect, measure, or otherwise capture a particular parameter (e.g., sound, images, or pressure).

[0037] Data capture devices 110 can include any type and form of a sensor for providing sensor data 174, including a positioning sensor, a biometric sensor, a velocity sensor, an acceleration sensor, a vibration sensor, a motion sensor, a pressure sensor, a light sensor, a distance sensor, a current sensor, a focus sensor, a temperature sensor, a haptic or tactile sensor or any other type and form of sensor used for providing data on medical tools 112, or data capture devices (e.g., optical devices). For example, a data capture device 110 can include a location sensor, a distance sensor or a positioning sensor providing coordinate locations of a medical tool 112 or a data capture device 110. Data capture device 110 can include a sensor providing information or data on a location, position or spatial orientation of an object (e.g., medical tool 112 or a lens of data capture device 110) with respect to a reference point. The reference point can include any fixed, defined location used as the starting point for measuringdistances and positions in a specific direction, serving as the origin from which all other points or locations can be determined.

[0038] Data stream 162 can include video data 178, which can include a series of video frames formed or organized into video fragments, such as video fragments of about 1, 2, 3, 4, 5, 10 or 15 seconds of a video. Each second of the video can include, for example, 30, 45, 60, 90 or 120 video frames 308 per second. Data stream 162 can include a stream of events data 176 which can include a stream of event data or information, such as packets, which identify or convey a state of the robotic medical system 120 or an event that occurred in association with the robotic medical system 120. Events data 176 can include information on the timing of calibration of a medical instrument 112 or timing of engagement or disengagement of a medical instrument 112 on a robotic arm of an RMS 120. Events data 176 can include information on a state of the RMS 120 indicating whether a medical instrument 112 is adjusted or if a manipulator arm is installed on an RMS 120. Event data 176 can include data on whether an RMS 120 is fully functional (e.g., without errors) during the procedure. For example, when a medical instrument 112 is installed on a manipulator arm of the RMS 120, a message, signal or data packet(s) can be generated indicating that the medical instrument 112 has been installed on the manipulator arm of the RMS 120.

[0039] Data stream 162 can include a stream of kinematics data 172, which can refer to or include data associated with one or more of the manipulator arms or medical tools 112 (e.g., instruments) attached to the manipulator arms, such as arm movements, motion, velocity, acceleration, directions of movement, locations or positioning. Data corresponding to medical tools 112 can be captured or detected by one or more displacement transducers, orientational sensors, positional sensors, or other types of sensors and devices to measure parameters or generate kinematics information. The kinematics data 172 can include sensor data along with time stamps and an indication of the medical tool 112 or type of medical tool 112 associated with the data stream 162.

[0040] Video data 178, including any images or videos captured by a medical tool 112 (e.g., endoscopic camera) can include any multimedia data (e.g., stream of images or videos). Video data 178 can include video images captured and sent for storage to the repository 160 or for use at a visualization tool 114. The robotic medical system 120 can include one or more input ports to receive direct or indirect connection of one or more auxiliary devices. For example, the visualization tool 114 can be connected to the RMS 120 to receive the images or videos from the medical instrument 112 when the medical instrument 112 is installed in theRMS 120 (e.g., on a manipulator arm of the RMS 120 that is used for moving, managing or otherwise handing medical instruments 112).

[0041] Display 116 can show, illustrate or play data streams 162, including video data 178, in which medical tools 112 are shown operating (e.g., implementing actions of steps or tasks) at or near surgical sites. For example, display 116 can display a rectangular image (e.g., a frame of a video data 178) of a surgical site along with at least a portion of medical instruments 112 being used to perform surgical tasks (e.g., actions or steps). Display 116 can provide compiled or composite images generated by the visualization tool 114 from a plurality of data capture devices 110 to provide visual feedback from one or more points of view.

[0042] The visualization tool 114 that can be configured or designed to receive any number of different data streams 162 from any number of data capture devices 110 and combine them into a single data stream displayed on a display 116. The visualization tool 114 can be configured to receive a plurality of data stream components and combine the plurality of data stream components into a single data stream 162. For instance, the visualization tool 114 can receive a visual sensor data from one or more medical tools 112, sensors or cameras with respect to a surgical site or an area in which a surgery is performed. The visualization tool 114 can incorporate, combine or utilize multiple types of data (e.g., positioning data of a medical tool 112 along sensor readings of pressure, temperature, vibration or any other data) to generate an output to present on a display 116. Visualization tool 114 can present locations of medical tools 112 along with locations of any reference points or surgical sites, including locations of anatomical parts of the patient.

[0043] Medical instruments or tools 112 can be any type and form of tool or instrument used for surgery, medical procedures or a tool in an operating room or environment. Medical tool 112 can be imaged by, associated with or include an image capture device. For instance, a medical tool 112 can be a tool for making incisions, a tool for suturing a wound, an endoscope for visualizing organs or tissues, an imaging device. Medical instrument 112 can include a needle and a thread for stitching a wound, a surgical scalpel, forceps, scissors, retractors, graspers, or any other tool or instrument to be used during a surgery. Medical tools 112 can include hemostats, trocars, surgical drills, suction devices or any instruments for use during a surgery. The medical tool 112 can include other or additional types of therapeutic or diagnostic medical imaging implements. The medical tool 112 can be configured to be installed in, coupled with, or manipulated by an RMS 120, such as by manipulator arms or other components for holding, using and manipulating the medical instruments or tools 112.

[0044] RMS 120 can include a robotic system that can be computer-assisted and including controllers and logic configured to perform a surgical or medical procedure or activity on a patient. RMS 120 can include the combination of hardware and software to allow a surgeon to implement various actions (e.g., movements or activities of medical instruments 112 and other tools) to implement any action of any step of any phase of any medical procedure, via one or more robotic components or medical tools 112. For instance, RMS 120 can include any number of manipulator arms for grasping, holding or manipulating various medical tools 112 and performing computer-assisted medical tasks using medical tools 112 controlled by the manipulator arms.

[0045] Data repository 160 can any combination of hardware and software for storing data. Data repository 160 can include a storage device for storing any type and form of data streams 162 generated by the robotic medical system (RMS) 120. Data streams 162 can include any combination of kinematics data 172, sensor data 174, events data 176 or video data 178. Data repository 160 can allow ML framework 132 to access and utilize data streams 162 for ML training by ML model trainer 142 or for use by one or more ML models, such as ML segment models 134 or unified ML model 140. Data repository 160 can store any determinations or data of any DPS 130 functions, including data characteristics 150, metrics 154 or indications 182.

[0046] Procedure segments 136 can include any classification of an RMP, including a type of a robotic medical procedure (RMP), a phase within an RMP, a task or step within a phase of the RMP, or an action within such a task or a step. Procedure segment 136 can include any procedure type, a phase, a task or a step or an action determined by a ML segment model 134 trained to determine such procedure segment 136. Procedure segment 136 can include any include a time duration or a period during which the particular procedure segment 136 starts and ends, as indicated by one or more data streams 162. For instance, a procedure segment 136 can include designations or classifications of a type of a medical procedure performed. A procedure segment 136 can include a designation or a classification of a type of a phase of a plurality of phases of a medical procedure. A procedure segment 136 can include a designation or a classification of a type of a task or a step of a plurality tasks or steps of a phase, of a plurality of phases of a medical procedure. A procedure segment 136 can include a designation or a classification of a type of an action of a plurality of actions to perform or complete a particular step or a task of a phase of a medical procedure. Procedure segments 136 can be determined or performed using ML segment models 134 trained using data sets (e.g., any number of RMPs of various kinds) to detect, recognize, identify or determine any particular orindividual procedure segment 136 (e.g., procedure types, phases, steps or tasks or actions) based on RMS 120 data (e.g., data stream 162) input into such ML segment model 134.

[0047] Procedure segment 136 can include, for example, a type of a medical procedure performed using the RMS 120. A procedure type can include any type of RMP that can be performed using RMS 120. A procedure type can include, as for example, a prostatectomy, a hysterectomy, a cardiac surgery (e.g., coronary artery bypass grafting), a colorectal surgery, a gynecologic surgery (e.g., myomectomy), a urologic surgery (e.g., nephrectomy), a bariatric surgery (e.g., gastric bypass), a thoracic surgery (e.g., lobectomy), or a head and neck surgery (e.g., thyroidectomy).

[0048] Procedure segment 136 can include any phase of any robotic medical procedure. For instance, procedure segment 136 can include a designation of a phase of a plurality of phases of an RMP. For instance, a procedure segment 136 pertaining to a phase, can include or identify an exposure phase, an extraction phase, a dissection phase, a reconstruction phase a transection phase, or any other phase of a medical procedure.

[0049] Procedure segment 136 can include any step or a task of a plurality of steps or tasks of any phase of a medical procedure. For instance, procedure segment 136 can include a designation of a step or a task, such as any type of a step or a task. A type of a step or a task can include, for example, a closure of peritoneum, a closure of hernia defect, a right peritoneal flap exploration, an excision of existing mesh, a removal of an adhesion, a closure of a hernia defect, a right excision of an existing mesh, a creation of a peritoneal incision, a dissection of a gallbladder, a reduction of a hernia sac, a mesh attachment, an excision, a lysis of adhesions or any other step or a task of any phase of any medical procedure. A plurality of steps or tasks or types of steps or tasks, in their individual distinction and arrangement (e.g., order of various individual steps or tasks) can be indicative of a phase or procedure type.

[0050] Procedure segment 136 can include an action of a plurality of actions to perform a step or a task of a plurality of steps or tasks of a phase of a medical procedure. For instance, an action can include a movement of a medical instrument 112, an application of an instrument on a portion of a patient’s body, an application of a treatment, or any other one or more actions to perform a particular task or a step. Procedure segment 136 can include a series of movements with a medical instrument (e.g., a needle) that can be indicative of an action, such as suturing of a wound or opening a cut. A plurality of actions or types of actions, in their individualdistinction and arrangement (e.g., order of various individual actions) can be indicative of a step or a task, and therefore be used to determine a phase or a procedure type.

[0051] Segment probability 138 can include any probability or a confidence level of a procedure segment 136 determination. For instance, a segment probability 138 can include any probability or a confidence level that a particular procedure segment 136 is the actual element of the procedure being performed, based on a segment or portion of data stream 162 being used as input to the ML segment model 134. Segment probability 138 can include any probability or confidence level determined by a ML segment model 134 trained to determine such segment probability. Segment probability 138 can include a value, from a range of values, such as between 0 and 1, or between 0% and 100%, indicative of the percent chance that the determined procedure segment 136 is the correct segment type. Segment probability 138 can include a probability value or a confidence level that a particular procedure type identified by a ML segment model 134 is the correct procedure type. Segment probability 138 can include a probability value or a confidence level that a particular phase of a procedure identified by a ML segment model 134 is the correct phase. Segment probability 138 can include a probability value or a confidence level that a particular step or a task of a phase of a procedure identified by a ML segment model 134 is the correct step or a task.

[0052] ML segment model 134 can include any model, such as a machine learning or an artificial intelligence model for determining procedure segment 136 or a segment probability 138. ML segment model 134 can include any ML model trained to detect or identify a feature of a RMP (e.g., a procedure type, a phase of a procedure, a step of a phase or an action of a step) based on one or more segments or portions of one or more data streams 162 input into the ML segment model 134. For instance, ML segment model 134 can be trained to determine a type of the medical procedure being performed or its corresponding procedure type (e.g., segment probability 138 corresponding to procedure type). Procedure segment 136 can be trained to determine a phase or a phase type and its corresponding segment probability 138, based on one or more portions of one or more data streams 162 input into the ML segment model 134. For example, a ML segment model 134 can be trained to determine a procedure segment 136 of a step or a task, or a segment probability 138 of a step or a task (e.g., of a particular phase of a RMP), based on one or more portions of one or more data streams 162 input into the ML segment model 134. For example, ML segment model 134 can be trained to determine or detect a procedure segment 136 of an action of a plurality of actions of a task, or asegment probability 138 of an action, based on one or more portions of one or more data streams 162 input into the ML segment model 134.

[0053] ML segment models 134 can be trained to make determinations based on various types of data, such as any combination of data streams 162. By being trained to make determinations based on different types of data (e.g., video, sensor, kinematics or events), ML segment models 134 can be selected to maximize the probability of an accurate determination, when faulty or erroneous data is detected. For instance, ML segment models 134 can be trained to make determinations based on any combination of video data 178, events data 176, sensor data 174 or kinematics data 174. For example, one ML segment model 134 can be trained to make determinations (e.g., of procedure types, phases, steps or actions) based on video data 178 and sensor data 174, while another ML segment model 134 can be trained to make same determinations using kinematics data 172 and sensor data 174. In such instances, if a data characteristics determiner 148 determines that video data 178 is faulty for a duration or segment of a data stream 162, the model selector 146 can select the ML segment model 134 trained on kinematics data 172 and sensor data 174 to make the determination, thereby avoiding usage of the ML segment model 134 trained on video data 178 for the duration of data stream 162 during which the erroneous or faulty video data 178 is detected by a data characteristics determiner 148.

[0054] Model selector 146 can include any combination of hardware and software for selecting machine learning models, such as ML segment models 134 and unified segment models 140. Model selector 146 can include the functionality to select, based on data characteristics 150 of the data streams 162, ML segment models 134 to use for determination of unified segment determinations 166 or determination of unified segment probabilities 168. Model selector 146 can select a particular one or more ML segment models 134 from a plurality of ML segment models 134 trained with machine learning. The model selector 146 can select ML segment models 134 based on prior determined probabilities (e.g., determinations of procedure type, phase, step or task or action) from a prior segment or section of data stream 162. The model selector 146 can select ML segment models 134 based at least in part on a characteristic of the data (e.g., 150) and one or more segment probabilities 138, such as a probability of a procedure type, a probability of a phase type, and a probability of a step type established for at least a segment of a plurality of segments of the medical procedure. The segment probabilities 138 can be determined with respect to a preceding set of data (e.g., datastream 162) used to make a prior determination of a procedure segment 136 or a procedure probability 138 for a preceding action, step or a task or a phase of a procedure.

[0055] The model selector 146 can select a second one or more ML segment models 134 from the plurality of ML segment models 134 based on the update made using the first one or more ML segment models 134 to one or more segment probabilities 138. For instance, the model selector 146 can select a second ML segment model 134 based on an update, adjustment or a correction (e.g., by unified ML model 140) to at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments determined by ML segment models 134.

[0056] The model selector 146 can receive, from a data characteristics determiner 148 determined type of the data (e.g., identified data type of a video, kinematics, system events) and select the first one or more ML segment models 134 based on the type of the data. The outputs of the ML segment models 134 can be used as inputs to the unified ML model 140. The unified ML model 140 can utilize Bayesian model or technique to refine or correct the determinations of the ML segment models 134 and generate unified segment determinations 166 and unified segment probabilities 168. For instance, the model selector 164 can identify or select an ML segment model 134 for determining a phase of a medical procedure and an ML segment model 134 for determining a step of a medical procedure to determine a phase or a procedure type of the RMP performed.

[0057] The model selector 146 can select the first one or more models based on the quality of each of the video stream, the kinematics stream, and the event stream. Such selections can be made based on data characteristics determiner 148 determining that the quality of the data (e.g., video stream, a kinematics stream, and an event stream) and determining a quality of each of the types of data. The model selector 146 can determine, using a data characteristics determiner 148, for a given segment of the plurality of segments of the medical procedure, that the quality of the event stream fails a quality threshold. The model selector 146 can select, responsive to the quality of the event stream failing the quality threshold for the given segment, a second one or more ML segment models 134 of the plurality of models 134. The unified segment analyzer 164 can determine, using the selected second one or more segment ML models 134, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more second segments.

[0058] The model selector 146 can select a confusion matrix 144 instead of a particular ML segment model 134 when a determination within a particular time frame is desired. For example, the model selector 146 can determine a first duration of time it takes to process the data using the first one or more models (e.g., 134) and a second duration to process the data using a second one or more models (e.g., 134) of the plurality of models (e.g., 134). The first duration can be less than the second duration. The model selector 146 can provide, for display via the graphical user interface 180, a graphical user interface element (e.g., indication 182) that indicates the first duration and the second duration. The model selector 146 can receive, via the graphical user interface, a selection of the first one or more models (e.g., 134) corresponding to the first duration less than the second duration. The model selector 146 can select, for a second segment of the medical procedure subsequent to the current segment, a second one or more models (e.g., 134) from the plurality of models using a confusion matrix 144 established based on performance of the plurality of models (e.g., 144) during training.

[0059] Performance determiner 152 can include the functionality for determining metrics 154 indicative of the performance of ML segment models 134. Metrics 154 can include any metrics or parameters indicative of the level of accuracy of determinations of the ML segment model 134. For instance, performance determiner 152 can operate together with unified segment analyzer 164 to determine confidence scores of any determinations made by the ML segment models 134 with respect to determination or identification of any phases, steps, actions or procedure types. Metrics 154 can identify whether or not an ML segment model 134 determination accuracy satisfies or does not satisfy an expected performance standard or a threshold. Metrics 154 can include a confidence score which can be determined for any particular procedure segment 136 or segment probability 138 determination. For example, performance determiner 152 can determine a confidence score (e.g., a metric 154) associated with one or more predictions (e.g., procedure segment 136, such as a step of a phase) made using a first one or more ML segment models 134. The metric 154 can be associated with a particular segment probability 138. The performance determiner 152 can determine a second confidence score associated with predictions made using the second one or more ML segment models 134 or a second segment probability. The performance determiner 152 can determine that the second confidence score is greater than the first confidence score. The performance determiner 152 can provide the metric 154 to the model selector 146. The model selector 146 can determine, based on the confidence scores (e.g., metrics 154) to utilize the ML segment model 134 with a higher metric 154.

[0060] Unified segment analyzer 164, also referred to as an integrated or a joint segment analyzer 164, can include any combination of hardware and software for determining unified segment determinations 166 from one or more procedure segments 136 or segment probabilities 138. Unified segment analyzer 164 can include any combination of hardware and software for determining unified segment probabilities 168 from one or more segment probabilities 138. Unified segment analyzer 164 can include the functionality to determine or compute a Bayesian function with respect to, or using, a plurality of outputs from a plurality of ML segment models 134 determining any procedure segments 136 or their segment probabilities 138. For instance, Unified segment analyzer 164 can analyze a plurality of determinations of procedure segments 136 (e.g., determinations of a procedure type, a phase of that procedure, a step of such a phase, or an action of such a step) in order to determine a unified segment determination 166 (e.g., a procedure type, a phase, a step or an action) considering all such segment type determinations with respect to the data stream 162 of that RMP. For example, the unified segment analyzer 164 can utilize a Bayesian inference or a function to calculate probabilities (e.g., 168) associated with various procedure segments 136 over the course of a data stream of the RMP.

[0061] Unified segment analyzer 164 can include the functionality to use a particular ML segment model 134 and an input data (e.g., from a data stream 162) to update the probability of one or more procedure segments 136, such as a procedure type, a probability of a phase type, and a probability of a step type for one or more segments of the plurality of segments of the medical procedure. For instance, unified segment analyzer can determine, using a data characteristics determiner 148, that the data streams 162 include some of the data having data characteristics 150 exceeding a data quality threshold, while other data falls below the threshold. The unified segment analyzer can utilize a model selector 146 to select the ML segment models 134 that are suitable for receiving as input the data that exceeds the quality threshold in order to make valid determinations. The unified segment analyzer 164 can utilize the selected ML segment models 134 to input the quality data into the models and determine the procedure segments 136 (e.g., procedure type, phase, steps or actions) and segment probabilities (e.g., probabilities or confidence levels for each such determination. Unified segment analyzer 164 can include a probabilistic graphical model to determine unified segment determinations 166 and unified segment probabilities 168.

[0062] Unified segment analyzer 164 can utilize Bayesian computation or techniques to combine the determinations from ML segment models 134 to generate improveddeterminations (e.g., unified segment determinations 166 and unified segment probabilities 168). The Bayesian computation or techniques can combine the outputs of one or more ML segment models 134 applied to a continuous one or more data streams 162 to generate an updated probability of various determinations from the ML segment models 134 based on new (e.g., combined) evidence. In doing so, the unified segment analyzer 164 can operate as a Bayesian model combining or unifying determinations to improve output accuracy. Bayesian computation can include unified analysis of segments and their probabilities and combine prior determinations (e.g., determinations of prior determined procedure segments 136 or their segment probabilities 138) with current evidence from data streams 162 to update probability estimates for currently determined segments. Using Bayesian approach, the prior probabilities for each segment type (such as procedure type, phase, task, or action) can be used along with the new data streams 162(e.g., incoming video data 178, kinematics data 172, sensor data 174 or events data 176) to be processed by ML segment models 134. The probability or likelihood of each segment type given the new data can be computed according to probabilities of the ordering or sequence of actions, tasks or phases from prior determinations along with the current determination. Using Bayes' theorem, the prior determined segment probabilities 138 can be updated with the likelihoods determined based on the updated likelihood of segment types given the new data (e.g., unified segment probabilities). For unified segment analysis, Bayesian theorem can be used to compute the unified probability of multiple segments occurring together, in accordance with the interdependencies between different segment types. The unified segment analyzer can integrate these posterior probabilities from multiple ML models trained to detect various segments, determining the most likely combination of segment types across the entire procedure. Using the Bayesian approach, the unified segment analyzer 164 can continually update the probabilities as new data is received, allowing for continually providing more accurate and reliable determinations (e.g., unified segment determinations 166 and unified segment probabilities 168).

[0063] Unified segment determination 166, also referred to as a joint or an integrated segment determination 166, can include any determination of segment type (e.g., a type of a procedure, a phase of a procedure, a step of a phase or an action of a step). Unified or a joint segment determination 166 can be determined based on a plurality of individual determinations of the segment types 136 or segment probabilities 138, for a set of data (e.g., data stream 162). Unified segment determination 166 can include any determination of an action, a step, a phase or a medical procedure determined based on a plurality of procedure segment 136determinations by ML segment models 134 trained to detect actions, steps, phases or procedure types. Unified segment determination 166 can be determined using plurality procedure segment 136 and segment probability 138 determinations over the course of a data stream 162. For instance, a unified segment analyzer 164 can determine that a particular unified segment determination 166 has occurred at a particular point in a RMP based on a series or order of procedure segments 136 (e.g., actions, steps or phases) over the course of a data set (e.g., 162).

[0064] Unified segment probability 168, which can also be referred to as a joint or an integrated segment probability 168, can correspond to the unified segment determination 166 of a probability of a particular segment (e.g., a probability of a type of a procedure, a phase of a procedure, a step of a phase or an action of a step) being performed. Unified segment probability 168 can be determined based on a plurality of outputs or determinations from various ML segment models 134 with respect to individual segment types 136 or segment probabilities 138 for a given data set. Unified segment determination 166 can include any determination of a probability of occurrence of an action, a step, a phase or a medical procedure determined based on a plurality of procedure segment 136 or segment probability 138 determinations by ML segment models 134. Combined segment determinations 168 can be determined using plurality procedure segment 136 and segment probability 138 determinations over the course of a data stream 162. For instance, a unified segment analyzer 164 can determine that a particular unified segment probability 168 for a particular determined segment type 166 based on a series or order of procedure segments 136 (e.g., actions, steps or phases) over the course of a data set (e.g., 162).

[0065] Confusion matrices 144 can include any one or more values or parameters including or representing determination or performance of a ML segment model 134. Confusion matrix 144 can correspond to an ML segment model 134 and can include values or parameters indicative of determinations (e.g., phases, steps, actions or procedure types) determined by, or on behalf of (e.g., instead of), an ML segment model 134. Confusion matrix 144 can include, for example, an average value for a phase determination with respect to prior one or more determinations for a data set. Confusion matrix 144 can include a parameter that estimates the determination of a phase, step, procedure type or action, based on prior ML segment model 134 determinations, which DPS 130 can utilize instead of, or while, waiting for the ML segment model 134 to complete its current determinations. Confusion matrix 144 can include any parameter or value indicative of a probability of procedure type, probability of procedure phase, a probability of a step type, or a probability of an action. Confusion matrix 144 caninclude a default value for each of the type of procedure segments 136 for any ML model (e.g., 134 or 140).

[0066] Data characteristics 150 can include any features or characteristics of data (e.g., any portion of data stream 162) used as input into ML models, such as ML segment models 134 or unified ML models 140. Data characteristics 150 can include any features, metadata or parameters indicative of the quality of data to use for ML segment model determinations. Data characteristics can include indications of presence or absence of data, range of values of data or data range coinciding with errors or erroneous function (e.g., events data 176 indicative of an error with respect to a particular data stream 162 portion). Data characteristics 150 can include information indicative of invalid, corrupt or missing data. Data characteristics 150 can indicate a particular data type (e.g., kinematics data 172, sensor data 174, events data 176 or video data 178).

[0067] Data characteristics determiner 148 can include any combination of hardware and software for detecting or determining data characteristics 150. Data characteristics determiner 148 can determine the characteristic of the data based at least in part on a quality of the data. The quality of the data can be determined based at least in part on a comparison of a count of invalid, corrupt, or missing samples in the data with a quality threshold. A quality threshold can include, for example, an amount of data that can be corrupted, missing or invalid within a particular range of a data stream 162. For instance, data characteristics determiner 148 can include thresholds for amount of data missing, number of erroneous data elements or number of invalid data instances, per a data range. Data characteristics determiner 148 can select the quality threshold based on at least one of a type of the data, the probability of the procedure type, the probability of the phase type, or the probability of the step type for the at least the segment of the plurality of segments. Data characteristics determiner 148 can determine a type of the data. The type of data can include at least one of video (e.g., 178), kinematics (e.g., 172), system events (e.g., 176) or sensor data (e.g., 174). Based on the data type, the data characteristics determiner 148 can determine the data comprises a video stream, a kinematics stream, and an event stream and determine a quality of each of the video stream, the kinematics stream, and the event stream; and

[0068] Machine learning (ML) framework 132 can include any combination of hardware and software for providing machine learning capability of the DPS 130. ML framework 132 can include ML models and ML infrastructure, such as ML segment models 134, unified ML models 140, ML trainers 142, confusion matrices 144, model selectors 146 and any otherfunctionality that can interface with or utilize machine learning (e.g., unified segment analyzer 164, data characteristics determiner 148 and performance determiner 152. ML framework 132 can include any combination of logic, circuitry and computer code, executables or instructions for training ML models (e.g., 136 or 140) or providing and using ML models 134 or 140. ML framework 132 can include the functionality to generate confusion matrices 144 and determine procedure segments 136 and segment probabilities 138 for the segment type determinations. ML framework 132 can include one or more ML segment models 134 for detecting, recognizing or identifying particular procedure segments 136 and their corresponding segment probabilities 138. ML framework 132 can include one or more unified ML models 140 for combining procedure segments 136 and segment probabilities 138 for unified segment analyzer 164 to generate unified segment determinations 166 and unified segment probabilities 168 with improved accuracy and reliability.

[0069] ML segment models 134 can include any machine learning models or functions trained to detect, identify or recognize procedure segments 136 or segment probabilities 138 of any RMPs. ML segment models 134 can be designed and trained to identify and detect any procedure segments 136, including a type of a procedure (e.g., a type of a RMP performed using a RMS 120), a phase of a RMP, a step or a task of a plurality steps or tasks of a phase of an RMP or an action of a plurality of actions of a step or a task. ML segment models 134 can be trained using any dataset of a plurality of data streams 162 of a plurality of RMPs.

[0070] Unified ML model 140, also referred to as an integrated or joint ML model 140, can include any machine learning model or a function for determining unified segment determinations 166 and unified segment probabilities 168. Unified ML model 140 can be a ML model using procedure segments 136 and segment probabilities 138 as inputs to generate unified segment determinations 166 and unified segment probabilities 168 as output. Unified segment determinations 166 can include any determination of a procedure segment 136 determined based on a combined plurality of determinations of one or more procedure segments 136 (e.g., procedure types, phases, steps or tasks or actions) from one or more ML segment models 134. Unified segment probabilities 168 can include any determination of a segment probability 138 determined based on a combined plurality of determinations of one or more segment probabilities 138 determined by one or more ML segment models 134. For instance, a unified segment probability 168 can include a probability of a procedure type being the actual procedure type performed, a probability of a phase being the actual phase performed, a probability of a step or a task being the actual step or a task or a probability of an action beingthe actual action being performed, each of which can be determined based on a plurality of individual ML segment model 134 determinations considered by the unified ML model 140.

[0071] ML framework 132 can include any ML architecture or techniques to implement the machine learning functionalities of the DPS 130. For instance, ML framework 132 can receive and use RMS 120 data streams 162 to inform determinations for the unified segment analyzer 164 using, for example, neural network models, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) to process video data and sequential timestamped data to identify surgical instruments and detect actions, steps, phases or procedure types. For instance, ML framework 132 can include long short-term memory networks (LSTMs) to handle kinematics data to track surgical tool movements over time or graph neural networks (GNNs) to model relationships between surgical actions, steps (e.g., tasks), phases and medical procedure types, based on sensor or other data. ML framework 132 can include attention mechanisms to facilitate the analysis of relevant features within the data streams 162 or generative adversarial networks (GNNs) to produce synthetic data to augment the training datasets and improve the robustness of the unified segment analyzer 164 and its unified ML models 140.

[0072] The ML trainer 142 can any combination of hardware and software for training ML models. Machine learning (ML) trainer 142 can include create or generate ML models (e.g., 134, 140), which can be trained using training datasets that can include various data streams 162. ML trainer 142 can include a framework or functionality for training different machine learning models, such as a neural network models, spatial-temporal attention mechanisms, transformer-based models or any other ML or artificial intelligence (Al) models or functions. ML trainer 142 can train ML models using any combination of data from data streams 162, including video data 178, kinematics data 172 and sensor data 174.

[0073] Data repository 160 of the DPS 130 can include one or more data streams 162, such as video data 178 including a stream of video frames. Data streams 162 can include measurements from sensors, which can be referred to as sensor data 174 and which can include various force, torque or biometric data, haptic feedback data, pressure or temperature data, vibration, tension or compression data, endoscopic images or data, ultrasound images or videos or communication and command data streams. Data repository 160 can include installation data, such as system files or logs including time stamps and data on installation, activation, calibration or use of particular medical instruments 112. ML models, including ML segmentmodels 134 or unified ML models 140, which can each be stored in a data repository 160, along with training data sets and data streams 162.

[0074] Data processing system (DPS) 130 can include any combination of hardware and software providing unified modeling of procedure segments and procedure segment probabilities based on data of a RMP implemented via an RMS 120. The data processing system 130 can be deployed in or associated with the medical environment 102, or it can be provided by a remote server or be cloud-based. The data processing system 130 can include an interface 180 designed, constructed and operational to communicate with one or more component of system 100 via network 101, including, for example, the robotic medical system 120. Data processing system 130 can be implemented using instructions stored in memory locations and processed by one or more processors, controllers or integrated circuitry. Data processing system 130 can include functionalities, computer codes or programs for executing or implementing any functionality of ML framework 132, including any ML models (e.g., 134 or 140) along with any associated functions or features (e.g., 146, 148, 152 or 164) for identification, detection and analysis RMP segments. DPS 130 can include the functionality to receive, via network 101, data (e.g., data streams 162) collected by a plurality of sensors of a robotic medical system 120 associated with performance of a medical procedure and process this data using the DPS 130 functionality.

[0075] The data repository 160 can include one or more data files, data structures, arrays, values, or other information that facilitates operation of the data processing system 130. The data repository 160 can include one or more local or distributed databases and can include a database management system. The data repository 160 can include, maintain, or manage a data stream 162. The data stream 162 can include or be formed from one or more of a video stream, image stream, stream of sensor measurements, event stream, or kinematics stream. The data stream 162 can include data collected by one or more data capture devices 110, such as a set of 3D sensors from a variety of angles or vantage points with respect to the procedure activity (e.g., point or area of surgery).

[0076] Interface 180 can include any combination of hardware and software to allow users to interact with a DPS 130 through visual elements like buttons, icons, and menus, facilitating control and navigation. Interface 180 can include the functionality for displaying or providing DPS 130 can include an interface 180 designed, constructed and operational to communicate with one or more component of system 100 via network 101, including, for example, the RMS 120 or another device, such as a client’s personal computer. The interface 180 can include anetwork interface. The interface 180 can include or provide a user interface, such as a graphical user interface. The graphical user interface can include, for example, a window for displaying video data 178, or indications 182. Indications 182 can be overlaid or displayed instead of, along with, or on top of the video data 178. Interface 180 can provide data for presentation via a display, such as a display 116, and can depict, illustrate, render, present, or otherwise provide indications 182.

[0077] Interface 180 can provide, for display (e.g., via a graphical user interface), an indication 182 of at least one of the probability (e.g., 138 or 168) of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments. Interface 180 can receive, via the graphical user interface, an indication 182 of one or more metrics 154 to compute for the medical procedure. The metric 154 can indicate performance of the medical procedure. Interface 180 can provide, for display via the graphical user interface, the graphical user interface element comprising the first confidence score and the second confidence score.

[0078] The data processing system 130 can interface with, communicate with, or otherwise receive or provide information with one or more component of system 100 via network 101, including, for example, the RMS 120. The data processing system 130, RMS 120 and devices in the medical environment 102 can each include at least one logic device such as a computing device having a processor to communicate via the network 101. The DPS 130, any portion of the ML framework 132, the RMS 120 or a client device that can be communicatively coupled with the DPS or the RMS 120 via the network 101, can each include at least one computation resource, server, processor or memory for processing data. For example, the data processing system 130 can include a plurality of computation resources or processors coupled with memory.

[0079] The data processing system 130, as well as any of its components (e.g., ML framework 132) can each be a part of or include a cloud computing environment functionality or features. The data processing system 130 can include multiple, logically grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can also be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous - one or more of the servers or machines can operate according to one or more type of operating system platform.

[0080] The data processing system 130, or components thereof can include a physical or virtual computer system operatively coupled, or associated with, the medical environment 102. In some embodiments, the data processing system 130, or components thereof (e.g., 110, 112, 114, 116, 120, 130) can be coupled, or associated with, the medical environment 102 via a network 101, either directly or directly through an intermediate computing device or system. The network 101 can be any type or form of network. The geographical scope of the network can vary widely and can include a body area network (BAN), a personal area network (PAN), a local-area network (LAN) (e.g., Intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 101 can assume any form such as point-to-point, bus, star, ring, mesh, tree, etc. The network 101 can utilize different techniques and layers or stacks of protocols, including, for example, the Ethernet protocol, the internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, the SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP / IP internet protocol suite can include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network 101 can be a type of a broadcast network, a telecommunications network, a data communication network, a computer network, a Bluetooth network, or other types of wired and wireless networks.

[0081] The data processing system 130, or components thereof, can be located at least partially at the location of the surgical facility associated with the medical environment 102 or remotely therefrom. Elements of the data processing system 130, or components thereof can be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The data processing system 130, or components thereof, can include other or additional elements that can be considered desirable to have in performing the functions described herein. The data processing system 130, or components thereof, can include, or be associated with, one or more components or functionality of a computing including, for example, one or more processors coupled with memory that can store instructions, data or commands for implementing the functionalities of the DPS 130 discussed herein.

[0082] For example, system 100 can be utilized for analysis of diverse medical procedures, such as laparoscopic surgeries. For example, video-based models can segment video frames by ontological multiclass categories, such as surgical phases, steps, and actions, which can help surgeons and researchers review and analyze medical procedures. Individual models can be trained to predict different multiclass categories (e.g., procedure segments 136 or segment probabilities 138). These determinations can correspond to procedure actions, steps, phases orprocedure types whose occurrence or data can be statistically dependent on the occurrence or data of other actions, steps, phases or procedure types that may precede or follow them, thereby providing leverage to be used to improve the accuracy and reliability of these determinations beyond merely the performance level of the individual ML segment models 134.

[0083] The technical solutions can incorporate such dependencies to take advantage of this leverage and combine outputs from individual models using unified Bayesian inferences. The surgical categories of the procedure type, phase and step can be represented by Markovian states, which can be determined using data sets of annotated surgical cases. For each advancing frame of a surgical case (e.g., video data 178), data likelihoods computed with video-based segmentation models can be combined with a prior probability distribution to update a following determination. The resulting unified ML model 140 can provide output that can represent a unified belief taking in consideration information on all surgical categories which can vary over the course of the procedure with the evidence accumulated. In doing so, the technical solutions can improve the accuracy of the modeled determinations outperforming the accuracy of the individual constituent models alone. Moreover, the unified ML model 140 of the technical solution can overcome scenarios in which one or more individual models fail (e.g., due to absent or faulty data), drawing inferences from other determinations (e.g., preceding identified procedure segments 136 and their probabilities 138) to provide a reliable output in the events of individual constituent model failure.

[0084] With the growing use of the robotic assisted surgeries (RASs), the data produced by the RMPs (e.g., high resolution video, surgeon commands and tool kinematics) can provide information on operating room workflow, surgeon skill assessment and provide outcome predictions using ML modeling. ML segment models 134, such as video segmentation models, can be trained on such RMP data to provide real-time and post-operative review by automatically parsing the information gathered from surgical procedures. ML segment models 134 can be trained to classify video frames into surgically relevant hierarchical categories, such as procedure type, surgical phase or surgical steps using variable or unstructured data over multiple time scales.

[0085] Procedure segments 136, such as a procedure type can be revealed over the course of a procedure through the implicit goals achieved, whereas surgical phases can include portions of the procedures, spanning over shorter periods of time. Surgical phases can include multiple surgical steps corresponding to, or including, various surgical actions. The knowledge of a current surgical phase can inform both the procedure type and the current surgical step, aswell as likely future phases. In part due to these complexities and interdependencies, the technical solutions combine determinations from multiple individual models trained on each of the categories to provide a more accurate modeled output.

[0086] The system 100 can utilize interdependent information from the multiple segmentation models to provide a probabilistic model that accounts for the conditional dependencies of the procedure types, surgical phases, surgical steps or any other procedure segments. Using large data sets of annotated surgeries, the technical solutions can estimate the statistical properties of this model. Predictions from trained segmentation models can be reinterpreted as data likelihoods, which can be combined with evolving prior distributions to compute a Bayesian update over the posterior beliefs. The resulting unified ML model 140 can provide determinations whose accuracies are improved over those of the individual segment models.

[0087] FIG. 2 illustrates an example representation of a probabilistic graphical model 200 to leverage statistical dependencies among different categories or procedure segments 136 (e.g. procedure types, phases and steps) in a probabilistic framework of the technical solution. The probabilistic graphical model 200 can include or reference dependencies between different representations of procedure types (r) 202, surgical phases (p) 204 and surgical steps (s) 206 that can be represented using Markovian model that can vary over time. The unified model (e.g., 140) can take advantage of the data used to generate predictions from the individually trained models (e.g., the constituent models, such as ML segment models 134) to compute or provide a unified distribution that can vary over time. Probabilistic graphical model 200 of FIG. 2 can be used to represent unified model inferences over time. For instance, procedure type (r) 202, surgical phase (p) 204 and surgical step (s) 206 can be expressed using Markovian process. Models trained to classify each of the individual categories or probability segments 136 (e.g., 202, 204 or 206) can be used to compute likelihoods. For instance, an ML segment model 134 trained to estimate the probability of a step, P(s) = f (ys) can be instead reinterpreted as a likelihood, P(ys|s) = f(ys).

[0088] The example unified distribution provided by an example probabilistic graphical model 200 can be expressed as follows:P ( , pl,sl, yr, yP, ys) = P^l "1) P(pt|rt, Pl') P(st|rt, pt, sl 1) P(yr|rt) P(yP|pt) P(ys|st), where P^l "1), P(pt|rt, pl 1) and P(st|rt, p sl 1) can refer to prior and conditional probabilities, and P(yr|rt), P(yP|pt) and P(ysIs1) can indicate the likelihoods. The technical solutions can utilize the lastthree terms to introduce the constituent models that can be trained to produce predictions, e.g., the probability of a surgical step as a function of the data P(s) = f(ys). For example, probability of a procedure type determination over time can be indicated as y? 208, probability of a phase type determination over time can be indicated as yPl210 and probability of a step determination over time can be indicated as ysl212. Instead, the technical solutions can utilize the constituent models to compute data likelihoods, e.g., the likelihood of the data given the step, P(ys|s) = f (ys).

[0089] The transitions of the time-varying prior and conditional probabilities can be linear, and can be tracked as:AsrpP(st-1|rt, pl) , where Ar, Ar, As17can be obtained from the transition probabilities. For example, As1*1= (Psrp)T where the (i, j) element of Ps'pis P (sl= j|sl 1= i, r, p).

[0090] The unified ontological model inferences (e.g., determinations of unified ML model 140) can be represented as follows: P ( , pl,sl, |yr, yP, ys) = [P (r^X1) P (p1]?, pl 1) P (s^ , p\ sl 1) P (yr|rl) P (yP|pl) P (ys| s1)] / z, where z, the partition function, can be found by summing for all permutations of r 202, p 204 and s 206.

[0091] The unified ontological model (e.g., 140) can allow for multiple ways of making predictions from inferences. For instance, the unified ontological model (e.g., 140) can predict categories using a triplet that maximizes the posterior probabilities (i.e., arg max P (r, p, s\yr, yP, )). For example, to obtain probabilities within each category, similar to the constituent models, the solutions can compute the marginals (i.e. P(r\yr, yP, ys) for the procedure type probabilities). For instance, the solutions can predict the class that had the maximum probability for each category separately.

[0092] The unified ontological model can assume that the procedure type, r 202, can be on one of a plurality of possible classes, such as for example three classes (nr= 3), which can include inguinal hernia, radical prostatectomy or hysterectomy. The phase, p 204, can also be one of a plurality of classes, such as one of six (nP= 6) classes, which can include exposure, dissection, transection, reconstruction, extraction, or other. Across all procedure types (e.g., all three procedure types) there can be ns = 69 possible steps (see Appendix). In total then, there are nr• nP• ns= 1,242 possible combinations of procedure type 202, phase 204 and step 206, that make up the state space of the technical solution. The state space can be the space over which the technical solutions can update its determinations and make inferences.

[0093] The transition matrices (e.g., confusion matrices 144), Ar, APr, As17can be approximated by counting the frequency of their respective transitions. The technical solutions can implement smoothing, such as Laplacian smoothing, for all approximated probabilities. Since the states can be discrete and finite, the probabilities can all be represented with arrays. For instance, P (r) can be an array with nrelements, and As17can be a 4-dimensional array of size, nsxnsxnP* nr.

[0094] The unified ML model 140 can use or combine determinations from the constituent models (e.g., 134) to provide or generate determinations. Constituent models (e.g., 134) can be trained to individually predict procedure type 202, surgical phase 204 and surgical steps 206. The procedure type, phase and step can be expressed using confusion matrices 144. To focus on the unified model (e.g., 140), and not the particulars of the constituent models (e.g., 134), the technical solutions can use approximate confusion matrices 144 as stand-ins for the likelihoods of the unified ML model 140 determinations. This can be utilized, for example, when determinations from unified ML model 140 are not generated in time and the system 100 prefers or seeks to utilize determinations within a shorter period of time. In such instances, for example, rather than use actual model predictions, which can vary from procedure to procedure and frame to frame, the technical solutions can use an average of the constituent ML model performance as measured or expressed with the confusion matrices 144. At each point in time, the technical solutions can use the ground truth (e.g., expressions for procedure type, phase and step), to grab the corresponding row from the constituent model’s confusion matrix 144, which can be utilized as a likelihood for the determination. This allows for examining model performance on average, while also allowing to artificially perturb the model when determinations within a shorter period of time are preferred.

[0095] FIG. 3 provides an illustration of an example of a phase confusion matrix 300 of an example set of determinations from an example ML segment model 134 (e.g., for identifying, determining or recognizing phases 204 of an RMP). To determine procedure type 202 based on detections of individual phases 204 (e.g., 204A-F) according to probability values, phase confusion matrix 300 can utilize a value scale 302, that can provide a gradient scale of values 0-1, such as a scale of steps of 0.1 from 0 to 1.0. For instance, for a procedure type 202, the confusion matrix can have the value of 0.75 on the diagonal (e.g., diagonal of phases 204 blocks from the upper left square towards the lower right of the matrix), and a value of 0.125 for all off-diagonal terms. Individual blocks can indicate phases 204A-F that can, in combination, indicate or identify a particular procedure type 202. The individual blocks 204can correspond to any particular arrangement of combination of a particular phase 204A, transaction phase 204B, reconstruction phase 204C, dissection phase 204D, extraction phase 204E and exposure phase 204F. The confusion matrix example 300 can represent or include an approximation based on several of the procedure type models (e.g., 134), and a way to keep the determination values (e.g., or their approximations over time) simple. With respect to the example phase confusion matrix 300, the diagonal elements can be approximately 0.8.

[0096] FIG. 4 shows an example of a step confusion matrix 400 corresponding to steps or tasks 206 of a phase 204 based on determinations or detections (e.g., from ML segment model 134). For the example step model corresponding to step confusion matrix 400, which can correspond to any confusion matrix 144, the average probabilities can be provided based on true / false prediction counts, as this can, in some instances, provide a more nuanced likelihood. The tasks 206 can be ordered (as shown in the table below) according to procedure type, which can include a hysterectomy, an inguinal hernia, and a radical prostatectomy. In some instances, the diagonal elements may not have the largest values, indicating that the wrong steps, or steps belonging to the incorrect procedure type, may have the highest likelihoods. The step confusion matrix 400 can have the steps or tasks 206 (e.g., 0-68) have their determinations graded according to probability values of a value scale 40. The phase confusion matrix 300 can utilize the value scale 402, that can provide a gradient scale of values 0-1, such as a scale of steps of 0.1 from 0 to 1.0.

[0097] For instance, a large data set of surgical procedures can include data on tool kinematics, system information and the accompanying video. Each data set (e.g., of each individual procedure) can be annotated by an expert for that specific procedure type, to identify the start and stop times of surgical phases 204 and steps 206. An example data set can include, for example, any collection of various types of RMPs, including tens, hundreds or thousands of data sets of individual hysterectomy RMPs, inguinal hernia RMPs, or radical prostatectomy RMPs. For testing or validation of the unified ontological model’s probabilities (e.g., 166 or 168) and transition matrices, data from the hysterectomy cases, the inguinal hernia cases, and the radical prostatectomy cases can be used.

[0098] To quantify comparisons between the unified and constituent models, the crossentropy and weighted Fl scores can be examined. Mean values within and across procedure types for each category can be computed along with the corresponding 95% confidence intervals for their differences, denoted as A. If this interval included zero or a statistically insignificant number, it can be denoted as a “f”. Any upper and lower confidence intervals thatwere equivalent up to the 3rd decimal point can be considered negligibly small and only the mean value, A can be reported.

[0099] In one example, to assess the performance of the unified ML model 140, four scenarios of various conditions can be examined. The first scenario (e.g., corresponding to FIGS. 5-7) can demonstrate the potential benefits when using relatively high-performing constituent models (e.g., 134). The remaining three scenarios (e.g., corresponding to FIGs. 9- 14) can demonstrate the benefits when various constituent models (e.g., 134) perform poorly, such as when the ML segment models 134 receive faulty or erroneous data or otherwise fail to provide determinations for a particular portion of data stream 162. This example can illustrate a proof-of-concept for the unified model’s advantages under various types of operations or settings. To safeguard this example analysis from the particulars of the constituent models, the technical solutions can substitute their likelihoods with their respective confusion matrices 144 for an easily interpretable substitution. By manipulating their confusion matrices, the example can illustrate arbitrarily poor performance, as an example of a constituent model failure.

[0100] FIGS 5-7 illustrate an example 500 of determinations of procedure type probabilities 202 (e.g., in FIG. 7), an example 600 of determinations of phase type probabilities 204 (e.g., in FIG. 6) and example 700 of determinations of step probabilities 206 (e.g., in FIG 5) of a particular RMP whose data is processed using a unified ML model 140. FIGs. 5-7 show the results of a unified ML model 140 utilizing determinations from the constituent ML segment models 134 to provide determinations of procedure segments 136 (e.g., 202, 204 or 206) with improved accuracy. The constituent model (e.g., 134 for procedure type 202) for procedure type can, for example, correctly predict inguinal hernia with a likelihood of 0.75, and a likelihood of 0.125 for hysterectomy and radical prostatectomy (Fig 7 top row, dashed lines). The unified ML model 140, using this likelihood, along with the phase and step likelihoods, can also predict this data to be indicative of an inguinal hernia case (e.g., solid lines in FIG. 7), though the unified ML model 140 can be more confident (e.g., have higher confidence level determination), with a probability of roughly one. As shown in FIG. 6, unlike the procedure type 202, the phase 204 can change multiple times during the case. For the 600 seconds of the case shown, the constituent model (e.g., 134) for phase can correctly identify each phase segment, with varying likelihoods of roughly 0.8 (e.g., FIG. 6, dashed lines). The unified ML model 140 can also correctly predict the phase 204, but with higher confidence (e.g., higher confidence interval of the determination) which can approach a value of 1, as shown in FIG. 6, solid lines.

[0101] FIGS. 5-7 further shows the surgical steps 206 change more frequently than phases 204 (e.g., multiple separate step instances shown in FIG. 6, bottom row) and the constituent step model (e.g., 134) can correctly predict these steps. However, during the first period, a step 206A corresponding to lysis of adhesions and bowel sweep can be determined by the unified ML model 140, while the step 206D corresponding to the homeostasis of uterine vessels (right) can be the step with the largest likelihood which can be an incorrect prediction from the constituent model (e.g., 134). Given other prior determinations of the unified ML model 140 with respect to the procedure type 202 (e.g., FIG. 7) and phase 204 (e.g., FIG. 6), the unified ML model 140 can determine that step 206D is unlikely and that step 206 A is more likely given prior determinations of the phases 204 and steps 206. For example, the unified ML model 140 can determine that the arrangement or order of the first 16 steps with the largest likelihoods (e.g., confidence levels) do not correspond to or match with an inguinal hernia procedure type 202 or with an exposure phase 204. The unified ML model 140 can determine that the step 206A of lysis of adhesions and bowel sweep does match or correspond to (e.g., with a confidence level exceeding a confidence level threshold) with another procedure type 202 and its exposure phase 204. In such an example, the constituent model (e.g., 134) can incorrectly predict a step 206, and the unified ML model 140 can determine, based on prior series of determined steps 206, phases 204 or procedure types 202, that this new determination by the ML segment model 134 is incorrect and select a more likely determination or option.

[0102] FIG. 8 illustrates an example plot 800 of cross entropy and weighted Fl determinations of procedure type 202, phase 204 and steps 206 for the unified ML model 140 and the ML segment models 134. For example, the results from the example cases, such as the one discussed in connection with FIGS. 5-7, can be summarized by comparing the weighted Fl scores and cross-entropy and the confidence intervals of their respective differences, across all cases and procedure. The constituent models (e.g., 134) may perform well in predicting procedure types with a mean Fl score of 1.0 and a mean cross-entropy of 0.288. The unified ML model 140 can make the same predictions with a mean Fl of 1.0, but it can make such predictions with higher confidence and a significantly smaller mean cross-entropy of 1.452E-4 (e.g., the lower and upper confidence bounds can be approximately equivalent, and A =0.288). The comparisons for phase 204 can be similar, and the constituent model has a mean Fl of 1.0 and cross-entropy 0.182, whereas the unified model can have a slightly smaller mean Fl of 0.996, and cross-entropy mean of 0.012. Both changes can be significant, as the confidence intervals for the differences can be A=[-0.006-0.003] and A=[0.164-0.178], respectively.

[0103] However, when the constituent model (e.g., 134) for particular steps 206 does not perform as well (e.g., often making errors with relatively low likelihoods, such as Fl of 0.803 and cross-entropy of 3.081), the unified ML model 140 can be able to improve these predictions by assigning higher probability to steps that are consistent with both the phase and procedure type. The result can be a marked and significant improvement in both the Fl score (0.982, A=[0.148-0.210]) and cross entropy (0.082, A=[2.878-3.120]).

[0104] The result of comparing the changes in cross-entropy and Fl scores (e.g., unified ML model 140 relative to constituent or ML segment model 134) within procedure types 202 can be displayed below (Table 1) for determinations with respect to hysterectomy (HYS) procedure type 202, inguinal hernia (IHR) procedure type 202 or radical prostatectomy (RP) procedure type 202. Most comparisons can be significant, so only those non-significant differences may be indicated using a “f”. As can be seen the cross-entropy can decrease under all conditions. The Fl scores can show a measure of some improvements, such as for steps 206, while in some instances no changes or small significant decreases in prediction accuracy can occur.Table 1 (above) presents results for the baseline scenario (e.g., example from FIGs. 5-7), comparing mean values and the confidence intervals of their differences for cross-entropy and weighted Fl values.

[0105] In some examples, the unified ML model 140 can have its performance evaluated using simulated results of poorly functioning or failing ML segment models 134. For instance, in first of three examples a unified ML model 140 can be tested using predictions from a procedure type ML segment model 134 whose output (e.g., procedure type 202 determination) is no better than random (e.g., 1 / 3 for three example procedure types).

[0106] Results from such a test with the failing ML segment model 134 for procedure type 202 determination is illustrated in example plot 900 of FIG. 9, showing procedure type, phase or step probability determinations associated with this example and in example 1200 of FIG. 12 showing cross-entropy and Fl weights of the same example. This example shows that the unified ML model 140 can be beneficial under even when the constituent ML segment model 134 for the procedure type 202 is replaced with just a constant value of l / nr= 1 / 3 (e.g., equal probability of the procedure being true for each of the three example procedure types). The ML segment models 134 for determining or detecting the phase 204 and steps 206 can remain as in the baseline scenario (e.g., models trained to predict outcomes as in FIGs. 5-8). Considering the same inguinal hernia case being the RMP performed, as shown in FIG. 9 the unified ML model’s probabilities for phase and step can remain similar as in the example discussed in connection with FIGs. 5-8, when all three ML segment models 134 (e.g., procedure type, phase and step) were functional and without data errors. In this example, despite having no functional procedure type ML segment model 134, the unified ML model 140 can still correctly detect or determine inguinal hernia procedure type 202 based on the evidence it accumulates for phases and steps from the phases and steps ML segment models 134.

[0107] As shown in FIG. 12, examining all the cases under similar error conditions the data shows similar improvements. For instance, for the procedure type predictions can have the cross entropy decreased from 1.099 to 0.002 (A =1.096), while the Fl can increase from 0.498 to 0.999 (A=[0.500-0.502]). Improvements for phase predictions can be more modest but stillsignificant, and cross entropy can be decreased from 0.182 to 0.011 (A=[0.164- 0.171]), and Fl changed from 1.0 to 0.996 (A=[-0.005- - 0.004]). The improvements in step predictions can be similar to what was found in the baseline scenario discussed in connection with FIGs. 2-3, the cross entropy decreasing from 3.081 to 0.083 (A=[2.877-2.998]), while the Fl increased from 0.803, to 0.982 (A=[0.148-0.179]).

[0108] Shown, for example, in example 1000 of FIG. 10, illustrating probability determinations when phase ML segment model 134 is non-functional and in example 1300 of FIG. 13, illustrating cross-entropy and Fl weighted data for the same example, the unified ML model 140 can predict phases 204 even when phase-determining ML segment model 134 fails. In examples 1000 and 1300, the phase model is non-functional (e.g., reduced to a random 1 / 6 probability prediction for six possible phase determinations). Dashed lines can in the plots in FIG. 1000 can correspond to category likelihoods while solid lines can correspond to unified model probabilities.

[0109] Unified ontological model (e.g., 140) predictions for a non-functional phase model (e.g., in FIG. 10 and 13) using procedure type 202 and steps 206 determining ML segment models 134 can be similar to the previous scenarios with marked improvements relative to the constituent models (e.g., 134). To predict phases 204, the unified ontological model (e.g., 140) can largely lean on the step ML segment model 134 predictions, which can be more informative of phase than procedure type ML segment model 134 predictions. During the first 100 seconds, the step homeostasis of uterine vessels (right) can have the largest likelihood but belong to the wrong procedure type so it can have a low probability.

[0110] The unified ontological model (e.g., 140) can be able to correctly infer the phases 204 as well as the steps 206 and procedure type 202 (Fig 10). For procedure type 202, the summary improvements in cross entropy can decrease from 0.288 to 1.1E-4 (A =0.288) and both models had a constant Fl score of 1.0. For phase the cross entropy improved from 1.792 to 0.143 (A=[1.605- 1.649]) and 0.213 to 0.969 (A=[0.746-0.756]) for Fl scores. The improvements in step predictions can be similar to other results, the cross entropy decreasing from 3.081 to 0.203 (A=[2.741-2.878]), while the Fl increasing from 0.803 to 0.960 (A=[0.126-0.157]).

[0111] FIGs. 11 and 14 can refer to examples 1100 and 1400 in which the step ML segment model’s likelihood is shown as erroneous or uninformative and having a constant value l / ns= 1 / 69. The unified model’s predictions for procedure type and phase can remain strong andshow improvements, just as in the baseline scenario (e.g., see FIG. 11). Forming accurate determinations about the steps 206 without a functional step model can be challenging. The procedure type 202 and phase 204 can each provide information, but only enough to predict the subset of steps 206 that typically occur during a phase 204. For example, initially the steps 206 of lysis of adhesions and bowel sweep and creation of peritoneal incision, both of which can occur during the exposure phase, can be identified as having large probabilities. Then, the model can determine that the steps 206 of left reduction of hernia sac and peritoneal flap exploration / reduction of hernia sac, both of which can occur during the dissection phase, have large probabilities.

[0112] The unified ML model 140 can provide improved predictions over the constituent models (e.g., 134) and the unified ML model 140 can improve the performance of constituent models even with erroneous steps ML segment modeling (e.g., FIGs. 11 and 14). With respect to the data and examples in FIGs. 9-14, for procedure type 202 the cross entropy changed from 0.288 to 5.9E-4 (A =0.287) whereas the Fl score didn’t change significantly from 1.0 for both models; for phase the cross entropy improved from 0.182 to 0.0175 (A=[0.157-0.165]) and from 1.0 to 0.994 (A=[-0.008 - -0.006]) for Fl scores; and for step predictions the cross entropy decreased from 4.234 to 1.563 (A=[2.569-2.671]), while the Fl increased from 0.025 to 0.365 (A=[0.301-0.340]).

[0113] Referring to FIGs. 15 and 16, examples of graphical models 1500 and 1600 trained in particular procedure types are illustrated. For example, technical solutions can include phase and step ML segment models 134 that are trained for a specific procedure type 202. Training procedure specific models can have an advantage of focusing an ML segment model 134 on a narrower set of training data and predictions and can have an improved performance. Such an approach can be employed with slight modifications to an example that utilizes one or more generative models (e.g., FIG. 15). The likelihoods for each phase and step model pair would implicitly provide evidence for their relevance, the likely current procedure type.

[0114] In graphical model examples of examples 1500 of FIG. 15 and 1600 of FIG. 16, phase and step ML segment models 134 can be trained specifically for a particular parent procedure type 202. The phase and step models 134 can operate as pairs making predictions or determinations in parallel, competing for relevance and providing evidence for the current procedure type. In FIG. 16, additional procedure segments or states, such as actions 1602, also referred to as “a1”, can be provided. Actions 1602 can include activities, such as a gesture, an act, or a movement of a medical instrument 112 (e.g., a scalpel), a gesture or a motion of amedical professional (e.g., surgeon), applying an instrument (e.g., a laser or an X-ray machine), or any other action that can be indicative of a step or a task 206. Actions 1602 can be represented or associated with a state and can be used to help train identification of steps 206 and phases 204, as a single step can be determined based on a series of actions 1602 (a), whose order or arrangement can uniquely identify a step or a task.

[0115] While the examples discussed in FIGs. 9-14 concern an example model architecture that combines three ML segment models 134 that map to a particular ontology (procedure type, phase and step), the technical solutions can involve any number of ML segment models 134 with any number of categories (e.g., actions, steps, tasks, phases, procedure types or other) which can be combined into a single unified distribution via a unified ML model 140. For example, by including another ontological category, such as surgical “actions” (e.g., a plurality of actions forming a step 206), a new graphical model could be employed with the attendant conditional dependencies (e.g., FIG. 16) to combine its predictions, and with minor assumptions result in a further improved unified estimate.

[0116] Likewise, there is the potential to capitalize on the data collected during a robotic assisted surgery. For example, there may be readily available information for tool type, which can be indicative of procedure type, surgical phase and surgical steps. With relatively minor changes to our model this information could be incorporated by including a new “tool” state. Though this can increase the space over which probabilities were estimated, the conceptual changes would be insignificant. Table 2, below, presents an example distribution of data from determinations discussed in connection with FIGs. 9-12.Table 2 illustrates a summary of results across the three examples with no evidence conditions (e.g., erroneous ML segment models) discussed in connection with FIGs. 9- 14.

[0117] Turning now to FIG. 17, an example flow diagram of a method 1700 for unified modeling based on multiple ML segment model predictions for a robotic medical procedure. Method 1700 can be implemented using system 100, or utilizing any features, functionalities or techniques discussed in connection with examples 100-1600 of FIGs. 1-16. The method 1700 can be performed using one or more processors 1910 coupled with memory or storage (e.g., 1915, 1920 or 1925) and executing instructions, commands, computer code or data stored to cause performance of actions, operations or aspects of the DPS of the system 100. Method 1700 can include actions 1705-1735. At 1705, the method can receive data from a robotic medical system. At 1710, the method can determine data characteristics. At 1715, the method can determine if the data characteristics trigger selection of one or more new ML models. At 1720, if the determination at act 1715 is in the negative, the method can utilize prior ML models to make determinations on procedure type, phase, step or action. At 1725, if the determination at act 1715 is in the affirmative, the method can select a new one or more ML models to make determinations on procedure type, phase, step or action. At 1730, the methodcan use a unified ML model to update the determinations. At 1735, the method can provide an indication for display.

[0118] At 1705, the method can receive data from a robotic medical system. The method can include one or more processors coupled with memory receiving data of a robotic medical system (RMS). The data can be collected by a plurality of sensors of the RMS that can be associated with performance of a medical procedure. For instance, the data can be a sensor data of one or more video cameras (e.g., video frames) or one or more signals or measurements of a sensor or a detector, such as signals or measurements of pressure, temperature, force, motion, rotation, velocity, acceleration, touch or any other sensor detection or measurement. The data can include kinematics data (e.g., data on motion of one or more medical instruments) or events data (e.g., data on calibration, activation, deactivation, engagement or disengagement) of one or more medical instruments.

[0119] The method can include the data processing system (DPS) determining a type of the data. For example, one or more processors can implement a data characteristics determiner to detect or determine a type of the data received by the RMS. The data can include, for example one or more portions of any one or more of a video data stream, a kinematics data stream, an events stream or a sensor data stream. The date type can include at least one of video, kinematics, system events, which the one or more processors can use to select or identify one or more ML segment models to use for processing of the data.

[0120] At 1710, the method can determine data characteristics. The method can include the one or more processors determining the characteristic of the received data, based at least in part on a quality of the data. For example, the one or more processors can implement a data characteristics determiner to monitor and analyze incoming data streams (e.g., kinematics data, sensor data, events data or video data). The data characteristics determiner can determine data quality over a period of time corresponding to a portion or a segment of one or more data streams to be processed. For example, data characteristics determiner can periodically (e.g., every one or more milliseconds or tens of milliseconds) capture a duration of one or more data streams (e.g., based on timestamps of the data) and determine the quality of the data.

[0121] The method can include the data characteristics determiner determining the quality of the data based at least in part on a comparison of a count of invalid, corrupt, or missing samples in the data with a quality threshold. For instance, the data characteristics determiner can identify any erroneous data, such as data including invalid, corrupted or missing fields orpayload. For instance, the data characteristics determiner can identify one or more data packets (e.g., packets of any of the data streams) that include erroneous, corrupted or missing portions. For instance, the data characteristics determiner can identify that a portion of video data is missing, corrupted or erroneous. For instance, the data characteristics determiner can identify that a portion of sensor data is missing, corrupted or erroneous. For instance, the data characteristics determiner can identify that a portion of kinematics data is missing, corrupted or erroneous. For instance, the data characteristics determiner can identify that a portion of events data is missing, corrupted or erroneous. The data characteristics determiner can identify as quality data, those data whose number of data packets that are faulty (e.g., missing, corrupted or with errors) are below a threshold. The threshold can be set based on the type of data, such as for example one or more video frames for video data (e.g., four video frames within a stream of 1 second of data having 120 video frames). The threshold can be set, for example, at four sensor readings within a stream of 1 second of 60 readings. The threshold can be set, for example, at five readings for kinematics data packets for kinematics data or one or more events data packets for events data.

[0122] The data characteristics determiner can select the quality threshold for determining the quality of the data streams. The quality threshold can be determined based on at least one of a type of the data or one or more procedure segments or probability segments. For example, the quality threshold can be determined based on at least one of a type of procedure detected, the probability of the procedure type, the type of the phase detected, the probability of the phase type, the type of the step type detected, or the probability of the step type for the at least the segment of the plurality of segments of a data stream. The quality threshold can be determined based at least one of a type of an action of a plurality of actions of a step or a task or a probability of the action.

[0123] The data characteristics determiner can determine whether the segment of data stream corresponding to a time duration or interval for which the data is to be utilized to determine procedure segments or segment probabilities, satisfies or does not satisfy the quality threshold. For instance, the data characteristics determiner can determine that a segment of data stream corresponding to video data does not satisfy the quality threshold, while other data streams (e.g., kinematics, sensor, or events data) satisfy the threshold. For instance, the data characteristics determiner can determine that a segment of data stream corresponding to sensor data does not satisfy the quality threshold, while other data streams (e.g., kinematics, video, or events data) satisfy the threshold. For instance, the data characteristics determiner candetermine that a segment of data stream corresponding to kinematics data does not satisfy the quality threshold, while other data streams (e.g., video, sensor, or events data) satisfy the threshold. The data characteristics determiner can provide the determinations to the model selector to select one or more ML segment models, based on these data characteristics determined.

[0124] At 1715, the method can determine if the data characteristics trigger selection of one or more new ML models. The method can include the one or more processors determining if the characteristics of the data determined by the data characteristics determiner trigger selection of one or more new ML models to be used for determining procedure segments and segment probabilities, or if the one or more ML models utilized in a prior determination can continue to be used. The method can include the one or more processors executing a model selector to determine the selection of one or more ML segment models to be utilized.

[0125] The one or more processors can determine, based on whether or not the data stream segment at 1710 satisfies a threshold for data quality, to utilize a particular one or more ML segment models. For example, the model selector can determine, responsive to a segment of data stream at 1710 satisfying a data quality threshold, that prior utilized one or more ML segment models can be utilized to determine the procedure segments (e.g., procedure type, phase, step or task or action) as well as any probabilities associated with such procedure segments. For example, the model selector can determine, responsive to a segment of data stream at 1710 not satisfying a data quality threshold, that a new one or more ML segment models are to be selected based on the data quality. For instance, responsive to one or more data streams (e.g., kinematics data, video data, sensor data or events data) experiencing errors that exceed a quality threshold, the model selector can identify or select one or more ML segment models whose performance with faults in such data is highest out of all ML segment models. For instance, if a video stream data is determined to be erroneous (e.g., does not satisfy a threshold at 1710 , a model selector can select a ML segment model that is trained to determine one or more procedure segments and segment probabilities based on kinematics data, sensor data and events data (e.g., without video data). For instance, if a sensor stream data is determined to be erroneous, a model selector can select a ML segment model that is trained to determine one or more procedure segments and segment probabilities based on kinematics data, sensor data and events data (e.g., without video data).

[0126] The model selector can determine, for example, based on the data characteristics, that one or more ML segment models utilized for a prior segment of data stream are to beutilized for the current segment of data stream. The model selector can determine, for example, based on the data characteristics, that the one or more ML segment models utilized for a prior segment of data stream are not to be utilized for a current segment of data stream. The model selector can select, for example, a first one or more models to be used for processing of the data segment, based on the quality of each of the video stream, the kinematics stream and the event stream. The model selector can select the first one or more models based on the quality (e.g., quality threshold being satisfied or not being satisfied) for each of the video stream, a kinematics stream or an event stream.

[0127] The method can include the one or more processors (e.g., the model selector) determining, for a second segment of the plurality of segments of the medical procedure (e.g., a second segment of a data stream), that the quality of the event stream fails a quality threshold. The model selector can select, responsive to the quality of the event stream failing the quality threshold for the second segment, a second one or more models of the plurality of models.

[0128] At 1720, if the determination at act 1715 is in the negative (e.g., a “no”), the method can utilize prior ML models to make determinations on procedure type, phase, step or action. For instance, the one or more processors can implement a performance determiner to utilize one or more ML segment models to determine or provide one or more determinations of one or more procedure segments and segment probabilities. For example, a selected ML segment model can determine a procedure type, a phase of a procedure type, a step or a task of a plurality of steps or tasks of the procedure type, or an action of a plurality of actions identifying a step or a task. For example, a selected ML segment model can determine a probability or a confidence level of a determination of a procedure type. A selected ML segment model can determine a probability or a confidence level of a determination of a phase of a procedure type. A selected ML segment model can determine a probability or a confidence level of a determination of a step or a task of a plurality of steps or tasks of the procedure type. A selected ML segment model can determine a probability or a confidence level of a determination of an action of a plurality of actions identifying a step or a task.

[0129] The one or more processors can implement the one or more ML segment models to update the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure. The one or more ML segment models or a unified ML model can update or modify the procedure type or the probability of the procedure type, a phase of the procedure type or a probability of the phase of the procedure type, a step or a task of the phase or aprobability of the step or the task of the phase, or an action of a procedure type, or a probability of the action of the procedure type. The updating of the determinations by the ML segment models or the unified ML model can be implemented using the first one or more ML segment models and the data (e.g., a segment of the data stream input into the one or more ML models).

[0130] The method can include the one or more processors implementing or utilizing one or more confusion matrices. The one or more confusion matrices can include one or more probabilities of the procedure type, the phase type, and the step type, which can be established for a first segment of the plurality of segments of the data stream corresponds to a default probability. The probabilities of the confusion matrices can include constant parameters (e.g., values between 0 and 1, inclusively) corresponding to an estimate value of a probability of a determination (e.g., procedure segment or a segment probability) based on one or more prior determinations (e.g., determinations of one or more prior segments of data stream).

[0131] For instance, upon selection of a second one or more ML segment models (e.g., based on the data characteristics satisfying or not satisfying the quality threshold), the selected second one or more ML segment models can determine, using the second one or more models, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more second segments. The method can include the one or more processors determining or estimating a first duration to process the data using the first one or more models and a second duration to process the data using a second one or more models of the plurality of models. The first duration can be less than the second duration.

[0132] At 1725, if the determination at act 1715 is in the affirmative, the method can select a new one or more ML models to make determinations on procedure type, phase, step or action. The one or more processors can implement a model selector. The model selector can select a first one or more ML segment models from a plurality of ML segment models trained with machine learning. The selection can be made based at least in part on a characteristic of the data. The selection can be made based at least in part on a probability of a procedure type, a probability of a phase type, or a probability of a step type established for at least a segment of a plurality of segments of the medical procedure (e.g., a segment of the data stream).

[0133] The model selector can select the first one or more ML segment models based on the type of data. The ML segment models can be used to determine a procedure type, a phase of a procedure, a step or a task of a phase or an action of the step or the task. For instance, in response to the type of data determined by the data characteristic determiner including at least one of video, kinematics, system events, the model selector can select the first one or more MLsegment models to use for the data stream. For instance, the model selector can determine the data comprises a video stream, a kinematics stream, and an event stream and determine, based on a quality of each of the video stream, the kinematics stream, and the event stream to select the first one or more models for the determined quality level (e.g., threshold) of each of the video stream, the kinematics stream, and the event stream.

[0134] The one or more processors can implement the model selector to select a second one or more models from the plurality of models based on the update made using the first one or more models to at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments. For example, the model selector can select a second one or more ML segment models based on a determination that a data characteristic of a data stream corresponds to one or more data streams failing to satisfy a quality threshold triggering a selection of the second ML segment model that is trained to use a portion of data stream that satisfies the quality threshold for the data stream. The model selector can select, for a second segment of the medical procedure subsequent to the segment, a second one or more models from the plurality of models using a confusion matrix established based on performance of the plurality of models during training. The method can include selecting, responsive to the quality of the event stream failing the quality threshold for the second segment, a second one or more models of the plurality of models and determining, using the second one or more models, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more second segments.

[0135] The method can include updating, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure. For example, the method can include determining the probability of the procedure type, the probability of the phase type, and the probability of the step type established for a first segment of the plurality of segments corresponds to a default probability.

[0136] At 1730, the method can use a unified ML model to update the determinations. The method can include the one or more processors implementing a unified ML model. The unified ML model can receive as inputs the outputs or determinations of the one or more ML segment models and determine updated outputs of the procedure type, a phase of the procedure, a step or a task or an action of the step or the task. The unified ML model can utilize a Bayesian determination to consider one or more prior determinations of the same RMP data (e.g., datafor the same medical procedure session) to make an updated determination of any procedure segment (e.g., procedure type, phase, step or task or action) and any probability of a segment.

[0137] For instance, the method can use a probabilistic graphical model to make determinations of any one or more of the probabilities of the procedure type, the probability of the phase type, and the probability of the step type or the probability of an action. These determinations can be provided to generate the indication to display. For instance, the method can include receiving, via the graphical user interface, an indication of a metric to compute for the medical procedure. The metric can indicate the performance of the medical procedure. The method can include selecting the first one or more models (e.g., ML segment models or a unified ML model) based on the indication of the metric.

[0138] At 1735, the method can provide an indication for display. The method can include the one or more processors providing, for display via a graphical user interface, an indication. The indication can be, for example, an indication of at least one of the probabilities of the procedure type, the probability of the phase type, the probability of the step type, or the probability of an action, for the one or more segments of the data stream. The indication can be, for example, an indication of the procedure type, a phase of a plurality of phases of the procedure type, a task or a step of a plurality of tasks or steps of the phase or an action of a plurality of actions of a step or a task. The indication can include a graphical or visual illustration or plot of any procedure segment or a segment probability. The indication can indicate a time duration for any determined procedure segment or segment probability, or a value for procedure segment or segment probability.

[0139] The method can include determining (e.g., by unified segment analyzer) a first confidence score associated with predictions made using the first one or more models, and a second confidence score associated with predictions made using the second one or more models. The second confidence score can be greater than the first confidence score. Based on the second score being greater than the first confidence score, the unified segment analyzer can provide, for display via the graphical user interface, the graphical user interface element comprising the first confidence score and the second confidence score.

[0140] The method can include providing, for display via the graphical user interface, a graphical user interface element that indicates the first duration and the second duration. The first and the second time durations can correspond to time durations for which the data streams were processed by the DPS. The method can include receiving, via the graphical user interface,a selection of the first one or more models corresponding to the first duration less than the second duration. The method can include generating, for display, a graphical user interface element based on the user selection.

[0141] FIG. 18 depicts a surgical system 1800, in accordance with some embodiments. The surgical system 1800 may be an example of the medical environment 102. The surgical system 1800 may include a robotic medical system 1805 (e.g., the robotic medical system 120), a user control system 1810, and an auxiliary system 1815 communicatively coupled one to another. A visualization tool 1820 (e.g., the visualization tool 114) may be connected to the auxiliary system 1815, which in turn may be connected to the robotic medical system 1805. Thus, when the visualization tool 1820 is connected to the auxiliary system 1815 and this auxiliary system is connected to the robotic medical system 1805, the visualization tool may be considered connected to the robotic medical system. In some embodiments, the visualization tool 1820 may additionally or alternatively be directly connected to the robotic medical system 1805.

[0142] The surgical system 1800 may be used to perform a computer-assisted medical procedure on a patient 1825. In some embodiments, surgical team may include a surgeon 1830A and additional medical personnel 1830B-1830D such as a medical assistant, nurse, and anesthesiologist, and other suitable team members who may assist with the surgical procedure or medical session. The medical session may include the surgical procedure being performed on the patient 1825, as well as any pre-operative (e.g., which may include setup of the surgical system 1800, including preparation of the patient 1825 for the procedure), and post-operative (e.g., which may include clean up or post care of the patient), or other processes during the medical session. Although described in the context of a surgical procedure, the surgical system 1800 may be implemented in a non-surgical procedure, or other types of medical procedures or diagnostics that may benefit from the accuracy and convenience of the surgical system.

[0143] The robotic medical system 1805 can include a plurality of manipulator arms 1835A-1835D to which a plurality of medical tools (e.g., the medical tool 112) can be coupled or installed. Each medical tool can be any suitable surgical tool (e.g., a tool having tissueinteraction functions), imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or other suitable instrument that can be used for a computer-assisted surgical procedure on the patient 1825 (e.g., by being at least partially inserted into the patient and manipulated to perform a computer-assisted surgical procedure on the patient). Although the robotic medical system1805 is shown as including four manipulator arms (e.g., the manipulator arms 1835A-1835D), in other embodiments, the robotic medical system can include greater than or fewer than four manipulator arms. Further, not all manipulator arms can have a medical tool installed thereto at all times of the medical session. Moreover, in some embodiments, a medical tool installed on a manipulator arm can be replaced with another medical tool as suitable.

[0144] One or more of the manipulator arms 1835A-1835D and / or the medical tools attached to manipulator arms can include one or more displacement transducers, orientational sensors, positional sensors, and / or other types of sensors and devices to measure parameters and / or generate kinematics information. One or more components of the surgical system 1800 can be configured to use the measured parameters and / or the kinematics information to track (e.g., determine poses of) and / or control the medical tools, as well as anything connected to the medical tools and / or the manipulator arms 1835A-1835D.

[0145] The user control system 1810 can be used by the surgeon 1830 A to control (e.g., move) one or more of the manipulator arms 1835A-1835D and / or the medical tools connected to the manipulator arms. To facilitate control of the manipulator arms 1835A-1835D and track progression of the medical session, the user control system 1810 can include a display (e.g., the display 1118 or 1130) that can provide the surgeon 1830A with imagery (e.g., high-definition 3D imagery) of a surgical site associated with the patient 1825 as captured by a medical tool (e.g., the medical tool 112, which can be an endoscope) installed to one of the manipulator arms 1835A-1835D. The user control system 1810 can include a stereo viewer having two or more displays where stereoscopic images of a surgical site associated with the patient 1825 and generated by a stereoscopic imaging system can be viewed by the surgeon 1830A. In some embodiments, the user control system 1810 can also receive images from the auxiliary system 1815 and the visualization tool 1820.

[0146] The surgeon 1830A can use the imagery displayed by the user control system 1810 to perform one or more procedures with one or more medical tools attached to the manipulator arms 1835A-1835D. To facilitate control of the manipulator arms 1835A-1835D and / or the medical tools installed thereto, the user control system 1810 can include a set of controls. These controls can be manipulated by the surgeon 1830A to control movement of the manipulator arms 1835A-1835D and / or the medical tools installed thereto. The controls can be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 1830A to allow the surgeon to intuitively perform a procedure on the patient 1825 using one or more medical tools installed to the manipulator arms 1835A-1835D.

[0147] The auxiliary system 1815 can include one or more computing devices configured to perform processing operations within the surgical system 1800. For example, the one or more computing devices can control and / or coordinate operations performed by various other components (e.g., the robotic medical system 1805, the user control system 1810) of the surgical system 1800. A computing device included in the user control system 1810 can transmit instructions to the robotic medical system 1805 by way of the one or more computing devices of the auxiliary system 1815. The auxiliary system 1815 can receive and process image data representative of imagery captured by one or more imaging devices (e.g., medical tools) attached to the robotic medical system 1805, as well as other data stream sources received from the visualization tool. For example, one or more image capture devices (e.g., the image capture devices 110) can be located within the surgical system 1800. These image capture devices can capture images from various viewpoints within the surgical system 1800. These images (e.g., video streams) can be transmitted to the visualization tool 1820, which can then passthrough those images to the auxiliary system 1815 as a single combined data stream. The auxiliary system 1815 can then transmit the single video stream (including any data stream received from the medical tool(s) of the robotic medical system 1805) to present on a display (e.g., the display 1118) of the user control system 1810.

[0148] In some embodiments, the auxiliary system 1815 can be configured to present visual content (e.g., the single combined data stream) to other team members (e.g., the medical personnel 1830B-1830D) who might not have access to the user control system 1810. Thus, the auxiliary system 1815 can include a display 1840 configured to display one or more user interfaces, such as images of the surgical site, information associated with the patient 1825 and / or the surgical procedure, and / or any other visual content (e.g., the single combined data stream). In some embodiments, display 1840 can be a touchscreen display and / or include other features to allow the medical personnel 1830A-1830D to interact with the auxiliary system 1815.

[0149] The robotic medical system 1805, the user control system 1810, and the auxiliary system 1815 can be communicatively coupled one to another in any suitable manner. For example, in some embodiments, the robotic medical system 1805, the user control system 1810, and the auxiliary system 1815 can be communicatively coupled by way of control lines 1845, which can represent any wired or wireless communication link that can serve a particular implementation. Thus, the robotic medical system 1805, the user control system 1810, and the auxiliary system 1815 can each include one or more wired or wireless communicationinterfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc. It is to be understood that the surgical system 1800 can include other or additional components or elements that can be needed or considered desirable to have for the medical session for which the surgical system is being used.

[0150] FIG. 19 depicts an example block diagram of an example computer system 1900 is shown, in accordance with some embodiments. The computer system 1900 can be any computing device used herein and can include or be used to implement a data processing system or its components. The computer system 1900 includes at least one bus 1905 or other communication component or interface for communicating information between various elements of the computer system. The computer system further includes at least one processor 1910 or processing circuit coupled to the bus 1905 for processing information. The computer system 1900 also includes at least one main memory 1915, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 1905 for storing information, and instructions to be executed by the processor 1910. The main memory 1915 can be used for storing information during execution of instructions by the processor 1910. The computer system 1900 can further include at least one read only memory (ROM) 1920 or other static storage device coupled to the bus 1905 for storing static information and instructions for the processor 1910. A storage device 1925, such as a solid-state device, magnetic disk or optical disk, can be coupled to the bus 1905 to persistently store information and instructions.

[0151] The computer system 1900 can be coupled via the bus 1905 to a display 1930, such as a liquid crystal display, or active-matrix display, for displaying information. An input device 1935, such as a keyboard or voice interface can be coupled to the bus 1905 for communicating information and commands to the processor 1910. The input device 1935 can include a touch screen display (e.g., the display 1930). The input device 1935 can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1910 and for controlling cursor movement on the display 1930.

[0152] The processes, systems and methods described herein can be implemented by the computer system 1900 in response to the processor 1910 executing an arrangement of instructions contained in the main memory 1915. Such instructions can be read into the main memory 1915 from another computer-readable medium, such as the storage device 1925. Execution of the arrangement of instructions contained in the main memory 1915 causes the computer system 1900 to perform the illustrative processes described herein. One or moreprocessors in a multi-processing arrangement can also be employed to execute the instructions contained in the main memory 1915. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0153] Although an example computing system has been described in FIG. 19, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0154] The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are illustrative, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable or physically interacting components or wirelessly interactable or wirelessly interacting components or logically interacting or logically interactable components.

[0155] With respect to the use of plural or singular terms herein, those having skill in the art can translate from the plural to the singular or from the singular to the plural as is appropriate to the context or application. The various singular / plural permutations can be expressly set forth herein for sake of clarity.

[0156] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limitedto,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).

[0157] Although the figures and description can illustrate a specific order of method steps, the order of such steps can differ from what is depicted and described, unless specified differently above. Also, two or more steps can be performed concurrently or with partial concurrence, unless specified differently above. Such variation can depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0158] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the following appended claims can contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

[0159] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general, such a construction is intended in the sense one having skill in the art would understand theconvention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0160] Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent.

[0161] The foregoing description of illustrative implementations has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or can be acquired from practice of the disclosed implementations. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: one or more processors, coupled with memory, to: receive data collected by a plurality of sensors of a robotic medical system associated with performance of a medical procedure; select, based at least in part on a characteristic of the data and a probability of a procedure type, a probability of a phase type, and a probability of a step type established for at least a segment of a plurality of segments of the medical procedure, a first one or more models from a plurality of models trained with machine learning; update, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure; and provide, for display via a graphical user interface, an indication of at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

2. The system of claim 1, comprising the one or more processors to: determine the characteristic of the data based at least in part on a quality of the data.

3. The system of claim 2, comprising the one or more processors to: determine the quality of the data based at least in part on a comparison of a count of invalid, corrupt, or missing samples in the data with a quality threshold.

4. The system of claim 3, comprising the one or more processors to: select the quality threshold based on at least one of a type of the data, the probability of the procedure type, the probability of the phase type, or the probability of the step type for the at least the segment of the plurality of segments.

5. The system of any one of claims 1-3, comprising the one or more processors to: determine a type of the data, wherein the type comprises at least one of video, kinematics, or system events; and select the first one or more models based on the type of the data.

6. The system of claim of claim 1, comprising the one or more processors to: determine the data comprises a video stream, a kinematics stream, and an event stream; determine a quality of each of the video stream, the kinematics stream, and the event stream; and select the first one or more models based on the quality of each of the video stream, the kinematics stream, and the event stream.

7. The system of claim 6, comprising the one or more processors to: determine, for a second segment of the plurality of segments of the medical procedure, that the quality of the event stream fails a quality threshold; select, responsive to the quality of the event stream failing the quality threshold for the second segment, a second one or more models of the plurality of models; and determine, using the second one or more models, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more second segments.

8. The system of claim 1, wherein the probability of the procedure type, the probability of the phase type, and the probability of the step type established for a first segment of the plurality of segments corresponds to a default probability.

9. The system of claim 1, comprising the one or more processors to: select a second one or more models from the plurality of models based on the update made using the first one or more models to at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

10. The system of claim 1, comprising the one or more processors to: receive, via the graphical user interface, an indication of a metric to compute for the medical procedure, wherein the metric indicates performance of the medical procedure; and select the first one or more models based on the indication of the metric.

11. The system of claim 1, comprising the one or more processors to:estimate a first duration to process the data using the first one or more models and a second duration to process the data using a second one or more models of the plurality of models, wherein the first duration is less than the second duration; provide, for display via the graphical user interface, a graphical user interface element that indicates the first duration and the second duration; and receive, via the graphical user interface, a selection of the first one or more models corresponding to the first duration less than the second duration.

12. The system of claim 11, comprising the one or more processors to: determine a first confidence score associated with predictions made using the first one or more models, and a second confidence score associated with predictions made using the second one or more models, wherein the second confidence score is greater than the first confidence score; and provide, for display via the graphical user interface, the graphical user interface element comprising the first confidence score and the second confidence score.

13. The system of claim 1, comprising the one or more processors to: select, for a second segment of the medical procedure subsequent to the segment, a second one or more models from the plurality of models using a confusion matrix established based on performance of the plurality of models during training.

14. The system of claim 1, comprising the one or more processors to: use a probabilistic graphical model and the probability of the procedure type, the probability of the phase type, and the probability of the step type to generate the indication.

15. A method, comprising: receiving, by one or more processors coupled with memory, data collected by a plurality of sensors of a robotic medical system associated with performance of a medical procedure; selecting, by the one or more processors, based on a probability of a procedure type, a probability of a phase type, and a probability of a step type established for at least a segment of a plurality of segments of the medical procedure, a first one or more models from a plurality of models trained with machine learning;updating, by the one or more processors, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure; and providing, by the one or more processors, for display via a graphical user interface, an indication of at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

16. The method of claim 15, comprising: selecting, by the one or more processors, the first one or more models based on a characteristic of the data, the characteristic comprising at least one of a quality of the data or a type of the data.

17. The method of claim 16, comprising: determining, by the one or more processors, for a second segment of the plurality of segments of the medical procedure, that a quality of the event stream fails a quality threshold; selecting, by the one or more processors, responsive to the quality of the event stream failing the quality threshold for the second segment, a second one or more models of the plurality of models; and determining, by the one or more processors, using the second one or more models, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more second segments.

18. The method of claim 15, wherein the probability of the procedure type, the probability of the phase type, and the probability of the step type established for a first segment of the plurality of segments corresponds to a default probability.

19. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to: receive data collected by a plurality of sensors of a robotic medical system associated with performance of a medical procedure; select, based at least in part on a number of corrupt samples in the data and a probability of at least a segment of the medical procedure corresponding to a procedure type, a phase type,or a step type, a first one or more models from a plurality of models trained with machine learning; update, using the first one or more models and the data, the probability of the procedure type, the probability of the phase type, and the probability of the step type for one or more segments of the plurality of segments of the medical procedure; and provide, for display via a graphical user interface, an indication of at least one of the probability of the procedure type, the probability of the phase type, or the probability of the step type for the one or more segments.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions further comprise instructions to cause the one or more processors to: select, for a second segment of the medical procedure subsequent to the segment, a second one or more models from the plurality of models using a probabilistic graphical model and based on the probability of the procedure type, the probability of the phase type, and the probability of the step updating using the first one or more models.

Images