Robotic medical system with machine learning predicted procedure outcomes for scenarios

Abstract

Machine learning predicted outcomes for scenarios of a procedure performed with a robotic medical system is provided. A system can identify state information related to a medical procedure to be performed by a robotic medical system. The system can generate, using the state information and with one or more models trained by machine learning, scenarios of operation of the robotic medical system to perform the medical procedure. The system can determine, using the one or more models, performance metrics for the scenarios. The system can select, based on a comparison of the performance metrics for the scenarios, a scenario of the scenarios for operation. The system can provide an indication of the selected scenario to cause the robotic medical system to perform at least a portion of the medical procedure in accordance with the selected scenario.

Description

ROBOTIC MEDICAL SYSTEM WITH MACHINE LEARNINGPREDICTED PROCEDURE OUTCOMES FOR SCENARIOSCROSS-REFERENCE TO RELATED PATENT APPLICATION

[0001] This application claims the benefit of, and priority to, under 35 U.S.C. § 119, U.S. Provisional Patent Application No. 63 / 644,253 filed May 8, 2024, the entirety of which is incorporated by reference herein.BACKGROUND

[0002] A robotic medical system can include an instrument for performing a medical session or procedure on a patient. The robotic medical system can perform surgery, therapy, or a medical evaluation. A medical facility can provide post-procedure care of the patient after the medical procedure is performed by the robotic medical system.SUMMARY

[0003] Technical solutions disclosed herein can include a robotic medical system with machine learning predicted procedure outcomes for scenarios. A computing system can generate synthetic variants or scenarios of a medical procedure using pre-procedure data of a patient. The computing system can generate intra-procedure data for each of the variants (e.g., metrics or indicators quantifying the performance of the different variants). The system can use a machine learning model (e.g., conditional generative adversarial model (cGAN)) to generate a large synthetic dataset (e.g., thousands of samples). Each sample of the synthetic dataset can represent a different potential workflow for the medical procedure, and include intra-procedure indicators for each workflow. The system can execute one or a set of machine learning models (e.g., surgical outcome prediction models) that predict surgical outcomes using the synthetic dataset. For example, for each sample of the synthetic dataset, the system can produce a different set of predicted outcomes for the workflow. The outcomes can be post-operative outcomes, e.g., time to recover from anesthesia, length-of- stay of a patient after the surgery, total cost of the post-operative care of the patient, or risk of infection from the procedure.

[0004] At least one aspect of the present disclosure is a system. The system can include one or more processors, coupled with memory, to identify state information related to amedical procedure to be performed by a robotic medical system. The one or more processors can generate, using the state information and with one or more models trained by machine learning, scenarios of operation of the robotic medical system to perform the medical procedure. The one or more processors can determine, using the one or more models, performance metrics for the scenarios. The one or more processors can select, based on a comparison of the performance metrics for the scenarios, a scenario of the scenarios for operation. The one or more processors can provide an indication of the selected scenario to cause the robotic medical system to perform at least a portion of the medical procedure in accordance with the selected scenario.

[0005] The one or more processors can generate, using a first conditional generative model and pre-procedure data of a patient, the scenarios. The one or more processors can generate, using a second conditional generative model, the pre-procedure data, and the scenarios, performance indicators for the scenarios. The one or more processors can construct a synthetic dataset including the scenarios and the performance indicators for the scenarios.

[0006] The one or more processors can receive a synthetic dataset including the scenarios and performance indicators for the scenarios. The one or more processors can execute, using the synthetic dataset, at least one model to generate the performance metrics for the scenarios.

[0007] The one or more processors can the one or more processors to generate the scenarios using a conditional generative model.

[0008] The one or more processors can determine the performance metrics using a model that predicts outcomes for the medical procedure from the scenarios.

[0009] The one or more processors can receive updated state information subsequent to execution of the at least the portion of the medical procedure in accordance with the selected scenario. The one or more processors can generate, using the one or more models, a second scenarios based on the updated state information. The one or more processors can determine, using the one or more models, second performance metrics for the second scenarios. The one or more processors can select, based on a comparison of the second performance metrics for the second scenarios, a second scenario of the second scenarios for operation. The one or more processors can provide an indication of the selected second scenario to cause the robotic medical system to perform at least a second portion of the medical procedure in accordancewith the selected second scenario.

[0010] The one or more processors can determine, using the one or more models, the performance metrics for the scenarios, wherein at least one of the performance metrics corresponds to an outcome of the medical procedure performed by the robotic medical system.

[0011] The one or more processors can determine a count of the scenarios to generate based on a type of an outcome of the medical procedure performed by the robotic medical system. The one or more processors can generate the scenarios in accordance with the determined count.

[0012] The one or more processors can generate the scenarios. The scenario of the scenarios can include a workflow of the medical procedure including a sequence of steps performed by the robotic medical system.

[0013] The one or more processors can generate the scenarios. The scenario of the scenarios can include a workflow of the medical procedure including a sequence of steps performed by the robotic medical system and indications of types of instrument for the robotic medical system to use at the steps.

[0014] At least one aspect of the present disclosure is a method. The method can include identifying, by one or more processors, state information related to a medical procedure to be performed by a robotic medical system. The method can include generating, by the one or more processors, using the state information and with one or more models trained by machine learning, scenarios of operation of the robotic medical system to perform the medical procedure. The method can include determining, by the one or more processors, using the one or more models, performance metrics for the scenarios. The method can include selecting, by the one or more processors, based on a comparison of the performance metrics for the scenarios, a scenario of the scenarios for operation. The method can include providing, by the one or more processors an indication of the selected scenario to cause the robotic medical system to perform at least a portion of the medical procedure in accordance with the selected scenario.

[0015] The method can include generating, by the one or more processors, using a first conditional generative model and pre-procedure data of a patient, the scenarios. The methodcan include generating, by the one or more processors, using a second conditional generative model, the pre-procedure data, and the scenarios, performance indicators for the scenarios. The method can include constructing, by the one or more processors, a synthetic dataset including the scenarios and the performance indicators for the scenarios.

[0016] The method can include receiving, by the one or more processors, a synthetic dataset including the scenarios and performance indicators for the scenarios. The method can include executing, by the one or more processors, using the synthetic dataset, at least one model to generate the performance metrics for the scenarios.

[0017] The method can include determining, by the one or more processors, the performance metrics using a model that predicts outcomes for the medical procedure from the scenarios.

[0018] The method can include receiving, by the one or more processors, updated state information subsequent to execution of the at least the portion of the medical procedure in accordance with the selected scenario. The method can include generating, by the one or more processors, using the one or more models, a second scenarios based on the updated state information. The method can include determining, by the one or more processors, using the one or more models, second performance metrics for the second scenarios. The method can include selecting, by the one or more processors, based on a comparison of the second performance metrics for the second scenarios, a second scenario of the second scenarios for operation. The method can include providing, by the one or more processors, an indication of the selected second scenario to cause the robotic medical system to perform at least a second portion of the medical procedure in accordance with the selected second scenario.

[0019] The method can include determining, by the one or more processors, a count of the scenarios to generate based on a type of an outcome of the medical procedure performed by the robotic medical system. The method can include generating, by the one or more processors, the scenarios in accordance with the determined count.

[0020] The method can include generating, by the one or more processors, the scenarios. The scenario of the scenarios can include a workflow of the medical procedure including a sequence of steps performed by the robotic medical system.

[0021] At least one aspect of the present disclosure is directed to one or more storagemedia storing instructions thereon, that, when executed by one or more processors, cause the one or more processors to perform operations, including identifying state information related to a medical procedure to be performed by a robotic medical system. The operations can include generating, using the state information and with one or more models trained by machine learning, scenarios of operation of the robotic medical system to perform the medical procedure. The method can include determining, using the one or more models, performance metrics for the scenarios. The method can include selecting, based on a comparison of the performance metrics for the scenarios, a scenario of the scenarios for operation. The method can include providing an indication of the selected scenario to cause the robotic medical system to perform at least a portion of the medical procedure in accordance with the selected scenario.

[0022] The operations can include generating, by the one or more processors, using a first conditional generative model and pre-procedure data of a patient, the scenarios. The operations can include generating, by the one or more processors, using a second conditional generative model, the pre-procedure data, and the scenarios, performance indicators for the scenarios. The method can include constructing, by the one or more processors, a synthetic dataset including the scenarios and the performance indicators for the scenarios. The operations can include executing, by the one or more processors, using the synthetic dataset, at least one model to generate the performance metrics for the scenarios.

[0023] The operations can include receiving, by the one or more processors, updated state information subsequent to execution of the at least the portion of the medical procedure in accordance with the selected scenario. The operations can include generating, by the one or more processors, using the one or more models, a second scenarios based on the updated state information. The operations can include determining, by the one or more processors, using the one or more models, second performance metrics for the second scenarios. The operations can include selecting, by the one or more processors, based on a comparison of the second performance metrics for the second scenarios, a second scenario of the second scenarios for operation. The operations can include providing, by the one or more processors, an indication of the selected second scenario to cause the robotic medical system to perform at least a second portion of the medical procedure in accordance with the selected second scenario.

[0024] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations, and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated in and constitute a part of this specification. The foregoing information and the following detailed description and drawings include illustrative examples and should not be considered as limiting.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0026] FIG. 1 depicts an example computing system to predict outcomes of medical procedures using machine learning.

[0027] FIG. 2 depicts an example computing system to generate synthetic medical procedure samples from pre-procedure data and predict outcomes for the synthetic medical procedure samples.

[0028] FIG. 3 depicts an example graphical user interface for a user to enter preprocedure data of a patient.

[0029] FIG. 4 depicts an example method of predicting outcomes of medical procedures using machine learning.

[0030] FIG. 5 depicts a chart comparing example machine learning models that predict different outcomes for different procedure types.

[0031] FIG. 6 depicts a chart of features and importance levels for the features for example machine learning models for various medical procedures that predict a postanesthesia care unit (PACU) time.

[0032] FIG. 7 depicts a chart of features and importance levels for the features for example machine learning models for various medical procedures that predict a length of stay (LOS) time.

[0033] FIG. 8 depicts a chart of features and importance levels for the features for example machine learning models for various medical procedures that predict patient readmittance flag.

[0034] FIG. 9 depicts an example computing architecture of a data processing system.DETAILED DESCRIPTION

[0035] Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems for a robotic medical system with machine learning predicted procedure outcomes for scenarios. The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.

[0036] This disclosure is generally directed to predicting outcomes of a medical procedure. A model, such as a surgical risk prediction model, can predict an outcome of a medical procedure based on pre-procedure data of the patient (e.g., weight of a patient, comorbidities of a patient, or results of past medical procedures of the patient). The model can predict probabilities of different surgical outcomes occurring based on the pre-procedure data. However, while the model can predict the outcome, the model may not be structured or designed to inform a robotic medical system on what actions to perform during the surgery to achieve a specific outcome. For example, the model can be executed after a medical procedure is performed to identify what the robotic medical system may have done incorrectly. However, the model may be unable to identify what actions the robotic medical system should take during the medical procedure to achieve a positive or successful outcome for the medical procedure. The model may not produce uncertainty in predictions and how surgical outcomes could be affected by intra-procedure choices, decisions, or actions. The model may not execute on, or be designed around, intra-procedure data. For example, the model may not be able to identify decisions or actions to be taken because the model may not execute on intra-procedure data (e.g., intra-operative data).

[0037] To solve these, and other technical problems, technical solutions of this disclosure can include a robotic medical system with machine learning predicted procedure outcomes for scenarios. A computing system can generate synthetic variants or scenarios of a medical procedure using pre-procedure data of a patient. A synthetic dataset or variants can refer to a collection of data that is artificially generated (e.g., using a model trained with machine learning), as opposed to a dataset that is collected from real-world observations or measurements. The computing system can generate intra-procedure data for each of the variants (e.g., metrics or indicators quantifying the performance of the different variants). The computing system can implement pre-procedure probabilistic risk analysis on the synthetically generated variants and intra-procedure data and predict outcomes of each of the variants using an outcome prediction model.

[0038] The computing system can use historical data to predict a variety of different potential variants or workflows that a medical practitioner can take to perform a medical procedure with a robotic medical system. For example, a variety of different orders or sequences of phases or steps can be taken to achieve a particular medical procedure. The robotic medical system can record intra-procedure data unobtrusively and the computing system can connect the intra-procedure information to pre-procedure information and postprocedure outcomes. The computing system can use this historical data to generate numerous possible variants of intra-procedure robotic medical system use for a single patient scheduled for a procedure and inform how best to proceed to optimize surgical outcomes. The system can use a machine learning model (e.g., conditional generative adversarial model (cGAN)) to generate a large synthetic dataset. Each sample of the synthetic dataset can represent a different potential workflow for the medical procedure, and include intra-procedure indicators for each workflow. The synthetic dataset can be generated to include a large number of samples (e.g., 1,000 or more) to capture a variety of different potential procedure outcomes.

[0039] The system can execute one or a set of machine learning models (e.g., surgical outcome prediction models) that predict surgical outcomes using the synthetic dataset. For example, for each sample of the synthetic dataset, the system can produce a different set of predicted outcomes for the workflow. The outcomes can be post-operative outcomes, e.g., time to recover from anesthesia, length-of-stay of a patient after the surgery, total cost of the post-operative care of the patient, or risk of infection from the procedure. The system can runa set of different machine learning models on the synthetic dataset, each model outputting a different metric for the workflows.

[0040] The resulting outputs of the models can be a set of sample workflows, each with a different set of surgical outcomes. The sample workflows and surgical results can be used to perform scenario analysis and planning. For example, the system can recommend one or a set of workflows to a surgeon to follow, either before the surgery or during the surgery. Furthermore, the system can inform post-operative care coordination for a patient and central planning for a hospital, such as scheduling surgeries or patient recoveries based on the workflows and outcomes.

[0041] Referring now to FIG. 1, among others, an example system 100 including computing system 105 to predict outcomes of medical procedures using machine learning is shown. Given a single sample of pre-procedure data 110 for a patient, the computing system 105 can produce a range of possible post-procedure outcomes 115 connected to intraoperative indicators of the robotic medical system 120 in order to provide pre-procedure, intra-procedure, or post-procedure scenario analysis and planning for the patient. The computing system 105 can be communicably coupled with, or integrated with, the robotic medical system 120. The computing system 105 can be separate from the robotic medical system 120, or can be a component of the robotic medical system 120.

[0042] The system 100 can include at least one computing system 105. The computing system 105 can be a data processing system, a computing system, a computer system, a computer, a desktop computer, a laptop computer, a tablet, a control system, a console system, an embedded system, a cloud computing system, a server system, or any other type of computing system. The computing system 105 can be an on-premises system or an off- premises system. The computing system 105 can be a hybrid system, where some components of the computing system 105 are located on-premises, and some components of the computing system 105 are located off-premises. The computing system 105 can be a back-end system or back-end server system that uses pre-procedure data 110 entered by a user to generate synthetic surgical scenarios and perform scenario analysis.

[0043] The system 100 can include at least one robotic medical system 120. The robotic medical system 120 can be a robotic system, apparatus, or assembly including at least one instrument 125. The instrument 125 can be or include a tip or end. The tip or end can beinstalled with or to the instrument 125. The tip can be removable or a permanent component of the instrument or the robotic medical system 120. For example, the tip can be a scalpel, a scissors, a monopolar curved scissors (MCS), a cautery hook tip, a cautery spatula tip, a needle driver, forceps, a round tooth retractor, a drill, or a clip applier. The instrument 125 can be or include a robotic arm, a robotic appendage, a robotic snake, or any other motor- controlled member that can be articulated by the robotic medical system 120. The instrument 125 can include at least one actuator, such as a motor, servo, or other actuator device. The instrument 125 can be manipulated by motors, servos, actuators, or other devices to perform a medical procedure.

[0044] The robotic medical system 120 can perform a medical session or medical procedure. For example, the robotic medical system 120 can articulate the instrument 125 to perform surgery (e.g., appendicitis, biopsy, hysterectomy, etc.), therapy, or a medical evaluation with the instrument 125. The medical procedure can be performed on a subject, e.g., a human, an adult, a child, or an animal. A medical practitioner, such as a surgeon, technician, nurse, or other operator can provide input via a user device or input apparatus (e.g., joystick, buttons, touchpad, keyboard, steering apparatus, etc.) to manipulate the instrument 125 to perform a medical procedure. The robotic medical system 120 can include an endoscope 130, in some implementations. The endoscope 130 can be an instrument that is manipulated by the medical practitioner and controlled via a motor, servo, or other input device. The endoscope 130 can record images, image frames, or videos of the medical procedure.

[0045] The computing system 105 can include at least one model 135. The model 135 can generate synthetic data 140. For example, the computing system 105 can include first models 135 (e.g., conditional generative models) and second models 155 (e.g., medical procedure outcome prediction models). The model 135 can be one or multiple generative models. For example, the generative models can be transformer models, variational autoencoders, encoder-decoder architectures, or gaussian generative models. For example, the model 135 can be a generative adversarial model (GAN), such as a conditional GAN (cGAN). The model 135 can be trained using at least one machine learning technique. The model 135 can be trained with a supervised technique, an unsupervised technique, or a selfsupervised technique.

[0046] The computing system 105 can train the model 135 using the historical data 175. The computing system 105 can train the model 135 with historical data 175 (e.g., surgical workflows, intra-operative OPIs, post-surgical outcomes) to output potential workflow / OPI samples for pre-procedure input data 110. The historical data 175 can include data for multiple different patients. The historical data 175 can include a set or multiple samples. For example, the samples of the historical data 175 can each include pre-procedure data 110, a workflow of the medical procedure (e.g., phases and steps) taken for the sample, performance indicators of the workflow, or post-procedure outcomes for the sample.

[0047] The model 135 can generate, using state information, at least one scenario 145. The model 135 can generate, using the state information, intra-procedure indicators 150 for at least one scenario 145. The model 135 can generate multiple possible scenarios 145 of procedure workflows and intra-procedure performance indicators 150 for the scenarios 145 using the pre-procedure data 110 of a patient. The scenarios 145 can be potential workflows or sequences of operations that the robotic medical system 120 can follow to perform the medical procedure. The model 135 can receive the pre-procedure data 110, and execute the model 135 with the pre-procedure data 110 as an input. The model 135 can output a variety of scenarios 145.

[0048] The scenarios 145 can be workflows of the medical procedure for the robotic medical system 120 to perform. Each scenario 145 can break the medical procedure down into parts or segments. For example, each scenario 145 can be made up of phases or steps. For example, each scenario 145 can be made up of multiple different phases. The phases can be made up of multiple different steps. The phases can include transection of an anatomical structure of a patient, extraction of an anatomical structure, reconstruction of an anatomical structure, dissection, or cutting or dissecting an anatomical structure or patient. The steps can include specimen removal, litigation or division of a cystic artery, dissection of a gallbladder of a liver off a liver bed, etc. The scenarios 145 can indicate what types of instruments 125 to be used at different phases or steps of the workflow. For example, the scenarios 145 can indicate at what point an instrument 125 should be replaced with another instrument, e.g., a forceps replaced with a scissors.

[0049] The model 135 can generate intra-procedure indicators 150 for the scenarios 145. The indicators 150 can be predicted intra-procedure performances of each scenario 145. For example, the indicators 150 can quantify the performance of the robotic medical system 120during the medical procedure. For example, the model 135 can generate an indicator 150 for each phase or step of the scenario 145. The model 135 can generate a set of different types of indicators 150 for each phase or step of the scenario 145. The indicators 150 can be metrics or objective performance indicators (OPIs) that can indicate or be indicative of performance of the robotic medical system performing the medical procedure. The model 135 can generate OPI indicators 150 for an entire workflow, a phase of a workflow, or a step of the workflow. The various indicator types can include an amount of energy or power consumed by the robotic medical system 120. The various indicator types can include a total time duration of the medical procedure, such as a length of time of a step, phase, or the entire medical procedure. The indicator types can include a total linear distance traveled by the instrument 125 of the robotic medical system 120 during the scenario 145. The various indicator types can include a total angular distance of the instrument 125 of the robotic medical system 120 during the segment. The angular distance can indicate an amount or distance that the instrument 125 is rotated during manipulations. The various indicator types can include a total number of operations of a clutch or brake. For example, the indicator 150 can indicate a total number of actuations or activations of a clutch or brake for the instrument 125 or the endoscope 130 of the robotic medical system 120. The clutch or brake can be operated to float joints of the instrument 125 or the endoscope 130.

[0050] The computing system 105 can generate or construct the synthetic data 140. For example, the computing system 105 can generate a dataset, a data structure, a data table, or a set of samples. The computing system 105 can generate one data structure, file, or set of components to store the scenarios 145 and the indicators 150. For example, the computing system 105 can combine the scenarios 145 and the indicators 150 into one file or set of files.

[0051] The computing system 105 can identify state information related to a medical procedure to be performed by a robotic medical system 120. For example, the state information can be the pre-procedure data 110. The pre-procedure data 110 can describe characteristics of a patient for a medical procedure to be performed on (e.g., a weight of the patient, a body mass index of the patient, disease states of a patient, comorbidities of the patient, an age of the patient, a sex of the patient, a race of the patient, demographics data of a patient, health levels of the patient, indications of past surgeries and outcomes of the past surgeries of the patient, pre-operation medication of the patient, lab results of the patient, or smoker status of the patient). The pre-procedure data 110 can indicate hospital proceduredata, e.g., the number of days the patient was in the hospital before the medical procedure, codes of the medical procedure, a disposition of the patient, or a date or time that the medical procedure is planned.

[0052] The computing system 105 can generate the synthetic data 140 before the medical procedure is performed, or regenerate the synthetic data 140 throughout the medical procedure. For example, the computing system 105 can regenerate the synthetic data 140 at predetermined time interval, at the end of certain steps or phases being completed by the robotic medical system 120, or responsive to detection of a condition or event. For example, the condition or event could be a likelihood of a predicted OPI 150 falling below threshold in a likely scenario 145 (e.g., 50% likelihood of an OPI falling below a threshold, identifying a likelihood of improving an OPI 150 by going along a different scenario 145, 50% likelihood of being able to improve OPI 150 by selecting a number of different scenarios), or responsive to request from surgeon to generate new scenarios 145.

[0053] The pre-procedure data 110 can describe characteristics of the robotic medical system 120 (e.g., the age of the robotic medical system 120, the model of the robotic medical system 120, the number of instruments 125 of the robotic medical system 120, or an indication of whether the robotic medical system 120 is a single-port or multi-port system). The pre-procedure data 110 can indicate characteristics of the medical professional performing the medical procedure, e.g., a name or identifier of a care team, a name of a doctor, surgeon, or other operator of the robotic medical system 120, a name of a principal practitioner or surgeon, names or numbers of secondary or support staff, a level of expertise of the operator, a number of past medical procedures performed by the operator, or a rating or success score of the operator.

[0054] The computing system 105 can include at least one interface manager 160. The interface manager 160 can receive the pre-procedure data 110 from a client device 165. The interface manager 160 can receive the pre-procedure data 110 from the client device 165 via a graphical user interface 170. The interface manager 160 can generate the graphical user interface 170 and cause the client device 165 to display the graphical user interface 170. The client device 165 can be a smartphone, a desktop computer, a laptop computer, a tablet, or any other mobile or stationary user device. The client device 165 can be accessed or utilized by a patient, a medical practitioner, a doctor, a nurse, a family member of a patient, or any other user. The interface manager 160 can cause the client device 165 to display thegraphical user interface 170. In some implementations, the computing system 105 can receive the pre-procedure data 110 or the historical data 175 from a hospital or medical system. The computing system 105 can integrate into an electronic health record (EHR) or electronic medical record (EMR) of a hospital or at an integrated delivery network (IDN) level. The computing system 105 can receive the pre-procedure data 110 or the historical data 175 in a programmatic manner as that information is available and synthetic surgical outcomes can be generated accordingly.

[0055] The model 135 can be configured with a parameter or setting to generate a predefined number of scenarios 145. The computing system 105 can determine or generate a count of the scenarios 145 to generate. The computing system 105 can determine the count based on the type of medical procedure outcomes to be predicted by the model 155. For example, if the computing system 105 is set or configured to predict a particular type of medical procedure outcome that is rare or happens infrequently, the computing system 105 can increase the count to generate enough scenarios 145 such that there will be scenarios 145 that have the type of outcome. The model 135 can generate the scenarios 145 in accordance with the determined count. The model 135 can generate a high number of samples (e.g., greater than 1,000) to capture a variety of different possible workflows. For example, the computing system 105 can determine the number of scenarios 145 to generate based on the type of outcome a surgeon wants to optimize to ensure that the number of scenarios 145 generated is statistically relevant.

[0056] The computing system 105 can update or change the number of scenarios 145 for the model 135 to generate for different patients. For example, for a first patient, the computing system 105 can generate a first number of scenarios 145. For a second patient, the computing system 105 can generate a second number of scenarios 145. The computing system 105 can determine different counts of scenarios 145 to generate based on different types of pre-procedure data 110 for the different patients or different outcome metrics 180 of interest for the second patient. The computing system 105 can update or change the number of scenarios 145 for the model 135 to generate for different types of medical procedures. For example, the model 135 can generate a first number of scenarios 145 for a first type of medical procedure (e.g., cholecystectomy) and a second number of scenarios 145 for a second type of medical procedure (e.g., appendectomy).

[0057] The computing system 105 can include at least one model 155. The model 155 can receive the synthetic data 140 including the scenarios 145 and performance indicators 150 for the scenarios 145. The computing system 105 can execute, using the synthetic data 140, at least one model 155 to generate the performance metrics 180 for the scenarios 145. The model 155 can predict the range of possible post-surgical outcomes for a patient, and connect those outcomes with different potential surgical workflows (e.g., sequence of steps and / or surgical phases) and objective performance indicators to help aide in pre-procedure, intra-procedure, and post-procedure care.

[0058] The model 155 can generate or predict outcomes 115. The outcomes 115 can be or include at least one dataset, data structure, file, or component that includes the scenarios 145 generated by the model 135. The outcomes 115 can include metrics 180. The metrics 180 can indicate predictions of outcomes for a medical procedure if a particular scenario 145 is followed. The computing system 105 can predict the performance metrics 180 for the scenarios 145. Each performance metric 180 can correspond to a different outcome of the medical procedure performed by the robotic medical system 120 if a particular scenario 145 would be followed by the robotic medical system 120. The metrics 180 can indicate a prediction of a length-of-stay of the patient after the medical procedure is performed if a particular scenario 145 is followed. The metrics 180 can indicate a total cost of care of a patient after the medical procedure is performed if a particular scenario 145 is followed. The metrics 180 can indicate a probability that the patient will be readmitted to a medical facility after the patient is released following the medical procedure if a particular scenario 145 is followed. The computing system 105 can include one model 155 that predicts each of multiple different metric types, or a set of models 155, each model 155 of the set predicting a different metric type.

[0059] The computing system 105 can build, construct, or train the models 155. The computing system 105 can build surgical outcome prediction models (SOPMs) that predict outcomes of surgical procedures performed by the robotic medical system 120. The computing system 105 can train the model 155 with historical data 175 to predict a medical procedure outcome. The model 155 can be trained to receive scenarios 145 and indicators 150 as an input, and produce metrics 180, such as a value or probability of a procedure outcome.

[0060] The computing system 105 can train the model 155 with a supervised technique, an unsupervised technique, or a self-supervised technique. The model 155 can be a predictive neural network that predicts a value or probability for a particular metric type or particular type of outcome of the medical procedure. The model 155 can be a feed forward neural network. The model 155 can be a boosted tree model, such as an ensemble classifier of an extreme gradient boost (XGBoost) model. The model 155 can be a modular neural network. The model 155 can be a Bayes network or a Hidden Markov model.

[0061] The computing system 105 can include at least one filter 185. The filter 185 can filter out or remove at least one scenario 145 from the predicted outcomes 115 based on the metrics 180. For example, the filter 185 can reduce the samples of the predicted outcomes 115 from a first number to a second smaller number. For example, if the predicted outcomes 115 have hundreds or thousands of samples, the filter 185 can reduce the number of samples to tens of samples. For example, the filter 185 can filter out the predicted outcomes 115 to a set of the predicted outcomes 115 with the best or highest metrics 180. For example, the filter 185 can compare the metrics 180 against each other, and select one scenario 145 of the scenarios 145 for operating the robotic medical system 120. The filter 185 can score each scenario 145 based on the metrics 180, and select a highest scored scenario 145. For example, the scores can be based on a combination of the metrics 180, e.g., length of stay, total cost of care, re-admittance, etc. The filter 185 can select a scenario 145 that reduces or minimizes length-of-stay, reduces or minimizes total cost of care, and reduces or minimizes the probability of re-admittance of the patient.

[0062] The filter 185 can filter each of the scenarios 145 to remove scenarios 145 with bad outcomes and retain scenarios 145 with good outcomes. The filter 185 can include or store a set of thresholds. The filter 185 can apply the thresholds to remove or retain scenarios 145. The filter 185 can store a different threshold for each metric type. For example, the filter 185 can apply a first threshold to length-of stay metrics 180, and remove scenarios 145 with length-of-stay metrics greater than the first threshold. The filter 185 can apply a second threshold to total cost of care metrics 180, and remove scenarios 145 with a total cost of care metric 180 greater than the second threshold.

[0063] In some implementations, the filter 185 can select a scenario 145 using the indicators 150. The filter 185 can select a scenario 145 using one or a multiple or combination of different types of indicators 150. The filter 185 can balance a set of group ofindicators 150 to select a scenario 145. For example the filter 185 can implement an optimization or mixed integer programming technique to optimize multiple metrics 180 simultaneously or concurrently. The filter 185 can identify which types of indicators 150 to use in selecting a scenario 145. For example, the filter 185 can select the types of indicators 150 based on pre-surgical state information, e.g., complexity of medical procedure, type of a medical procedure, health of a patient, or surgeon skill level.

[0064] The filter 185 can execute based on user input data. For example, a user can provide input data via the robotic medical system 120 or the client device 165. The computing system 105 can analyze the predicted outcomes 115 in a user-led manner. The user can input a range of acceptable outcomes for the medical procedure over a set of possible outcomes. For example, the user can input thresholds for comparing the metrics 180 to. The computing system 105, such as the filter 185, can determine what set of surgical workflow scenarios 145 and OPIs 150 lead to the outcomes specified by the user.

[0065] The computing system 105 can include at least one planner 190. The planner 190 can receive a scenario 145 selected by the filter 185. The planner 190 can implement the selected scenario 145. The planner 190 can provide an indication of the selected scenario 145 to cause the robotic medical system 120 to perform at least a portion of the medical procedure in accordance with the selected scenario 145. The planner 190 can communicate with the robotic medical system 120 to cause the robotic medical system 120 to perform the workflow of the selected scenario 145. The planner 190 or the robotic medical system 120 can execute control algorithms with the workflow of the selected scenario 145 to operate the instruments 125 or the endoscope 130. For example, the planner 190 or the robotic medical system 120 can manipulate, move, or control the instrument 125 or the endoscope 130 by controlling motors of the robotic medical system 120 to perform the workflow of the robotic medical system 120. The robotic medical system 120 can perform the workflow of the selected scenariol 45 autonomously, semi -autonomously with user input, or based on user input or control. For example, instead of autonomously or semi-autonomously performing the workflow of the selected scenario 145, the robotic medical system 120 can display alerts or guided indicators to cause an operator to operate the robotic medical system 120 according to the workflow of the selected scenario 145.

[0066] The planner 190 can perform post-procedure care. For example, the planner 190 can integrate with systems of a hospital, such as scheduling systems, patient recovery roombooking and management systems, or surgery or procedure scheduling systems. The planner 190 can perform central planning using the metrics 180 predicted for a scenario 145 that the robotic medical system 120 executes with. For example, the planner 190 can identify multiple predicted metrics 180 for multiple different patient procedures at a hospital, and then select the date or time of the procedure and book post-operation care based on the predicted metrics 180, e.g., based on a predicted length-of-stay of each patient after the operation, a predicted total time that the procedure will take, etc.

[0067] The planner 190 can use the predicted outcomes 115 to avoid un-desirable outcomes and create a surgical plan for the best possible set of outcomes. The planner 190 can generate a report with recommended scenarios 145. The information in the report can be used to avoid un-desirable outcomes and create a surgical plan for the best possible set of outcomes. If certain adverse outcomes are not likely to be avoided, then the report can also be used for post-surgical care planning.

[0068] The computing system 105 can include at least one scenario updater 195. The computing system 105 can run intra-operatively, in real-time, or near real-time at an operating room, or in a remote location as the robotic medical system 120 performs a medical procedure. As the robotic medical system 120 performs the procedure, certain values of the intra-operative data become fixed. For example, certain phases or steps are completed, and these phases or steps become fixed. The computing system 105 can use the fixed data to increase the set of conditional priors used by the model 135 to generate the remaining possible workflow scenarios 145 and OPIs 150. These updated workflow scenarios 145 and OPIs 150 can be used to update the range of surgical outcomes 115 and scenario analysis.The results of this analysis can be used to inform intra-procedure decision by highlighting the next most favorable workflow scenario 145 and patterns of system use from the current surgical state.

[0069] The updater 195 can be communicably coupled with the robotic medical system 120. The robotic medical system 120 can transmit or communicate system data that indicates the actions taken by the robotic medical system 120. For example, the system data can be indications that a particular phase or step began, is being performed, or is completed. The robotic medical system 120 can provide kinematics data indicating the motions of the instrument 125 or the endoscope 130 or frames or video captured from the endoscope 130. The scenario updater 195 can determine, based on the received kinematics data or video data,that a particular phase or step was begun, is being performed, or is completed. The scenario updater 195 can use the determination to determine updated feasible scenarios 145.

[0070] In some implementations, the scenario updater 195 receives the data from the robotic medical system 120 intraoperatively, in real-time, or in near real time. The scenario updater 195 can make real-time or intra-procedure recommendations while the medical procedure is performed by the robotic medical system 120. The scenario updater 195 can use the real-time data collected from the robotic medical system 120 to narrow down the set of predicted workflows 145 as the surgery is performed to track which potential workflows 145 can still be followed. For example, the scenario updater 195 can track what phases or steps the robotic medical system 120 has taken during the medical procedure. The scenario updater 195 can identify which synthesized scenarios 145 begin with the same phases and steps that the robotic medical system 120 has followed. The scenario updater 195 can discard any scenarios 145 that do not have the same order of phases and steps as taken by the robotic medical system 120. The scenario updater 195 can take the remaining scenarios 145 and rank or sort the remaining scenarios 145 based on the metrics 180 quantifying the predicted outcomes of the remaining scenarios 145. The scenario updater 195 can make recommendations to operators of the robotic medical system 120 during the medical procedure to indicate what phase or step to transition to next based on the remaining possible scenarios 145 that could be followed.

[0071] For example, there can be three scenarios 145, a first scenario 145 including a workflow Step A, then Step B, and then Step C, a second scenario 145 including a workflow Step A, then Step B, and then Step D, and a third scenario 145 including a workflow Step A, then Step C, and then Step E. If the scenario updater 195 identifies that the robotic medical system 120 implemented Step A, and then Step B, but has not yet implemented a third step, the scenario updater 195 can determine that the first and second scenarios 145 should be retained, but the third scenario 145 should be discarded. The scenario updater 195 can compare the metrics 180 for the first scenario 145 and the second scenario 145, e.g., the predicted outcomes that result from performing Step C or Step D respectively. For example, if the predicted outcomes of the first scenario 145 are better than the predicted outcomes of the second scenario 145, the scenario updater 195 can recommend that the robotic medical system 120 perform Step C next, or the planner 190 can cause the robotic medical system 120 to perform Step C next. The computing system 105 can use the predicted outcomes 115 ofeach remaining medical procedure scenario 145 to drive the medical procedure to a positive patient outcome, an ideal patient outcome, or a best patient outcome.

[0072] The scenario updater 195 can receive updated state information from the robotic medical system 120. The updated state information can indicate what actions, steps, or phases the robotic medical system 120 has already performed for a particular medical procedure. The updated state information can be received after the robotic medical system 120 has begun or is operating based on the selected scenario 145. With the updated state information, the scenario updater 195 can cause the model 135 to generate second scenarios 145. The model 135 can generate second scenarios 145 based on the updated state information. For example, the model 135 can generate the second scenarios 145 to represent workflows that the robotic medical system 120 could still take. The model 135 can generate indicators 150 for the second scenarios 145. With the second synthetic data 140, e.g., second scenarios 145 and indicators 150 for the second scenarios 145, the model 155 can generate the predicted outcomes 115 for the second scenarios 145. For example, the model 155 can generate metrics 180 predicting the outcomes of the second scenarios 145. The scenario updater 195 can compare the various metrics 180 for the second scenarios 145 against one another. The scenario updater 195 can identify one or a set of second scenarios 145 to recommend that the robotic medical system 120 could still follow. For example, the scenario updater 195 can identify and select a second scenario 145 with metrics 180 greater than or less than thresholds. The scenario 145 can identify and select a second scenario 145 that has the best metrics 180 compared to the other second scenarios 145. The scenario updater 195 can generate, send, or provide an indication of the selected second scenario 145 to the robotic medical system 120 to cause the robotic medical system 120 to perform at least a second portion of the medical procedure in accordance with the selected second scenario 145.

[0073] The scenario updater 195 can generate recommendations to recommend the second scenario 145 to the robotic medical system 120 or an operator of the robotic medical system 120. For example, the scenario updater 195 can generate data to cause the graphical user interface 170 to display the selected scenario 145. The scenario updater 195 can display the workflow as a workflow, e.g., actions, phases, or steps for the robotic medical system 120 to perform. The scenario updater 195 can provide the recommendation to the client device 165 or the robotic medical system 120. The recommendation can be displayed in a display, along with an option to approve or reject the scenario 145. The user can provide input via thedisplay or via an input device to accept the scenario 145, and cause the robotic medical system 120 to follow the selected scenario 145.

[0074] In some implementations, before the medical procedure is performed, a user can adjust or change the pre-procedure data 110 to see how the outcomes 115 of the medical procedure change with different pre-procedure data 110. For example, each time the user changes the pre-procedure data 110, the computing system 105 can execute the model 135 to generate the synthetic data 140, and execute the model 155 can execute to generate the predicted outcomes 115. The user could also alter certain aspects of the pre-procedure data 110 that is actionable or changeable, like the procedure sub-type (e.g., radical vs. partial, different procedure approaches, etc.), the care team, the surgeon, pre-procedure medication, etc. to see what effect the altered aspects have on the range of possible surgical outcomes.

[0075] In some implementations, the computing system 105 can run or integrate with a simulation tool. The simulation tool can simulate different kinds of exercises to help acclimate early career surgeons to the robotic medical system 120 and get practice outside of the operating room. These exercises can range from games (like peg-placement) to portions or entire simulated surgical procedures. The computing system 105 can track the simulation usage and exercises for surgeons and generate scores based on OPIs (e.g., the OPIs can be the same or similar to those computed for clinical cases). In some implementations, the preprocedure data 110 can include simulation data collected from the simulation tool.Simulation tool usage can be connected to changes in OPIs in clinical cases. For example, how surgical outcome distributions change with changed patterns of simulation usage for various surgeons can be implemented.

[0076] Referring now to FIG. 2, among others, an example computing system 105 to generate synthetic medical procedure samples from pre-procedure data 110 and predict outcomes 180 for the synthetic medical procedure samples is shown. The computing system 105 can receive the historical data 175. The historical data 175 can include pre-procedure data 210. The pre-procedure data 210 can include the same or similar attributes or values as the pre-procedure data 110, but are for a variety of historical medical procedures for a variety of different patients. The historical data 175 can include workflows 215. The workflows 215 can indicate the various phases, steps, or actions taken for each historical medical procedure. The historical data 175 can include indicators 220. The indicators 220 can be the same as or similar to the indicators 150. The indicators 220 can be indicators determined for the variousphases or steps of the historical workflows 215. The historical data 175 can include postprocedure outcomes 225. The outcomes 225 can be metrics for each workflow 215. The outcomes 225 can be the same as or similar to the metrics 180.

[0077] The computing system 105 can ingest the historical data 175 from a database, a user device, another computing system, or another server system. The historical data 175 can be received in a CSV file, an XML file, a JSON file, a PARQUET file. The file can include pre-defined column names for various types of pre-procedure data 210, workflows 215, indicators 220, or post-procedure outcomes 225. The file can include populated values, indicators, or entries for the columns. Each row of the file can represent a different sample. Each sample can be a different medical procedure or case. The file can include a row for each case or medical procedure. The clinical data (pre-procedure data 210 and postprocedure outcomes 225), the surgical workflow 215, and the OPIs 220 for all historical cases can be represented in a tabular format, each row can be a case, and each column can be a property of that case. The file can indicate formatting procedures describing how to retrieve the clinical data from an electronic health record system. The computing system 105 can receive pre-procedure data 210, indicator data 220, or post-procedure outcome data 225 and ingest the retrieved data into the historical data 175 according to the formatting or retrieval procedures of the file. The computing system 105 can train the cGANs 135 or the outcome prediction models 155 using the historical data 175.

[0078] The surgical workflow data 215 can be created for each case through annotation (e.g., manual or machine-driven annotation) of surgical activities as recognized in endoscopic video recordings of each case. The computing system 105 can compute the OPIs 220 from the retrieved data using video recordings of the endoscope 130 of the robotic medical system 120, kinematics data of the robotic medical system 120, or events recorded by the robotic medical system 120 for each medical procedure case.

[0079] The computing system 105 can train the cGANs 135 using the historical data 175. The computing system 105 can train the cGANs 135 from all possible historical data 175 without any further sub-setting. The computing system 105 can receive the pre-procedure data 110 from a user interface. The pre-procedure data 110 can take certain pre-defined pre- surgical information fields with pre-set values and ranges to enforce expected use. These fields and ranges map to the specific columns of the historical data 175 relevant to preprocedure data 110 and the previously seen / defined values for those columns.

[0080] With the cGANs 135 trained, the cGANs 135 can execute to generate the synthetic data 140 using the pre-procedure data 110. The pre-procedure data 110 can be input to the cGANs 135 and used as a constraint or an initial condition by the cGANs 135. The pre-procedure data 110 can be a conditional prior for the cGANs 135 to execute on. The synthetic data 140 can be a posterior of the cGANs. The computing system 105 can generate, using a first cGAN model 135 and pre-procedure data 110 of a patient, workflow scenarios 145. For example, the cGANs 135 can include a workflow cGAN 135. The computing system 105 can generate, using a second cGAN 135, the pre-procedure data 110, and the scenarios 145, performance indicators 150 for the workflow scenarios 145. The workflow cGAN 135 can use the pre-procedure data 110 as a prior, and the workflow scenarios 145 can be a posterior of the workflow cGAN 135. The intra-procedure indicator cGAN 135 can use the pre-procedure data 110 as a prior, and the intra-procedure indicators 150 can be a posterior of the intra-procedure indicator cGAN 135.

[0081] With a sample of pre-procedure data 110 from one patient scheduled for surgery, the computing system 105 use cGANs to conditionally sample N number of samples 205 of workflows 145 and intraoperative OPIs 150 given the pre-procedure data 110 as a conditional prior. The values of N can be on the order of thousands (e.g., greater than 1,000) for results to capture low-likelihood but high-cost events or outcomes 180. With the samples 205 of the synthetic data 140, models 155 can predict outcomes 180 for each sample 205 of the synthetic data 140.

[0082] The computing system 105 can execute the outcome prediction models 155 to predict outcomes for the potential surgical workflows 145. For example, the predicted outcomes 155 can be inferences of the outcome prediction models 155. The synthetic data 140 can be input data or predictors that the outcome prediction models 155 execute on to generate the inferences. The outcome prediction models 155 can output or add data into a dataset 115 that predicts different outcomes 180 of the surgical workflows 145. The outcome prediction models 135 can each predict a different outcome 180 for each sample 205. The outcome prediction models 155 can each receive the synthetic data 140 including different potential surgical workflows 145 and output a different predicted post-operative indicators 150. One model 135 can be a length-of-stay model that predicts the length of time a patient will stay at a hospital after completion of the medical procedure. Another model 135 can predict a length of post anesthesia care time for a patient after the medical procedure.Another model 135 can predict the likelihood of re-admittance within a time frame after the medical procedure (e.g., 30 days after the procedure). Another model 135 can predict a total cost of care of a patient after a medical procedure. Another model 135 can predict surgical site patient infection.

[0083] With the samples 205 of the predicted outcomes, the planner 190 can perform a scenario analysis to highlight the best, most likely, and worst-case scenarios 145 based on the outcomes 180. In some implementations, the filter 185 can reduce the predicted outcomes 115 to a set of scenarios 145 with the metrics 180 greater or less than a threshold. From the categories of scenarios 145, the planner 190 can use the associated intraoperative surgical workflows 145 and OPIs 150 to inform surgical planning and post-operative care coordination.

[0084] Referring now to FIG. 3, among others, an example graphical user interface 170 for a user to enter pre-procedure data 110 of a patient into is shown. The computing system 105 can generate a graphical user interface 170 to depict the various surgical workflows to a user or surgeon. The graphical user interface 170 can enable scenario analysis by generating thousands of fictive surgical episodes, each with OPIs and surgical outcomes, using pre- surgical patient data as conditions passed to generative and supervised machine learning models. The results generated can be the possible set of outcomes and OPIs specific to the pre-surgical patient features.

[0085] The graphical user interface 170 can include a first element 305. The first element 305 can receive different types of pre-procedure data 110 to run the models 135 or 155 on. The element 305 can include elements, drop down menus, graphic based inputs, or other interface portions to receive the pre-procedure data 110. For example, a user can input an age, sex, body mass index (BMI), race, smoker status, an American Society of Anesthesiology (ASA) class, pre-procedure length of stay (LOS), chronic steroid use, ventilator dependency, dyspnea, diabetes, malignant cancer, hypertension, chronic obstructive pulmonary disease (COPD), renal failure, or 30-day heart status. In some implementations, a user can select a surgery type or sub-type in order to obtain results tailored directly to the surgery type or sub-type.

[0086] The graphical user interface 170 can include a second element 310. The element 310 can include a set of visualizations that summarize best-outcomes, most likely-outcomes,and worst-case outcomes for a given set of pre-procedure data 110. The graphical user interface 170 can include an element 315. A user, via the element 315 can select different outcome metric types to rank or sort the workflow scenarios 145 according to. For example, a user can select operating room (OR) time, post anesthesia care unit (PACU) time, length of stay (LOS) days, intensive care unit (ICU) days, surgical site infection (SSI), or readmittance prediction. The element 310 can include an element 320 depicting percentiles of the selected metric type. For example, if operating room time is selected via the element 315, the graphical user interface 170, the top 20% operating room time for the scenarios 145 can be displayed, a most-likely or middle 50% percentile for the scenarios 145 can be displayed, and a worst or bottom 20% operating room time can be displayed.

[0087] The graphical user interface 170 can be an outcome distribution 325, steps or phases of workflows 330, or OPIs 335. The graphical user interface 170 can show OPIs, likelihood of OPIs, or provide alerts or warnings. The graphical user interface 170 can include a graphic flow of the different workflow scenarios 145 in terms of the phases or steps of the workflow scenario 145. For example, the graphical user interface 170 can display a time-line or graphic that depicts each phase, and the steps within each phase.

[0088] Referring now to FIG. 4, among others, an example method 400 of predicting outcomes of medical procedures using machine learning is shown. At least a portion of the method 400 can be performed by the computing system 105, the robotic medical system 120, or the client device 165. The method 400 can include an ACT 405 of identifying state information. The method 400 can include an ACT 410 of generating scenarios. The method 400 can include an ACT 415 of determining performance metrics. The method 400 can include an ACT 420 of selecting scenarios. The method 400 can include an ACT 425 of providing an indication.

[0089] At ACT 405, the method 400 can include identifying, by the computing system 105, state information. For example, the method 400 can include receiving, by the computing system 105 the pre-procedure data 110. The method 400 can include identifying the state information from the pre-procedure data 110. The method 400 can include receiving the preprocedure data from the client device 165 via a graphical user interface 170. The method 400 can include receiving the pre-procedure data 110 from the robotic medical system 120. The method 400 can include determining, by the computing system 105, at least a portion of thepre-procedure data 110. In some implementations, the method 400 can include receiving the pre-procedure data 110 from an EHR or EMR database or system.

[0090] At ACT 410, the method 400 can include generating, by the computing system 105, a scenario 145. The method 400 can include generating synthetic data 140. The method 400 can include generating the scenarios 145 from the pre-procedure data 110. For example, the method 400 can include executing the model 135 to generate the scenarios 145. For example, the method 400 can include executing the model 135 with the pre-procedure data 110 as a prior to generate a posterior including the workflow scenarios 145. The method 400 can include executing a cGAN 135 that performs conditional sampling of the pre-procedure data 110 to generate the workflow scenarios 145. The method 400 can include generating intra-procedure indicators 150 with the model 135 for the workflow scenarios 145. The method 400 can include executing another model 135 to generate intra-procedure indicators 150 for each of the generated scenarios 145. For example, the method 400 can include executing, a cGAN that conditionally samples the pre-procedure data 110 and / or the workflow scenarios 145 to generate the intra-procedure indicators 150.

[0091] At ACT 415, the method 400 can include determining, by the computing system 105, performance metrics 180. For example, the method 400 can include executing a model 155 to generate the metrics 180 for the scenarios 145. The method 400 can include executing the model 155 with the synthetic data 140 to generate the metrics 180. The method 400 can include executing the model 155 with the synthetic data 140 as an input to output predictions for the metrics 180. The model 155 can output different predictions for different types of metrics 180. In this regard, for a particular scenario 145, the method 400 can include generating different types of metrics 180 for each scenario 145. In some implementations, the method 400 can include executing a set or group of different models 155. Each model 155 can output a different prediction for a metric 180 (e.g., length-of-stay, total cost of care, re-admittance). The method 400 can include executing each of the set of different models 155 on the synthetic data 140 to generate a different metric 180. The computing system 105 can include executing the models 155 with the workflow scenarios 145 as inputs and the intra-procedure indicators 150 as inputs.

[0092] At ACT 420, the method 400 can include selecting, by the computing system 105, a scenario 145. For example, the method 400 can include analyzing the metrics 180 of the scenarios 145 to select one or a group of scenarios 145. For example, the method 400 caninclude comparing the metrics 180 of the scenarios 145 to thresholds, and select scenarios 145 with metrics 180 that satisfy the thresholds (e.g., are greater than or less than the thresholds). The computing system 105 can run an optimization or objective function optimization to determine a scenario 145 with an optimal mix of metric types.

[0093] At ACT 425, the method 400 can include providing, by the computing system 105, an indication. The method 400 can include providing the selected scenario 145 to the robotic medical system 120. The method 400 can include causing the robotic medical system 120 to follow or perform the workflow indicated by the selected scenario 145. For example, the robotic medical system 120 can be controlled to implement the various phases and steps of the selected scenario 145. The method 400 can include causing the selected scenario 145 to be displayed in the graphical user interface 170.

[0094] Referring now to FIG. 5, among others, a chart 500 comparing example machine learning models 155 that predict different outcomes for different procedure types is shown. The chart 500 includes different procedure types along a vertical axis (e.g., inguinal hernia, sigmoid colectomy, prostatectomy, lobectomy, hysterectomy (benign), sleeve gastrectomy, ventral hernia, cholecystectomy, hysterectomy (malignant), gastric bypass). For each procedure type, a different model 155 can be trained to generate a metric 180, e.g., a flag indicating predicting patient readmittance after the procedure, a post surgery length of stay (days), a post-anesthesia care unit (PACU) time (minutes), operating room time (minutes). Along a horizontal axis, the chart 500 indicates the number of different models 155 for each procedure type predicting the various metrics 180 that have a balanced accuracy over a threshold (e.g., 0.69).

[0095] Referring now to FIGS. 6-8, among others, charts 600-800 indicating importance levels for features of example machine learning models 155 for various medical procedures are shown. For example, chart 600 indicates the importance levels of predictors or features of pre-procedure data 110 used by a model 155 to predict a particular metric 180, postanesthesia care unit (PACU) time (minutes). The chart 700 indicates the importance levels of predictors or features of pre-procedure data 110 used by a model 155 to predict a particular metric 180, a length of stay (LOS) time. The chart 800 indicates the importance levels of predictors or features of pre-procedure data 110 used by a model 155 to predict a particular metric 180, patient readmittance flag.

[0096] Referring now to FIG. 9, among others, an example block diagram of a computing system 105 is shown. The computing system 105 can include or be used to implement a data processing system or its components. The architecture described in FIG. 9 can be used to implement the computing system 105, the endoscope 130, the instrument 125, the robotic medical system 120. The computing system 105 can include at least one bus 925 or other communication component for communicating information and at least one processor 930 or processing circuit coupled to the bus 925 for processing information. The computing system 105 can include one or more processors 930 or processing circuits coupled to the bus 925 for processing information. The computing system 105 can include at least one main memory 910, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 925 for storing information, and instructions to be executed by the processor 930. The main memory 910 can be used for storing information during execution of instructions by the processor 930. The computing system 105 can further include at least one read only memory (ROM) 915 or other static storage device coupled to the bus 925 for storing static information and instructions for the processor 930. A storage device 920, such as a solid state device, magnetic disk or optical disk, can be coupled to the bus 925 to persistently store information and instructions.

[0097] The computing system 105 can be coupled via the bus 925 to a display 900, such as a liquid crystal display, or active matrix display. The display 900 can display information to a user. An input device 905, such as a keyboard or voice interface can be coupled to the bus 925 for communicating information and commands to the processor 930. The input device 905 can include a touch screen of the display 900. The input device 905 can include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 930 and for controlling cursor movement on the display 900.

[0098] The processes, systems and methods described herein can be implemented by the computing system 105 in response to the processor 930 executing an arrangement of instructions contained in main memory 910. Such instructions can be read into main memory 910 from another computer-readable medium, such as the storage device 920. Execution of the arrangement of instructions contained in main memory 910 causes the computing system 105 to perform the illustrative processes described herein. One or more processors in a multiprocessing arrangement can be employed to execute the instructions contained in mainmemory 910. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software.

[0099] Although an example computing system has been described in FIG. 9, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0100] Some of the description herein emphasizes the structural independence of the aspects of the system components or groupings of operations and responsibilities of these system components. Other groupings that execute similar overall operations are within the scope of the present application. Modules can be implemented in hardware or as computer instructions on a non-transient computer readable storage medium, and modules can be distributed across various hardware or computer based components.

[0101] The systems described above can provide multiple ones of any or each of those components and these components can be provided on either a standalone system or on multiple instantiations in a distributed system. In addition, the systems and methods described above can be provided as one or more computer-readable programs or executable instructions embodied on or in one or more articles of manufacture. The article of manufacture can be cloud storage, a hard disk, a CD-ROM, a flash memory card, a PROM, a RAM, a ROM, or a magnetic tape. In general, the computer-readable programs can be implemented in any programming language, such as LISP, PERL, C, C++, C#, PROLOG, Python, or in any byte code language such as JAVA. The software programs or executable instructions can be stored on or in one or more articles of manufacture as object code.

[0102] Example and non-limiting module implementation elements include sensors providing any value determined herein, sensors providing any value that is a precursor to a value determined herein, datalink or network hardware including communication chips, oscillating crystals, communication links, cables, twisted pair wiring, coaxial wiring, shielded wiring, transmitters, receivers, or transceivers, logic circuits, hard-wired logic circuits, reconfigurable logic circuits in a particular non-transient state configured according to themodule specification, any actuator including at least an electrical, hydraulic, or pneumatic actuator, a solenoid, an op-amp, analog control elements (springs, filters, integrators, adders, dividers, gain elements), or digital control elements.

[0103] The subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices including cloud storage). The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0104] The terms “computing device”, “component” or “data processing apparatus” or the like encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more ofthem. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0105] A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0106] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Devices suitable for storing computer program instructions and data can include non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0107] The subject matter described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back end, middleware, or front end components. Thecomponents of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0108] While operations are depicted in the drawings in a particular order, such operations are not required to be performed in the particular order shown or in sequential order, and all illustrated operations are not required to be performed. Actions described herein can be performed in a different order.

[0109] Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. ACTs, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations.

[0110] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including” “comprising” “having” “containing” “involving” “characterized by” “characterized in that” and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.

[0111] Any references to implementations or elements or acts of the systems and methods herein referred to in the singular may also embrace implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein may also embrace implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations. References to any ACT or element being based on any information, act or element may includeimplementations where the act or element is based at least in part on any information, act, or element.

[0112] Any implementation disclosed herein may be combined with any other implementation or example, and references to “an implementation,” “some implementations,” “one implementation” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation or example. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0113] References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. References to at least one of a conjunctive list of terms may be construed as an inclusive OR to indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Such references used in conjunction with “comprising” or other open terminology can include additional items.

[0114] Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.

[0115] Modifications of described elements and acts such as variations in sizes, dimensions, structures, shapes and proportions of the various elements, values of parameters, mounting arrangements, use of materials, colors, orientations can occur without materially departing from the teachings and advantages of the subject matter disclosed herein. For example, elements shown as integrally formed can be constructed of multiple parts or elements, the position of elements can be reversed or otherwise varied, and the nature or number of discrete elements or positions can be altered or varied. Other substitutions, modifications, changes and omissions can also be made in the design, operating conditions and arrangement of the disclosed elements and operations without departing from the scope of the present disclosure.

Claims

CLAIMSWhat is claimed is:

1. A system, comprising: one or more processors, coupled with memory, to: identify state information related to a medical procedure to be performed by a robotic medical system; generate, using the state information and with one or more models trained by machine learning, a plurality of scenarios of operation of the robotic medical system to perform the medical procedure; determine, using the one or more models, performance metrics for the plurality of scenarios; select, based on a comparison of the performance metrics for the plurality of scenarios, a scenario of the plurality of scenarios for operation; and provide an indication of the selected scenario to cause the robotic medical system to perform at least a portion of the medical procedure in accordance with the selected scenario.

2. The system of claim 1, comprising the one or more processors to: generate, using a first conditional generative model and pre-procedure data of a patient, the plurality of scenarios; generate, using a second conditional generative model, the pre-procedure data, and the plurality of scenarios, performance indicators for the plurality of scenarios; and construct a synthetic dataset comprising the plurality of scenarios and the performance indicators for the plurality of scenarios.

3. The system of claim 1, comprising the one or more processors to: receive a synthetic dataset comprising the plurality of scenarios and performance indicators for the plurality of scenarios; and execute, using the synthetic dataset, at least one model to generate the performance metrics for the plurality of scenarios.

4. The system of claim 1, comprising: the one or more processors to generate the plurality of scenarios using a conditional generative model.

5. The system of claim 1, comprising: the one or more processors to determine the performance metrics using a model that predicts outcomes for the medical procedure from the plurality of scenarios.

6. The system of claim 1, comprising the one or more processors to: receive updated state information subsequent to execution of the at least the portion of the medical procedure in accordance with the selected scenario; generate, using the one or more models, a second plurality of scenarios based on the updated state information; determine, using the one or more models, second performance metrics for the second plurality of scenarios; select, based on a comparison of the second performance metrics for the second plurality of scenarios, a second scenario of the second plurality of scenarios for operation; and provide an indication of the selected second scenario to cause the robotic medical system to perform at least a second portion of the medical procedure in accordance with the selected second scenario.

7. The system of claim 1, comprising the one or more processors to: determine, using the one or more models, the performance metrics for the plurality of scenarios, wherein at least one of the performance metrics corresponds to an outcome of the medical procedure performed by the robotic medical system.

8. The system of claim 1, comprising the one or more processors to: determine a count of the plurality of scenarios to generate based on a type of an outcome of the medical procedure performed by the robotic medical system; and generate the plurality of scenarios in accordance with the determined count.

9. The system of claim 1, comprising the one or more processors to: generate the plurality of scenarios, the scenario of the plurality of scenarios comprising: a workflow of the medical procedure comprising a sequence of a plurality of steps performed by the robotic medical system.

10. The system of claim 1, comprising the one or more processors to: generate the plurality of scenarios, the scenario of the plurality of scenarios comprising: a workflow of the medical procedure comprising a sequence of a plurality of steps performed by the robotic medical system; and indications of types of instrument for the robotic medical system to use at the plurality of steps.

11. A method, comprising: identifying, by one or more processors, state information related to a medical procedure to be performed by a robotic medical system; generating, by the one or more processors, using the state information and with one or more models trained by machine learning, a plurality of scenarios of operation of the robotic medical system to perform the medical procedure; determining, by the one or more processors, using the one or more models, performance metrics for the plurality of scenarios; selecting, by the one or more processors, based on a comparison of the performance metrics for the plurality of scenarios, a scenario of the plurality of scenarios for operation; and providing, by the one or more processors, an indication of the selected scenario to cause the robotic medical system to perform at least a portion of the medical procedure in accordance with the selected scenario.

12. The method of claim 11, comprising: generating, by the one or more processors, using a first conditional generative model and pre-procedure data of a patient, the plurality of scenarios; generating, by the one or more processors, using a second conditional generative model, the pre-procedure data, and the plurality of scenarios, performance indicators for the pluralityof scenarios; and constructing, by the one or more processors, a synthetic dataset comprising the plurality of scenarios and the performance indicators for the plurality of scenarios.

13. The method of claim 11, comprising: receiving, by the one or more processors, a synthetic dataset comprising the plurality of scenarios and performance indicators for the plurality of scenarios; and executing, by the one or more processors, using the synthetic dataset, at least one model to generate the performance metrics for the plurality of scenarios.

14. The method of claim 11, comprising: determining, by the one or more processors, the performance metrics using a model that predicts outcomes for the medical procedure from the plurality of scenarios.

15. The method of claim 11, comprising: receiving, by the one or more processors, updated state information subsequent to execution of the at least the portion of the medical procedure in accordance with the selected scenario; generating, by the one or more processors, using the one or more models, a second plurality of scenarios based on the updated state information; determining, by the one or more processors, using the one or more models, second performance metrics for the second plurality of scenarios; selecting, by the one or more processors, based on a comparison of the second performance metrics for the second plurality of scenarios, a second scenario of the second plurality of scenarios for operation; and providing, by the one or more processors, an indication of the selected second scenario to cause the robotic medical system to perform at least a second portion of the medical procedure in accordance with the selected second scenario.

16. The method of claim 11, comprising: determining, by the one or more processors, a count of the plurality of scenarios togenerate based on a type of an outcome of the medical procedure performed by the robotic medical system; and generating, by the one or more processors, the plurality of scenarios in accordance with the determined count.

17. The method of claim 11, comprising: generating, by the one or more processors, the plurality of scenarios, the scenario of the plurality of scenarios comprising: a workflow of the medical procedure comprising a sequence of a plurality of steps performed by the robotic medical system.

18. One or more storage media storing instructions thereon, that, when executed by one or more processors, cause the one or more processors to perform operations, comprising: identifying state information related to a medical procedure to be performed by a robotic medical system; generating, using the state information and with one or more models trained by machine learning, a plurality of scenarios of operation of the robotic medical system to perform the medical procedure; determining, using the one or more models, performance metrics for the plurality of scenarios; selecting, based on a comparison of the performance metrics for the plurality of scenarios, a scenario of the plurality of scenarios for operation; and providing an indication of the selected scenario to cause the robotic medical system to perform at least a portion of the medical procedure in accordance with the selected scenario.

19. The one or more storage media of claim 18, the operations comprising: generating, by the one or more processors, using a first conditional generative model and pre-procedure data of a patient, the plurality of scenarios; generating, by the one or more processors, using a second conditional generative model, the pre-procedure data, and the plurality of scenarios, performance indicators for the plurality of scenarios;constructing, by the one or more processors, a synthetic dataset comprising the plurality of scenarios and the performance indicators for the plurality of scenarios; and executing, by the one or more processors, using the synthetic dataset, at least one model to generate the performance metrics for the plurality of scenarios.

20. The one or more storage media of claim 18, the operations comprising: receiving, by the one or more processors, updated state information subsequent to execution of the at least the portion of the medical procedure in accordance with the selected scenario; generating, by the one or more processors, using the one or more models, a second plurality of scenarios based on the updated state information; determining, by the one or more processors, using the one or more models, second performance metrics for the second plurality of scenarios; selecting, by the one or more processors, based on a comparison of the second performance metrics for the second plurality of scenarios, a second scenario of the second plurality of scenarios for operation; and providing, by the one or more processors, an indication of the selected second scenario to cause the robotic medical system to perform at least a second portion of the medical procedure in accordance with the selected second scenario.

Images